Human factors validation study of an artificial neural network‑based preoperative decision‑support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN99301435).

backgroundThe integration of clinical decision support tools in medical practice is challenging and must be carefully undertaken, especially in cancer management. Noninvasive Lymph Node Status (NILS) is a web-based tool designed to estimate the probability of benign axillary lymph nodes in female patients with breast cancer scheduled for primary surgery. The aim was to identify barriers to NILS adoption in a clinical setting and assess whether intended users can operate the tool without significant errors or difficulties. Additionally, the study aimed to evaluate the appropriateness of result interpretation for decisions to abstain from sentinel lymph node biopsy (SNLB) and measure overall user satisfaction with the tool.

methodsThis mixed-methods multicenter on-site qualitative human factor validation study was conducted in a simulated clinical environment, replicating both real-world physical and digital conditions. Based on the identified target user population for the NILS model, twenty physicians comprised the cohort. The study used simulated clinical cases, the System Usability Scale (SUS), and the After-Scenario Questionnaire (ASQ) to evaluate usability and satisfaction. An oral interview was conducted after the evaluations. The study followed a structured protocol, with distinct roles assigned to the test participants, leader, and observer.

resultsTwenty physicians from four hospitals, with a median of 9.5 years of specialist practice, participated. Most participants were surgeons (75%, N = 15), while the remaining 25% (N = 5) were oncologists. Usability scores were high, with a mean SUS score of 89.5 ("excellent") and a mean ASQ score of 6.3 (Likert scale 1-7). Several interface challenges were identified, including difficulty in locating the reset and information buttons, the ability to enter values outside valid ranges (age and tumor size), and the risk of unintentionally modifying the entered values while scrolling.

conclusionsThis study identified the key usability factors, barriers, and facilitators affecting the NILS model implementation. Physicians found the NILS interface easy to use and valued the presented risk estimates of benign axillary lymph node status in their decisions to perform or abstain from SLNB. Study-informed interface redesigns have been implemented. Iterative usability work in clinical settings is crucial for successful implementation.

trial registrationISRCTN99301435.