Usability and Usefulness of Machine Learning-Based Clinical Decision Support Software in Primary Care: Survey of Users in a Prospective Observational Study.

Background: The successful implementation of decision support systems promises to enhance high-quality care. However, the successful implementation of a clinical decision support system (CDSS) depends on user acceptance and adoption. A machine learning (ML)-based CDSS to assist primary care professionals treating urinary tract infections (UTIs) was implemented, and usability and usefulness were assessed through a questionnaire. Objective: This study aimed to assess the system's usability by examining users' experiences with the software. A secondary goal was to assess users' attitudes toward evidence-based practice and innovation in health care. Methods: In collaboration with the Netherlands Institute for Health Services Research (NIVEL) and Leiden University Medical Center (LUMC), Pacmed Ltd developed the CDSS. The cohort was mostly recruited at the care group level; practices within participating care groups were required to participate. Health insurers partly funded the research. Practitioners participated in the implementation study for 4 months. A survey based on the Unified Theory of Acceptance and Use of Technology (UTAUT) was sent to 263 general practitioners and assistants shortly after the implementation period. Furthermore, usage data were analyzed. Results: Of the 34 participating practices that used the software, 30 (88%) submitted at least one survey response, with a mean of 2.23 responses per practice (SD 1.43). The CDSS was used throughout the pilot period, and 31 practices continued using the tool, with 9% dropping out during the first 8 weeks. Sixty-seven percent of respondents trusted the tool's output, and 73% found it understandable how the algorithm came to predictions. Sixty-five percent of respondents indicated that the information provided was useful in addition to the available guidelines, and 52% agreed that it supported their decision-making. However, many respondents were uncertain whether the tool improved patient care (46%) or patient outcomes (66%). Forty-eight percent of respondents found the software easy to integrate into their clinical workflow. Conclusions: The CDSS was perceived as trustworthy and easy to use. However, users were unable to determine whether the CDSS improved patient outcomes. In addition, the CDSS development could have benefited from including assistants as well as general practitioners more in the design phase of the software. Because assistants play an important role in UTI care, designing the software to better fit existing workflows may reduce the perceived time investment associated with using the tool. Finally, respondents reported strong motivation to contribute to further research in this field and indicated willingness to embrace change in health care delivery, which may also reflect selection bias in our sample.