Observational studyInternational journal of medical informatics2017
"Think aloud" and "Near live" usability testing of two complex clinical decision support tools.
Observational study in International journal of medical informatics, 2017. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 53 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
53 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- The implementation challenge of computerised clinical decision support systems for the detection of disease in primary care: systematic review and recommendations.Implementation science : IS · 2025Pooled it
- Optimizing Clinical Decision Support System Functionality by Leveraging Specific Human-Computer Interaction Elements: Insights From a Systematic Review.JMIR human factors · 2025Pooled it
- Factors influencing the implementation of decision support systems for antibiotic prescription in hospitals: a systematic review.BMC medical informatics and decision making · 2023Pooled it
- Evaluation of an Antimicrobial Stewardship Decision Support for Pediatric Infections.Applied clinical informatics · 2023Trial
- Impact of interactive computerised decision support for hospital antibiotic use (COMPASS): an open-label, cluster-randomised trial in three Swiss hospitals.The Lancet. Infectious diseases · 2022Trial
- Impact of Clinical Decision Support on Antibiotic Prescribing for Acute Respiratory Infections: a Cluster Randomized Implementation Trial.Journal of general internal medicine · 2020Trial
- Clinical decision support system for child and adolescent mental health services: a formative usability study.BMC medical informatics and decision making · 2026Article
- Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review.Journal of medical systems · 2026Article
- Evaluating a Shared Decision Support Tool for Pediatric Cardiopulmonary Arrest: Mixed Methods Usability Study.JMIR human factors · 2026Article
- SAFER-Dem: generating co-designed adaptations to a discharge care planning bundle for people living with dementia.BMJ open · 2026Article
- Optimizing pharmacogenomic clinical decision support: a usability study of epic's Genomic-Indicators.BMC pharmacology & toxicology · 2026Article
- Nursing Performance Using Clinical Prediction Rules for Acute Respiratory Infection Management: A Case-Based Simulation.Applied clinical informatics · 2025Article
- "My Back Exercise App"-mHealth for Low Back Pain: Development and Usability Testing.Journal of healthcare informatics research · 2025Article
- Snowball Group Usability Testing for Rapid and Iterative Multisite Tool Development: Method Development Study.JMIR formative research · 2025Article
- User-Centered Design of a Preference-Driven Patient Activation Tool for Optimizing Depression Treatment in Integrated Primary Care Settings (The Transform DepCare Study).Journal of general internal medicine · 2025Article
- Article
- A qualitative, theory-based exploration of facilitators and barriers for implementation of pharmacist prescribing in chronic kidney disease.International journal of clinical pharmacy · 2024Article
- Exploring the User Acceptability and Feasibility of a Clinical Decision Support Tool Designed to Facilitate Timely Diagnosis of New-Onset Type 1 Diabetes in Children: Qualitative Interview Study Among General Practitioners.JMIR formative research · 2024Article
- Digital clinical empathy in a live chat: multiple findings from a formative qualitative study and usability tests.BMC health services research · 2024Article
- Exploring the feasibility of an artificial intelligence based clinical decision support system for cutaneous melanoma detection in primary care - a mixed method study.Scandinavian journal of primary health care · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
objectivesLow provider adoption continues to be a significant barrier to realizing the potential of clinical decision support. "Think Aloud" and "Near Live" usability testing were conducted on two clinical decision support tools. Each was composed of an alert, a clinical prediction rule which estimated risk of either group A Streptococcus pharyngitis or pneumonia and an automatic order set based on risk. The objective of this study was to further understanding of the facilitators of usability and to evaluate the types of additional information gained from proceeding to "Near Live" testing after completing "Think Aloud".
methodsThis was a qualitative observational study conducted at a large academic health care system with 12 primary care providers. During "Think Aloud" testing, participants were provided with written clinical scenarios and asked to verbalize their thought process while interacting with the tool. During "Near Live" testing participants interacted with a mock patient. Morae usability software was used to record full screen capture and audio during every session. Participant comments were placed into coding categories and analyzed for generalizable themes. Themes were compared across usability methods.
results"Think Aloud" and "Near Live" usability testing generated similar themes under the coding categories visibility, workflow, content, understand-ability and navigation. However, they generated significantly different themes under the coding categories usability, practical usefulness and medical usefulness. During both types of testing participants found the tool easier to use when important text was distinct in its appearance, alerts were passive and appropriately timed, content was up to date, language was clear and simple, and each component of the tool included obvious indicators of next steps. Participant comments reflected higher expectations for usability and usefulness during "Near Live" testing. For example, visit aids, such as automatically generated order sets, were felt to be less useful during "Near-Live" testing because they would not be all inclusive for the visit.
conclusionsThese complementary types of usability testing generated unique and generalizable insights. Feedback during "Think Aloud" testing primarily helped to improve the tools' ease of use. The additional feedback from "Near Live" testing, which mimics a real clinical encounter, was helpful for eliciting key barriers and facilitators to provider workflow and adoption.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.