SynthesisPloS one2022
Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review.
Synthesis in PloS one, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed.
- Evaluating digital triage symptom checker with historical triage-related adverse events.Scandinavian journal of primary health care · 2026Article
- Artificial Intelligence Models for Predicting Triage in Emergency Departments: Seven-Month Retrospective Comparative Study of Natural Language Processing, Large Language Model, and Joint Embedding Predictive Architectures.JMIR medical informatics · 2026Article
- Artificial Intelligence in Eating Disorder Treatment: A Qualitative Analysis of Clinical Opportunities, Barriers, and Ethical Considerations From Multi-Disciplinary Focus Groups.The International journal of eating disorders · 2026Article
- Understanding Acceptability of AI Triage Tools Amongst Underserved Populations: Lessons From the Early Phases of Co-Production With Bangladeshi Communities in Birmingham, UK.Health expectations : an international journal of public participation in health care and health policy · 2025Article
- Artificial Intelligence in Digital Self-Diagnosis Tools: A Narrative Overview of Reviews.Mayo Clinic proceedings. Digital health · 2025Review
- Opportunities for Artificial Intelligence in Operational Medicine: Lessons from the United States Military.Bioengineering (Basel, Switzerland) · 2025Review
- Improving Safety, Efficiency, Cost, and Satisfaction Across a Musculoskeletal Pathway Using the Digital Assessment Routing Tool for Triage: Quality Improvement Study.Journal of medical Internet research · 2025Article
- Exploring Laypersons' Experiences With a Mobile Symptom Checker App as an Interface Between eHealth Literacy, Health Literacy, and Health-Related Behavior: Qualitative Interview Study.JMIR formative research · 2025Article
- Mapping and Summarizing the Research on AI Systems for Automating Medical History Taking and Triage: Scoping Review.Journal of medical Internet research · 2025Article
- Influences on trust in the use of AI-based triage-an interview study with primary healthcare professionals and patients in Sweden.Frontiers in digital health · 2025Article
- Who is more trustworthy? The influence mechanism of AI vs. human doctor triage on user trust: testing the mediating effect of psychological distance, and a multiple moderating effects analysis.Frontiers in psychology · 2025Article
- Evaluating Artificial Intelligence in Clinical Settings-Let Us Not Reinvent the Wheel.Journal of medical Internet research · 2024Article
- Evaluation of a Musculoskeletal Digital Assessment Routing Tool (DART): Crossover Noninferiority Randomized Pilot Trial.JMIR formative research · 2024Article
- Drinking from the Holy Grail-Does a Perfect Triage System Exist? And Where to Look for It?Journal of personalized medicine · 2024Review
- Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics.JMIR formative research · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
introductionPatient-operated digital triage systems with AI components are becoming increasingly common. However, previous reviews have found a limited amount of research on such systems' accuracy. This systematic review of the literature aimed to identify the main challenges in determining the accuracy of patient-operated digital AI-based triage systems.
methodsA systematic review was designed and conducted in accordance with PRISMA guidelines in October 2021 using PubMed, Scopus and Web of Science. Articles were included if they assessed the accuracy of a patient-operated digital triage system that had an AI-component and could triage a general primary care population. Limitations and other pertinent data were extracted, synthesized and analysed. Risk of bias was not analysed as this review studied the included articles' limitations (rather than results). Results were synthesized qualitatively using a thematic analysis.
resultsThe search generated 76 articles and following exclusion 8 articles (6 primary articles and 2 reviews) were included in the analysis. Articles' limitations were synthesized into three groups: epistemological, ontological and methodological limitations. Limitations varied with regards to intractability and the level to which they can be addressed through methodological choices. Certain methodological limitations related to testing triage systems using vignettes can be addressed through methodological adjustments, whereas epistemological and ontological limitations require that readers of such studies appraise the studies with limitations in mind. DISCUSSION: The reviewed literature highlights recurring limitations and challenges in studying the accuracy of patient-operated digital triage systems with AI components. Some of these challenges can be addressed through methodology whereas others are intrinsic to the area of inquiry and involve unavoidable trade-offs. Future studies should take these limitations in consideration in order to better address the current knowledge gaps in the literature.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.