ArticleJournal of medical Internet research2020
Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study.
Article in Journal of medical Internet research, 2020. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 23 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
23 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- The Accuracy of Artificial Intelligence in the Endoscopic Diagnosis of Early Gastric Cancer: Pooled Analysis Study.Journal of medical Internet research · 2022Pooled it
- Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review.PloS one · 2022Pooled it
- Impact of a Symptom Checker App on Patient-Physician Interaction Among Self-Referred Walk-In Patients in the Emergency Department: Multicenter, Parallel-Group, Randomized, Controlled Trial.Journal of medical Internet research · 2025Trial
- AI-Based Diagnostic Platform Capabilities With Lyme Disease as a Use Case: Integrative Exploration.Online journal of public health informatics · 2026Article
- The last human touch: why AI doesn't belong in life's defining moments.BMC medical ethics · 2026Article
- Improving the Readability of Institutional Heart Failure-Related Patient Education Materials Using GPT-4: Observational Study.JMIR cardio · 2025Observational
- Accuracy of online symptom assessment applications, large language models, and laypeople for self-triage decisions.NPJ digital medicine · 2025Article
- Contrasting rule and machine learning based digital self triage systems in the USA.NPJ digital medicine · 2024Review
- Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics.JMIR formative research · 2024Article
- Article
- Existing Barriers Faced by and Future Design Recommendations for Direct-to-Consumer Health Care Artificial Intelligence Apps: Scoping Review.Journal of medical Internet research · 2023Article
- Assessment of a Digital Symptom Checker Tool's Accuracy in Suggesting Reproductive Health Conditions: Clinical Vignettes Study.JMIR mHealth and uHealth · 2023Article
- Empowering Preventive Care withHealthcare (Basel, Switzerland) · 2023Article
- Characteristics of Users and Nonusers of Symptom Checkers in Germany: Cross-Sectional Survey Study.Journal of medical Internet research · 2023Article
- Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study.JMIR mHealth and uHealth · 2022Observational
- Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation.Journal of medical Internet research · 2022Article
- Determinants of Laypersons' Trust in Medical Decision Aids: Randomized Controlled Trial.JMIR human factors · 2022Article
- Periodic Manual Algorithm Updates and Generalizability: A Developer's Response. Comment on "Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study".Journal of medical Internet research · 2021Article
- Author's Reply to: Periodic Manual Algorithm Updates and Generalizability: A Developer's Response. Comment on "Evaluation of Four Artificial Intelligence-Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study".Journal of medical Internet research · 2021Article
- Constructing High-Fidelity Phenotype Knowledge Graphs for Infectious Diseases With a Fine-Grained Semantic Information Model: Development and Usability Study.Journal of medical Internet research · 2021Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundConsumer-oriented mobile self-diagnosis apps have been developed using undisclosed algorithms, presumably based on machine learning and other artificial intelligence (AI) technologies. The US Food and Drug Administration now discerns apps with learning AI algorithms from those with stable ones and treats the former as medical devices. To the author's knowledge, no self-diagnosis app testing has been performed in the field of ophthalmology so far.
objectiveThe objective of this study was to test apps that were previously mentioned in the scientific literature on a set of diagnoses in a deliberate time interval, comparing the results and looking for differences that hint at "nonlocked" learning algorithms.
methodsFour apps from the literature were chosen (Ada, Babylon, Buoy, and Your.MD). A set of three ophthalmology diagnoses (glaucoma, retinal tear, dry eye syndrome) representing three levels of urgency was used to simultaneously test the apps' diagnostic efficiency and treatment recommendations in this specialty. Two years was the chosen time interval between the tests (2018 and 2020). Scores were awarded by one evaluating physician using a defined scheme.
resultsTwo apps (Ada and Your.MD) received significantly higher scores than the other two. All apps either worsened in their results between 2018 and 2020 or remained unchanged at a low level. The variation in the results over time indicates "nonlocked" learning algorithms using AI technologies. None of the apps provided correct diagnoses and treatment recommendations for all three diagnoses in 2020. Two apps (Babylon and Your.MD) asked significantly fewer questions than the other two (P<.001).
conclusions"Nonlocked" algorithms are used by self-diagnosis apps. The diagnostic efficiency of the tested apps seems to worsen over time, with some apps being more capable than others. Systematic studies on a wider scale are necessary for health care providers and patients to correctly assess the safety and efficacy of such apps and for correct classification by health care regulating authorities.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.