ArticleJMIR formative research2019
Accuracy of a Chatbot (Ada) in the Diagnosis of Mental Disorders: Comparative Case Study With Lay and Expert Users.
Article in JMIR formative research, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 53 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
53 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Challenges in evaluating the accuracy of AI-containing digital triage systems: A systematic review.PloS one · 2022Pooled it
- Changes to the Psychiatric Chatbot Landscape: A Systematic Review of Conversational Agents in Serious Mental Illness: Changements du paysage psychiatrique des chatbots: une revue systématique des agents conversationnels dans la maladie mentale sérieuse.Canadian journal of psychiatry. Revue canadienne de psychiatrie · 2021Pooled it
- Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial.Journal of medical Internet research · 2024Trial
- Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students.Arthritis research & therapy · 2021Trial
- Evaluating digital triage symptom checker with historical triage-related adverse events.Scandinavian journal of primary health care · 2026Article
- Enhancing patient care with AI agents: integrating advanced chatbot technologies for improved healthcare delivery.BMC medical informatics and decision making · 2026Article
- A scoping review of the use of artificial intelligence as a psychological assessment tool.Translational psychiatry · 2026Article
- The last human touch: why AI doesn't belong in life's defining moments.BMC medical ethics · 2026Article
- Effects of action instruction source on the sense of agency: humans vs. artificial agents.Experimental brain research · 2026Article
- "But it sounded confident": the role of accuracy, tone, and disclaimers in users' medical decision-making.Frontiers in artificial intelligence · 2026Article
- A Prompt Engineering Framework for Large Language Model-Based Mental Health Chatbots: Conceptual Framework.JMIR mental health · 2025Article
- Evaluating privacy policies of AI-powered mHealth iOS applications.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Artificial intelligence in healthcare and medicine: clinical applications, therapeutic advances, and future perspectives.European journal of medical research · 2025Review
- Artificial Intelligence in Health Promotion and Disease Reduction: Rapid Review.Journal of medical Internet research · 2025Review
- Novel Blended Learning on Artificial Intelligence for Medical Students: Qualitative Interview Study.JMIR medical education · 2025Article
- Revolutionizing Medicine: Chatbots as Catalysts for Improved Diagnosis, Treatment, and Patient Support.Cureus · 2025Review
- Increasing the Impact and Value of Laboratory Medicine Through Effective and AI-Assisted Communication.EJIFCC · 2025Review
- Developing Effective Frameworks for Large Language Model-Based Medical Chatbots: Insights From Radiotherapy Education With ChatGPT.JMIR cancer · 2025Article
- Strategic technological innovation through ChatMu: transforming information accessibility in Muhammadiyah.Frontiers in artificial intelligence · 2025Article
- Contrasting rule and machine learning based digital self triage systems in the USA.NPJ digital medicine · 2024Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundHealth apps for the screening and diagnosis of mental disorders have emerged in recent years on various levels (eg, patients, practitioners, and public health system). However, the diagnostic quality of these apps has not been (sufficiently) tested so far.
objectiveThe objective of this pilot study was to investigate the diagnostic quality of a health app for a broad spectrum of mental disorders and its dependency on expert knowledge.
methodsTwo psychotherapists, two psychology students, and two laypersons each read 20 case vignettes with a broad spectrum of mental disorders. They used a health app (Ada-Your Health Guide) to get a diagnosis by entering the symptoms. Interrater reliabilities were computed between the diagnoses of the case vignettes and the results of the app for each user group.
resultsOverall, there was a moderate diagnostic agreement (kappa=0.64) between the results of the app and the case vignettes for mental disorders in adulthood and a low diagnostic agreement (kappa=0.40) for mental disorders in childhood and adolescence. When psychotherapists applied the app, there was a good diagnostic agreement (kappa=0.78) regarding mental disorders in adulthood. The diagnostic agreement was moderate (kappa=0.55/0.60) for students and laypersons. For mental disorders in childhood and adolescence, a moderate diagnostic quality was found when psychotherapists (kappa=0.53) and students (kappa=0.41) used the app, whereas the quality was low for laypersons (kappa=0.29). On average, the app required 34 questions to be answered and 7 min to complete.
conclusionsThe health app investigated here can represent an efficient diagnostic screening or help function for mental disorders in adulthood and has the potential to support especially diagnosticians in their work in various ways. The results of this pilot study provide a first indication that the diagnostic accuracy is user dependent and improvements in the app are needed especially for mental disorders in childhood and adolescence.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.