ArticleJMIR formative research2024
Evaluating ChatGPT-4's Accuracy in Identifying Final Diagnoses Within Differential Diagnoses Compared With Those of Physicians: Experimental Study for Diagnostic Cases.
Article in JMIR formative research, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 12 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
12 citing papers in PubMed.
- A Cautious Integration With AI in the Clinic: A Standardized-Patient Pilot Study of ChatGPT's Reliability in Hamilton Depression Rating Scale Scoring.Alpha psychiatry · 2026Article
- Large language models in respiratory care: safety and clinical integration.ERJ open research · 2026Article
- ChatGPT in Orthopedic Trauma: Consistency, Accuracy, and Agreement With Textbook and Expert Opinion.Cureus · 2026Article
- Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications - A Systematic Review.Journal of healthcare leadership · 2026Review
- Article
- Evaluating Large Language Models in Interpreting MRI Reports and Recommending Treatment for Vestibular Schwannoma.Diagnostics (Basel, Switzerland) · 2025Article
- Virtual case reasoning and AI-assisted diagnostic instruction: an empirical study based on body interact and large language models.BMC medical education · 2025Article
- Rapidly Benchmarking Large Language Models for Diagnosing Comorbid Patients: Comparative Study Leveraging the LLM-as-a-Judge Method.JMIRx med · 2025Article
- Can AI match emergency physicians in managing common emergency cases? A comparative performance evaluation.BMC emergency medicine · 2025Article
- Computerized diagnostic decision support systems-Isabel Pro versus ChatGPT-4 part II.JAMIA open · 2025Article
- Comparative evaluation of artificial intelligence models GPT-4 and GPT-3.5 in clinical decision-making in sports surgery and physiotherapy: a cross-sectional study.BMC medical informatics and decision making · 2025Article
- Transforming clinical reasoning-the role of AI in supporting human cognitive limitations.Frontiers in digital health · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThe potential of artificial intelligence (AI) chatbots, particularly ChatGPT with GPT-4 (OpenAI), in assisting with medical diagnosis is an emerging research area. However, it is not yet clear how well AI chatbots can evaluate whether the final diagnosis is included in differential diagnosis lists.
objectiveThis study aims to assess the capability of GPT-4 in identifying the final diagnosis from differential-diagnosis lists and to compare its performance with that of physicians for case report series.
methodsWe used a database of differential-diagnosis lists from case reports in the American Journal of Case Reports, corresponding to final diagnoses. These lists were generated by 3 AI systems: GPT-4, Google Bard (currently Google Gemini), and Large Language Models by Meta AI 2 (LLaMA2). The primary outcome was focused on whether GPT-4's evaluations identified the final diagnosis within these lists. None of these AIs received additional medical training or reinforcement. For comparison, 2 independent physicians also evaluated the lists, with any inconsistencies resolved by another physician.
resultsThe 3 AIs generated a total of 1176 differential diagnosis lists from 392 case descriptions. GPT-4's evaluations concurred with those of the physicians in 966 out of 1176 lists (82.1%). The Cohen κ coefficient was 0.63 (95% CI 0.56-0.69), indicating a fair to good agreement between GPT-4 and the physicians' evaluations.
conclusionsGPT-4 demonstrated a fair to good agreement in identifying the final diagnosis from differential-diagnosis lists, comparable to physicians for case report series. Its ability to compare differential diagnosis lists with final diagnoses suggests its potential to aid clinical decision-making support through diagnostic feedback. While GPT-4 showed a fair to good agreement for evaluation, its application in real-world scenarios and further validation in diverse clinical environments are essential to fully understand its utility in the diagnostic process.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.