ArticleJournal of the American Medical Informatics Association : JAMIA2024
The potential and pitfalls of using a large language model such as ChatGPT, GPT-4, or LLaMA as a clinical assistant.
Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 43 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
43 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Performance and improvement strategies for adapting generative large language models for electronic health record applications: A systematic review.International journal of medical informatics · 2026Pooled it
- Comparing Diagnostic Accuracy of Clinical Professionals and Large Language Models: Systematic Review and Meta-Analysis.JMIR medical informatics · 2025Pooled it
- Applications of digital health technologies and artificial intelligence algorithms in COPD: systematic review.BMC medical informatics and decision making · 2025Pooled it
- Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance.Journal of global health · 2026Article
- Evidence, use cases, and implementation safeguards of large language models in primary care.Communications medicine · 2026Review
- Large language models for interpretation of health checkup results.NPJ digital medicine · 2026Article
- The Role of Large Language Models in Hormonal Contraception Consultation.Journal of clinical medicine · 2026Article
- Open-Source Large Language Models Distilled DeepSeek-R1 Pose Challenges for On-Premises Clinical Deployment in Medical Diagnosis: A Comparative Study of Performance.Journal of medical systems · 2026Article
- A Dataset for Evaluating Large Language Models on Chinese National Medical Licensing Examinations.Scientific data · 2026Article
- Benchmarking GPT-5, LLaMA, and Mistral for Clinical Named Entity Recognition in Ophthalmology Progress Notes.Translational vision science & technology · 2026Article
- Empowering liver cancer diagnosis and treatment with foundation models: technological innovation and clinical practice.Clinical and experimental medicine · 2026Review
- Systematic literature review and narrative synthesis of the use of natural language processing to triage outpatient referrals.Frontiers in health services · 2026Review
- Large Language Models and Primary Care: A Scoping Review.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2026Article
- Comparison of local large language models for extraction of signs and symptoms data from electronic health records.PloS one · 2026Article
- Evaluating few-shot prompting for spectrogram-based lung sound classification using a multimodal language model.PLOS digital health · 2026Article
- Artificial Intelligence Applications in Chronic Obstructive Pulmonary Disease: A Global Scoping Review of Diagnostic, Symptom-Based, and Outcome Prediction Approaches.Biomedicines · 2025Review
- Evaluating large language models' performance in answering common questions on drug-induced liver injury.JHEP reports : innovation in hepatology · 2025Article
- Using large language models to automate summarization of CT simulation orders in radiation oncology.Journal of applied clinical medical physics · 2025Article
- Comparison of rule- and large language model-based phenotype extraction from clinical notes for neurofibromatosis type 1.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- Performance of Large Language Models in Diagnosing Rare Hematologic Diseases and the Impact of Their Diagnostic Outputs on Physicians: Combined Retrospective and Prospective Study.Journal of medical Internet research · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
Abstract
objectivesThis study aims to evaluate the utility of large language models (LLMs) in healthcare, focusing on their applications in enhancing patient care through improved diagnostic, decision-making processes, and as ancillary tools for healthcare professionals. MATERIALS AND
methodsWe evaluated ChatGPT, GPT-4, and LLaMA in identifying patients with specific diseases using gold-labeled Electronic Health Records (EHRs) from the MIMIC-III database, covering three prevalent diseases-Chronic Obstructive Pulmonary Disease (COPD), Chronic Kidney Disease (CKD)-along with the rare condition, Primary Biliary Cirrhosis (PBC), and the hard-to-diagnose condition Cancer Cachexia.
resultsIn patient identification, GPT-4 had near similar or better performance compared to the corresponding disease-specific Machine Learning models (F1-score ≥ 85%) on COPD, CKD, and PBC. GPT-4 excelled in the PBC use case, achieving a 4.23% higher F1-score compared to disease-specific "Traditional Machine Learning" models. ChatGPT and LLaMA3 demonstrated lower performance than GPT-4 across all diseases and almost all metrics. Few-shot prompts also help ChatGPT, GPT-4, and LLaMA3 achieve higher precision and specificity but lower sensitivity and Negative Predictive Value. DISCUSSION: The study highlights the potential and limitations of LLMs in healthcare. Issues with errors, explanatory limitations and ethical concerns like data privacy and model transparency suggest that these models would be supplementary tools in clinical settings. Future studies should improve training datasets and model designs for LLMs to gain better utility in healthcare.
conclusionThe study shows that LLMs have the potential to assist clinicians for tasks such as patient identification but false positives and false negatives must be mitigated before LLMs are adequate for real-world clinical assistance.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.