ArticleNature medicine2024
Evaluation and mitigation of the limitations of large language models in clinical decision-making.
Article in Nature medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 344 papers, 4 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
344 citing papers in PubMed, 4 syntheses or guidelines pooled it.
- Fine-Tuning, Retrieval-Augmented Generation, and Hybrid Adaptation of Language Models for Clinical Decision-Making in Health Care: Systematic Review.Journal of medical Internet research · 2026Pooled it
- Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.International dental journal · 2026Pooled it
- Generative large language models in the clinical management of Alzheimer's disease and mild cognitive impairment.Neurological sciences : official journal of the Italian Neurological Society and of the Italian Society of Clinical Neurophysiology · 2026Pooled it
- Large Language Models in Colorectal Cancer Care and Clinical Decision Support: Systematic Review.Journal of medical Internet research · 2026Pooled it
- Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial.Nature medicine · 2026Trial
- A cognitive layer architecture to support large-language model performance in psychotherapy interactions.Nature medicine · 2026Trial
- An LLM chatbot to facilitate primary-to-specialist care transitions: a randomized controlled trial.Nature medicine · 2026Trial
- A large language model for complex cardiology care.Nature medicine · 2026Trial
- Ontological deskilling: clinical expertise and the changing disease object in precision medicine.Medicine, health care, and philosophy · 2026Article
- LLM-supported self-diagnosis and self-treatment of familial Mediterranean fever: a case report.EULAR rheumatology open · 2026Article
- Determinants of Artificial Intelligence Adoption in Aesthetic Practice: A Cross-Sectional Study.Aesthetic plastic surgery · 2026Article
- Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.Rheumatology international · 2026Article
- On-premise medical AI agents for reliable clinical decision-making.Nature medicine · 2026Article
- Large language models' performance in answering common patient questions about colonoscopy: an expert-based evaluation.BMC gastroenterology · 2026Observational
- Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury.Journal of experimental orthopaedics · 2026Article
- Prospective evaluation of a large language model clinical decision support system in the emergency department.Nature medicine · 2026Article
- Performance of large language models in ambiguous treatment scenarios of early breast cancer.Breast (Edinburgh, Scotland) · 2026Article
- Agentic AI in Medicine: Challenges for Responsible Development and the Case for Clinical Testing Harnesses.Annals of biomedical engineering · 2026Article
- The Alignment Paradox of Medical Large Language Models in Infertility Care: Decoupling Algorithmic Improvement From Clinical Decision-Making Quality.Journal of medical Internet research · 2026Article
- [Application effect of LLMs as decision-support tools in emergency burn treatment].Zhonghua shao shang yu chuang mian xiu fu za zhi · 2026Article
284 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Clinical decision-making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from artificial intelligence solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills necessary for deployment in a realistic clinical decision-making environment, including gathering information, adhering to guidelines, and integrating into clinical workflows. Here we have created a curated dataset based on the Medical Information Mart for Intensive Care database spanning 2,400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for autonomous clinical decision-making while providing a dataset and framework to guide future studies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.