ArticleNature communications2024
Evaluating the use of large language models to provide clinical recommendations in the Emergency Department.
Article in Nature communications, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07632859 (Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes), which is not on this map. Cited by 72 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Diagnostic Accuracy of Two Large Language Models Against a Blinded Specialist Consensus Standard in Turkish Emergency Department Notes: A Retrospective Study of 600 Cases
Who cites it
72 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Large Language Models in Colorectal Cancer Care and Clinical Decision Support: Systematic Review.Journal of medical Internet research · 2026Pooled it
- Performance and improvement strategies for adapting generative large language models for electronic health record applications: A systematic review.International journal of medical informatics · 2026Pooled it
- LungGPT: A unified multimodal system for interpretable diagnosis and clinical decision support of respiratory diseases.Cell reports. Medicine · 2026Article
- From assistant to collaborator: A systematic review of the evolution of artificial intelligence in end-stage renal disease care and management.PLOS digital health · 2026Article
- The Utility of Large Language Models to Assist With Emergency Triage Decisions Within Otolaryngology.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2026Article
- Searching the druggable genome using large language models.Bioinformatics (Oxford, England) · 2026Article
- Performance of Large Language Models in Oral Cancer Patient Education: An Evaluation of Reliability, Readability, and Patient Communication Quality.Oral health & preventive dentistry · 2026Article
- Prospective evaluation of a large language model clinical decision support system in the emergency department.Nature medicine · 2026Article
- DETERIO-LLM: enhancing traditional deterioration risk scores with clinical context and advanced reasoning.JAMIA open · 2026Article
- The Role of Large Language Models in Hormonal Contraception Consultation.Journal of clinical medicine · 2026Article
- From Promising Capabilities to Pervasive Bias: Assessing Large Language Models for Emergency Department Triage.Journal of healthcare informatics research · 2026Article
- Spine patient history-taking: a two phase study of a prompt-engineered large language model assistant.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2026Article
- A real-world feasibility evaluation of LLM-based clinical prediction: emergency department return visit admission across two academic medical centers.Research square · 2026Article
- Screening for Missed Opportunities for Diagnosis in the ED Using eTriggers and Large Language Models.JAMA network open · 2026Article
- Susceptibility of Large Language Models to User-Driven Factors in Medical Queries.Journal of healthcare informatics research · 2026Article
- Performance of Large Language Models in Answering Healthcare Delivery Questions: A Quantitative Cross-Sectional Study.Health science reports · 2026Article
- Large language models for post-discharge follow-up in erectile dysfunction care: a narrative review.Translational andrology and urology · 2026Review
- Application of Sparse Autoencoders to Enhance Mechanistic Interpretability of Large Language Models in Medicine.JMIR AI · 2026Article
- A workflow utilizing general-purpose large language models for efficient structuring and data mining of bone scintigraphy narratives.Scientific reports · 2026Article
- Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review.Journal of medical systems · 2026Article
12 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on byBeyond the2025
Authors and funding
4 authors.
Funding
Abstract
The release of GPT-4 and other large language models (LLMs) has the potential to transform healthcare. However, existing research evaluating LLM performance on real-world clinical notes is limited. Here, we conduct a highly-powered study to determine whether LLMs can provide clinical recommendations for three tasks (admission status, radiological investigation(s) request status, and antibiotic prescription status) using clinical notes from the Emergency Department. We randomly selected 10,000 Emergency Department visits to evaluate the accuracy of zero-shot, GPT-3.5-turbo- and GPT-4-turbo-generated clinical recommendations across four different prompting strategies. We found that both GPT-4-turbo and GPT-3.5-turbo performed poorly compared to a resident physician, with accuracy scores 8% and 24%, respectively, lower than physician on average. Both LLMs tended to be overly cautious in its recommendations, with high sensitivity at the cost of specificity. Our findings demonstrate that, while early evaluations of the clinical use of LLMs are promising, LLM performance must be significantly improved before their deployment as decision support systems for clinical recommendations and other complex tasks.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.