SynthesisBMC medical informatics and decision making2025
A systematic review of large language model (LLM) evaluations in clinical medicine.
Synthesis in BMC medical informatics and decision making, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 165 papers, 7 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
165 citing papers in PubMed, 7 syntheses or guidelines pooled it.
- [Empowering narrative medicine with artificial intelligence: a new perspective in pediatric medical education].Zhongguo dang dai er ke za zhi = Chinese journal of contemporary pediatrics · 2026Pooled it
- Accuracy of the large language model ChatGPT in adult emergency department triage: a systematic review and meta-analysis.BMC emergency medicine · 2026Pooled it
- Testing and evaluation of generative large language models in electronic health record applications: a systematic review.Journal of the American Medical Informatics Association : JAMIA · 2026Pooled it
- Large language models in healthcare: applications, evaluation frameworks, and governance pathways - a scoping review and multidimensional framework.Frontiers in digital health · 2026Pooled it
- Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks.Journal of medical Internet research · 2025Pooled it
- A systematic review of ethical considerations of large language models in healthcare and medicine.Frontiers in digital health · 2025Pooled it
- Performance of large language models in medical licensing examinations: a systematic review and meta-analysis.Journal of educational evaluation for health professions · 2025Pooled it
- Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial.Nature medicine · 2026Trial
- Integrating Large Language Models Into Trauma Education for Medical Students: Randomized Controlled Pilot Trial.JMIR medical education · 2026Trial
- A large language model for complex cardiology care.Nature medicine · 2026Trial
- Assessing the performance of large language models when used to determine ASA status of cats and dogs and generate anaesthetic protocols.The Veterinary record · 2026Article
- Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury.Journal of experimental orthopaedics · 2026Article
- Attitudes Toward Large Language Models in Health Care and Preferences for Their Adoption and Oversight Among Health Care Professionals: Cross-Sectional Survey.Journal of medical Internet research · 2026Article
- EmmaEmb: A quantitative framework for analyzing embedding spaces in molecular biology.Patterns (New York, N.Y.) · 2026Article
- Artificial Intelligence for Alzheimer's Disease Diagnosis: From Traditional Machine Learning to Large Language Models.Biosensors · 2026Review
- Privacy Leakage in Federated Learning in Radiology Reports: Comparative Evaluation of Tokenizer and Batch-Size Privacy Risks.JMIR medical informatics · 2026Article
- Large Language Models for Heterogeneous Data Mining in Liver Disease: Framework Development and Retrospective Validation Study.Journal of medical Internet research · 2026Article
- Evaluating the Accuracy, Empathy, and Readability of Generative AI Versus Registered Nurses in Discharge Planning: A Vignette-Based Study.Nursing open · 2026Observational
- Novelty to necessity: large language models in clinical practice-a call to action for physicians.Internal and emergency medicine · 2026Review
- A Rasch-based analysis comparing the performance of updated multimodal large language models and surgeons on the Japanese surgical specialist examination.Surgery today · 2026Article
105 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge Language Models (LLMs), advanced AI tools based on transformer architectures, demonstrate significant potential in clinical medicine by enhancing decision support, diagnostics, and medical education. However, their integration into clinical workflows requires rigorous evaluation to ensure reliability, safety, and ethical alignment.
objectiveThis systematic review examines the evaluation parameters and methodologies applied to LLMs in clinical medicine, highlighting their capabilities, limitations, and application trends.
methodsA comprehensive review of the literature was conducted across PubMed, Scopus, Web of Science, IEEE Xplore, and arXiv databases, encompassing both peer-reviewed and preprint studies. Studies were screened against predefined inclusion and exclusion criteria to identify original research evaluating LLM performance in medical contexts.
resultsThe results reveal a growing interest in leveraging LLM tools in clinical settings, with 761 studies meeting the inclusion criteria. While general-domain LLMs, particularly ChatGPT and GPT-4, dominated evaluations (93.55%), medical-domain LLMs accounted for only 6.45%. Accuracy emerged as the most commonly assessed parameter (21.78%). Despite these advancements, the evidence base highlights certain limitations and biases across the included studies, emphasizing the need for careful interpretation and robust evaluation frameworks.
conclusionsThe exponential growth in LLM research underscores their transformative potential in healthcare. However, addressing challenges such as ethical risks, evaluation variability, and underrepresentation of critical specialties will be essential. Future efforts should prioritize standardized frameworks to ensure safe, effective, and equitable LLM integration in clinical practice.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.