SynthesisJournal of the American Medical Informatics Association : JAMIA2026
Testing and evaluation of generative large language models in electronic health record applications: a systematic review.
Synthesis in Journal of the American Medical Informatics Association : JAMIA, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
14 citing papers in PubMed.
- Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial.Nature medicine · 2026Trial
- From promise to practice: digital health through the lens of neonatal care.Lancet regional health. Americas · 2026Review
- A Patient Simulation Framework for Risk Assessment of Conversational Health Care AI: Development and Evaluation Study.JMIR AI · 2026Article
- Large Language Models for Clinical Data Extraction in Workplace Injury Rehabilitation: Protocol for a Retrospective Pilot Study of Accuracy, Fairness, and Methodological Considerations in Workers' Compensation Medical Chart Review.JMIR research protocols · 2026Article
- Generative large language models in medicine: a scoping review of recent methodological advances.npj health systems · 2026Review
- Precision pharmacology: deep learning infused ontological framework with E-GRU enhancement for tailored medicine prescriptions.Scientific reports · 2026Article
- Precision Grounding: augmenting large language models with evidence-based databases for trustworthy genetic variant summarization.International journal of medical informatics · 2026Article
- A Language Model for Pediatric Occupational Therapy Documentation: Model Development and Pilot Study.JMIR AI · 2026Article
- Structured taxonomy and framework for developing medical benchmark in large language models derived from scoping review.NPJ digital medicine · 2026Article
- Large Language Models Using Clinical Text in Pediatrics: A Scoping Review.JAMA network open · 2026Article
- Textbook-level medical knowledge in large language models: comparative evaluation using Japanese National Medical Examination.BMC medical informatics and decision making · 2026Article
- AI-driven healthcare: a trend toward better healthcare or the emergence of public health burden.Frontiers in digital health · 2026Article
- Safety of a large language model-based clinical decision support system in African primary healthcare.Nature health · 2026Article
- Exploring multimodal large language models on transthoracic Echocardiogram (TTE) tasks for cardiovascular decision support.Journal of biomedical informatics · 2025Article
Corrections and comments
- Update of
Authors and funding
14 authors.
Funding
Abstract
backgroundThe use of generative large language models (LLMs) with electronic health record (EHR) data is rapidly expanding to support clinical and research tasks. This systematic review characterizes the clinical fields and use cases that have been studied and evaluated to date.
methodsWe followed the Preferred Reporting Items for Systematic Review and Meta-Analyses guidelines to conduct a systematic review of articles from PubMed and Web of Science published between January 1, 2023, and November 9, 2024. Studies were included if they used generative LLMs to analyze real-world EHR data and reported quantitative performance evaluations. Through data extraction, we identified clinical specialties and tasks for each included article, and summarized evaluation methods.
resultsOf the 18 735 articles retrieved, 196 met our criteria. Most studies focused on radiology (26.0%), oncology (10.7%), and emergency medicine (6.6%). Regarding clinical tasks, clinical decision support made up the largest proportion of studies (62.2%), while summarizations and patient communications made up the smallest, at 5.6% and 5.1%, respectively. In addition, GPT-4 and GPT-3.5 were the most commonly used generative LLMs, appearing in 60.2% and 57.7% of studies, respectively. Across these studies, we identified 22 unique non-NLP metrics and 35 unique NLP metrics. While NLP metrics offer greater scalability, none demonstrated a strong correlation with gold-standard human evaluations.
conclusionOur findings highlight the need to evaluate generative LLMs on EHR data across a broader range of clinical specialties and tasks, as well as the urgent need for standardized, scalable, and clinically meaningful evaluation frameworks.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.