ArticleJournal of the American Medical Informatics Association : JAMIA2025
Large language models are less effective at clinical prediction tasks than locally trained machine learning models.
Article in Journal of the American Medical Informatics Association : JAMIA, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 27 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
27 citing papers in PubMed.
- A Multidisciplinary Team-Based Large Language Model Framework for Predicting Postoperative Neurological Complications in Acute Type A Aortic Dissection: Model Development and Validation Study.Journal of medical Internet research · 2026Article
- Privacy and Security Throughout the Health Data Life Cycle: From Primary Care to Research Networks.Annual review of biomedical data science · 2026Review
- PHO-Agents: A Large Language Model-Powered Multi-Agent System for Predicting Health Outcomes.medRxiv : the preprint server for health sciences · 2026Article
- Functional Outcome Prediction in Young Adults With Mental Health Symptoms Using Machine Learning and Large Language Models: Longitudinal Observational Study.JMIR mental health · 2026Observational
- An Evaluation of Pretrained Generative Models for Augmenting Small Health Data: Comparative Modeling Study.Journal of medical Internet research · 2026Article
- A real-world feasibility evaluation of LLM-based clinical prediction: emergency department return visit admission across two academic medical centers.Research square · 2026Article
- Benchmarking reliability and calibration of LLMs for multi-cancer early detection test communication.JAMIA open · 2026Article
- Spatial and telehealth accessibility to eating disorder treatment in the United States: evidence from registry and LLM-augmented data.International journal of health geographics · 2026Article
- Pitfalls and risks of generative AI in machine learning.Patterns (New York, N.Y.) · 2026Review
- Community medical centers struggle to produce well-calibrated clinical prediction models: Data augmentation can help.Journal of biomedical informatics · 2026Article
- Disease Risk Prediction Using Structured EHR Data: Can Generalist Large Language Models Match Specialized Clinical Foundation Models? A Comparative Evaluation with Fine-Tuning.medRxiv : the preprint server for health sciences · 2026Article
- Classifying American Society of Anesthesiologists Physical Status With a Low-Rank-Adapted Large Language Model: Development and Validation Study.Journal of medical Internet research · 2026Article
- Increasing Large Language Model Accuracy for Care-Seeking Advice Using Prompts Reflecting Human Reasoning Strategies in the Real World: Validation Study.JMIR biomedical engineering · 2026Article
- ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical prediction tasks.NPJ digital medicine · 2026Article
- Comparative performance of LLMs and machine learning in predicting complications after percutaneous kyphoplasty for osteoporotic vertebral compression fractures.NPJ digital medicine · 2026Article
- A multimodal generative model for structured and unstructured electronic health records.npj health systems · 2026Article
- Article
- Traditional Cox regression outperforms large language models in predicting long-term progression of intermediate to advanced hepatocellular carcinoma.Frontiers in oncology · 2026Article
- Using generative artificial intelligence in clinical practice: a narrative review and proposed agenda for implementation.The Medical journal of Australia · 2025Review
- AI-augmented frameworks for enhancing Alzheimer's disease clinical trials: A memory clinic perspective.The journal of prevention of Alzheimer's disease · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors.
Funding
Abstract
objectivesTo determine the extent to which current large language models (LLMs) can serve as substitutes for traditional machine learning (ML) as clinical predictors using data from electronic health records (EHRs), we investigated various factors that can impact their adoption, including overall performance, calibration, fairness, and resilience to privacy protections that reduce data fidelity. MATERIALS AND
methodsWe evaluated GPT-3.5, GPT-4, and traditional ML (as gradient-boosting trees) on clinical prediction tasks in EHR data from Vanderbilt University Medical Center (VUMC) and MIMIC IV. We measured predictive performance with area under the receiver operating characteristic (AUROC) and model calibration using Brier Score. To evaluate the impact of data privacy protections, we assessed AUROC when demographic variables are generalized. We evaluated algorithmic fairness using equalized odds and statistical parity across race, sex, and age of patients. We also considered the impact of using in-context learning by incorporating labeled examples within the prompt.
resultsTraditional ML [AUROC: 0.847, 0.894 (VUMC, MIMIC)] substantially outperformed GPT-3.5 (AUROC: 0.537, 0.517) and GPT-4 (AUROC: 0.629, 0.602) (with and without in-context learning) in predictive performance and output probability calibration [Brier Score (ML vs GPT-3.5 vs GPT-4): 0.134 vs 0.384 vs 0.251, 0.042 vs 0.06 vs 0.219)]. DISCUSSION: Traditional ML is more robust than GPT-3.5 and GPT-4 in generalizing demographic information to protect privacy. GPT-4 is the fairest model according to our selected metrics but at the cost of poor model performance.
conclusionThese findings suggest that non-fine-tuned LLMs are less effective and robust than locally trained ML for clinical prediction tasks, but they are improving across releases.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.