ArticleJournal of the American Medical Informatics Association : JAMIA2024
Evaluating the accuracy of a state-of-the-art large language model for prediction of admissions from the emergency room.
Article in Journal of the American Medical Informatics Association : JAMIA, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 39 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
39 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Performance and improvement strategies for adapting generative large language models for electronic health record applications: A systematic review.International journal of medical informatics · 2026Pooled it
- Improving large language model applications in biomedicine with retrieval-augmented generation: a systematic review, meta-analysis, and clinical development guidelines.Journal of the American Medical Informatics Association : JAMIA · 2025Pooled it
- Hierarchical multi-label structuring of Japanese SOAP clinical notes with large language models.Scientific reports · 2026Article
- From nonagentic large language models to multiagent systems in emergency medicine: a scoping review.Clinical and experimental emergency medicine · 2026Article
- Simulation of covariate and concept drift in machine learning hospital admission prediction from emergency triage.NPJ digital medicine · 2026Article
- Leveraging Large Language Models to Integrate Clinical Knowledge and Machine Learning Predictions for Lymph Node Metastasis Prediction: Development of a Knowledge-Augmented Framework.JMIR medical informatics · 2026Article
- Explainable AI for mental health emergency returns: integrating large language models with predictive modeling.JAMIA open · 2026Article
- Large language models are poor clinical administrators: An evaluation of structured queries in real-world electronic health records.PLOS digital health · 2026Article
- Interpretable depressive symptoms screening via statistical reasoning-augmented large language models using wearable and environmental data.Scientific reports · 2026Article
- Comparative performance of LLMs and machine learning in predicting complications after percutaneous kyphoplasty for osteoporotic vertebral compression fractures.NPJ digital medicine · 2026Article
- Predicting emergency department disposition using machine learning and large language models to support proactive capacity management: a multicenter retrospective study.BMC emergency medicine · 2026Article
- Evaluation of Large Language Models as Emergency Department Revisit Predictors.Pacific Symposium on Biocomputing. Pacific Symposium on Biocomputing · 2026Article
- Enhancing the prediction of hospital discharge disposition with extraction-based language model classification.npj health systems · 2026Article
- Authors' Reply: Predicting the Emergency Department Patient Journey Using a Machine Learning Approach.JMIR AI · 2025Article
- A scalable framework for benchmark embedding models in semantic health-care tasks.Journal of the American Medical Informatics Association : JAMIA · 2025Article
- AI-generated neurology consultation summaries improve efficiency and reduce documentation burden in the emergency department.Scientific reports · 2025Article
- Evaluating the performance of large language models versus human researchers on real world complex medical queries.Scientific reports · 2025Article
- AI Based Clinical Decision-Making Tool for Neurologists in the Emergency Department.Journal of clinical medicine · 2025Article
- Comparing Machine Learning and Nurse Predictions for Hospital Admissions in a Multisite Emergency Care System.Mayo Clinic proceedings. Digital health · 2025Article
- Clinical applications and limitations of large language models in nephrology: a systematic review.Clinical kidney journal · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Funding
Abstract
backgroundArtificial intelligence (AI) and large language models (LLMs) can play a critical role in emergency room operations by augmenting decision-making about patient admission. However, there are no studies for LLMs using real-world data and scenarios, in comparison to and being informed by traditional supervised machine learning (ML) models. We evaluated the performance of GPT-4 for predicting patient admissions from emergency department (ED) visits. We compared performance to traditional ML models both naively and when informed by few-shot examples and/or numerical probabilities.
methodsWe conducted a retrospective study using electronic health records across 7 NYC hospitals. We trained Bio-Clinical-BERT and XGBoost (XGB) models on unstructured and structured data, respectively, and created an ensemble model reflecting ML performance. We then assessed GPT-4 capabilities in many scenarios: through Zero-shot, Few-shot with and without retrieval-augmented generation (RAG), and with and without ML numerical probabilities.
resultsThe Ensemble ML model achieved an area under the receiver operating characteristic curve (AUC) of 0.88, an area under the precision-recall curve (AUPRC) of 0.72 and an accuracy of 82.9%. The naïve GPT-4's performance (0.79 AUC, 0.48 AUPRC, and 77.5% accuracy) showed substantial improvement when given limited, relevant data to learn from (ie, RAG) and underlying ML probabilities (0.87 AUC, 0.71 AUPRC, and 83.1% accuracy). Interestingly, RAG alone boosted performance to near peak levels (0.82 AUC, 0.56 AUPRC, and 81.3% accuracy).
conclusionsThe naïve LLM had limited performance but showed significant improvement in predicting ED admissions when supplemented with real-world examples to learn from, particularly through RAG, and/or numerical probabilities from traditional ML models. Its peak performance, although slightly lower than the pure ML model, is noteworthy given its potential for providing reasoning behind predictions. Further refinement of LLMs with real-world data is necessary for successful integration as decision-support tools in care settings.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.