SynthesisBMC medical informatics and decision making2025
External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis.
Synthesis in BMC medical informatics and decision making, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 33 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
33 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Artificial Intelligence for Diagnosis, Risk Stratification, and Prognosis of Neuroblastoma - A Systematic Review and Meta-Analysis.Current oncology reports · 2026Pooled it
- External validation of AI-based scoring systems in the ICU: a systematic review and meta-analysis.BMC medical informatics and decision making · 2025Pooled it
- A systematic review on the generative AI applications in human medical genetics.Frontiers in genetics · 2025Pooled it
- Artificial Intelligence for Predicting Postoperative Outcomes in Anterior Segment and Ocular Adnexa Surgery: A Systematic Review.Ophthalmology science · 2026Review
- Reimagining the Intensive Care Unit as a Complex Adaptive System: A Systems Thinking Perspective.Cureus · 2026Review
- Process Improvement Before Artificial Intelligence and Automation: Building Trust With the Understand-Transform-Sustain Framework.Mayo Clinic proceedings. Innovations, quality & outcomes · 2026Article
- Bias and Fairness Across the Healthcare AI Lifecycle: A Clinician-Oriented Review.Balkan medical journal · 2026Review
- Toward universal representations of the living: Physiological invariance for transportable medical AI.PLOS digital health · 2026Review
- From Electrocardiography to the Catheterization Laboratory: A Multimodal Artificial Intelligence Framework for Acute Coronary Syndrome Detection and Risk Stratification.Diagnostics (Basel, Switzerland) · 2026Review
- Artificial intelligence in chronic kidney disease: Early detection, risk prediction, and personalized treatment strategies.World journal of nephrology · 2026Review
- Predicting ICU in-hospital mortality from text-encoded structured EHR data using adaptive transformer layer fusion.iScience · 2026Article
- Beyond Model Development in Healthcare AI: Post-Development Robustness, Post-Deployment Monitoring, and Lifecycle Governance-A Scoping Review of Reviews.Healthcare (Basel, Switzerland) · 2026Review
- Multi-source data integration through pooling and transfer learning improves generalizability and specialization of deep learning models for ICU mortality and length of stay prediction: a four-database external validation study.Critical care (London, England) · 2026Article
- Artificial Intelligence For 6P Medicine: Consolidating AI Needs of Predictive, Preventive, Personalized, Participatory, Precision, and Public Health Trajectories.Journal of medical systems · 2026Review
- The Impact of Evaluation Strategy on Sepsis Prediction Model Performance Metrics in Intensive Care Data: Retrospective Cohort Study.Journal of medical Internet research · 2026Observational
- Scalable and Privacy-Conscious End-to-End Processing of Large-Scale Clinical Data for Precision Medicine: Empirical Evaluation Study.JMIR medical informatics · 2026Article
- Machine Learning Model for Sepsis Prediction in Prolonged and Chronic Critical Illness: Development and Validation Using Retrospective Real-World ICU Data.Journal of clinical medicine · 2026Article
- Combining machine learning and physiological network models for sepsis prediction.Frontiers in network physiology · 2026Article
- Artificial intelligence in experimental and clinical in vitro analysis: applications, limitations, and future directions.Biotechnology notes (Amsterdam, Netherlands) · 2026Review
- Artificial Intelligence in Intensive Care: An Overview of Systematic Reviews with Clinical Maturity and Readiness Mapping.Journal of clinical medicine · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundMachine learning (ML) is increasingly used to predict clinical deterioration in intensive care unit (ICU) patients through scoring systems. Although promising, such algorithms often overfit their training cohort and perform worse at new hospitals. Thus, external validation is a critical - but frequently overlooked - step to establish the reliability of predicted risk scores to translate them into clinical practice. We systematically reviewed how regularly external validation of ML-based risk scores is performed and how their performance changed in external data.
methodsWe searched MEDLINE, Web of Science, and arXiv for studies using ML to predict deterioration of ICU patients from routine data. We included primary research published in English before December 2023. We summarised how many studies were externally validated, assessing differences over time, by outcome, and by data source. For validated studies, we evaluated the change in area under the receiver operating characteristic (AUROC) attributable to external validation using linear mixed-effects models.
resultsWe included 572 studies, of which 84 (14.7%) were externally validated, increasing to 23.9% by 2023. Validated studies made disproportionate use of open-source data, with two well-known US datasets (MIMIC and eICU) accounting for 83.3% of studies. On average, AUROC was reduced by -0.037 (95% CI -0.052 to -0.027) in external data, with more than 0.05 reduction in 49.5% of studies. DISCUSSION: External validation, although increasing, remains uncommon. Performance was generally lower in external data, questioning the reliability of some recently proposed ML-based scores. Interpretation of the results was challenged by an overreliance on the same few datasets, implicit differences in case mix, and exclusive use of AUROC.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.