ArticleNPJ digital medicine2024
Distribution shift detection for the postmarket surveillance of medical AI algorithms: a retrospective simulation study.
Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 28 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
28 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Research on machine learning-based clinical prediction models: a bibliometric analysis.Frontiers in oncology · 2026Pooled it
- Refined Exclusion in Medical AI: Data Justice and Patient Safety Governance.Journal of medical Internet research · 2026Article
- Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation.IEEE transactions on bio-medical engineering · 2026Review
- AI-based augmentation of oncology clinical trials.Nature reviews. Clinical oncology · 2026Review
- PROACTIVE-AI: An Intelligent Dashboard for Assessing and Predicting of FDA-approved AI Software Performance in Real-World Clinical Scenarios.Research square · 2026Article
- Medical AI Agents for Clinical Decision Support: Viewpoint Using the Planning, Action, Reflection, and Memory (PARM) Analytical Lens.JMIR medical informatics · 2026Article
- Detecting Performance Drift in AI Models for Medical Image Analysis Using CUSUM Chart.Journal of imaging informatics in medicine · 2026Article
- Alignment of Policy, Practice, and Patient Safety for Trustworthy AI in Radiology.Radiology. Artificial intelligence · 2026Article
- Navigating AI and machine learning in cancer research: an end-to-end translational framework.Journal of translational medicine · 2026Review
- Beyond Validation: Operationalising Post-Deployment Surveillance of AI Medical Devices in Clinical Practice.Journal of medical systems · 2026Article
- Reliable Uncertainty Under Class Imbalance and Distribution Shift: Class-Conditional Conformal Prediction of Multiple Sclerosis.medRxiv : the preprint server for health sciences · 2026Article
- Fair and robust early readmission risk prediction from electronic health records via diffusion-based data augmentation and causal-invariant representation learning.Scientific reports · 2026Article
- Application of artificial intelligence in schizophrenia rehabilitation management: a systematic scoping review.Translational psychiatry · 2026Article
- Neuro-symbolic LLM Integration in Clinical Medicine: A Systematic Review and Taxonomy.Research square · 2026Article
- A roadmap for medical large language models: a review of foundations, applications, and challenges.Military Medical Research · 2026Review
- An End-to-End Overview of Clinical Speech AI.IEEE transactions on audio, speech, and language processing (2025) · 2026Article
- Foundation Model-Guided Synthetic EHR Release: Performance Enhancement with Privacy Preservation.AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science · 2026Article
- AI without representation is just inequity at scale: on the exportation of unrepresentative artificial intelligence models to the Global South.Frontiers in artificial intelligence · 2026Article
- Continuous assurance for AI-driven clinical decision support systems.Frontiers in artificial intelligence · 2026Review
- Tumor-anchored deep feature random forests for out-of-distribution detection in lung cancer segmentation.Transactions on machine learning research · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Distribution shifts remain a problem for the safe application of regulated medical AI systems, and may impact their real-world performance if undetected. Postmarket shifts can occur for example if algorithms developed on data from various acquisition settings and a heterogeneous population are predominantly applied in hospitals with lower quality data acquisition or other centre-specific acquisition factors, or where some ethnicities are over-represented. Therefore, distribution shift detection could be important for monitoring AI-based medical products during postmarket surveillance. We implemented and evaluated three deep-learning based shift detection techniques (classifier-based, deep kernel, and multiple univariate kolmogorov-smirnov tests) on simulated shifts in a dataset of 130'486 retinal images. We trained a deep learning classifier for diabetic retinopathy grading. We then simulated population shifts by changing the prevalence of patients' sex, ethnicity, and co-morbidities, and example acquisition shifts by changes in image quality. We observed classification subgroup performance disparities w.r.t. image quality, patient sex, ethnicity and co-morbidity presence. The sensitivity at detecting referable diabetic retinopathy ranged from 0.50 to 0.79 for different ethnicities. This motivates the need for detecting shifts after deployment. Classifier-based tests performed best overall, with perfect detection rates for quality and co-morbidity subgroup shifts at a sample size of 1000. It was the only method to detect shifts in patient sex, but required large sample sizes (
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.