ArticleNature communications2024
A framework for evaluating clinical artificial intelligence systems without ground-truth annotations.
Article in Nature communications, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 13 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
13 citing papers in PubMed.
- RCAF for patient-level thyroid ultrasound malignancy prediction under leakage-free evaluation and calibration.Scientific reports · 2026Article
- Frameworks encompassing intersectional perspective of artificial intelligence in healthcare. Scoping review.Public health in practice (Oxford, England) · 2026Review
- Application of Sparse Autoencoders to Enhance Mechanistic Interpretability of Large Language Models in Medicine.JMIR AI · 2026Article
- Reply to letter to the editor on "Forecasting Anesthetic Depth Using Auto-Regressive Transformer in Propofol Infusion during Induction Phase".Journal of anesthesia · 2026Article
- Clinical Plausibility in Large Language Model Robustness Testing for Medicine: A Scoping Review.Journal of medical systems · 2026Article
- Important Ethical, Technical, and Epidemiological Considerations in an AI Tool Production (ETEPAI): Scoping Review.JMIR AI · 2026Review
- Ensuring Reliability of Curated Electronic Health Record-Derived Data: The Validation of Accuracy for Large Language Model-/Machine Learning-Extracted Information and Data (VALID) Framework.JCO clinical cancer informatics · 2026Article
- From behavioral signals to therapy management: a real-world pilot implementation of human-in-the-loop AI workflow in proactive patient care.Frontiers in digital health · 2026Article
- Automated real-time assessment of intracranial hemorrhage detection AI using an ensembled monitoring model (EMM).NPJ digital medicine · 2025Article
- Multimodal model for the diagnosis of biliary atresia based on sonographic images and clinical parameters.NPJ digital medicine · 2025Article
- Artificial Intelligence in Orthodontics: Concerns, Conjectures, and Ethical Dilemmas.International dental journal · 2025Article
- A practical guide to implementing artificial intelligence in traditional East Asian medicine research.Integrative medicine research · 2024Article
- Deep Learning for MRI Segmentation and Molecular Subtyping in Glioblastoma: Critical Aspects from an Emerging Field.Biomedicines · 2024Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
A clinical artificial intelligence (AI) system is often validated on data withheld during its development. This provides an estimate of its performance upon future deployment on data in the wild; those currently unseen but are expected to be encountered in a clinical setting. However, estimating performance on data in the wild is complicated by distribution shift between data in the wild and withheld data and the absence of ground-truth annotations. Here, we introduce SUDO, a framework for evaluating AI systems on data in the wild. Through experiments on AI systems developed for dermatology images, histopathology patches, and clinical notes, we show that SUDO can identify unreliable predictions, inform the selection of models, and allow for the previously out-of-reach assessment of algorithmic bias for data in the wild without ground-truth annotations. These capabilities can contribute to the deployment of trustworthy and ethical AI systems in medicine.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.