ArticleJournal of the American College of Radiology : JACR2021
Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage.
Article in Journal of the American College of Radiology : JACR, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 50 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
50 citing papers in PubMed, 5 syntheses or guidelines pooled it, 86 citations in OpenAlex.
- Systematic review of commercial artificial intelligence tools for the detection and volume quantification in intracerebral hemorrhage.European radiology · 2026Pooled it
- Deep learning in neurosurgery: a systematic literature review with a structured analysis of applications across subspecialties.Frontiers in neurology · 2025Pooled it
- Software with artificial intelligence-derived algorithms for analysing CT brain scans in people with a suspected acute stroke: a systematic review and cost-effectiveness analysis.Health technology assessment (Winchester, England) · 2024Pooled it
- Diagnostic test accuracy of machine learning algorithms for the detection intracranial hemorrhage: a systematic review and meta-analysis study.Biomedical engineering online · 2023Pooled it
- Systematic Review of Artificial Intelligence for Abnormality Detection in High-volume Neuroimaging and Subgroup Meta-analysis for Intracranial Hemorrhage Detection.Clinical neuroradiology · 2023Pooled it
- Deep learning algorithm in detecting intracranial hemorrhages on emergency computed tomographies.PloS one · 2021Trial
- Real-world performance and misclassification patterns of deep learning for intracranial hemorrhage detection.Neuroradiology · 2026Article
- Acute-on-chronic versus isolated acute subdural hematoma in complicated mild traumatic brain injury: association with radiological mass effect markers in a multicenter cohort.European journal of trauma and emergency surgery : official publication of the European Trauma Society · 2026Article
- Performance of AI vs radiology residents in the detection of intracranial hemorrhage on emergency CT: a real-world analysis.European radiology · 2026Article
- Review
- ESR Essentials: common performance metrics in AI-practice recommendations by the European Society of Medical Imaging Informatics.European radiology · 2026Review
- Revolutionizing non-traumatic acute care: a review of the role of artificial intelligence and machine learning in triaging and diagnosis.Acute and critical care · 2026Article
- Real-World Integration of an Automated Tool for Intracranial Hemorrhage Detection in an Unselected Cohort of Emergency Department Patients-An External Validation Study.Diagnostics (Basel, Switzerland) · 2026Article
- Real-world performance evaluation of a commercial deep learning model for intracranial hemorrhage detection.NPJ digital medicine · 2025Article
- High sensitivity in spontaneous intracranial hemorrhage detection from emergency head CT scans using ensemble-learning approach.Scientific reports · 2025Article
- Standalone AI Versus AI-Assisted Radiologists in Emergency ICH Detection: A Prospective, Multicenter Diagnostic Accuracy Study.Journal of clinical medicine · 2025Article
- Article
- A deployment safety case for AI-assisted prostate cancer diagnosis.Computers in biology and medicine · 2025Article
- Artificial Intelligence in Vascular Neurology: Applications, Challenges, and a Review of AI Tools for Stroke Imaging, Clinical Decision Making, and Outcome Prediction Models.Current neurology and neuroscience reports · 2025Review
- Diagnostic Accuracy of Deep Learning for Intracranial Hemorrhage Detection in Non-Contrast Brain CT Scans: A Systematic Review and Meta-Analysis.Journal of clinical medicine · 2025Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors at 1 institution in 1 country.
Funding
Abstract
objectiveTo determine the institutional diagnostic accuracy of an artificial intelligence (AI) decision support systems (DSS), Aidoc, in diagnosing intracranial hemorrhage (ICH) on noncontrast head CTs and to assess the potential generalizability of an AI DSS.
methodsThis retrospective study included 3,605 consecutive, emergent, adult noncontrast head CT scans performed between July 1, 2019, and December 30, 2019, at our institution (51% female subjects, mean age of 61 ± 21 years). Each scan was evaluated for ICH by both a certificate of added qualification certified neuroradiologist and Aidoc. We determined the diagnostic accuracy of the AI model and performed a failure mode analysis with quantitative CT radiomic image characterization.
resultsOf the 3,605 scans, 349 cases of ICH (9.7% of studies) were identified. The neuroradiologist and Aidoc interpretations were concordant in 96.9% of cases and the overall sensitivity, specificity, positive predictive value, and negative predictive value were 92.3%, 97.7%, 81.3%, and 99.2%, respectively, with positive predictive values unexpectedly lower than in previously reported studies. Prior neurosurgery, type of ICH, and number of ICHs were significantly associated with decreased model performance. Quantitative image characterization with CT radiomics failed to reveal significant differences between concordant and discordant studies. DISCUSSION: This study revealed decreased diagnostic accuracy of an AI DSS at our institution. Despite extensive evaluation, we were unable to identify the source of this discrepancy, raising concerns about the generalizability of these tools with indeterminate failure modes. These results further highlight the need for standardized study design to allow for rigorous and reproducible site-to-site comparison of emerging deep learning technologies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.