ArticleEuropean radiology2023
Can incorrect artificial intelligence (AI) results impact radiologists, and if so, what can we do about it? A multi-reader pilot study of lung cancer detection with chest radiography.
Article in European radiology, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 56 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
56 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- The impact of generative artificial intelligence on clinical skills and knowledge acquisition in medical undergraduates: a systematic review and meta-analysis.BMC medical education · 2026Pooled it
- Revolution or routine? Comparing AI and traditional imaging in thoracic surgery outcomes: a systematic review.Journal of medicine and life · 2025Pooled it
- Comparative accuracy of artificial intelligence versus manual interpretation in detecting pulmonary hypertension across chest imaging modalities: a diagnostic test accuracy meta-analysis.Frontiers in artificial intelligence · 2025Pooled it
- AI-based chest X-ray prioritization in the lung cancer diagnostic pathway: the LungIMPACT randomized controlled trial.Nature medicine · 2026Trial
- Human-AI collaboration in large language model-assisted brain MRI differential diagnosis: a usability study.European radiology · 2025Trial
- Patient Reactions to Artificial Intelligence-Clinician Discrepancies: Web-Based Randomized Experiment.Journal of medical Internet research · 2025Trial
- Performance Comparison Between Two Versions of a Commercial Artificial Intelligence System for Chest Radiograph Interpretation: A Multicenter Study.Journal of imaging informatics in medicine · 2026Article
- A Detection, Diagnostic, and Triage AI Buyer's Guide: Questions that Matter.Journal of imaging informatics in medicine · 2026Article
- Effects of Type and Timing of Clinician-Facing AI Support on Patient Trust in Medical Consultations: 2 Vignette Experiments.Journal of medical Internet research · 2026Article
- Diagnostic performance with sensitivity/specificity optimized computer-aided detection to detect common abnormalities in chest radiography.Japanese journal of radiology · 2026Article
- Toward trustworthy healthcare AI: designing academic research for translation readiness.Molecular systems biology · 2026Review
- Radiologist Perceptions of an AI Tool for Intracranial Hemorrhage Detection in Teleradiology: Cross-Sectional Survey Study.JMIR human factors · 2026Article
- Letter to the Editor: Radiologist burnout-AI's true black box.European radiology · 2026Article
- Patient perceptions and attitudes towards the use of artificial intelligence in the symptomatic breast unit.European radiology · 2026Article
- [Artificial intelligence in surgical disciplines: Clinical application, advantages, and potential-a Delphi expert consensus].Urologie (Heidelberg, Germany) · 2026Article
- Evaluation of an artificial intelligence generated Brock score to determine malignancy risk of screen-detected pulmonary nodules.European journal of radiology open · 2026Article
- Artificial Intelligence in Lung Cancer: From Early Detection to Personalized Therapy.Current oncology reports · 2026Review
- RAPID CTA versus JLK LVO for large vessel occlusion detection: a pragmatic comparison of performance and common pitfalls.Neuroradiology · 2026Article
- Review
- A novel statistical framework for quantifying risks and benefits of AI automation in screening mammography.PLOS digital health · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveTo examine whether incorrect AI results impact radiologist performance, and if so, whether human factors can be optimized to reduce error.
methodsMulti-reader design, 6 radiologists interpreted 90 identical chest radiographs (follow-up CT needed: yes/no) on four occasions (09/20-01/22). No AI result was provided for session 1. Sham AI results were provided for sessions 2-4, and AI for 12 cases were manipulated to be incorrect (8 false positives (FP), 4 false negatives (FN)) (0.87 ROC-AUC). In the Delete AI (No Box) condition, radiologists were told AI results would not be saved for the evaluation. In Keep AI (No Box) and Keep AI (Box), radiologists were told results would be saved. In Keep AI (Box), the ostensible AI program visually outlined the region of suspicion. AI results were constant between conditions.
resultsRelative to the No AI condition (FN = 2.7%, FP = 51.4%), FN and FPs were higher in the Keep AI (No Box) (FN = 33.0%, FP = 86.0%), Delete AI (No Box) (FN = 26.7%, FP = 80.5%), and Keep AI (Box) (FN = to 20.7%, FP = 80.5%) conditions (all ps < 0.05). FNs were higher in the Keep AI (No Box) condition (33.0%) than in the Keep AI (Box) condition (20.7%) (p = 0.04). FPs were higher in the Keep AI (No Box) (86.0%) condition than in the Delete AI (No Box) condition (80.5%) (p = 0.03).
conclusionIncorrect AI causes radiologists to make incorrect follow-up decisions when they were correct without AI. This effect is mitigated when radiologists believe AI will be deleted from the patient's file or a box is provided around the region of interest. CLINICAL RELEVANCE STATEMENT: When AI is wrong, radiologists make more errors than they would have without AI. Based on human factors psychology, our manuscript provides evidence for two AI implementation strategies that reduce the deleterious effects of incorrect AI. KEY POINTS: • When AI provided incorrect results, false negative and false positive rates among the radiologists increased. • False positives decreased when AI results were deleted, versus kept, in the patient's record. • False negatives and false positives decreased when AI visually outlined the region of suspicion.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.