ArticleJournal of applied clinical medical physics2024
Gross failure rates and failure modes for a commercial AI-based auto-segmentation algorithm in head and neck cancer patients.
Article in Journal of applied clinical medical physics, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
11 citing papers in PubMed.
- Potential of E-Learning Interventions and Artificial Intelligence-Assisted Contouring Skills in Radiotherapy: The ELAISA Study.JCO global oncology · 2024Trial
- Sustaining High Reliability Amid Artificial Intelligence Adoption in Oncology.Journal of clinical oncology : official journal of the American Society of Clinical Oncology · 2026Article
- Real-world clinical impact of implementing and updating a deep learning-based automatic contouring system in rectal cancer radiotherapy.Journal of applied clinical medical physics · 2026Article
- Clinical evaluation of novel deep learning-based auto-segmentation software: Utility and potential pitfalls.Journal of applied clinical medical physics · 2026Article
- Transparency of medical artificial intelligence systems.Nature reviews bioengineering · 2026Article
- Geometric, dosimetric and psychometric evaluation of three commercial AI software solutions for OAR auto-segmentation in head and neck radiotherapy.Scientific reports · 2025Article
- The Limitations of Artificial Intelligence in Head and Neck Oncology.Advances in therapy · 2025Review
- Automated contouring for breast cancer radiotherapy in the isocentric lateral decubitus position: a neural network-based solution for enhanced precision and efficiency.Strahlentherapie und Onkologie : Organ der Deutschen Rontgengesellschaft ... [et al] · 2025Article
- Radiation Treatment Planning After Minimum Metallic Instrumentation for Patients with Spinal Metastases: A Case Series.Medicina (Kaunas, Lithuania) · 2025Article
- Gross failure rates and failure modes for a commercial AI-based auto-segmentation algorithm in head and neck cancer patients.Journal of applied clinical medical physics · 2024Article
- Automated Patient-specific Quality Assurance for Automated Segmentation of Organs at Risk in Nasopharyngeal Carcinoma Radiotherapy.Cancer control : journal of the Moffitt Cancer CenterArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
2 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeArtificial intelligence (AI) based commercial software can be used to automatically delineate organs at risk (OAR), with potential for efficiency savings in the radiotherapy treatment planning pathway, and reduction of inter- and intra-observer variability. There has been little research investigating gross failure rates and failure modes of such systems.
method50 head and neck (H&N) patient data sets with "gold standard" contours were compared to AI-generated contours to produce expected mean and standard deviation values for the Dice Similarity Coefficient (DSC), for four common H&N OARs (brainstem, mandible, left and right parotid). An AI-based commercial system was applied to 500 H&N patients. AI-generated contours were compared to manual contours, outlined by an expert human, and a gross failure was set at three standard deviations below the expected mean DSC. Failures were inspected to assess reason for failure of the AI-based system with failures relating to suboptimal manual contouring censored. True failures were classified into 4 sub-types (setup position, anatomy, image artefacts and unknown).
resultsThere were 24 true failures of the AI-based commercial software, a gross failure rate of 1.2%. Fifteen failures were due to patient anatomy, four were due to dental image artefacts, three were due to patient position and two were unknown. True failure rates by OAR were 0.4% (brainstem), 2.2% (mandible), 1.4% (left parotid) and 0.8% (right parotid).
conclusionTrue failures of the AI-based system were predominantly associated with a non-standard element within the CT scan. It is likely that these non-standard elements were the reason for the gross failure, and suggests that patient datasets used to train the AI model did not contain sufficient heterogeneity of data. Regardless of the reasons for failure, the true failure rate for the AI-based system in the H&N region for the OARs investigated was low (∼1%).
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.