ArticleIEEE access : practical innovations, open solutions2023
Assessing Inter-Annotator Agreement for Medical Image Segmentation.
Article in IEEE access : practical innovations, open solutions, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 31 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
31 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Advancements in Caries Diagnostics Using Bitewing Radiography: A Systematic Review of Deep Learning Approaches.Caries research · 2026Pooled it
- SAM-CLIP-Thermal: Leveraging large multimodal models for reliable and scalable annotation in thermal image segmentation for field plant phenotyping.Plant phenomics (Washington, D.C.) · 2026Article
- Automatic Kidney Image Segmentation During Robot-Assisted Partial Nephrectomy Using a Deep Learning Model Based on a Multiannotator Dataset: Model Development and Validation Study.JMIR medical informatics · 2026Article
- Article
- Reply to the Letter to the Editor: GPT-4o in radiology-a review of label extraction accuracy and clinical applications in upper extremity imaging.European radiology · 2026Article
- An Open Dataset of Yangtze River Docks Based on OSM and Google Satellite Imagery (2024).Scientific data · 2026Article
- Benchmarking variability in semantic segmentation in minimally invasive abdominal surgery.International journal of computer assisted radiology and surgery · 2026Article
- Semi-automated end-to-end quality control system for annotation data based on automated Cobb angle measurement.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2026Article
- Pediatric Personalized Deep Learning Models for Segmentation of Hepatoblastoma at CT and MRI.Radiology. Imaging cancer · 2026Article
- Tomato Multi-Angle Multi-Pose Dataset for Fine-Grained Phenotyping.Scientific data · 2026Article
- AI-driven healthcare: a trend toward better healthcare or the emergence of public health burden.Frontiers in digital health · 2026Article
- Reliability of radiotherapist assessment for radiation induced skin-toxicity: investigational study.Frontiers in oncology · 2026Article
- Structuring medication safety narratives: development and evaluation of the medication-related incident reports annotation scheme.Frontiers in digital health · 2026Article
- Performance comparison of YOLO, Faster R-CNN, and HRNet architectures for bull sperm viability assessment.Frontiers in veterinary science · 2026Article
- Global and regional accuracy of deep learning-based tumor segmentation from whole-body [EJNMMI research · 2025Article
- Precise detection of Eimeria oocysts in sheep: a deep learning model based on microscopic images.Parasites & vectors · 2025Article
- A Multi-Dimensional Framework for Data Quality Assurance in Cancer Imaging Repositories.Cancers · 2025Article
- Early detection of pancreatic cancer on computed tomography: advancements with deep learning.Radiology advances · 2025Review
- Artificial Intelligence Approaches for Geographic Atrophy Segmentation: A Systematic Review and Meta-Analysis.Bioengineering (Basel, Switzerland) · 2025Review
- FOVEA: Preoperative and intraoperative retinal fundus images with optic disc and retinal vessel annotations.Scientific data · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
Artificial Intelligence (AI)-based medical computer vision algorithm training and evaluations depend on annotations and labeling. However, variability between expert annotators introduces noise in training data that can adversely impact the performance of AI algorithms. This study aims to assess, illustrate and interpret the inter-annotator agreement among multiple expert annotators when segmenting the same lesion(s)/abnormalities on medical images. We propose the use of three metrics for the qualitative and quantitative assessment of inter-annotator agreement: 1) use of a common agreement heatmap and a ranking agreement heatmap; 2) use of the extended Cohen's kappa and Fleiss' kappa coefficients for a quantitative evaluation and interpretation of inter-annotator reliability; and 3) use of the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm, as a parallel step, to generate ground truth for training AI models and compute Intersection over Union (IoU), sensitivity, and specificity to assess the inter-annotator reliability and variability. Experiments are performed on two datasets, namely cervical colposcopy images from 30 patients and chest X-ray images from 336 tuberculosis (TB) patients, to demonstrate the consistency of inter-annotator reliability assessment and the importance of combining different metrics to avoid bias assessment.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.