ArticleJournal of medical imaging (Bellingham, Wash.)2018
Large-scale medical image annotation with crowd-powered algorithms.
Article in Journal of medical imaging (Bellingham, Wash.), 2018. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Crowdsourcing in health and medical research: a systematic review.Infectious diseases of poverty · 2020Pooled it
- Applications of artificial intelligence in nuclear medicine.Zeitschrift fur medizinische Physik · 2026Review
- Using Generative AI to Appraise the Quality of Medical Education Research Studies: Agreement Between AI-Generated and Human MERSQI Scores.AEM education and training · 2026Article
- Noninvasive Diagnosis of Melanoma Using Machine Learning and Reflectance Confocal Microscopy.The Journal of investigative dermatology · 2026Article
- Self-supervised learning for label-free segmentation in cardiac ultrasound.Nature communications · 2025Article
- AI-Powered Object Detection in Radiology: Current Models, Challenges, and Future Direction.Journal of imaging · 2025Review
- Real-Time Ultrasound Diagnosis of Developmental Dysplasia of the Hip Using an Attention-Enhanced YOLOv11 Model.International journal of medical sciences · 2025Article
- Harnessing Advanced Machine Learning Techniques for Microscopic Vessel Segmentation in Pulmonary Fibrosis Using Novel Hierarchical Phase-Contrast Tomography Images.Methods of information in medicine · 2024Article
- Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Data Set Labeling: Prospective Analysis.Journal of medical Internet research · 2024Article
- A vascularized breast cancer spheroid platform for the ranked evaluation of tumor microenvironment-targeted drugs by light sheet fluorescence microscopy.Nature communications · 2024Article
- Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.Modern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc · 2024Article
- Crowdsourcing Skin Demarcations of Chronic Graft-Versus-Host Disease in Patient Photographs: Training Versus Performance Study.JMIR dermatology · 2023Article
- Large scale crowdsourced radiotherapy segmentations across a variety of cancer anatomic sites.Scientific data · 2023Article
- Article
- Agreement Between Experts and an Untrained Crowd for Identifying Dermoscopic Features Using a Gamified App: Reader Feasibility Study.JMIR medical informatics · 2023Article
- Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency.Physics in medicine and biology · 2022Review
- Fully Automated Wound Tissue Segmentation Using Deep Learning on Mobile Devices: Cohort Study.JMIR mHealth and uHealth · 2022Article
- Preparing Medical Imaging Data for Machine Learning.Radiology · 2020Article
- Millimeter-Level Plant Disease Detection From Aerial PhotographsFrontiers in plant science · 2019Article
- Observer Variability in CT Angiography Carotid Segmentation: Assessing Variability to Set Minimum Clinical Performance.Journal of neuroimaging : official journal of the American Society of NeuroimagingArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
15 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Accurate segmentations in medical images are the foundations for various clinical applications. Advances in machine learning-based techniques show great potential for automatic image segmentation, but these techniques usually require a huge amount of accurately annotated reference segmentations for training. The guiding hypothesis of this paper was that crowd-algorithm collaboration could evolve as a key technique in large-scale medical data annotation. As an initial step toward this goal, we evaluated the performance of untrained individuals to detect and correct errors made by three-dimensional (3-D) medical segmentation algorithms. To this end, we developed a multistage segmentation pipeline incorporating a hybrid crowd-algorithm 3-D segmentation algorithm integrated into a medical imaging platform. In a pilot study of liver segmentation using a publicly available dataset of computed tomography scans, we show that the crowd is able to detect and refine inaccurate organ contours with a quality similar to that of experts (engineers with domain knowledge, medical students, and radiologists). Although the crowds need significantly more time for the annotation of a slice, the annotation rate is extremely high. This could render crowdsourcing a key tool for cost-effective large-scale medical image annotation.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.