ArticleNature communications2018
Why rankings of biomedical image analysis competitions should be interpreted with care.
Erratum issuedAbstract read
In one paragraph
Article in Nature communications, 2018. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. Cited by 135 papers, 4 of them syntheses that pooled it.
0numbers the graph read from it
0cells of the map it votes in
135citing papers in PubMed, 4 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
135 citing papers in PubMed, 4 syntheses or guidelines pooled it.
- Vasculature segmentation in 3D hierarchical phase-contrast tomography images of human kidneys.Nature communications · 2026Pooled it
- Systematic review of artificial intelligence competitions in radiology: a focus on design, evaluation, and trends.Diagnostic and interventional radiology (Ankara, Turkey) · 2026Pooled it
- Advancements in automated nuclei segmentation for histopathology using you only look once-driven approaches: A systematic review.Computers in biology and medicine · 2025Pooled it
- Public Covid-19 X-ray datasets and their impact on model bias - A systematic review of a significant problem.Medical image analysis · 2021Pooled it
- Replication and Refinement of an Algorithm for Automated Drusen Segmentation on Optical Coherence Tomography.Scientific reports · 2020Trial
- Consensus operational definitions of surgical meta-competencies for computer vision-driven performance assessment: a multi-institutional Delphi study.Surgical endoscopy · 2026Article
- Beyond the LUMIR challenge: The pathway to foundational registration models.Medical image analysis · 2026Article
- An nnU-Net-based framework with adaptive feature representation for 3D brain tumor segmentation.Quantitative imaging in medicine and surgery · 2026Article
- AFFUNet: adaptive feature fusion Transformer U-Net with joint loss function for medical image segmentation.Quantitative imaging in medicine and surgery · 2026Article
- From PhysioNet to foundation models-a history and potential futures.Physiological measurement · 2026Article
- From Label-Free Multiphoton Imaging to Pathological Reports: A Vision-Language Breast Cancer Margin Pathological Diagnosis System.Advanced science (Weinheim, Baden-Wurttemberg, Germany) · 2026Article
- Quantifying User Satisfaction: Weighted Metric Approach for Evaluating Deep Learning-Based Thigh MRI Segmentations.Diagnostics (Basel, Switzerland) · 2026Article
- An Agentic Multimodal Sensing Architecture for CT-Guided Wearable and Respiratory Monitoring in Oncology Care.Sensors (Basel, Switzerland) · 2026Article
- Deep Learning-Assisted Three-Dimensional Segmentation of Vertebrobasilar Artery Calcification in Cone Beam Computed Tomography.Journal of imaging informatics in medicine · 2026Article
- Medical Image Segmentation Methods: A Decision-Guided Survey Covering 2D/3D CNNs, Transformers, VLMs, SAM-Based Models and Diffusion Approaches.Bioengineering (Basel, Switzerland) · 2026Review
- AI and the digital pathology revolution: clinical applications in cancer diagnosis and assessment.Expert review of molecular diagnostics · 2026Review
- A generalizable pathology foundation model using a unified knowledge distillation pretraining framework.Nature biomedical engineering · 2026Article
- Generalisation of automatic tumour segmentation in histopathological whole-slide images across multiple cancer types.NPJ precision oncology · 2026Article
- Generating Training Data for Ureter Segmentation Using Dual-Energy CT Two-Material Decomposition.Journal of imaging informatics in medicine · 2026Article
- BaselineFrontiers in oncology · 2026Article
75 more citing papers are in PubMed but not listed here.
Corrections and comments
- Erratum issued
Authors and funding
38 authors.
Lena Maier-HeinDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany. l.maier-hein@dkfz.de.ORCID http://orcid.org/0000-0003-4910-9368
Matthias EisenmannDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-0713-8761
Annika ReinkeDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Sinan OnogurDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Marko StankovicDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Patrick ScholzDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Tal ArbelCentre for Intelligent Machines, McGill University, Montreal, QC, H3A0G4, Canada.
Hrvoje BogunovicChristian Doppler Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology, Medical University Vienna, 1090, Vienna, Austria.
Andrew P BradleyScience and Engineering Faculty, Queensland University of Technology, Brisbane, QLD, 4001, Australia.
Aaron CarassDepartment of Electrical and Computer Engineering, Department of Computer Science, Johns Hopkins University, Baltimore, MD, 21218, USA.ORCID http://orcid.org/0000-0003-4939-5085
Carolin FeldmannDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Alejandro F FrangiCISTIB - Center for Computational Imaging & Simulation Technologies in Biomedicine, The University of Leeds, Leeds, Yorkshire, LS2 9JT, UK.ORCID http://orcid.org/0000-0002-2675-528X
Peter M FullDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.ORCID http://orcid.org/0000-0003-4326-8026
Bram van GinnekenDepartment of Radiology and Nuclear Medicine, Medical Image Analysis, Radboud University Center, 6525 GA, Nijmegen, The Netherlands.
Allan HanburyInstitute of Information Systems Engineering, TU Wien, 1040, Vienna, Austria.
Katrin HonauerHeidelberg Collaboratory for Image Processing (HCI), Heidelberg University, 69120, Heidelberg, Germany.
Michal KozubekCentre for Biomedical Image Analysis, Masaryk University, 60200, Brno, Czech Republic.ORCID http://orcid.org/0000-0001-7902-589X
Bennett A LandmanElectrical Engineering, Vanderbilt University, Nashville, TN, 37235-1679, USA.ORCID http://orcid.org/0000-0001-5733-2127
Keno MärzDivision of Computer Assisted Medical Interventions (CAMI), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Oskar MaierInstitute of Medical Informatics, Universität zu Lübeck, 23562, Lübeck, Germany.
Klaus Maier-HeinDivision of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Bjoern H MenzeInstitute for Advanced Studies, Department of Informatics, Technical University of Munich, 80333, Munich, Germany.
Henning MüllerInformation System Institute, HES-SO, Sierre, 3960, Switzerland.ORCID http://orcid.org/0000-0001-6800-9878
Peter F NeherDivision of Medical Image Computing (MIC), German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-5285-7554
Wiro NiessenDepartments of Radiology, Nuclear Medicine and Medical Informatics, Erasmus MC, 3015 GD, Rotterdam, The Netherlands.
Nasir RajpootDepartment of Computer Science, University of Warwick, Coventry, CV4 7AL, UK.
Gregory C SharpDepartment of Radiation Oncology, Massachusetts General Hospital, Boston, MA, 02114, USA.
Korsuk SirinukunwattanaInstitute of Biomedical Engineering, University of Oxford, Oxford, OX3 7DQ, UK.
Stefanie SpeidelDivision of Translational Surgical Oncology (TCO), National Center for Tumor Diseases Dresden, 01307, Dresden, Germany.
Christian StockDivision of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Danail StoyanovCentre for Medical Image Computing (CMIC) & Department of Computer Science, University College London, London, W1W 7TS, UK.
Abdel Aziz TahaData Science Studio, Research Studios Austria FG, 1090, Vienna, Austria.ORCID http://orcid.org/0000-0002-7604-9041
Fons van der SommenDepartment of Electrical Engineering, Eindhoven University of Technology, 5600 MB, Eindhoven, The Netherlands.ORCID http://orcid.org/0000-0002-3593-2356
Ching-Wei WangAIExplore, NTUST Center of Computer Vision and Medical Imaging, Graduate Institute of Biomedical Engineering, National Taiwan University of Science and Technology, Taipei, 106, Taiwan.
Marc-André WeberInstitute of Diagnostic and Interventional Radiology, University Medical Center Rostock, 18051, Rostock, Germany.
Guoyan ZhengInstitute for Surgical Technology and Biomechanics, University of Bern, Bern, 3014, Switzerland.ORCID http://orcid.org/0000-0003-4173-0379
Pierre JanninUniv Rennes, Inserm, LTSI (Laboratoire Traitement du Signal et de l'Image) - UMR_S 1099, Rennes, 35043, Cedex, France.ORCID http://orcid.org/0000-0002-7415-071X
Annette Kopp-SchneiderDivision of Biostatistics, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
Funding
Controlling Quality and Capturing Uncertainty in Advanced Diffusion Weighted MRIR01EB017230 · NIBIB · VANDERBILT UNIVERSITY · PI LANDMAN, BENNETT A. · 2015 to 2024
$5.1MBrain Image Analysis Tools for Quantitative Longitudinal Assessment of MSR01NS070906 · NINDS · HENRY M. JACKSON FDN FOR THE ADV MIL/MED · PI PHAM, DZUNG L · 2010 to 2014
$1.7MMedical Research Council MR/P015476/1NIBIB NIH HHS R01 EB017230NINDS NIH HHS R01 NS070906Wellcome Trust
Abstract
International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.
Indexed as
Biomedical ResearchBiomedical TechnologyDiagnostic ImagingHumansImage Processing, Computer-AssistedReproducibility of ResultsSurveys and QuestionnairesTechnology Assessment, Biomedical
Identifiers
What OpenQuestion holds
Textmetadata
LicenceCC BY
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.