ReviewNature methods2024
Understanding metric-related pitfalls in image analysis validation.
Abstract readReview
In one paragraph
Review in Nature methods, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 74 papers, 2 of them syntheses that pooled it.
0numbers the graph read from it
0cells of the map it votes in
74citing papers in PubMed, 2 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
74 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- PRIME 2.0: Proposed Requirements for Cardiovascular Imaging-Related Multimodal-AI Evaluation: An Updated Checklist.JACC. Cardiovascular imaging · 2026Guideline
- Deep learning for MRI-based acute and subacute ischaemic stroke lesion segmentation-a systematic review, meta-analysis, and pilot evaluation of key results.Frontiers in medical technology · 2025Pooled it
- Feasibility of Deep Learning Segmentation in Anterior Quadratus Lumborum Ultrasound Imaging: Challenges in Identifying the Injection Plane.Journal of imaging informatics in medicine · 2026Article
- TOPOS: target organ prediction on scout views for automated CT scan planning.European radiology · 2026Article
- Quantifying the Resolution Ceiling of Prototype-Based Few-Shot Segmentation on Sub-Centimetre Pulmonary Nodules: A Training-Free Geometric Analysis.Diagnostics (Basel, Switzerland) · 2026Article
- Ethical and Legal Concerns of Deepfake Technology in Biomedical Imaging: A Comprehensive Survey.Annals of biomedical engineering · 2026Review
- Toward trustworthy healthcare AI: designing academic research for translation readiness.Molecular systems biology · 2026Review
- Knowledge and Awareness of Performance Metrics in Dental Image Analysis With Artificial Intelligence: A Cross-Sectional Survey of Researchers and Clinicians.International dental journal · 2026Article
- Language-Guided Segmentation of Medical Images: A Review of Foundation Models.Bioengineering (Basel, Switzerland) · 2026Review
- Quantifying cortical lesions in multiple sclerosis MRI datasets using multi-contrast post-processing and deep learning.Communications medicine · 2026Article
- Risk and liability in the deployment of AI systems for surgery: a SAGES white paper.Surgical endoscopy · 2026Review
- Uncertainty quantification of U-Net based segmentation tool using conformal prediction.Journal of applied clinical medical physics · 2026Article
- Regulating digital surgery to balance safety and innovation: a SAGES white paper.Surgical endoscopy · 2026Article
- Automatic lesion segmentation in ⁶⁸Ga-PSMA PET/CT and ¹⁷⁷Lu-PSMA SPECT/CT: added value of PET-guided SPECT in a bicentric study.EJNMMI research · 2026Article
- Multimodal deep learning to support delineation of neuro-oncology organs based on the EPTN international neurological contouring atlas.Physics and imaging in radiation oncology · 2026Article
- PIGMENT: A deep learning framework for Porcine Immunohistochemistry seGMENTation.bioRxiv : the preprint server for biology · 2026Article
- Super-resolution of periapical and bitewing digital radiographs using convolutional neural network.BMC medical imaging · 2026Article
- A Generalized Deep Learning Pipeline for Stain-Invariant Ultrastructural Segmentation in Peripheral Nerves.Journal of imaging · 2026Article
- Markerless 3D ultrasound reconstruction for augmented reality assisted minimally invasive spine surgery.International journal of computer assisted radiology and surgery · 2026Article
- Development of Human Conjunctival Goblet Cell Segmentation Datasets to Improve Quantitation.Scientific data · 2026Article
14 more citing papers are in PubMed but not listed here.
Corrections and comments
- Update of
Authors and funding
70 authors.
Annika Reinke *German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. a.reinke@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0003-4363-1876
Minu D Tizabi *German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. m.tizabi@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0003-3687-6381
Michael BaumgartnerGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.ORCID http://orcid.org/0000-0003-4455-9917
Matthias EisenmannGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Doreen Heckmann-NötzelGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
A Emre KavurGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-9328-8140
Tim RädschGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0003-3518-0315
Carole H SudreMRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK.
Laura AcionInstituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina.ORCID http://orcid.org/0000-0001-5213-6012
Michela AntonelliSchool of Biomedical Engineering and Imaging Science, King's College London, London, UK.ORCID http://orcid.org/0000-0002-3005-4523
Tal ArbelCentre for Intelligent Machines and MILA (Quebec Artificial Intelligence Institute), McGill University, Montréal, Quebec, Canada.ORCID http://orcid.org/0000-0001-8870-3007
Spyridon BakasDivision of Computational Pathology, Dept of Pathology & Laboratory Medicine, Indiana University School of Medicine, Indianapolis, IN, USA.ORCID http://orcid.org/0000-0001-8734-6482
Arriel BenisDepartment of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel.ORCID http://orcid.org/0000-0002-9125-8300
Florian BuettnerGerman Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany.
M Jorge CardosoSchool of Biomedical Engineering and Imaging Science, King's College London, London, UK.ORCID http://orcid.org/0000-0003-1284-2558
Veronika CheplyginaDepartment of Computer Science, IT University of Copenhagen, Copenhagen, Denmark.ORCID http://orcid.org/0000-0003-0176-9324
Jianxu ChenLeibniz-Institut für Analytische Wissenschaften - ISAS - e.V., Dortmund, Germany.ORCID http://orcid.org/0000-0002-8500-1357
Evangelia ChristodoulouGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Beth A CiminiImaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA.ORCID http://orcid.org/0000-0001-9640-9318
Keyvan FarahaniCenter for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA.
Luciana FerrerInstituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina.
Adrian GaldranUniversitat Pompeu Fabra, Barcelona, Spain.
Bram van GinnekenFraunhofer MEVIS, Bremen, Germany.
Ben GlockerDepartment of Computing, Imperial College London, South Kensington Campus, London, UK.ORCID http://orcid.org/0000-0002-4897-9356
Patrick GodauGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-0365-7265
Daniel A HashimotoDepartment of Surgery, Perelman School of Medicine, Philadelphia, PA, USA.ORCID http://orcid.org/0000-0003-4725-3104
Michael M HoffmanPrincess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada.ORCID http://orcid.org/0000-0002-4517-1562
Merel HuismanDepartment of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands.
Fabian IsenseeGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Pierre JanninLaboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France.ORCID http://orcid.org/0000-0002-7415-071X
Charles E KahnDepartment of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA.ORCID http://orcid.org/0000-0002-6654-7434
Dagmar KainmuellerMax-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany.
Bernhard KainzDepartment of Computing, Faculty of Engineering, Imperial College London, London, UK.
Alexandros KarargyrisIHU Strasbourg, Strasbourg, France.ORCID http://orcid.org/0000-0002-1930-3410
Jens KleesiekTranslational Image-guided Oncology (TIO), Institute for AI in Medicine (IKIM), University Medicine Essen, Essen, Germany.ORCID http://orcid.org/0000-0001-8686-0682
Florian KoflerHelmholtz AI, Oberschleißheim, Germany.
Thijs KooiLunit, Seoul, South Korea.
Annette Kopp-SchneiderGerman Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-1810-0267
Michal KozubekCentre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic.ORCID http://orcid.org/0000-0001-7902-589X
Anna KreshukCell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany.ORCID http://orcid.org/0000-0003-1334-6388
Tahsin KurcDepartment of Biomedical Informatics, Stony Brook University, Health Science Center, Stony Brook, NY, USA.
Bennett A LandmanElectrical Engineering, Vanderbilt University, Nashville, TN, USA.ORCID http://orcid.org/0000-0001-5733-2127
Geert LitjensDepartment of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands.ORCID http://orcid.org/0000-0003-1554-1291
Amin MadaniDepartment of Surgery, University Health Network, Philadelphia, PA, USA.
Klaus Maier-HeinGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Anne L MartelDepartment of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada.ORCID http://orcid.org/0000-0003-1375-5501
Erik MeijeringSchool of Computer Science and Engineering, University of New South Wales, UNSW Sydney, Kensington, New South Wales, Australia.ORCID http://orcid.org/0000-0001-8015-8358
Bjoern MenzeDepartment of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland.ORCID http://orcid.org/0000-0003-4136-5690
Karel G M MoonsJulius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands.
Henning MüllerInformation Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland.ORCID http://orcid.org/0000-0001-6800-9878
Brennan NichyporukMILA (Quebec Artificial Intelligence Institute), Montréal, Quebec, Canada.ORCID http://orcid.org/0009-0006-8087-6089
Felix NickelDepartment of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
Jens PetersenGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Susanne M RafelskiAllen Institute for Cell Science, Seattle, WA, USA.ORCID http://orcid.org/0000-0002-1399-5970
Nasir RajpootTissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK.ORCID http://orcid.org/0000-0001-6760-1271
Mauricio ReyesARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland.
Michael A RieglerSimula Metropolitan Center for Digital Engineering, Oslo, Norway.ORCID http://orcid.org/0000-0002-3153-2064
Nicola RiekeNVIDIA GmbH, München, Germany.ORCID http://orcid.org/0000-0003-0241-9334
Julio Saez-RodriguezInstitute for Computational Biomedicine, Heidelberg University, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-8552-8976
Clara I SánchezInformatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, the Netherlands.
Shravya ShettyGoogle Health, Google, Palo Alto, CA, USA.
Ronald M SummersNational Institutes of Health Clinical Center, Bethesda, MD, USA.ORCID http://orcid.org/0000-0001-8081-7376
Abdel A TahaInstitute of Information Systems Engineering, TU Wien, Vienna, Austria.
Aleksei TiulpinResearch Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland.ORCID http://orcid.org/0000-0002-7852-4141
Sotirios A TsaftarisSchool of Engineering, The University of Edinburgh, Edinburgh, Scotland.
Ben Van CalsterDepartment of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium.
Gaël VaroquauxParietal project team, INRIA Saclay-Île de France, Palaiseau, France.ORCID http://orcid.org/0000-0003-1076-5122
Ziv R YanivNational Institute of Allergy and Infectious Diseases, Bethesda, MD, USA.ORCID http://orcid.org/0000-0003-0315-7727
Paul F JägerGerman Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany. p.jaeger@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0002-6243-2568
Lena Maier-HeinGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. l.maier-hein@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0003-4910-9368
Funding
Statistical atlases of brain tumor MRI:do imaging phenotypes predict progression?R01NS042645 · NINDS · UNIVERSITY OF PENNSYLVANIA · PI DAVATZIKOS, CHRISTOS · 2002 to 2024
$8.7MTCIA Sustainment and Scalability - Platforms for Quantitative Imaging Informatics in Precision MedicineU24CA215109 · NCI · UNIV OF ARKANSAS FOR MED SCIS · PI BANERJEE, IMON, PRIOR, FRED WILLIAM · 2017 to 2021
$8.5MCenter for Open Bioimage AnalysisP41GM135019 · NIGMS · BROAD INSTITUTE, INC. · PI CARPENTER, ANNE E. · 2020 to 2024
$6.9MTools to Analyze Morphology and Spatially Mapped Molecular DataU24CA180924 · NCI · STATE UNIVERSITY NEW YORK STONY BROOK · PI SALTZ, JOEL H. · 2014 to 2019
$3.6MPrivacy-Aware Federated Learning for Breast Cancer Risk AssessmentU24CA279629 · NCI · INDIANA UNIVERSITY INDIANAPOLIS · PI Spyridon Bakas · 2023 to 2026
$2.9MMethods and Tools for Integrating Pathomics Data into Cancer RegistriesUH3CA225021 · NCI · STATE UNIVERSITY NEW YORK STONY BROOK · PI CLIFFORD, GARI DAVID, DURBIN, ERIC B. · 2020 to 2022
$1.9MThe Federated Tumor Segmentation (FeTS) platform: An intuitive tool facilitating secure multi-institutional collaborationU01CA242871 · NCI · UNIVERSITY OF PENNSYLVANIA · PI BAKAS, SPYRIDON · 2019 to 2021
$1.1MComputer Aided Detection for Radiologic ImagesZIACL040004 · CLC · CLINICAL CENTER · PI SUMMERS, RONALD M. · 2009 to 2025
$0kEPA EP-W-17-011NCI NIH HHS U01 CA242871NCI NIH HHS U24 CA180924NCI NIH HHS U24 CA215109NCI NIH HHS U24 CA279629NCI NIH HHS UH3 CA225021NIGMS NIH HHS P41 GM135019NINDS NIH HHS R01 NS042645Wellcome Trust 203148Wellcome Trust 213038
Abstract
Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.
Indexed as
Artificial Intelligence
Identifiers
What OpenQuestion holds
Textmetadata
LicenceTDM
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.