ReviewNature methods2024
Metrics reloaded: recommendations for image analysis validation.
Abstract readReview
In one paragraph
Review in Nature methods, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 252 papers, 4 of them syntheses that pooled it.
0numbers the graph read from it
0cells of the map it votes in
252citing papers in PubMed, 4 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
252 citing papers in PubMed, 4 syntheses or guidelines pooled it.
- Artificial intelligence-powered renal pathology for glomerular disease classification: a systematic review and meta-analysis.Renal failure · 2026Pooled it
- Diagnostic accuracy of AI-assisted versus independent physician interpretation for bone fractures: a systematic review and meta-analysis.European radiology experimental · 2026Pooled it
- PRIME 2.0: Proposed Requirements for Cardiovascular Imaging-Related Multimodal-AI Evaluation: An Updated Checklist.JACC. Cardiovascular imaging · 2026Guideline
- Application of artificial intelligence in esophageal surgery: a systematic review.Journal of robotic surgery · 2025Pooled it
- Application of artificial intelligence in head and neck squamous cell carcinoma.Annals of medicine · 2026Review
- In search of truth: evaluating concordance of AI-based anatomy segmentation models.Journal of medical imaging (Bellingham, Wash.) · 2026Article
- Fully automated diaphragm segmentation on routine 2D-cine and 3D thoracic MRI: multireader validation.European radiology experimental · 2026Article
- Automatic Generation of Liver Virtual Models With Artificial Intelligence: Application to Liver Resection Complexity Prediction.Annals of surgery · 2026Article
- Deep learning-based 3D forensic facial reconstruction: a volumetric morphological inference approach using a skull-conditioned U-Net.International journal of legal medicine · 2026Article
- Artificial Intelligence for Automated Recognition of Hepatocystic Anatomy During Laparoscopic Cholecystectomy: Current Evidence, Clinical Readiness, and Future Directions.Medicina (Kaunas, Lithuania) · 2026Review
- Cross-fraction prior learning for scalable organ-at-risk segmentation in abdominal MR-guided radiotherapy.Medical physics · 2026Article
- A comparative analysis of YOLOv8 and nnU-Net v2 based pipelines for sex and age estimation from maxillary sinus morphometry on panoramic radiographs.International journal of legal medicine · 2026Article
- Beyond benchmarking: an expert-guided consensus approach to spatially aware clustering.Nature methods · 2026Article
- Predicting brain tumour enhancement from non-contrast MRI with artificial intelligence: a multicohort, retrospective, diagnostic accuracy study.The Lancet medical imaging & theranostics · 2026Article
- OrgLine: A versatile pipeline for organoid morphometry using detector-guided prompts.Cell reports methods · 2026Article
- Mel-DEPTHS: a benchmark dataset for epidermis and tumor segmentation for melanoma staging.Medical & biological engineering & computing · 2026Article
- Do not smile when acquiring consumer selfies for multi-attribute skin profiling and baseline deep learning evaluation.Scientific reports · 2026Article
- Benchmarking Deep Learning for NSCLC PET/CT Segmentation on a Histologically Confirmed Vietnamese Dataset: Validation and Generalization.Journal of imaging · 2026Article
- AI-Assisted Papillary Thyroid Carcinoma Localization on Cervical Non-contrast CT: A Multicenter Validation and Two-Reader Pilot Study of a Standalone Executable nnU-Net Software.Journal of imaging informatics in medicine · 2026Article
- Toward trustworthy healthcare AI: designing academic research for translation readiness.Molecular systems biology · 2026Review
192 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
73 authors.
Lena Maier-Hein *German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. l.maier-hein@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0003-4910-9368
Annika Reinke *German Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany. a.reinke@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0003-4363-1876
Patrick GodauGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-0365-7265
Minu D TizabiGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Florian BuettnerGerman Cancer Consortium (DKTK), partner site Frankfurt/Mainz, a partnership between DKFZ and UCT Frankfurt-Marburg, Frankfurt am Main, Germany.
Evangelia ChristodoulouGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Ben GlockerDepartment of Computing, Imperial College London, South Kensington Campus, London, UK.ORCID http://orcid.org/0000-0002-4897-9356
Fabian IsenseeGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Jens KleesiekInstitute for AI in Medicine, University Medicine Essen, Essen, Germany.
Michal KozubekCentre for Biomedical Image Analysis and Faculty of Informatics, Masaryk University, Brno, Czech Republic.ORCID http://orcid.org/0000-0001-7902-589X
Mauricio ReyesARTORG Center for Biomedical Engineering Research, University of Bern, Bern, Switzerland.
Michael A RieglerSimula Metropolitan Center for Digital Engineering, Oslo, Norway.ORCID http://orcid.org/0000-0002-3153-2064
Manuel WiesenfarthGerman Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany.
A Emre KavurGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Carole H SudreMRC Unit for Lifelong Health and Ageing at UCL and Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK.
Michael BaumgartnerGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Matthias EisenmannGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-0713-8761
Doreen Heckmann-NötzelGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.
Tim RädschGerman Cancer Research Center (DKFZ) Heidelberg, Division of Intelligent Medical Systems, Heidelberg, Germany.ORCID http://orcid.org/0000-0003-3518-0315
Laura AcionInstituto de Cálculo, CONICET - Universidad de Buenos Aires, Buenos Aires, Argentina.ORCID http://orcid.org/0000-0001-5213-6012
Michela AntonelliSchool of Biomedical Engineering and Imaging Science, King's College London, London, UK.ORCID http://orcid.org/0000-0002-3005-4523
Tal ArbelCentre for Intelligent Machines and MILA (Québec Artificial Intelligence Institute), McGill University, Montréal, Quebec, Canada.ORCID http://orcid.org/0000-0001-8870-3007
Spyridon BakasDivision of Computational Pathology, Department of Pathology & Laboratory Medicine, Indiana University School of Medicine, IU Health Information and Translational Sciences Building, Indianapolis, IN, USA.ORCID http://orcid.org/0000-0001-8734-6482
Arriel BenisDepartment of Digital Medical Technologies, Holon Institute of Technology, Holon, Israel.ORCID http://orcid.org/0000-0002-9125-8300
Matthew B BlaschkoCenter for Processing Speech and Images, Department of Electrical Engineering, KU Leuven, Leuven, Belgium.
M Jorge CardosoSchool of Biomedical Engineering and Imaging Science, King's College London, London, UK.ORCID http://orcid.org/0000-0003-1284-2558
Veronika CheplyginaDepartment of Computer Science, IT University of Copenhagen, Copenhagen, Denmark.ORCID http://orcid.org/0000-0003-0176-9324
Beth A CiminiImaging Platform, Broad Institute of MIT and Harvard, Cambridge, MA, USA.ORCID http://orcid.org/0000-0001-9640-9318
Gary S CollinsCentre for Statistics in Medicine, University of Oxford, Nuffield Orthopaedic Centre, Oxford, UK.ORCID http://orcid.org/0000-0002-2772-2316
Keyvan FarahaniCenter for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, MD, USA.
Luciana FerrerInstituto de Investigación en Ciencias de la Computación (ICC), CONICET-UBA, Ciudad Autónoma de Buenos Aires, Buenos Aires, Argentina.
Adrian GaldranBCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain.
Bram van GinnekenFraunhofer MEVIS, Bremen, Germany.
Robert HaaseTechnische Universität (TU) Dresden, DFG Cluster of Excellence 'Physics of Life', Dresden, Germany.ORCID http://orcid.org/0000-0001-5949-2327
Daniel A HashimotoDepartment of Surgery, Perelman School of Medicine, Philadelphia, PA, USA.ORCID http://orcid.org/0000-0003-4725-3104
Michael M HoffmanPrincess Margaret Cancer Centre, University Health Network, Toronto, Ontario, Canada.ORCID http://orcid.org/0000-0002-4517-1562
Merel HuismanDepartment of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands.
Pierre JanninLaboratoire Traitement du Signal et de l'Image - UMR_S 1099, Université de Rennes 1, Rennes, France.ORCID http://orcid.org/0000-0002-7415-071X
Charles E KahnDepartment of Radiology and Institute for Biomedical Informatics, University of Pennsylvania, Philadelphia, PA, USA.ORCID http://orcid.org/0000-0002-6654-7434
Dagmar KainmuellerMax-Delbrück Center for Molecular Medicine in the Helmholtz Association (MDC), Biomedical Image Analysis and HI Helmholtz Imaging, Berlin, Germany.
Bernhard KainzDepartment of Computing, Faculty of Engineering, Imperial College London, London, UK.
Alexandros KarargyrisIHU Strasbourg, Strasbourg, France.ORCID http://orcid.org/0000-0002-1930-3410
Alan KarthikesalingamGoogle Health DeepMind, London, UK.
Florian KoflerHelmholtz AI, Oberschleißheim, Germany.
Annette Kopp-SchneiderGerman Cancer Research Center (DKFZ) Heidelberg, Division of Biostatistics, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-1810-0267
Anna KreshukCell Biology and Biophysics Unit, European Molecular Biology Laboratory (EMBL), Heidelberg, Germany.ORCID http://orcid.org/0000-0003-1334-6388
Tahsin KurcDepartment of Biomedical Informatics, Stony Brook University, Health Science Center, Stony Brook, NY, USA.
Bennett A LandmanElectrical Engineering, Vanderbilt University, Nashville, TN, USA.ORCID http://orcid.org/0000-0001-5733-2127
Geert LitjensDepartment of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands.ORCID http://orcid.org/0000-0003-1554-1291
Amin MadaniDepartment of Surgery, University Health Network, Philadelphia, PA, USA.
Klaus Maier-HeinGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Anne L MartelDepartment of Medical Biophysics, University of Toronto, Toronto, Ontario, Canada.ORCID http://orcid.org/0000-0003-1375-5501
Peter MattsonGoogle, 1600 Amphitheatre Pkwy, Mountain View, CA, USA.ORCID http://orcid.org/0000-0002-5984-238X
Erik MeijeringSchool of Computer Science and Engineering, University of New South Wales, UNSW Sydney, Kensington, New South Wales, Australia.ORCID http://orcid.org/0000-0001-8015-8358
Bjoern MenzeDepartment of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland.ORCID http://orcid.org/0000-0003-4136-5690
Karel G M MoonsJulius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands.
Henning MüllerInformation Systems Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland.ORCID http://orcid.org/0000-0001-6800-9878
Brennan NichyporukMILA (Québec Artificial Intelligence Institute), Montréal, Quebec, Canada.ORCID http://orcid.org/0009-0006-8087-6089
Felix NickelDepartment of General, Visceral and Thoracic Surgery, University Medical Center Hamburg-Eppendorf, Hamburg, Germany.
Jens PetersenGerman Cancer Research Center (DKFZ) Heidelberg, Division of Medical Image Computing, Heidelberg, Germany.
Nasir RajpootTissue Image Analytics Laboratory, Department of Computer Science, University of Warwick, Coventry, UK.ORCID http://orcid.org/0000-0001-6760-1271
Nicola RiekeNVIDIA, München, Germany.ORCID http://orcid.org/0000-0003-0241-9334
Julio Saez-RodriguezInstitute for Computational Biomedicine, Heidelberg University, Heidelberg, Germany.ORCID http://orcid.org/0000-0002-8552-8976
Clara I SánchezInformatics Institute, Faculty of Science, University of Amsterdam, Amsterdam, the Netherlands.
Shravya ShettyGoogle Health, Google, Palo Alto, CA, USA.
Maarten van SmedenJulius Center for Health Sciences and Primary Care, UMC Utrecht, Utrecht University, Utrecht, the Netherlands.
Ronald M SummersNational Institutes of Health Clinical Center, Bethesda, MD, USA.ORCID http://orcid.org/0000-0001-8081-7376
Abdel A TahaInstitute of Information Systems Engineering, TU Wien, Vienna, Austria.ORCID http://orcid.org/0000-0002-7604-9041
Aleksei TiulpinResearch Unit of Health Sciences and Technology, Faculty of Medicine, University of Oulu, Oulu, Finland.ORCID http://orcid.org/0000-0002-7852-4141
Sotirios A TsaftarisSchool of Engineering, The University of Edinburgh, Edinburgh, Scotland.
Ben Van CalsterDepartment of Development and Regeneration and EPI-centre, KU Leuven, Leuven, Belgium.
Gaël VaroquauxParietal project team, INRIA Saclay-Île de France, Palaiseau, France.ORCID http://orcid.org/0000-0003-1076-5122
Paul F JägerGerman Cancer Research Center (DKFZ) Heidelberg, HI Helmholtz Imaging, Heidelberg, Germany. p.jaeger@dkfz-heidelberg.de.ORCID http://orcid.org/0000-0002-6243-2568
Funding
Statistical atlases of brain tumor MRI:do imaging phenotypes predict progression?R01NS042645 · NINDS · UNIVERSITY OF PENNSYLVANIA · PI DAVATZIKOS, CHRISTOS · 2002 to 2024
$8.7MTCIA Sustainment and Scalability - Platforms for Quantitative Imaging Informatics in Precision MedicineU24CA215109 · NCI · UNIV OF ARKANSAS FOR MED SCIS · PI BANERJEE, IMON, PRIOR, FRED WILLIAM · 2017 to 2021
$8.5MCenter for Open Bioimage AnalysisP41GM135019 · NIGMS · BROAD INSTITUTE, INC. · PI CARPENTER, ANNE E., CIMINI, BETH · 2020 to 2024
$6.9MTools to Analyze Morphology and Spatially Mapped Molecular DataU24CA180924 · NCI · STATE UNIVERSITY NEW YORK STONY BROOK · PI SALTZ, JOEL H. · 2014 to 2019
$3.6MPrivacy-Aware Federated Learning for Breast Cancer Risk AssessmentU24CA279629 · NCI · INDIANA UNIVERSITY INDIANAPOLIS · PI Spyridon Bakas · 2023 to 2026
$2.9MMethods and Tools for Integrating Pathomics Data into Cancer RegistriesUH3CA225021 · NCI · STATE UNIVERSITY NEW YORK STONY BROOK · PI CLIFFORD, GARI DAVID, DURBIN, ERIC B. · 2020 to 2022
$1.9MThe Federated Tumor Segmentation (FeTS) platform: An intuitive tool facilitating secure multi-institutional collaborationU01CA242871 · NCI · UNIVERSITY OF PENNSYLVANIA · PI BAKAS, SPYRIDON · 2019 to 2021
$1.1MComputer Aided Detection for Radiologic ImagesZIACL040004 · CLC · CLINICAL CENTER · PI SUMMERS, RONALD M. · 2009 to 2025
$0kEPA EP-W-17-011NCI NIH HHS U01 CA242871NCI NIH HHS U24 CA180924NCI NIH HHS U24 CA215109NCI NIH HHS U24 CA279629NCI NIH HHS UH3 CA225021NIGMS NIH HHS P41 GM135019NINDS NIH HHS R01 NS042645
Abstract
Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.
Indexed as
AlgorithmsImage Processing, Computer-AssistedMachine LearningSemantics
Identifiers
What OpenQuestion holds
Textmetadata
LicenceTDM
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.