Evidence map›Paper›PMID 36821110›Full record

ArticleJAMA network open2023

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis.

Nicholas Konz, Mateusz Buda, Hanxue Gu, Ashirbani Saha, Jichen Yang, Jakub Chledowski, Jungkyu Park, Jan Witowski, Krzysztof J Geras, Yoel Shoshan and 24 more

Open access · goldAbstract read
In one paragraph

Article in JAMA network open, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 14 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
14citing papers in PubMed, 1 pooled it
4.5field-weighted citation impact, top 4% of its field
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

14 citing papers in PubMed, 1 synthesis or guideline pooled it, 26 citations in OpenAlex.

  1. Pooled it
  2. Article
  3. Article
  4. Article
  5. Review
  6. Review
  7. Article
  8. Breast cancer: pathogenesis and treatments.Signal transduction and targeted therapy · 2025
    Review
  9. Article
  10. Article
  11. Article
  12. Article
  13. Review
  14. Article
4 · The record

Corrections and comments

5 · Who and what money

Authors and funding

34 authors at 17 institutions in 5 countries.

Nicholas KonzDepartment of Electrical and Computer Engineering, Duke University, Durham, North Carolina.
Mateusz BudaDepartment of Radiology, Duke University Medical Center, Durham, North Carolina.
Hanxue GuDepartment of Electrical and Computer Engineering, Duke University, Durham, North Carolina.
Ashirbani SahaDepartment of Radiology, Duke University Medical Center, Durham, North Carolina.
Jichen Yang
Jakub ChledowskiJagiellonian University, Kraków, Poland.
Jungkyu ParkDepartment of Radiology, NYU Grossman School of Medicine, New York, New York.
Jan WitowskiDepartment of Radiology, NYU Grossman School of Medicine, New York, New York.
Krzysztof J GerasDepartment of Radiology, NYU Grossman School of Medicine, New York, New York.
Yoel ShoshanMedical Image Analytics, IBM Research, Haifa, Israel.
Flora Gilboa-SolomonMedical Image Analytics, IBM Research, Haifa, Israel.
Daniel KhapunMedical Image Analytics, IBM Research, Haifa, Israel.
Vadim RatnerMedical Image Analytics, IBM Research, Haifa, Israel.
Ella BarkanMedical Image Analytics, IBM Research, Haifa, Israel.
Michal Ozery-FlatoMedical Image Analytics, IBM Research, Haifa, Israel.
Robert MartíInstitute of Computer Vision and Robotics, University of Girona, Girona, Spain.
Akinyinka OmigbodunMedical and Imaging Informatics Group, Department of Radiological Sciences, David Geffen School of Medicine, University of California Los Angeles.
Chrysostomos MarasinouMedical and Imaging Informatics Group, Department of Radiological Sciences, David Geffen School of Medicine, University of California Los Angeles.
Noor NakhaeiMedical and Imaging Informatics Group, Department of Radiological Sciences, David Geffen School of Medicine, University of California Los Angeles.
William HsuMedical and Imaging Informatics Group, Department of Radiological Sciences, David Geffen School of Medicine, University of California Los Angeles.
Pranjal SahuDepartment of Computer Science, Stony Brook University, Stony Brook, New York.
Md Belayat HossainDepartment of Radiology, University of Pittsburgh, Pittsburgh, Pennsylvania.
Juhun LeeDepartment of Radiology, University of Pittsburgh, Pittsburgh, Pennsylvania.
Carlos SantosDepartment of Radiology, Duke University Medical Center, Durham, North Carolina.
Artur PrzelaskowskiFaculty of Mathematics and Information Science, Warsaw University of Technology, Warsaw, Poland.
Jayashree Kalpathy-CramerDepartment of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown.
Benjamin BearceDepartment of Radiology, Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown.
Kenny ChaUS Food and Drug Administration, Silver Spring, Maryland.
Keyvan FarahaniCenter for Biomedical Informatics and Information Technology, National Cancer Institute, Bethesda, Maryland.
Nicholas PetrickUS Food and Drug Administration, Silver Spring, Maryland.
Lubomir HadjiiskiDepartment of Radiology, University of Michigan, Ann Arbor.
Karen DrukkerDepartment of Radiology, University of Chicago, Chicago, Illinois.
Samuel G ArmatoDepartment of Radiology, University of Chicago, Chicago, Illinois.
Maciej A MazurowskiDepartment of Electrical and Computer Engineering, Duke University, Durham, North Carolina.
IBM Research - Haifa · ILNew York University · USDuke University · USUniversity of California, Los Angeles · USUnited States Food and Drug Administration · USUniversity of Chicago · USUniversity of Pittsburgh · USWarsaw University of Technology · PLAthinoula A. Martinos Center for Biomedical Imaging · USDuke Medical Center · USMassachusetts General Hospital · USMcMaster University · CANational Cancer Institute · USSamueli Institute · USStony Brook University · USUniversity of Girona · ESUniversity of Michigan–Ann Arbor · US

Funding

WORK ORDER 126643 B539 EXPAND IC SUITE75N91019D00024 · NIAID · LEIDOS BIOMEDICAL RESEARCH, INC. · PI PENDLETON, RICHARD · 2019 to 2025
$3932.6M
TR&D 4: Revealing Microstructure: Biophysical modeling and validation for discovery and clinical careP41EB017183 · NIBIB · NEW YORK UNIVERSITY SCHOOL OF MEDICINE · PI Hersh Chandarana · 2014 to 2026
$19.3M
Developing a personalized breast cancer screening tool using sequential mammogramsR37CA248207 · NCI · UNIVERSITY OF PITTSBURGH AT PITTSBURGH · PI LEE, JUHUN · 2020 to 2025
$2.5M
Machine learning and collaborative filtering tools for personalized education in digital breast tomosynthesisR01EB021360 · NIBIB · DUKE UNIVERSITY · PI MAZUROWSKI, MACIEJ A. · 2016 to 2019
$1.6M
NCI NIH HHS 75N91019D00024NCI NIH HHS R37 CA248207NIBIB NIH HHS P41 EB017183NIBIB NIH HHS R01 EB021360
6 · The paper itself

Abstract

Importance: An accurate and robust artificial intelligence (AI) algorithm for detecting cancer in digital breast tomosynthesis (DBT) could significantly improve detection accuracy and reduce health care costs worldwide. Objectives: To make training and evaluation data for the development of AI algorithms for DBT analysis available, to develop well-defined benchmarks, and to create publicly available code for existing methods. Design, Setting, and Participants: This diagnostic study is based on a multi-institutional international grand challenge in which research teams developed algorithms to detect lesions in DBT. A data set of 22 032 reconstructed DBT volumes was made available to research teams. Phase 1, in which teams were provided 700 scans from the training set, 120 from the validation set, and 180 from the test set, took place from December 2020 to January 2021, and phase 2, in which teams were given the full data set, took place from May to July 2021. Main Outcomes and Measures: The overall performance was evaluated by mean sensitivity for biopsied lesions using only DBT volumes with biopsied lesions; ties were broken by including all DBT volumes. Results: A total of 8 teams participated in the challenge. The team with the highest mean sensitivity for biopsied lesions was the NYU B-Team, with 0.957 (95% CI, 0.924-0.984), and the second-place team, ZeDuS, had a mean sensitivity of 0.926 (95% CI, 0.881-0.964). When the results were aggregated, the mean sensitivity for all submitted algorithms was 0.879; for only those who participated in phase 2, it was 0.926. Conclusions and Relevance: In this diagnostic study, an international competition produced algorithms with high sensitivity for using AI to detect lesions on DBT images. A standardized performance benchmark for the detection task using publicly available clinical imaging data was released, with detailed descriptions and analyses of submitted algorithms accompanied by a public release of their predictions and code for selected methods. These resources will serve as a foundation for future research on computer-assisted diagnosis methods for DBT, significantly lowering the barrier of entry for new researchers.

Indexed as

Artificial IntelligenceBreast NeoplasmsAlgorithmsBenchmarkingFemaleHumansMammographyRadiographic Image Interpretation, Computer-Assisted

Identifiers

PMID36821110
PMCPMC9951043
OpenAlexW4321612729

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.