Evidence map›Paper›PMID 38286221›Full record

ArticleModern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc2024

Reproducible Reporting of the Collection and Evaluation of Annotations for Artificial Intelligence Models.

Katherine Elfer, Emma Gardecki, Victor Garcia, Amy Ly, Evangelos Hytopoulos, Si Wen, Matthew G Hanna, Dieter J E Peeters, Joel Saltz, Anna Ehinger and 10 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Modern pathology : an official journal of the United States and Canadian Academy of Pathology, Inc, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 8 papers.

0numbers the graph read from it
0cells of the map it votes in
8citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

8 citing papers in PubMed.

  1. Review
  2. Review
  3. Review
  4. Article
  5. Article
  6. Article
  7. Article
  8. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

20 authors.

Katherine ElferUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland. Electronic address: Katherine.Elfer@fda.hhs.gov.
Emma GardeckiUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
Victor GarciaUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
Amy LyDepartment of Pathology, Massachusetts General Hospital, Boston, Massachusetts.
Evangelos HytopoulosSystem Development, iRhythm Technologies Inc, San Francisco, California.
Si WenUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
Matthew G HannaDepartment of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York.
Dieter J E PeetersDepartment of Pathology, University Hospital Antwerp/University of Antwerp, Antwerp, Belgium; Department of Pathology, Sint-Maarten Hospital, Mechelen, Belgium.
Joel SaltzDepartment of Biomedical Informatics, Stony Brook University, Stony Brook, New York.
Anna EhingerDepartment of Clinical Genetics, Pathology and Molecular Diagnostics, Laboratory Medicine, Lund University, Lund, Sweden.
Sarah N DudgeonDepartment of Laboratory Medicine, Yale School of Medicine, New Haven, Connecticut.
Xiaoxian LiDepartment of Pathology and Laboratory Medicine, Emory University School of Medicine, Atlanta, Georgia.
Kim R M BlenmanDepartment of Internal Medicine, Section of Medical Oncology, Yale School of Medicine and Yale Cancer Center, Yale University, New Haven, Connecticut; Department of Computer Science, School of Engineering and Applied Science, Yale University, New Haven, Connecticut.
Weijie ChenUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.
Ursula GreenDepartment of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
Ryan BirminghamUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland; Department of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
Tony PanDepartment of Biomedical Informatics, Emory University School of Medicine, Atlanta, Georgia.
Jochen K LennerzDepartment of Pathology, Center for Integrated Diagnostics, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts.
Roberto SalgadoDivision of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia; Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium.
Brandon D GallasUnited States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics and Software Reliability, Silver Spring, Maryland.

Funding

IMMUNOHEMATOLOGY/TRANSFUSION MEDICINE RESEARCH TRAININGT32HL007974 · NHLBI · YALE UNIVERSITY · PI JEANNE E HENDRICKSON, Diane S Krause · 2001 to 2026
$8.1M
Illuminating the evolutionary history of colorectal cancer metastasis: basic principles and clinical applicationsR37CA225655 · NCI · MASSACHUSETTS GENERAL HOSPITAL · PI NAXEROVA, KAMILA · 2018 to 2024
$3.0M
FDA Office of Women's HealthNCI NIH HHS R37 CA225655NHLBI NIH HHS T32 HL007974
6 · The paper itself

Abstract

This work puts forth and demonstrates the utility of a reporting framework for collecting and evaluating annotations of medical images used for training and testing artificial intelligence (AI) models in assisting detection and diagnosis. AI has unique reporting requirements, as shown by the AI extensions to the Consolidated Standards of Reporting Trials (CONSORT) and Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) checklists and the proposed AI extensions to the Standards for Reporting Diagnostic Accuracy (STARD) and Transparent Reporting of a Multivariable Prediction model for Individual Prognosis or Diagnosis (TRIPOD) checklists. AI for detection and/or diagnostic image analysis requires complete, reproducible, and transparent reporting of the annotations and metadata used in training and testing data sets. In an earlier work by other researchers, an annotation workflow and quality checklist for computational pathology annotations were proposed. In this manuscript, we operationalize this workflow into an evaluable quality checklist that applies to any reader-interpreted medical images, and we demonstrate its use for an annotation effort in digital pathology. We refer to this quality framework as the Collection and Evaluation of Annotations for Reproducible Reporting of Artificial Intelligence (CLEARR-AI).

Indexed as

Artificial IntelligenceChecklistHumansImage Processing, Computer-AssistedPrognosisResearch DesignAnnotation StudyArtificial Intelligence ValidationData setdigital pathologyReference StandardReproducible Research

Identifiers

PMID38286221
PMCPMC12908141

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.