Evidence map›Paper›PMID 35157950›Full record

ArticleOphthalmology2022

Artificial Intelligence for Retinopathy of Prematurity: Validation of a Vascular Severity Scale against International Expert Diagnosis.

J Peter Campbell, Michael F Chiang, Jimmy S Chen, Darius M Moshfeghi, Eric Nudleman, Paisan Ruambivoonsuk, Hunter Cherwek, Carol Y Cheung, Praveer Singh, Jayashree Kalpathy-Cramer and 5 more

Abstract read
In one paragraph

Article in Ophthalmology, 2022. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 38 papers, 2 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
38citing papers in PubMed, 2 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

38 citing papers in PubMed, 2 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Article
  4. Article
  5. Current Trends in AI and Eye Disease Diagnostics.Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists) · 2026
    Review
  6. Impact of image enhancement on grading agreement of plus disease features in retinopathy of prematurity.Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie · 2026
    Article
  7. Article
  8. Article
  9. Article
  10. Review
  11. Article
  12. Review
  13. Article
  14. Article
  15. Article
  16. Article
  17. Article
  18. Article
  19. Article
  20. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

15 authors.

J Peter CampbellCasey Eye Institute, Department of Ophthalmology, Oregon Health & Science University, Portland, Oregon. Electronic address: campbelp@ohsu.edu.
Michael F ChiangNational Eye Institute, National Institutes of Health, Bethesda, Maryland.
Jimmy S ChenCasey Eye Institute, Department of Ophthalmology, Oregon Health & Science University, Portland, Oregon.
Darius M MoshfeghiByers Eye Institute, Horngren Family Vitreoretinal Center, Department of Ophthalmology, Stanford University, Palo Alto, California.
Eric NudlemanDepartment of Ophthalmology, University of California, San Diego, California.
Paisan RuambivoonsukDepartment of Ophthalmology, Rajavithi Hospital, Bangkok, Thailand.
Hunter CherwekOrbis International, New York, New York.
Carol Y CheungDepartment of Ophthalmology and Visual Sciences, Faculty of Medicine, The Chinese University of Hong Kong, Hong Kong, China.
Praveer SinghDepartment of Radiology, MGH/Harvard Medical School, Charlestown, Massachusetts; Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Massachusetts.
Jayashree Kalpathy-CramerDepartment of Radiology, MGH/Harvard Medical School, Charlestown, Massachusetts; Massachusetts General Hospital & Brigham and Women's Hospital Center for Clinical Data Science, Boston, Massachusetts.
Susan OstmoCasey Eye Institute, Department of Ophthalmology, Oregon Health & Science University, Portland, Oregon.
Malvina EydelmanCenter for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland.
R V Paul ChanDepartment of Ophthalmology and Visual Sciences, University of Illinois at Chicago, Chicago, Illinois.
Antonio CaponeAssociated Retinal Consultants, Oakland University William Beaumont School of Medicine, Royal Oak, Michigan.
Collaborative Community in Ophthalmic Imaging Executive Committee and the Collaborative Community in Ophthalmic Imaging Retinopathy of Prematurity Workgroup

Funding

Proteomics CoreP30EY010572 · NEI · OREGON HEALTH & SCIENCE UNIVERSITY · PI Kate E Keller · 1995 to 2026
$19.4M
Novel Glaucoma Diagnostics for Structure and Function - Renewal - 1R01EY013178 · NEI · WILLS EYE HEALTH SYSTEM · PI Joel S Schuman · 2000 to 2026
$17.5M
Stanford Vision Research CoreP30EY026877 · NEI · STANFORD UNIVERSITY · PI Jeffrey L Goldberg · 2017 to 2026
$8.0M
Endothelial Transmigration in Neovascular Age-related Macular DegenerationR01EY017011 · NEI · UNIV OF NORTH CAROLINA CHAPEL HILL · PI HARTNETT, MARY ELIZABETH RUTH · 2007 to 2025
$7.7M
Mechanisms of Angiogenesis of Retinopathy of PrematurityR01EY015130 · NEI · UNIV OF NORTH CAROLINA CHAPEL HILL · PI Mary Elizabeth Ruth Hartnett · 2005 to 2026
$7.6M
Clinical and Genetic Analysis of Retinopathy of PrematurityR01EY019474 · NEI · OREGON HEALTH & SCIENCE UNIVERSITY · PI CAMPBELL, JOHN PETER · 2010 to 2020
$7.0M
Artificial Intelligence Assisted Panoramic Optical Coherence Tomography Angiography for Retinopathy of PrematurityR01HD107494 · NICHD · OREGON HEALTH & SCIENCE UNIVERSITY · PI John Peter Campbell, Yifan Jian · 2021 to 2026
$2.4M
Clinical and genetic analysis of retinopathy of prematurityR01HD107493 · NICHD · OREGON HEALTH & SCIENCE UNIVERSITY · PI CAMPBELL, JOHN PETER · 2021 to 2023
$1.8M
Artificial intelligence assisted panoramic Optical Coherence Tomography Angiography for Retinopathy of PrematurityR01EY031331 · NEI · OREGON HEALTH & SCIENCE UNIVERSITY · PI CAMPBELL, JOHN PETER, JIAN, YIFAN · 2020 to 2020
$342k
NEI NIH HHS P30 EY010572NEI NIH HHS P30 EY026877NEI NIH HHS R01 EY013178NEI NIH HHS R01 EY015130NEI NIH HHS R01 EY017011NEI NIH HHS R01 EY019474NEI NIH HHS R01 EY031331NICHD NIH HHS R01 HD107493NICHD NIH HHS R01 HD107494
6 · The paper itself

Abstract

purposeTo validate a vascular severity score as an appropriate output for artificial intelligence (AI) Software as a Medical Device (SaMD) for retinopathy of prematurity (ROP) through comparison with ordinal disease severity labels for stage and plus disease assigned by the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), committee.

designValidation study of an AI-based ROP vascular severity score.

participantsA total of 34 ROP experts from the ICROP3 committee.

methodsTwo separate datasets of 30 fundus photographs each for stage (0-5) and plus disease (plus, preplus, neither) were labeled by members of the ICROP3 committee using an open-source platform. Averaging these results produced a continuous label for plus (1-9) and stage (1-3) for each image. Experts were also asked to compare each image to each other in terms of relative severity for plus disease. Each image was also labeled with a vascular severity score from the Imaging and Informatics in ROP deep learning system, which was compared with each grader's diagnostic labels for correlation, as well as the ophthalmoscopic diagnosis of stage.

main outcome measuresWeighted kappa and Pearson correlation coefficients (CCs) were calculated between each pair of grader classification labels for stage and plus disease. The Elo algorithm was also used to convert pairwise comparisons for each expert into an ordered set of images from least to most severe.

resultsThe mean weighted kappa and CC for all interobserver pairs for plus disease image comparison were 0.67 and 0.88, respectively. The vascular severity score was found to be highly correlated with both the average plus disease classification (CC = 0.90, P < 0.001) and the ophthalmoscopic diagnosis of stage (P < 0.001 by analysis of variance) among all experts.

conclusionsThe ROP vascular severity score correlates well with the International Classification of Retinopathy of Prematurity committee member's labels for plus disease and stage, which had significant intergrader variability. Generation of a consensus for a validated scoring system for ROP SaMD can facilitate global innovation and regulatory authorization of these technologies.

Indexed as

Retinopathy of PrematurityArtificial IntelligenceDiagnostic ImagingGestational AgeHumansInfant, NewbornOphthalmoscopyReproducibility of ResultsArtificial intelligenceDeep learningDisease classificationInterobserver agreementRetinopathy of prematuritySeverity score

Identifiers

PMID35157950
PMCPMC9232863

What OpenQuestion holds

Textmetadata
LicenceTDM
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.