Evidence map›Paper›PMID 30799571›Full record

SynthesisKorean journal of radiology2019

Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers.

Dong Wook Kim, Hye Young Jang, Kyung Won Kim, Youngbin Shin, Seong Ho Park

Abstract readMeta-AnalysisSystematic Review
PubMed Full text at PMC Publisher
In one paragraph

Synthesis in Korean journal of radiology, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 212 papers, 21 of them syntheses that pooled it.

0numbers the graph read from it
0cells of the map it votes in
212citing papers in PubMed, 21 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

212 citing papers in PubMed, 21 syntheses or guidelines pooled it.

  1. Pooled it
  2. Pooled it
  3. Pooled it
  4. Pooled it
  5. Pooled it
  6. Pooled it
  7. Pooled it
  8. Pooled it
  9. Pooled it
  10. Pooled it
  11. Are current machine learning applications comparable to radiologist classification of degenerate and herniated discs and Modic change? A systematic review and meta-analysis.European spine journal : official publication of the European Spine Society, the European Spinal Deformity Society, and the European Section of the Cervical Spine Research Society · 2023
    Pooled it
  12. Pooled it
  13. Pooled it
  14. Pooled it
  15. Pooled it
  16. Pooled it
  17. Pooled it
  18. Pooled it
  19. Empirical assessment of bias in machine learning diagnostic test accuracy studies.Journal of the American Medical Informatics Association : JAMIA · 2020
    Pooled it
  20. Pooled it

152 more citing papers are in PubMed but not listed here.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

5 authors.

Dong Wook Kim *Department of Radiology, Taean-gun Health Center and County Hospital, Taean-gun, Korea.ORCID 0000-0001-7887-657X
Hye Young Jang *Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.ORCID 0000-0002-2420-8709
Kyung Won KimDepartment of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.ORCID 0000-0002-1532-5970
Youngbin ShinDepartment of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea.ORCID 0000-0003-2753-9586
Seong Ho ParkDepartment of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Korea. seongho@amc.seoul.kr.ORCID 0000-0002-1257-8315

Funding

No grant is acknowledged in the PubMed record.

6 · The paper itself

Abstract

objectiveTo evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images. MATERIALS AND

methodsPubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journals.

resultsOf 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals.

conclusionNearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

Indexed as

AlgorithmsCase-Control StudiesDatabases, FactualHumansImage Processing, Computer-AssistedAccuracyAppropriatenessArtificial intelligenceClinical trialClinical validationDeep learningMachine learningMeta-analysisQualityStudy designSystematic review

Identifiers

PMID30799571
PMCPMC6389801

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.