Evidence map›Paper›PMID 38625446›Full record

ArticleInternational journal of computer assisted radiology and surgery2024

Performance changes due to differences among annotating radiologists for training data in computerized lesion detection.

Yukihiro Nomura, Shouhei Hanaoka, Naoto Hayashi, Takeharu Yoshikawa, Saori Koshino, Chiaki Sato, Momoko Tatsuta, Yuya Tanaka, Shintaro Kano, Moto Nakaya and 8 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in International journal of computer assisted radiology and surgery, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.

0numbers the graph read from it
0cells of the map it votes in
0citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

0 citing papers in PubMed.

No citing paper in PubMed yet.

4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

18 authors.

Yukihiro NomuraCenter for Frontier Medical Engineering, Chiba University, 1-33 Yayoi-cho, Inage-ku, Chiba, 263-8522, Japan. ynomura@chiba-u.jp.ORCID http://orcid.org/0000-0001-6471-9936
Shouhei HanaokaDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
Naoto HayashiDepartment of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
Takeharu YoshikawaDepartment of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
Saori KoshinoDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
Chiaki SatoDepartment of Radiology, Tokyo Metropolitan Bokutoh Hospital, Tokyo, Japan.
Momoko TatsutaDepartment of Diagnostic Radiology, Kitasato University Hospital, Sagamihara, Kanagawa, Japan.
Yuya TanakaDivision of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
Shintaro KanoDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
Moto NakayaDivision of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
Shohei InuiDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
Masashi KusakabeDepartment of Radiology, NTT Medical Center Tokyo, Tokyo, Japan.
Takahiro NakaoDepartment of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
Soichiro MikiDepartment of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, Tokyo, Japan.
Takeyuki WatadaniDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.
Ryusuke NakaokaDivision of Medical Devices, National Institute of Health Sciences, Kawasaki, Kanagawa, Japan.
Akinobu ShimizuInstitute of Engineering, Tokyo University of Agriculture and Technology, Tokyo, Japan.
Osamu AbeDepartment of Radiology, The University of Tokyo Hospital, Tokyo, Japan.

Funding

Japan Agency for Medical Research and Development 20mk0102148h0202Japan Agency for Medical Research and Development 21mk0102148h0203
6 · The paper itself

Abstract

purposeThe quality and bias of annotations by annotators (e.g., radiologists) affect the performance changes in computer-aided detection (CAD) software using machine learning. We hypothesized that the difference in the years of experience in image interpretation among radiologists contributes to annotation variability. In this study, we focused on how the performance of CAD software changes with retraining by incorporating cases annotated by radiologists with varying experience.

methodsWe used two types of CAD software for lung nodule detection in chest computed tomography images and cerebral aneurysm detection in magnetic resonance angiography images. Twelve radiologists with different years of experience independently annotated the lesions, and the performance changes were investigated by repeating the retraining of the CAD software twice, with the addition of cases annotated by each radiologist. Additionally, we investigated the effects of retraining using integrated annotations from multiple radiologists.

resultsThe performance of the CAD software after retraining differed among annotating radiologists. In some cases, the performance was degraded compared to that of the initial software. Retraining using integrated annotations showed different performance trends depending on the target CAD software, notably in cerebral aneurysm detection, where the performance decreased compared to using annotations from a single radiologist.

conclusionsAlthough the performance of the CAD software after retraining varied among the annotating radiologists, no direct correlation with their experience was found. The performance trends differed according to the type of CAD software used when integrated annotations from multiple radiologists were used.

Indexed as

Intracranial AneurysmRadiologistsSoftwareTomography, X-Ray ComputedClinical CompetenceDiagnosis, Computer-AssistedHumansImage Interpretation, Computer-AssistedLung NeoplasmsMachine LearningMagnetic Resonance AngiographyObserver VariationSolitary Pulmonary NoduleAnnotationComputer-aided detection (CAD)Machine learningRetraining

Identifiers

PMID38625446
PMCPMC11329536

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.