Evidence map›Paper›PMID 38551771›Full record

ArticleJapanese journal of radiology2024

Comparison of natural language processing algorithms in assessing the importance of head computed tomography reports written in Japanese.

Tomohiro Wataya, Azusa Miura, Takahisa Sakisuka, Masahiro Fujiwara, Hisashi Tanaka, Yu Hiraoka, Junya Sato, Miyuki Tomiyama, Daiki Nishigaki, Kosuke Kita and 3 more

Abstract readComparative Study
In one paragraph

Article in Japanese journal of radiology, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 3 papers.

0numbers the graph read from it
0cells of the map it votes in
3citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

3 citing papers in PubMed.

  1. Article
  2. Article
  3. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

13 authors.

Tomohiro WatayaDepartment of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-1224-6601
Azusa MiuraDepartment of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.
Takahisa SakisukaDepartment of Diagnostic Imaging, Osaka General Medical Center, 3-1-56. Mandai Higashi, Sumiyoshi, Osaka, 558-8558, Japan.
Masahiro FujiwaraDepartment of Diagnostic Radiology, Sakai City Medical Center, 1-1-1, Ebaracho, Sakai, Osaka, 593-8304, Japan.ORCID http://orcid.org/0000-0003-3423-7658
Hisashi TanakaDivision of Health Science, Osaka University Graduate School of Medicine, 1-7, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-6148-0672
Yu HiraokaDepartment of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.
Junya SatoDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-1206-2036
Miyuki TomiyamaDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.
Daiki NishigakiDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-7006-3408
Kosuke KitaDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-5825-3348
Yuki SuzukiDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0002-1572-5771
Shoji KidoDepartment of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan. kido@radiol.med.osaka-u.ac.jp.ORCID http://orcid.org/0000-0001-5293-4110
Noriyuki TomiyamaDepartment of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan.ORCID http://orcid.org/0000-0001-9918-7327

Funding

KAKENHI 21H03840
6 · The paper itself

Abstract

purposeTo propose a five-point scale for radiology report importance called Report Importance Category (RIC) and to compare the performance of natural language processing (NLP) algorithms in assessing RIC using head computed tomography (CT) reports written in Japanese. MATERIALS AND

methods3728 Japanese head CT reports performed at Osaka University Hospital in 2020 were included. RIC (category 0: no findings, category 1: minor findings, category 2: routine follow-up, category 3: careful follow-up, and category 4: examination or therapy) was established based not only on patient severity but also on the novelty of the information. The manual assessment of RIC for the reports was performed under the consensus of two out of four neuroradiologists. The performance of four NLP models for classifying RIC was compared using fivefold cross-validation: logistic regression, bidirectional long-short-term memory (BiLSTM), general bidirectional encoder representations of transformers (general BERT), and domain-specific BERT (BERT for medical domain).

resultsThe proportion of each RIC in the whole data set was 15.0%, 26.7%, 44.2%, 7.7%, and 6.4%, respectively. Domain-specific BERT showed the highest accuracy (0.8434 ± 0.0063) in assessing RIC and significantly higher AUC in categories 1 (0.9813 ± 0.0011), 2 (0.9492 ± 0.0045), 3 (0.9637 ± 0.0050), and 4 (0.9548 ± 0.0074) than the other models (p < .05). Analysis using layer-integrated gradients showed that the domain-specific BERT model could detect important words, such as disease names in reports.

conclusionsDomain-specific BERT has superiority over the other models in assessing our newly proposed criteria called RIC of head CT radiology reports. The accumulation of similar and further studies of has a potential to contribute to medical safety by preventing missed important findings by clinicians.

Indexed as

Natural Language ProcessingTomography, X-Ray ComputedAlgorithmsEast Asian PeopleFemaleHeadHumansJapanMaleRadiology Information SystemsArtificial IntelligenceNatural Language ProcessingRadiology ReportReport ImportanceRisk Management

Identifiers

PMID38551771
PMCPMC11217108

What OpenQuestion holds

Textmetadata
LicenceCC BY
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.