SynthesisBMC medical imaging2021
The reporting quality of natural language processing studies: systematic review of studies of radiology reports.
Synthesis in BMC medical imaging, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed, 2 syntheses or guidelines pooled it, 37 citations in OpenAlex.
- Large language models for simplifying radiology reports: a systematic review and meta-analysis of patient, public, and clinician evaluations.The Lancet. Digital health · 2026Pooled it
- Using natural language processing to analyze unstructured patient-reported outcomes data derived from electronic health records for cancer populations: a systematic review.Expert review of pharmacoeconomics & outcomes research · 2024Pooled it
- Introducing a New Research Design: The Multidimensional Interpretative Textual (MIT) Design. Epistemological and Methodological Foundations.Nursing inquiry · 2026Article
- Radiological Reporting Beyond Written Text: Evaluation of the Dynamic Video Reporting Model in Venous Doppler Ultrasonography from Patient and Surgeon Perspectives.Journal of imaging informatics in medicine · 2026Article
- Beyond Validation: Operationalising Post-Deployment Surveillance of AI Medical Devices in Clinical Practice.Journal of medical systems · 2026Article
- Clinically reported covert cerebrovascular disease and risk of neurological disease: a whole-population cohort of 367 988 people using natural language processing.Journal of neurology, neurosurgery, and psychiatry · 2026Article
- US-derived Pediatric Kidney Length and Volume Percentiles by Age: A Big Data Approach.Radiology. Artificial intelligence · 2026Article
- A Crosstalk Between Periodontal Disease and Human Immunodeficiency Virus: Application of Artificial Intelligence and Machine Learning in Risk Assessment and Diagnosis-A Narrative Review.Dentistry journal · 2025Review
- Improving the Reporting Quality of Studies on Information Extraction From Clinical Texts: Protocol for the Development of a Consensus-Based Reporting Guideline.JMIR research protocols · 2025Article
- Revolutionizing Radiology with Natural Language Processing and Chatbot Technologies: A Narrative Umbrella Review on Current Trends and Future Directions.Journal of clinical medicine · 2024Review
- A scoping review of large language model based approaches for information extraction from radiology reports.NPJ digital medicine · 2024Article
- Revolutionizing Pulmonary Diagnostics: A Narrative Review of Artificial Intelligence Applications in Lung Imaging.Cureus · 2024Review
- Comparison of an Ensemble of Machine Learning Models and the BERT Language Model for Analysis of Text Descriptions of Brain CT Reports to Determine the Presence of Intracranial Hemorrhage.Sovremennye tekhnologii v meditsine · 2024Article
- Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?The British journal of radiology · 2023Article
- Structured reporting of computed tomography in the polytrauma patient assessment: a Delphi consensus proposal.La Radiologia medica · 2023Article
- A scoping review of natural language processing of radiology reports in breast cancer.Frontiers in oncology · 2023Article
- A survey on clinical natural language processing in the United Kingdom from 2007 to 2022.NPJ digital medicine · 2022Review
- Rule-based natural language processing for automation of stroke data extraction: a validation study.Neuroradiology · 2022Article
- Text Analysis of Radiology Reports with Signs of Intracranial Hemorrhage on Brain CT Scans Using the Decision Tree Algorithm.Sovremennye tekhnologii v meditsine · 2022Article
- A Hybrid Matching Network for Fault Diagnosis under Different Working Conditions with Limited Data.Computational intelligence and neuroscience · 2022Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
13 authors at 3 institutions in 1 country.
Funding
Abstract
backgroundAutomated language analysis of radiology reports using natural language processing (NLP) can provide valuable information on patients' health and disease. With its rapid development, NLP studies should have transparent methodology to allow comparison of approaches and reproducibility. This systematic review aims to summarise the characteristics and reporting quality of studies applying NLP to radiology reports.
methodsWe searched Google Scholar for studies published in English that applied NLP to radiology reports of any imaging modality between January 2015 and October 2019. At least two reviewers independently performed screening and completed data extraction. We specified 15 criteria relating to data source, datasets, ground truth, outcomes, and reproducibility for quality assessment. The primary NLP performance measures were precision, recall and F1 score.
resultsOf the 4,836 records retrieved, we included 164 studies that used NLP on radiology reports. The commonest clinical applications of NLP were disease information or classification (28%) and diagnostic surveillance (27.4%). Most studies used English radiology reports (86%). Reports from mixed imaging modalities were used in 28% of the studies. Oncology (24%) was the most frequent disease area. Most studies had dataset size > 200 (85.4%) but the proportion of studies that described their annotated, training, validation, and test set were 67.1%, 63.4%, 45.7%, and 67.7% respectively. About half of the studies reported precision (48.8%) and recall (53.7%). Few studies reported external validation performed (10.8%), data availability (8.5%) and code availability (9.1%). There was no pattern of performance associated with the overall reporting quality.
conclusionsThere is a range of potential clinical applications for NLP of radiology reports in health services and research. However, we found suboptimal reporting quality that precludes comparison, reproducibility, and replication. Our results support the need for development of reporting standards specific to clinical NLP studies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.