ArticleJournal of the American College of Radiology : JACR2019
Use of Machine Learning to Identify Follow-Up Recommendations in Radiology Reports.
Article in Journal of the American College of Radiology : JACR, 2019. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 30 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
30 citing papers in PubMed.
- The role of artificial intelligence in pediatric cardiovascular imaging: clinical applications and future directions in computed tomography and magnetic resonance imaging.Pediatric radiology · 2026Review
- Automatic Abstraction of Computed Tomography Imaging Indication Using Natural Language Processing for Evaluation of Surveillance Patterns in Long-Term Lung Cancer Survivors.JCO clinical cancer informatics · 2025Article
- Clinical applications of large language models in medicine and surgery: A scoping review.The Journal of international medical research · 2025Article
- Cumulative Effect of Targeted Interventions on Radiologist Recommendations for Additional Imaging.Radiology · 2025Article
- Use of ChatGPT Large Language Models to Extract Details of Recommendations for Additional Imaging From Free-Text Impressions of Radiology Reports.AJR. American journal of roentgenology · 2025Article
- Artificial intelligence in musculoskeletal applications: a primer for radiologists.Diagnostic and interventional radiology (Ankara, Turkey) · 2025Review
- How AI May Transform Musculoskeletal Imaging.Radiology · 2024Review
- Advancements in Standardizing Radiological Reports: A Comprehensive Review.Medicina (Kaunas, Lithuania) · 2023Review
- Development and External Validation of an Artificial Intelligence Model for Identifying Radiology Reports Containing Recommendations for Additional Imaging.AJR. American journal of roentgenology · 2023Article
- Artificial Intelligence to Improve Patient Understanding of Radiology Reports.The Yale journal of biology and medicine · 2023Review
- Impact of an Automated Closed-Loop Communication and Tracking Tool on the Rate of Recommendations for Additional Imaging in Thoracic Radiology Reports.Journal of the American College of Radiology : JACR · 2023Article
- Application of Deep Learning in Generating Structured Radiology Reports: A Transformer-Based Technique.Journal of digital imaging · 2023Article
- The promise and limitations of artificial intelligence in musculoskeletal imaging.Frontiers in radiology · 2023Review
- Identifying stroke diagnosis-related features from medical imaging reports to improve clinical decision-making support.BMC medical informatics and decision making · 2022Article
- Machine Learning Model Drift: Predicting Diagnostic Imaging Follow-Up as a Case Example.Journal of the American College of Radiology : JACR · 2022Article
- The role of artificial intelligence in paediatric cardiovascular magnetic resonance imaging.Pediatric radiology · 2022Review
- Tempering Expectations on the Medical Artificial Intelligence Revolution: The Medical Trainee Viewpoint.JMIR medical informatics · 2022Article
- Automated vetting of radiology referrals: exploring natural language processing and traditional machine learning approaches.Insights into imaging · 2022Article
- Automatic text classification of actionable radiology reports of tinnitus patients using bidirectional encoder representations from transformer (BERT) and in-domain pre-training (IDPT).BMC medical informatics and decision making · 2022Article
- Analysis of Radiology Report Recommendation Characteristics and Rate of Recommended Action Performance.JAMA network open · 2022Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
4 authors.
Funding
Abstract
purposeThe aims of this study were to assess follow-up recommendations in radiology reports, develop and assess traditional machine learning (TML) and deep learning (DL) models in identifying follow-up, and benchmark them against a natural language processing (NLP) system.
methodsThis HIPAA-compliant, institutional review board-approved study was performed at an academic medical center generating >500,000 radiology reports annually. One thousand randomly selected ultrasound, radiography, CT, and MRI reports generated in 2016 were manually reviewed and annotated for follow-up recommendations. TML (support vector machines, random forest, logistic regression) and DL (recurrent neural nets) algorithms were constructed and trained on 850 reports (training data), with subsequent optimization of model architectures and parameters. Precision, recall, and F1 score were calculated on the remaining 150 reports (test data). A previously developed and validated NLP system (iSCOUT) was also applied to the test data, with equivalent metrics calculated.
resultsFollow-up recommendations were present in 12.7% of reports. The TML algorithms achieved F1 scores of 0.75 (random forest), 0.83 (logistic regression), and 0.85 (support vector machine) on the test data. DL recurrent neural nets had an F1 score of 0.71; iSCOUT also had an F1 score of 0.71. Performance of both TML and DL methods by F1 scores appeared to plateau after 500 to 700 samples while training.
conclusionsTML and DL are feasible methods to identify follow-up recommendations. These methods have great potential for near real-time monitoring of follow-up recommendations in radiology reports.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.