ArticleRadiology. Artificial intelligence2024
Performance of an Open-Source Large Language Model in Extracting Information from Free-Text Radiology Reports.
Article in Radiology. Artificial intelligence, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 61 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
61 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Performance of Natural Language Processing Model in Extracting Information from Free-Text Radiology Reports: A Systematic Review and Meta-Analysis.Journal of imaging informatics in medicine · 2026Pooled it
- Human researchers are superior to large language models in writing a medical systematic review in a comparative multitask assessment.Scientific reports · 2025Pooled it
- The performance of large language models in dentomaxillofacial radiology: a systematic review.Dento maxillo facial radiology · 2025Pooled it
- Automated Identification of Complex Percutaneous Coronary Intervention from Cardiac Catheterization Reports using Large Language Models.medRxiv : the preprint server for health sciences · 2026Article
- Transparent, Reproducible Text-Based Phenotyping of Lumbar Intervertebral Disc Degeneration from 500 Consecutive MRI Reports, with a Pre-Specified Image Analysis Framework.Diagnostics (Basel, Switzerland) · 2026Article
- Incremental Diagnostic Value of Clinical Information for Large Language Models Across Multiple Organs: Retrospective Study.Journal of medical Internet research · 2026Article
- Real vs virtual dissection: a region-based comparison of medical students' perceptions in a French faculty.Surgical and radiologic anatomy : SRA · 2026Article
- Large language models and prostate MRI reporting: a stringent testbed for safe deployment under evolving AI, health-data, and cybersecurity regulation.Abdominal radiology (New York) · 2026Review
- Pre-Imaging Clinical Factors Associated With Cardiac MR Image Quality Using Large Language Model-Enabled Data Extraction.Journal of magnetic resonance imaging : JMRI · 2026Article
- Reasoning Model-Assisted Second-Reader Quality Control of Chinese-Language Ultrasound Reports: A Retrospective Imaging Informatics Study.Journal of imaging informatics in medicine · 2026Article
- Implementing a Resource-Light and Low-Code Large Language Model System for Information Extraction from Mammography Reports: A Pilot Study.Journal of imaging informatics in medicine · 2026Article
- Adaptive Fast-Slow Large Language Model Framework for Multidimensional Classification of Prenatal Ultrasound Reports: Comparative Study.Journal of medical Internet research · 2026Article
- How green are large language models for radiology report labelling? Comparing human, rule-based and hybrid workflows.Insights into imaging · 2026Article
- Who labels best? Radiologists, rules, or large language models for CT reports on pulmonary embolism.European radiology experimental · 2026Article
- BI-RADS-compliant structured mammography reporting using locally deployed large language models under privacy constraints.European radiology · 2026Article
- Leveraging Large Language Models for Accurate AO Fracture Classification from CT Text Reports.Journal of imaging informatics in medicine · 2026Article
- Privacy-Preserving Generation of Structured Lymphoma Progression Reports from Cross-sectional Imaging: A Comparative Analysis of Llama 3.3 and Llama 4.Journal of imaging informatics in medicine · 2026Article
- Review
- AI-powered surveillance of bronchiolitis in the Nirsevimab era: comparative performance of machine learning, deep learning, and large language models on free-text ED records.BMC emergency medicine · 2026Article
- Natural language processing and LLMs in liver imaging: a practical review of clinical applications.Abdominal radiology (New York) · 2026Review
1 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
Authors and funding
10 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Purpose To assess the performance of a local open-source large language model (LLM) in various information extraction tasks from real-life emergency brain MRI reports. Materials and Methods All consecutive emergency brain MRI reports written in 2022 from a French quaternary center were retrospectively reviewed. Two radiologists identified MRI scans that were performed in the emergency department for headaches. Four radiologists scored the reports' conclusions as either normal or abnormal. Abnormalities were labeled as either headache-causing or incidental. Vicuna (LMSYS Org), an open-source LLM, performed the same tasks. Vicuna's performance metrics were evaluated using the radiologists' consensus as the reference standard. Results Among the 2398 reports during the study period, radiologists identified 595 that included headaches in the indication (median age of patients, 35 years [IQR, 26-51 years]; 68% [403 of 595] women). A positive finding was reported in 227 of 595 (38%) cases, 136 of which could explain the headache. The LLM had a sensitivity of 98.0% (95% CI: 96.5, 99.0) and specificity of 99.3% (95% CI: 98.8, 99.7) for detecting the presence of headache in the clinical context, a sensitivity of 99.4% (95% CI: 98.3, 99.9) and specificity of 98.6% (95% CI: 92.2, 100.0) for the use of contrast medium injection, a sensitivity of 96.0% (95% CI: 92.5, 98.2) and specificity of 98.9% (95% CI: 97.2, 99.7) for study categorization as either normal or abnormal, and a sensitivity of 88.2% (95% CI: 81.6, 93.1) and specificity of 73% (95% CI: 62, 81) for causal inference between MRI findings and headache. Conclusion An open-source LLM was able to extract information from free-text radiology reports with excellent accuracy without requiring further training.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.