ArticleRadiology2024
BI-RADS Category Assignments by GPT-3.5, GPT-4, and Google Bard: A Multilanguage Study.
Abstract read
In one paragraph
Article in Radiology, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 57 papers, 1 of them a synthesis that pooled it.
0numbers the graph read from it
0cells of the map it votes in
57citing papers in PubMed, 1 pooled it
–field-weighted citation impact
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
57 citing papers in PubMed, 1 synthesis or guideline pooled it.
- ESR Essentials: artificial intelligence in breast imaging-practice recommendations by the European Society of Breast Imaging.European radiology · 2026Guideline
- Incremental Diagnostic Value of Clinical Information for Large Language Models Across Multiple Organs: Retrospective Study.Journal of medical Internet research · 2026Article
- Error Detection and Correction in Chinese Radiology Reports Using Large Language Models: Real-World Clinical Validation Study.Journal of medical Internet research · 2026Article
- Automatic extraction of structured information from brain MRI reports using an open-weight large language model.European radiology · 2026Article
- Large Language Models Versus Human Readers in CAD-RADS 2.0 Categorization of Coronary CT Angiography Reports.Journal of imaging informatics in medicine · 2026Article
- Fine-tuned multimodal GPT-4o for generating diagnostic impressions in breast magnetic resonance imaging: insights into non-mass enhancement lesions.Quantitative imaging in medicine and surgery · 2026Article
- Review
- A comparative study of large language models (ChatGPT/DeepSeek) in generating structured thyroid ultrasound reports.Scientific reports · 2026Article
- Evaluation of ChatGPT-4o in Breast Cancer Screening: Insights from the 5th Edition BI-RADS Atlas and ACR Guidelines.Journal of imaging informatics in medicine · 2026Article
- A workflow utilizing general-purpose large language models for efficient structuring and data mining of bone scintigraphy narratives.Scientific reports · 2026Article
- Multiparameter concept-based interpretable model for early breast cancer diagnosis and structured reporting: a multi-center, multi-reader, radiologist-in-the-loop study.BMC medicine · 2026Article
- Error Detection in Emergency Radiology Reports Using a Large Language Model: Multistage Evaluation Study.Journal of medical Internet research · 2026Article
- Enhancing bone metastasis CT report analysis: a comparison of local and proprietary large language models for privacy and resource efficiency.BMC health services research · 2026Article
- Insufficient reporting quality in large language model studies in the field of radiology.Insights into imaging · 2026Article
- Retrieval-augmented generation-enhanced large language models for comprehensive CAD-RADS 2.0 categorization from structured coronary CTA reports.Radiologie (Heidelberg, Germany) · 2026Article
- Role of Model Size and Prompting Strategies in Extracting Labels from Free-Text Radiology Reports with Open-Source Large Language Models.Journal of imaging informatics in medicine · 2026Article
- Reliability of Gemini 2.5 Pro, ChatGPT 4.1, DeepSeek V3, and Claude Opus 4 in generating standardized CMR protocols.European radiology experimental · 2026Article
- [Research progress of large language models in tumor diagnosis: applications in textual reports and medical imaging].Nan fang yi ke da xue xue bao = Journal of Southern Medical University · 2026Review
- A randomized controlled trial of a WeChat-based artificial intelligence agent for postoperative care in orthopedic patients.NPJ digital medicine · 2026Article
- Evaluation of large language models in assigning PI-RADS v2.1 categories for prostate MRI reports.BMC urology · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
12 authors.
Andrea Cozzi *From the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0003-4922-7065
Katja Pinker *From the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0002-2722-7331
Andri HidberFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0009-0005-6239-3377
Tianyu ZhangFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0001-9891-6874
Luca BonomoFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0002-6978-9033
Roberto Lo GulloFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).
Blake ChristiansonFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0009-0004-9327-2626
Marco CurtiFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).
Stefania RizzoFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0002-5151-0866
Filippo Del GrandeFrom the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).
Ritse M Mann *From the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).ORCID 0000-0001-8111-1930
Simone Schiaffino *From the Imaging Institute of Southern Switzerland (IIMSI), Ente Ospedaliero Cantonale, Via Tesserete 46, 6900 Lugano, Switzerland (A.C., L.B., M.C., S.R., F.D.G., S.S.); Breast Imaging Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, NY (K.P., R.L.G., B.C.); Faculty of Biomedical Sciences, Università della Svizzera Italiana, Lugano, Switzerland (A.H., S.R., F.D.G., S.S.); Department of Radiology, Netherlands Cancer Institute, Amsterdam, the Netherlands (T.Z., R.M.M.); Department of Diagnostic Imaging, Radboud University Medical Center, Nijmegen, the Netherlands (T.Z., R.M.M.); and GROW Research Institute for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands (T.Z.).
Funding
X-RAY CRYSTALLOGRAPHYP30CA008748 · NCI · SLOAN-KETTERING INSTITUTE FOR CANCER RES · PI Hongbiao Carl Lekaye · 1985 to 2026
$347.4MNCI NIH HHS P30 CA008748
Abstract
Background The performance of publicly available large language models (LLMs) remains unclear for complex clinical tasks. Purpose To evaluate the agreement between human readers and LLMs for Breast Imaging Reporting and Data System (BI-RADS) categories assigned based on breast imaging reports written in three languages and to assess the impact of discordant category assignments on clinical management. Materials and Methods This retrospective study included reports for women who underwent MRI, mammography, and/or US for breast cancer screening or diagnostic purposes at three referral centers. Reports with findings categorized as BI-RADS 1-5 and written in Italian, English, or Dutch were collected between January 2000 and October 2023. Board-certified breast radiologists and the LLMs GPT-3.5 and GPT-4 (OpenAI) and Bard, now called Gemini (Google), assigned BI-RADS categories using only the findings described by the original radiologists. Agreement between human readers and LLMs for BI-RADS categories was assessed using the Gwet agreement coefficient (AC1 value). Frequencies were calculated for changes in BI-RADS category assignments that would affect clinical management (ie, BI-RADS 0 vs BI-RADS 1 or 2 vs BI-RADS 3 vs BI-RADS 4 or 5) and compared using the McNemar test. Results Across 2400 reports, agreement between the original and reviewing radiologists was almost perfect (AC1 = 0.91), while agreement between the original radiologists and GPT-4, GPT-3.5, and Bard was moderate (AC1 = 0.52, 0.48, and 0.42, respectively). Across human readers and LLMs, differences were observed in the frequency of BI-RADS category upgrades or downgrades that would result in changed clinical management (118 of 2400 [4.9%] for human readers, 611 of 2400 [25.5%] for Bard, 573 of 2400 [23.9%] for GPT-3.5, and 435 of 2400 [18.1%] for GPT-4;
Indexed as
Breast NeoplasmsAdultAgedBreastFemaleHumansLanguageMagnetic Resonance ImagingMammographyMiddle AgedRadiology Information SystemsRetrospective StudiesUltrasonography, Mammary
Identifiers
What OpenQuestion holds
Textmetadata
Read underepoch 390
Registered trials
None linked
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.