ArticleJournal of the American College of Radiology : JACR2023
Evaluating GPT as an Adjunct for Radiologic Decision Making: GPT-4 Versus GPT-3.5 in a Breast Imaging Pilot.
Article in Journal of the American College of Radiology : JACR, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. It is linked to trial NCT07199231 (A Comparative Performance Evaluation of Four Publicly Available Large Language Models Against Gold Standard Medical References), which is not on this map. Cited by 189 papers, 4 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
A Comparative Performance Evaluation of Four Publicly Available Large Language Models Against Gold Standard Medical References
Who cites it
189 citing papers in PubMed, 4 syntheses or guidelines pooled it, 265 citations in OpenAlex.
- Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis.Journal of biomedical science · 2025Pooled it
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.JAMA · 2025Pooled it
- Utilizing large language models in breast cancer management: systematic review.Journal of cancer research and clinical oncology · 2024Pooled it
- Large language models and conditional rules in clinical decision support systems.Health information science and systems · 2026Article
- Appropriateness of Lumbar Spine Magnetic Resonance Imaging Requests: Guideline Concordance, Diagnostic Yield, Artificial-Intelligence Assessment and Derivation of a Simple Clinical Decision Rule.Diagnostics (Basel, Switzerland) · 2026Article
- Attitudes Toward Large Language Models in Health Care and Preferences for Their Adoption and Oversight Among Health Care Professionals: Cross-Sectional Survey.Journal of medical Internet research · 2026Article
- Comparison of AI-based Chatbot Performance in Analyzing Clinical Scenarios versus Medical Residents: A Novel Approach in Chest Diseases Education.Thoracic research and practice · 2026Article
- Large Language Models for Patient Education in Cardiovascular Imaging: Prospective Observational Comparative Study.JMIR formative research · 2026Observational
- Fine-tuned multimodal GPT-4o for generating diagnostic impressions in breast magnetic resonance imaging: insights into non-mass enhancement lesions.Quantitative imaging in medicine and surgery · 2026Article
- Voice-controlled super-resolution ultrasound imaging and reporting powered by multimodal large language models.NPJ digital medicine · 2026Article
- Evaluation of ChatGPT-4o's and DeepSeek R1's responses to urological problems: a comparative study.BMC urology · 2026Article
- Authoritative Textbook-Augmented Large Language Models for High-Altitude Public Health Medical Education in the Xizang Autonomous Region: Cross-Sectional Comparative Evaluation Study.Journal of medical Internet research · 2026Article
- Evaluation of ChatGPT-4o in Breast Cancer Screening: Insights from the 5th Edition BI-RADS Atlas and ACR Guidelines.Journal of imaging informatics in medicine · 2026Article
- Understanding uncertainty in large language model predictions of early death in critically ill patients: a conformal prediction approach.JAMIA open · 2026Article
- Integrated CT pipeline for automatic intracranial hemorrhage evaluation with GPT-enhanced clinical decision support.European radiology · 2026Article
- Use of Generative Artificial Intelligence in the Management of Low Back Pain: a Scoping Review.Journal of medical systems · 2026Article
- Artificial Intelligence Chatbots as Sources of Implant Dentistry Information for the Public: Validity and Reliability Assessment.European journal of dentistry · 2026Article
- Tracing the Pen: Electronic Health Records Amid the Rise of Generative AI.NPJ digital medicine · 2026Review
- GPT-4o for Automated Determination of Follow-up Examinations Based on Radiology Reports from Clinical Routine.Scientific reports · 2026Article
129 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
- Commented on by
- Commented on by
- Update of
Authors and funding
7 authors at 1 institution in 1 country.
Funding
Abstract
objectiveDespite rising popularity and performance, studies evaluating the use of large language models for clinical decision support are lacking. Here, we evaluate ChatGPT (Generative Pre-trained Transformer)-3.5 and GPT-4's (OpenAI, San Francisco, California) capacity for clinical decision support in radiology via the identification of appropriate imaging services for two important clinical presentations: breast cancer screening and breast pain.
methodsWe compared ChatGPT's responses to the ACR Appropriateness Criteria for breast pain and breast cancer screening. Our prompt formats included an open-ended (OE) and a select all that apply (SATA) format. Scoring criteria evaluated whether proposed imaging modalities were in accordance with ACR guidelines. Three replicate entries were conducted for each prompt, and the average of these was used to determine final scores.
resultsBoth ChatGPT-3.5 and ChatGPT-4 achieved an average OE score of 1.830 (out of 2) for breast cancer screening prompts. ChatGPT-3.5 achieved a SATA average percentage correct of 88.9%, compared with ChatGPT-4's average percentage correct of 98.4% for breast cancer screening prompts. For breast pain, ChatGPT-3.5 achieved an average OE score of 1.125 (out of 2) and a SATA average percentage correct of 58.3%, as compared with an average OE score of 1.666 (out of 2) and a SATA average percentage correct of 77.7%. DISCUSSION: Our results demonstrate the eventual feasibility of using large language models like ChatGPT for radiologic decision making, with the potential to improve clinical workflow and responsible use of radiology services. More use cases and greater accuracy are necessary to evaluate and implement such tools.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.