ArticleNPJ digital medicine2025
Benchmarking the diagnostic performance of open source LLMs in 1933 Eurorad case reports.
Article in NPJ digital medicine, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 23 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
23 citing papers in PubMed.
- The Utility of Large Language Models to Assist With Emergency Triage Decisions Within Otolaryngology.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2026Article
- Diagnostic accuracy and consistency of ChatGPT-4o in radiology: influence of image, clinical data, and answer options on performance.Diagnostic and interventional radiology (Ankara, Turkey) · 2026Article
- Do Accuracy Gains Reflect Genuine Visual Understanding? A Multi-Model Evaluation of Vision-Language Models in Thoracic Imaging.Journal of imaging informatics in medicine · 2026Article
- Incremental Diagnostic Value of Clinical Information for Large Language Models Across Multiple Organs: Retrospective Study.Journal of medical Internet research · 2026Article
- Zero-Shot Classification of Postoperative Complications From Real-World Discharge Letters According to the Clavien-Dindo System Using Large Language Models in Liver Surgery: Comparative Study.Journal of medical Internet research · 2026Article
- Simplifying radiology reports with large language models: privacy-compliant open- versus closed-weight models.European radiology · 2026Article
- Predicting molecular types of adult-type diffuse gliomas based on MRI reports with large language models.European radiology · 2026Article
- Rethinking radiology AI benchmarks.Radiology advances · 2026Article
- Independent and collaborative performance of large language models and healthcare professionals in diagnosis and triage.NPJ digital medicine · 2026Article
- Benchmarking Open-Source Vision Language Models in Orthopedic In-Training Examination: A Comparison with Residents, Domain-Specific Evaluation, and Parameter Scaling.Clinics in orthopedic surgery · 2026Article
- A Comparative Assessment of ChatGPT, Gemini, and DeepSeek Accuracy: Examining Visual Medical Assessment in Internal Medicine Cases with and Without Clinical Context.Diagnostics (Basel, Switzerland) · 2026Article
- Exposing the fragility of LLM reasoning through bias-inducing prompts: evidence from BiasMedQA.BMJ digital health & AI · 2026Article
- Large Language Models for Preoperative Microvascular Invasion Prediction in Hepatocellular Carcinoma: A Multicenter Comparison with Radiologists and Treatment Outcomes.Journal of hepatocellular carcinoma · 2026Article
- Visual Large Language Models in Radiology: A Systematic Multimodel Evaluation of Diagnostic Accuracy and Hallucinations.Life (Basel, Switzerland) · 2026Article
- A review for navigating the trade-offs: evaluating open-source and proprietary large language models for clinical and biomedical information extraction.Frontiers in digital health · 2026Review
- AI-Automated Risk Operative Stratification for Severe Aortic Stenosis: A Proof-of-Concept Study.Journal of clinical medicine · 2025Article
- Accuracy of Large Language Models in Detecting Cases Requiring Immediate Reporting in Pediatric Radiology: A Feasibility Study Using Publicly Available Clinical Vignettes.Korean journal of radiology · 2025Article
- Accuracy of large language models in generating differential diagnosis from clinical presentation and imaging findings in pediatric cases.Pediatric radiology · 2025Article
- Evaluating prompt and data perturbation sensitivity in large language models for radiology reports classification.JAMIA open · 2025Article
- Benchmarking vision-language models for diagnostics in emergency and critical care settings.NPJ digital medicine · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Recent advancements in large language models (LLMs) have created new ways to support radiological diagnostics. While both open-source and proprietary LLMs can address privacy concerns through local or cloud deployment, open-source models provide advantages in continuity of access, and potentially lower costs. This study evaluated the diagnostic performance of fifteen open-source LLMs and one closed-source LLM (GPT-4o) in 1,933 cases from the Eurorad library. LLMs provided differential diagnoses based on clinical history and imaging findings. Responses were considered correct if the true diagnosis appeared in the top three suggestions. Models were further tested on 60 non-public brain MRI cases from a tertiary hospital to assess generalizability. In both datasets, GPT-4o demonstrated superior performance, closely followed by Llama-3-70B, revealing how open-source LLMs are rapidly closing the gap to proprietary models. Our findings highlight the potential of open-source LLMs as decision support tools for radiological differential diagnosis in challenging, real-world cases.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.