ArticleJapanese journal of radiology2024
Diagnostic accuracy of vision-language models on Japanese diagnostic radiology, nuclear medicine, and interventional radiology specialty board examinations.
Article in Japanese journal of radiology, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- A Rasch-based analysis comparing the performance of updated multimodal large language models and surgeons on the Japanese surgical specialist examination.Surgery today · 2026Article
- Artificial superintelligence alignment in healthcare.Japanese journal of radiology · 2026Review
- Insufficient reporting quality in large language model studies in the field of radiology.Insights into imaging · 2026Article
- Assessing accuracy and legitimacy of multimodal large language models on Japan Diagnostic Radiology Board Examination.Japanese journal of radiology · 2026Article
- Vision-language model performance on the Japanese Nuclear Medicine Board Examination: high accuracy in text but challenges with image interpretation.Annals of nuclear medicine · 2025Article
- Comparative performance of large language models in structuring head CT radiology reports: multi-institutional validation study in Japan.Japanese journal of radiology · 2025Article
- Evaluating GPT-4's visual interpretation and clinical reasoning on emergency settings: A 5-year analysis.Journal of the Chinese Medical Association : JCMA · 2025Observational
- Role of Artificial Intelligence in Surgical Training by Assessing GPT-4 and GPT-4o on the Japan Surgical Board Examination With Text-Only and Image-Accompanied Questions: Performance Evaluation Study.JMIR medical education · 2025Article
- Evaluation of radiology residents' reporting skills using large language models: an observational study.Japanese journal of radiology · 2025Observational
- Influence of prior probability information on large language model performance in radiological diagnosis.Japanese journal of radiology · 2025Article
- Article
- The critical need for an open medical imaging database in Japan: implications for global health and AI development.Japanese journal of radiology · 2025Article
- A Novel Multimodal Large Language Model for Interpreting Image-Based Ophthalmology Case Questions: Comparative Analysis of Multiple-Choice and Open-Ended Response.Beyoglu eye journal · 2025Article
- Artificial intelligence-assisted radiation imaging pathways for distinguishing uterine fibroids and malignant lesions in patients presenting with cancer pain: a literature review.Frontiers in oncology · 2025Review
- Generative AI and large language models in nuclear medicine: current status and future prospects.Annals of nuclear medicine · 2024Review
- "This Is a Quiz" Premise Input: A Key to Unlocking Higher Diagnostic Accuracy in Large Language Models.Cureus · 2024Article
- Comparison of coronary CT angiography and invasive coronary angiography results.Irish journal of medical science · 2024Article
- Custom GPTs Enhancing Performance and Evidence Compared with GPT-3.5, GPT-4, and GPT-4o? A Study on the Emergency Medicine Specialist Examination.Healthcare (Basel, Switzerland) · 2024Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
14 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeThe performance of vision-language models (VLMs) with image interpretation capabilities, such as GPT-4 omni (GPT-4o), GPT-4 vision (GPT-4V), and Claude-3, has not been compared and remains unexplored in specialized radiological fields, including nuclear medicine and interventional radiology. This study aimed to evaluate and compare the diagnostic accuracy of various VLMs, including GPT-4 + GPT-4V, GPT-4o, Claude-3 Sonnet, and Claude-3 Opus, using Japanese diagnostic radiology, nuclear medicine, and interventional radiology (JDR, JNM, and JIR, respectively) board certification tests. MATERIALS AND
methodsIn total, 383 questions from the JDR test (358 images), 300 from the JNM test (92 images), and 322 from the JIR test (96 images) from 2019 to 2023 were consecutively collected. The accuracy rates of the GPT-4 + GPT-4V, GPT-4o, Claude-3 Sonnet, and Claude-3 Opus were calculated for all questions or questions with images. The accuracy rates of the VLMs were compared using McNemar's test.
resultsGPT-4o demonstrated the highest accuracy rates across all evaluations with the JDR (all questions, 49%; questions with images, 48%), JNM (all questions, 64%; questions with images, 59%), and JIR tests (all questions, 43%; questions with images, 34%), followed by Claude-3 Opus with the JDR (all questions, 40%; questions with images, 38%), JNM (all questions, 42%; questions with images, 43%), and JIR tests (all questions, 40%; questions with images, 30%). For all questions, McNemar's test showed that GPT-4o significantly outperformed the other VLMs (all P < 0.007), except for Claude-3 Opus in the JIR test. For questions with images, GPT-4o outperformed the other VLMs in the JDR and JNM tests (all P < 0.001), except Claude-3 Opus in the JNM test.
conclusionThe GPT-4o had the highest success rates for questions with images and all questions from the JDR, JNM, and JIR board certification tests.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.