ArticleJMIR medical education2024
Performance of GPT-4V in Answering the Japanese Otolaryngology Board Certification Examination Questions: Evaluation Study.
Article in JMIR medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
15 citing papers in PubMed, 27 citations in OpenAlex.
- Clinical Applications of Multimodal Artificial Intelligence in Otolaryngology: A State-of-the-Art Review.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2026Review
- Critical Limitations in Comparing ChatGPT and DeepSeek for Orthopedic Assessment.JMIR formative research · 2026Article
- Guideline-based benchmarking of large language models in otorhinolaryngology using 250 real-world cases.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Article
- Multimodal large language models in ultrasound diagnosis of breast masses: a multicenter comparative analysis based on GPT-4o, radiologists, and convolutional neural network (CNN).Quantitative imaging in medicine and surgery · 2025Article
- Data Contamination in AI Evaluation.JMIR medical informatics · 2025Article
- Applications of Natural Language Processing in Otolaryngology: A Scoping Review.The Laryngoscope · 2025Article
- Exploring Generative Pre-Trained Transformer-4-Vision for Nystagmus Classification: Development and Validation of a Pupil-Tracking Process.JMIR formative research · 2025Article
- Customized GPT-4V(ision) for radiographic diagnosis: can large language model detect supernumerary teeth?BMC oral health · 2025Article
- Article
- Evaluating AI proficiency in nuclear cardiology: Large language models take on the board preparation exam.Journal of nuclear cardiology : official publication of the American Society of Nuclear Cardiology · 2025Article
- Custom GPTs Enhancing Performance and Evidence Compared with GPT-3.5, GPT-4, and GPT-4o? A Study on the Emergency Medicine Specialist Examination.Healthcare (Basel, Switzerland) · 2024Article
- Reforming China's Secondary Vocational Medical Education: Adapting to the Challenges and Opportunities of the AI Era.JMIR medical education · 2024Article
- Evaluating AI Proficiency in Nuclear Cardiology: Large Language Models take on the Board Preparation Exam.medRxiv : the preprint server for health sciences · 2024Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors at 3 institutions in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundArtificial intelligence models can learn from medical literature and clinical cases and generate answers that rival human experts. However, challenges remain in the analysis of complex data containing images and diagrams.
objectiveThis study aims to assess the answering capabilities and accuracy of ChatGPT-4 Vision (GPT-4V) for a set of 100 questions, including image-based questions, from the 2023 otolaryngology board certification examination.
methodsAnswers to 100 questions from the 2023 otolaryngology board certification examination, including image-based questions, were generated using GPT-4V. The accuracy rate was evaluated using different prompts, and the presence of images, clinical area of the questions, and variations in the answer content were examined.
resultsThe accuracy rate for text-only input was, on average, 24.7% but improved to 47.3% with the addition of English translation and prompts (P<.001). The average nonresponse rate for text-only input was 46.3%; this decreased to 2.7% with the addition of English translation and prompts (P<.001). The accuracy rate was lower for image-based questions than for text-only questions across all types of input, with a relatively high nonresponse rate. General questions and questions from the fields of head and neck allergies and nasal allergies had relatively high accuracy rates, which increased with the addition of translation and prompts. In terms of content, questions related to anatomy had the highest accuracy rate. For all content types, the addition of translation and prompts increased the accuracy rate. As for the performance based on image-based questions, the average of correct answer rate with text-only input was 30.4%, and that with text-plus-image input was 41.3% (P=.02).
conclusionsExamination of artificial intelligence's answering capabilities for the otolaryngology board certification examination improves our understanding of its potential and limitations in this field. Although the improvement was noted with the addition of translation and prompts, the accuracy rate for image-based questions was lower than that for text-based questions, suggesting room for improvement in GPT-4V at this stage. Furthermore, text-plus-image input answers a higher rate in image-based questions. Our findings imply the usefulness and potential of GPT-4V in medicine; however, future consideration of safe use methods is needed.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.