ArticleEuropean archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery2023
ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions.
Article in European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 82 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
82 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Clinical decision support using large language models in otolaryngology: a systematic review.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2025Pooled it
- Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care: Systematic Review.Journal of medical Internet research · 2024Pooled it
- Turing problems in otolaryngology: a scoping review of the principal challenges of artificial ıntelligence and large language models.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Review
- Physician Perspectives on ChatGPT-4o as a Patient Resource for Abdominal Cancer Surgeries: Cross-Sectional Survey.JMIR perioperative medicine · 2026Article
- Benchmarking next-generation large language models in otorhinolaryngology: a prospective, guideline-anchored, blinded evaluation of ChatGPT-5, Gemini-3, Grok-2, Llama-4, and DeepSeek-4 across case scenarios.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- The effectiveness of large language models in dental specialty questions: a comparative study in the field of prosthodontics.BMC medical education · 2026Article
- Comparative performance of ChatGPT, Gemini, and Deepseek on endodontic exam questions in Turkish and English.BMC oral health · 2026Article
- Performance of large language models on sleep medicine certification examination: a comprehensive multi-model analysis.Frontiers in medicine · 2026Article
- The reliability of answers from four different AI chatbots on periodontology theoretical exam questions: an evaluation in dental education.BMC oral health · 2025Article
- ChatGPT as an information source for cochlear implants: A study on accuracy and reproducibility.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2025Article
- Artificial Intelligence Models for Dysphonia Patient Education.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2025Article
- Ten tips to harnessing generative AI for high-quality MCQS in medical education assessment.Medical education online · 2025Article
- Benchmarking proprietary and open-source language and vision-language models for gastroenterology clinical reasoning.NPJ digital medicine · 2025Article
- Applications and Future Perspectives of Large Language Models in Otolaryngology-Head and Neck Surgery: A Comprehensive Survey.Clinical and experimental otorhinolaryngology · 2025Review
- An Evaluation of Current Trends in AI-Generated Text in Otolaryngology Publications.The Laryngoscope · 2025Article
- Are ChatGPT's Answers to Questions About Salivary Gland Diseases Accurate and Reliable?Turkish archives of otorhinolaryngology · 2025Article
- ChatGPT versus DeepSeek in head and neck cancer staging and treatment planning: guideline-based study.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2025Article
- Applications of Natural Language Processing in Otolaryngology: A Scoping Review.The Laryngoscope · 2025Article
- Article
22 more citing papers are in PubMed but not listed here.
Corrections and comments
- Commented on by
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeWith the increasing adoption of artificial intelligence (AI) in various domains, including healthcare, there is growing acceptance and interest in consulting AI models to provide medical information and advice. This study aimed to evaluate the accuracy of ChatGPT's responses to practice quiz questions designed for otolaryngology board certification and decipher potential performance disparities across different otolaryngology subspecialties.
methodsA dataset covering 15 otolaryngology subspecialties was collected from an online learning platform funded by the German Society of Oto-Rhino-Laryngology, Head and Neck Surgery, designed for board certification examination preparation. These questions were entered into ChatGPT, with its responses being analyzed for accuracy and variance in performance.
resultsThe dataset included 2576 questions (479 multiple-choice and 2097 single-choice), of which 57% (n = 1475) were answered correctly by ChatGPT. An in-depth analysis of question style revealed that single-choice questions were associated with a significantly higher rate (p < 0.001) of correct responses (n = 1313; 63%) compared to multiple-choice questions (n = 162; 34%). Stratified by question categories, ChatGPT yielded the highest rate of correct responses (n = 151; 72%) in the field of allergology, whereas 7 out of 10 questions (n = 65; 71%) on legal otolaryngology aspects were answered incorrectly.
conclusionThe study reveals ChatGPT's potential as a supplementary tool for otolaryngology board certification preparation. However, its propensity for errors in certain otolaryngology areas calls for further refinement. Future research should address these limitations to improve ChatGPT's educational use. An approach, with expert collaboration, is recommended for the reliable and accurate integration of such AI models.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.