ArticleJMIR medical education2024
A Novel Evaluation Model for Assessing ChatGPT on Otolaryngology-Head and Neck Surgery Certification Examinations: Performance Study.
Article in JMIR medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- ChatGPT Versus DeepSeek in Resectability Categorization of Chronic Rhinosinusitis: A Comparison Based on EPOS 2020.World journal of otorhinolaryngology - head and neck surgery · 2026Article
- Turing problems in otolaryngology: a scoping review of the principal challenges of artificial ıntelligence and large language models.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Review
- Large language models versus human examinee performance on Israeli anesthesiology board examinations.Scientific reports · 2026Article
- Guideline-based benchmarking of large language models in otorhinolaryngology using 250 real-world cases.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2026Article
- Is It Getting Better? An Evaluation of Two Successive Generations of ChatGPT in Answering Specialized Vascular Surgery Questions.Vascular specialist international · 2025Article
- Applications and Future Perspectives of Large Language Models in Otolaryngology-Head and Neck Surgery: A Comprehensive Survey.Clinical and experimental otorhinolaryngology · 2025Review
- An Evaluation of Current Trends in AI-Generated Text in Otolaryngology Publications.The Laryngoscope · 2025Article
- Are ChatGPT's Answers to Questions About Salivary Gland Diseases Accurate and Reliable?Turkish archives of otorhinolaryngology · 2025Article
- Evaluation of the performance of different large language models on head and neck anatomy questions in the dentistry specialization exam in Turkey.Surgical and radiologic anatomy : SRA · 2025Article
- Applications of Natural Language Processing in Otolaryngology: A Scoping Review.The Laryngoscope · 2025Article
- Parental Perception on Usage of AI Chatbot to Understand Paediatric Otorhinolaryngology Condition: A Survey.Indian journal of otolaryngology and head and neck surgery : official publication of the Association of Otolaryngologists of India · 2025Article
- Chasing sleep physicians: ChatGPT-4o on the interpretation of polysomnographic results.European archives of oto-rhino-laryngology : official journal of the European Federation of Oto-Rhino-Laryngological Societies (EUFOS) : affiliated with the German Society for Oto-Rhino-Laryngology - Head and Neck Surgery · 2025Article
- ChatGPT's Attitude, Knowledge, and Clinical Application in Geriatrics Practice and Education: Exploratory Observational Study.JMIR formative research · 2025Observational
- Opportunities and Challenges of Chatbots in Ophthalmology: A Narrative Review.Journal of personalized medicine · 2024Review
- Exploring the Ability of ChatGPT to Act as a Research Aid in Otolaryngology.Indian journal of otolaryngology and head and neck surgery : official publication of the Association of Otolaryngologists of India · 2024Article
- Artificial intelligence large language model scores highly on focused practice designation in metabolic and bariatric surgery board practice questions.Surgical endoscopy · 2024Article
- Evaluating the Performance of ChatGPT, Gemini, and Bing Compared with Resident Surgeons in the Otorhinolaryngology In-service Training Examination.Turkish archives of otorhinolaryngology · 2024Article
- The Evaluation of Generative AI Should Include Repetition to Assess Stability.JMIR mHealth and uHealth · 2024Article
- Is ChatGPT 3.5 smarter than Otolaryngology trainees? A comparison study of board style exam questions.PloS one · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundChatGPT is among the most popular large language models (LLMs), exhibiting proficiency in various standardized tests, including multiple-choice medical board examinations. However, its performance on otolaryngology-head and neck surgery (OHNS) certification examinations and open-ended medical board certification examinations has not been reported.
objectiveWe aimed to evaluate the performance of ChatGPT on OHNS board examinations and propose a novel method to assess an AI model's performance on open-ended medical board examination questions.
methodsTwenty-one open-ended questions were adopted from the Royal College of Physicians and Surgeons of Canada's sample examination to query ChatGPT on April 11, 2023, with and without prompts. A new model, named Concordance, Validity, Safety, Competency (CVSC), was developed to evaluate its performance.
resultsIn an open-ended question assessment, ChatGPT achieved a passing mark (an average of 75% across 3 trials) in the attempts and demonstrated higher accuracy with prompts. The model demonstrated high concordance (92.06%) and satisfactory validity. While demonstrating considerable consistency in regenerating answers, it often provided only partially correct responses. Notably, concerning features such as hallucinations and self-conflicting answers were observed.
conclusionsChatGPT achieved a passing score in the sample examination and demonstrated the potential to pass the OHNS certification examination of the Royal College of Physicians and Surgeons of Canada. Some concerns remain due to its hallucinations, which could pose risks to patient safety. Further adjustments are necessary to yield safer and more accurate answers for clinical implementation.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.