Benchmarking next-generation large language models in otorhinolaryngology: a prospective, guideline-anchored, blinded evaluation of ChatGPT-5, Gemini-3, Grok-2, Llama-4, and DeepSeek-4 across case scenarios.

purposeLarge language models (LLMs) are increasingly utilized in medicine; however, their reliability in surgical and diagnostic specialties requiring guideline-anchored reasoning remains underexplored. The study systematically evaluated the performance of five contemporary LLMs in otorhinolaryngology (ENT).

methodsA prospective, two-phase, cross-sectional analysis compared ChatGPT-5 (OpenAI), Gemini-3 (Google), Grok 2 (xAI), Llama 4 (Meta), and DeepSeek-4 (DeepSeek) using 100 standardized ENT clinical scenarios derived from AAO-HNS, ERS, and IFOS guidelines. Each model answered 900 questions (4 open-ended and 5 multiple-choice questions per case), and the process was repeated one month later to assess temporal stability. Two board-certified otorhinolaryngologists, blinded to the study, rated all 9,000 responses for accuracy, comprehensiveness, and adherence to guidelines on a 6-point scale. Statistical analyses included Kruskal–Wallis, Dunn’s post-hoc, Wilcoxon, and ICC reliability testing.

resultsInter-rater reliability was excellent (ICC = 0.91; Cronbach’s α = 0.89). Significant inter-model differences were found across all domains (p < 0.001). ChatGPT-5 achieved the highest composite score (5.58 ± 0.24), outperforming Gemini-3 (5.42 ± 0.28; p < 0.01) and other models within a narrow, clinically credible band (5.05–5.30). Accuracy was higher for multiple-choice than open-ended items (p = 0.027), with ChatGPT-5 showing the smallest format gap. All models demonstrated strong alignment with international ENT guidelines and adhered to clinical safety standards. Temporal stability was greatest in ChatGPT-5 and Gemini-3, with minimal drift between test phases (Δ < 0.1).

conclusionAll evaluated LLMs delivered high factual and guideline-based reasoning in otolaryngology, with ChatGPT-5 exhibiting superior precision, completeness, and longitudinal stability. These findings support the supervised integration of LLMs as adjunctive tools for clinical education and decision support in ENT.