Mechanism-aware benchmarking of large language models as learning aids in pediatric difficult airway training: true hallucinations vs. false contraindications.

backgroundLarge language models (LLMs) are being investigated for their potential applications in perioperative decision-making and as informal "just-in-time" learning tools for trainees. This study evaluated five LLMs in pediatric difficult-airway management using a mechanism-aware scoring approach that distinguishes true hallucinations from false contraindications and examines their suitability for supervised learning.

methodsIn a simulation study involving a comparative blinded evaluation, 50 cases of pediatric difficult airway management were presented to five LLMs using a standardized prompt based on the 2022 American Society of Anesthesiologists Difficult Airway Guidelines. Two anesthesiologists evaluated the responses on a modified 0-5 scale.

resultsThe inter-rater agreement was deemed to be of the highest caliber, as evidenced by a quadratic weighted κ value of 0.982. A statistically significant variation in overall performance was observed among the models (Friedman χ²(4) = 100.12, p < 0.001). In this single-pass evaluation, GPT-5.2 Thinking had the highest mean score and proportion of acceptable responses, whereas Claude Opus 4.5 had the highest combined critical-error rate (23.0% vs. 6.5% for both GPT models). True hallucinations were most frequent with Claude Opus 4.5 (11.8%; other models 0.5-2.2%), while false contraindications occurred across all models (5.0-13.0%) and clustered around rocuronium. Scenario inadequacy (score ≤ 2) increased with complexity: 32-98% for difficult intubation, 46-94% for difficult ventilation, and 52-92% for CICO.

conclusionThe contemporary generation of LLMs exhibits model-specific performance characteristics and error patterns in pediatric difficult airway management. Dosing-related failures manifest through different mechanisms of true hallucination vs. false contraindication. The findings reveal a marked inadequacy of the scenarios employed in these models, particularly as complexity and critical error rates increase. Current LLMs may have value as supervised supplementary learning tools, but not as autonomous sources of clinical or educational guidance.