"MELMA" in otolaryngology: Medical evaluation of large language model answers. Clinician-rated scoring (MELMA-Q) and web-based auditing (MELMA-W) novel tools for AI assessment.

purposeTo propose a novel, standardized, and safety-centered tool for Large Language Models (LLMs) evaluation.

methodsThe Medical Evaluation of Large Language Model Answers Questionnaire (MELMA-Q) was developed as a 30-item clinician-rated instrument spanning seven domains: Medical Accuracy/Groundedness, Clinical Reasoning/Management, Safety/Ethics/Trustworthiness, Linguistic Quality/Semantic Fidelity, Understandability/Literacy Adaptation, Usefulness/Decision Support, and Performance/Answer Behavior. The MELMA Clinical Acceptability Framework (MELMA-CAF) is a two-tier system that incorporates a non-compensatory safety gate and weighted scoring. Five standardized otolaryngology scenarios were posed to three LLMs (ChatGPT 5.2, Gemini Flash 3, DeepSeek v3.2), generating 15 responses, which were independently scored by five blinded ENT specialists. A web-based implementation (MELMA-W) operationalized rubric-based scoring and was compared with clinician ratings.

resultsAll responses passed Tier A safety screening. Mean total MELMA-Q scores ranged from 72.4 to 85.6 across models; inter-rater reliability was excellent (ICC 0.89; 95% CI 0.84-0.93). MELMA-W validation using paired model × domain observations showed systematically higher clinician scores (bias of 0.804 Likert points).

conclusionsMELMA-Q and MELMA-W provide a structured, safety-centered pilot framework for evaluating LLM-generated medical responses in otolaryngology; however, broader validation across larger datasets, additional raters, and other clinical specialties remains required.