Response Quality of AI-Generated Answers to Orthodontic Patient Concerns Across Empathy, Accuracy, Comprehensiveness, Personalisation, Safety, and Patient-Centeredness: A Cross-Model, Bilingual Evaluation of Claude, GPT-4o, and Gemini.

backgroundPatients now consult large language models (LLMs) for orthodontic concerns, but most evaluations sit in English and focus on factual accuracy. We compared three current LLMs (Claude Opus 4.5, GPT-4o, Gemini 2.5 Flash) on patient-facing responses in English and Turkish, and tested whether model differences held up after adjusting for response length.

methodsSixty standardised patient scenarios were prompted in both languages to each model via stateless single-turn API calls with identical token limits, yielding 360 responses. Three blinded orthodontists scored every response on empathy, accuracy, comprehensiveness, personalisation, safety, and patient-centeredness (1-5 Likert). Primary between-model contrasts used mixed-effects linear models with scenario as a random effect to accommodate the paired scenario structure, with length-adjusted specifications adding response word count as a covariate; one-way ANOVA and ANCOVA on the balanced English sample were reported concordantly as descriptive checks. Reliability was assessed with ICC(2,3) and Krippendorff's α.

resultsForty-eight Turkish Gemini outputs (80%) were truncated mid-generation and were excluded, leaving 312 successfully completed responses. Between-model ranking is therefore reported on the balanced English cell (

conclusionsRubric-based scoring of LLM outputs is strongly associated with response length; unadjusted between-model comparisons may mainly reflect length differences rather than architectural differences. The English-Turkish gap and the Turkish Gemini truncation are robust to length adjustment. Per-language quality assurance-for both completeness and length-normalised content quality-is a prerequisite for patient-facing LLM deployment; English-only evaluation is not a safe proxy.