Risk-centered benchmarking of large language models for AI-enabled counseling in chronic autoimmune thyroid eye disease.

Background: Thyroid eye disease (TED) is a chronic autoimmune inflammatory orbital disease requiring activity assessment, risk stratification, and triage. As patients increasingly consult large language models (LLMs), evidence on their quality and safety for TED counseling remains limited. Methods: We conducted a cross-sectional benchmark using a prespecified 35-question Chinese TED counseling bank covering symptom recognition, activity assessment, treatment, daily management, follow-up, and care-seeking. The bank was developed from guideline-/consensus-derived scenarios, expert discussion, and recurrent patient-inquiry themes, then applied under a unified single-turn protocol. Five web-based LLM chatbots were evaluated: Gemini 3 Pro, ChatGPT-5.2, DeepSeek-V3.1, Doubao, and Qwen3-Max. Systems were accessed through official interfaces in Quzhou, China, during 27-29 December 2025, with identifiers recorded. Automated text analysis extracted output features, and response time was measured. Two blinded expert raters assessed alignment with a guideline-/consensus-informed reference standard using 5-point Likert scales for Accuracy, Logic, Coherence, Safety, and Content Accessibility. Between-model comparisons used repeated-measures methods, and correlations used Spearman analysis. Results: Response time differed significantly across models (Friedman χ Conclusion: LLMs show marked heterogeneity in efficiency, output structure, and clinical quality for TED counseling. Longer or slower responses do not necessarily indicate better performance. The absence of between-model Safety differences should not be interpreted as absolute safety or equivalence. Risk-centered, structured outputs emphasizing red-flag symptoms and care-seeking thresholds warrant validation through multi-turn dialogues, repeated sampling, patient/lay-user evaluation, and finer-grained safety endpoints.