Performance evaluation of mainstream large language models in autoimmune hepatitis patient education: a comparative study of readability, quality, and reliability.

Background: Autoimmune hepatitis (AIH) is a chronic immune-mediated liver disease that requires long-term management, in which effective patient education plays a critical role. With the rapid development of large language models (LLMs), AI-generated health information is increasingly accessed by patients; however, the readability, quality, and educational suitability of LLM-generated AIH-related content remain insufficiently evaluated. Methods: Five widely used LLMs-ChatGPT, Doubao, DeepSeek, Wenxin Yiyan, and Tongyi Qianwen-were assessed based on their responses to 20 frequently asked AIH patient education questions covering five thematic categories. Text readability was evaluated using multiple indices, including the Automated Readability Index, Flesch Reading Ease Score, Gunning Fog Index, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG, and Linsear Write formula. Information quality and educational suitability were assessed using the Global Quality Score (GQS) and the Chinese version of the Patient Education Materials Assessment Tool (C-PEMAT). Clinical Intent Alignment (CIA) was used to evaluate the coverage of guideline-defined medical key points based on the 2025 EASL Clinical Practice Guidelines. Inter-rater reliability was analyzed using Cohen's kappa, and comparative and correlation analyses were performed. Results: Significant differences were observed among the LLMs in readability, information quality, and educational suitability (all Conclusion: Substantial variability exists among LLMs in generating AIH patient education materials. Model selection critically influences information quality and educational suitability, whereas content theme primarily affects linguistic complexity. Although most models produced moderate-to-good quality information, relatively high readability levels suggest that further simplification may be needed for general patient populations. A multidimensional evaluation framework integrating readability, quality, educational suitability, and clinical intent alignment is essential for the responsible use of LLMs in AIH patient education.