Large Language Models Show Comparable Response Performance but Vary in Readability Regarding Patient Questions on Hip Arthroscopy.

Purpose: To compare the quality of large language model (LLM) responses to frequently asked questions regarding hip arthroscopy, assess the incorrect response rate of LLMs, and compare the readability among different LLM outputs. Methods: Three LLMs, including OpenAI Chat Generative Pre-Trained Transformer (ChatGPT) 3.5, Microsoft Co-Pilot, and Google Gemini, were each queried with 10 frequently asked questions regarding hip arthroscopy. Two high-volume hip arthroscopists graded the responses on a 4-point Likert scale (1 = excellent, requiring no clarification; 2 = satisfactory, requiring minimal clarification; 3 = satisfactory, requiring moderate clarification; and 4 = unsatisfactory, requiring substantial clarification). Additionally, the 2 graders ranked the responses from the 3 different LLMs for each of the 10 questions on a 3-point Likert scale (1 = best, 2 = intermediate, 3 = worst). Readability was assessed using the Flesch-Kincaid Grade Level and Flesch Reading Ease metrics. Results: Commonly used LLMs performed on a similar level of response accuracy and adequacy (mean ± SD: ChatGPT: 3.0 ± 1.0 vs Microsoft: 2.9 ± 1.1 vs Gemini: 2.6 ± 1.1, Conclusions: Hip arthroscopists find LLM outputs on patient questions regarding hip arthroscopy satisfactory but requiring moderate clarification and show no preference for one LLM's responses over another. LLMs produce a substantial number of incorrect responses. ChatGPT outputs had a significantly worse reading level compared to those of Microsoft and Gemini. Clinical Relevance: This study provides insights into the accuracy and readability of LLM-generated responses to commonly asked questions about hip arthroscopy. As patients increasingly turn to artificial intelligence tools for health information, understanding the quality and potential risks of misinformation becomes essential.