How well do large language models answer postoperative lumbar fusion questions? A blinded comparative analysis of accuracy, quality, readability, and safety-related content.

Background: Patients recovering from lumbar fusion increasingly seek guidance from large language model (LLM) chatbots when their surgeon is unavailable, but the accuracy, completeness, and safety-related content of such responses have not been compared across models for the postoperative period. Objective: To compare the accuracy, quality, safety-related content, and readability of responses from ChatGPT, Gemini, and Claude to frequently asked postoperative lumbar fusion questions, and to identify the question types for which responses are weakest. Methods: Thirty questions, compiled by three neurosurgeons and grouped into six clinical categories, were submitted to each model, each with a standardized, patient-oriented prompt requesting warning signs. Ninety anonymized, randomized responses were rated independently by three blinded neurosurgeons for accuracy and four quality subdomains (clarity, completeness, relevance, and patient safety) on 4-point scales. Readability (Flesch-Kincaid, Flesch Reading Ease, and Gunning Fog), word count, and responses meeting an operational screening criterion for potential harm were also assessed. Models were compared with Friedman and Holm-corrected pairwise tests. Results: Accuracy was high for all models without significant differences. Quality, clarity, completeness, relevance, and patient safety differed globally, but effect sizes were small (Kendall's Conclusion: Responses from contemporary LLMs to common postoperative lumbar fusion questions were generally rated as accurate, with small between-model differences and a model-level divergence between readability and completeness. Because safety-related content was explicitly prompted and no patient outcomes were assessed, these findings describe the quality of safety-related information rather than clinical safety. LLMs may support postoperative patient education but should not replace physician guidance, particularly for warning signs and patient-specific recovery decisions.