Quality, transparency, and readability of AI chatbot responses for patient counseling in robot-assisted colorectal cancer resection: a cross-sectional evaluation.

Robot-assisted colorectal cancer resection (RACR) has emerged as a preferred surgical approach for mid-to-low rectal tumors, yet patients often struggle to find accessible, accurate information about this complex procedure. Artificial intelligence (AI) chatbots increasingly serve as lay-friendly health information sources, but whether they deliver counseling of sufficient quality, transparency, and readability for RACR-specific queries remains unknown. To evaluate the quality, transparency, and readability of patient counseling information generated by four leading AI chatbots-ChatGPT (GPT-5.5), Google Gemini (Gemini 3.1 Pro), Anthropic Claude (Claude Sonnet 5), and DeepSeek (DeepSeek-V4)-in response to common questions about robot-assisted colorectal cancer resection. Thirty frequently asked patient questions spanning five clinical domains were posed to each chatbot on a single day (June 20, 2026), yielding 120 question-paired responses. Two independent raters assessed quality using the DISCERN instrument and the Patient Education Materials Assessment Tool (PEMAT). Transparency was measured with a 5-item checklist. Readability was computed via four validated formulas (Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, SMOG), with three syllable-free metrics (Coleman-Liau, Automated Readability Index, Dale-Chall) as a sensitivity analysis. Because the same 30 questions were posed to every chatbot, between-platform comparisons used the Friedman test with question as the blocking factor, with Bonferroni-corrected pairwise Wilcoxon signed-rank tests and Kendall's W effect sizes. Inter-rater reliability was estimated using intraclass correlation coefficients (ICC). ChatGPT achieved the highest mean DISCERN score (64.7 ± 7.0), the only platform whose mean reached the high-quality threshold (≥ 63), followed by Claude (61.2 ± 7.5), Gemini (59.4 ± 7.7), and DeepSeek (56.8 ± 8.2) (Friedman χ²(3) = 13.62, p = 0.003, Kendall's W = 0.15) (Fig. 1); after Bonferroni-corrected paired comparisons, only the ChatGPT-DeepSeek difference remained significant (p = 0.004). Transparency was uniformly deficient: only 27.5% of responses cited sources, and a mere 5.0% indicated information currency (Fig. 3). All chatbots produced text far exceeding the NIH-recommended sixth-grade level (mean Flesch-Kincaid Grade Level 12.3; platform means 11.4-13.7) (Fig. 4). DeepSeek yielded the most readable output (Flesch Reading Ease: 42.1), while ChatGPT generated the most complex text (Flesch Reading Ease: 26.4). A significant negative correlation between quality and readability was observed (ρ = -0.29, p = 0.001) (Fig. 5). AI chatbots furnish moderately good-quality counseling for RACR, yet persistent gaps in transparency and readability render them inadequate as standalone patient education tools. Between-platform quality differences were modest and, once the paired design and multiple comparisons were accounted for, remained significant only between ChatGPT and DeepSeek. No platform met recommended readability benchmarks. Because factual accuracy was not evaluated, no conclusion on clinical safety can be drawn. Clinicians should guide patients toward verified resources and consider supplying optimized prompt templates to enhance accessibility of AI-generated content.