Evaluation of ChatGPT-4o

This study aims to evaluate the reliability, quality and readability of ChatGPT-4o responses regarding pediatric urolithiasis. Forty frequently asked questions about pediatric urinary stones were posed to ChatGPT-4o twice, one week apart. The reliability of ChatGPT-4o's responses was assessed using the five-point DISCERN tool (mDISCERN). The overall quality of the responses was evaluated using the Global Quality Scale (GQS). To assess the readability of ChatGPT-4o's responses, multiple metrics were employed, including the Flesch Reading Ease (FRE) score, the Flesch-Kincaid Grade Level (FKGL), the Gunning Fog Index (GFI), the Coleman-Liau Index (CLI), and the Simple Measure of Gobbledygook (SMOG). The median mDISCERN score was 5 (range: 4-5), and the median GQS score was 5 (range: 3-5), indicating high reliability and quality. However, readability metrics suggested a high level of difficulty: FRE (27.98 ± 13.65), FKGL (11.46 ± 1.88), SMOG (14.96 ± 1.64), GFI (17.27 ± 2.37), and CLI (15.60 ± 1.95). Only 2.5% of responses were understandable to individuals with reading skills at a 10-12-year-old level, 37.5% were suitable for college-level readers, and 60% required professional-level comprehension. A moderate correlation was observed between mDISCERN and GQS scores (r = 0.42, p = 0.007), but neither correlated significantly with readability metrics. ChatGPT-4o may provide reliable and high-quality information about pediatric urinary stones; however, the advanced reading level of its responses presents a significant barrier to accessibility for patients and caregivers. Therefore, despite its potential utility, the readability challenge must be addressed to ensure equitable patient education.