Assessing retina-specific ophthalmic counseling generated by an early public large language model across different levels of clinical urgency.

Purpose: To evaluate how the quality of retina-specific ophthalmology counseling provided by an early publicly available large language model (LLM) differs when advising patients with varying clinical characteristics and risk factors. Design: Prospective, cross-sectional study. Subjects: 18 ophthalmologists. Methods: Six patient vignettes were constructed with high- and low-urgency clinical scenarios for diabetic retinopathy (DR), retinal detachment (RD), and age-related macular degeneration (AMD). Based on these vignettes, an LLM (ChatGPT-3.5) was asked to provide written medical counseling in February 2024. Each response was rated on several metrics via 5-point Likert scale by 18 independent reviewers. Notably, readability was assessed both qualitatively via survey and quantitatively via Readable (an online readability tool that incorporates 5 different metrics). Main outcome measures: Counseling generated by the LLM was graded on accuracy, appropriate communication of urgency and empathy, readability, and potential for clinically significant harm. Results: Counseling accuracy differed across levels of clinical urgency ( Conclusions: The evaluated LLM-generated counseling outputs were largely similar across the sampled retinal vignettes with differing clinical urgency. Future studies should investigate the optimization of LLM prompting needed to garner counseling of consistent/appropriate accuracy, readability, empathy, and communication of urgency for specific conditions.