ArticleiScience2023
Popular large language model chatbots' accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries.
Article in iScience, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 43 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
43 citing papers in PubMed, 5 syntheses or guidelines pooled it.
- Evaluating Large Language Models in Ophthalmology: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Transformative applications of oculomics-based AI approaches in the management of systemic diseases: A systematic review.Journal of advanced research · 2025Pooled it
- Evaluating the Performance of ChatGPT on Board-Style Examination Questions in Ophthalmology: A Meta-Analysis.Journal of medical systems · 2025Pooled it
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Testing and Evaluation of Health Care Applications of Large Language Models: A Systematic Review.JAMA · 2025Pooled it
- Evaluating large language models vs residents in cataract and refractive surgery: comparative analysis using the American Academy of Ophthalmology Self-Assessment Program.Journal of cataract and refractive surgery · 2026Trial
- Large Language Models in Ophthalmology: A Bibliographic Analysis.Turkish journal of ophthalmology · 2026Article
- Comparative performance of large language models in answering cornea and cataract surgery questions for resident training.BMC ophthalmology · 2026Article
- Enhancing Large Language Models for Improved Accuracy and Safety in Medical Question Answering: Comparative Study.JMIR medical education · 2025Article
- Feasibility analysis of generative artificial intelligence tools as a means of medical science popularization on urological diseases.Translational andrology and urology · 2025Article
- Enhancing diabetic retinopathy query responses: assessing large language model in ophthalmology.The British journal of ophthalmology · 2025Article
- Predicting Intraocular Collamer Lens Vault in Myopic Patients With Shallow Anterior Chamber Using FedEYE Platform and ChatGPT.Translational vision science & technology · 2025Article
- Large language models in ophthalmology: a scoping review on their utility for clinicians, researchers, patients, and educators.Eye (London, England) · 2025Article
- Performance of ChatGPT on optometry and vision science exam questions.Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists) · 2025Article
- Ophthalmological Question Answering and Reasoning Using OpenAI o1 vs Other Large Language Models.JAMA ophthalmology · 2025Article
- Evaluating Large Language Models in Ptosis-Related inquiries: A Cross-Lingual Study.Translational vision science & technology · 2025Article
- Enhancing the Accuracy of Human Phenotype Ontology Identification: Comparative Evaluation of Multimodal Large Language Models.Journal of medical Internet research · 2025Article
- Comparative assessment of artificial intelligence chatbots' performance in responding to healthcare professionals' and caregivers' questions about Dravet syndrome.Epilepsia open · 2025Article
- Integrated visual and text-based analysis of ophthalmology clinical cases using a large language model.Scientific reports · 2025Article
- Is Information About Musculoskeletal Malignancies From Large Language Models or Web Resources at a Suitable Reading Level for Patients?Clinical orthopaedics and related research · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
In light of growing interest in using emerging large language models (LLMs) for self-diagnosis, we systematically assessed the performance of ChatGPT-3.5, ChatGPT-4.0, and Google Bard in delivering proficient responses to 37 common inquiries regarding ocular symptoms. Responses were masked, randomly shuffled, and then graded by three consultant-level ophthalmologists for accuracy (poor, borderline, good) and comprehensiveness. Additionally, we evaluated the self-awareness capabilities (ability to self-check and self-correct) of the LLM-Chatbots. 89.2% of ChatGPT-4.0 responses were 'good'-rated, outperforming ChatGPT-3.5 (59.5%) and Google Bard (40.5%) significantly (all p < 0.001). All three LLM-Chatbots showed optimal mean comprehensiveness scores as well (ranging from 4.6 to 4.7 out of 5). However, they exhibited subpar to moderate self-awareness capabilities. Our study underscores the potential of ChatGPT-4.0 in delivering accurate and comprehensive responses to ocular symptom inquiries. Future rigorous validation of their performance is crucial to ensure their reliability and appropriateness for actual clinical use.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.