Trial reportJournal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society2025
A Comparative Study of Large Language Models, Human Experts, and Expert-Edited Large Language Models to Neuro-Ophthalmology Questions.
Trial report in Journal of neuro-ophthalmology : the official journal of the North American Neuro-Ophthalmology Society, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 13 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
13 citing papers in PubMed, 1 synthesis or guideline pooled it, 17 citations in OpenAlex.
- Evaluating Large Language Models in Ophthalmology: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Generative artificial intelligence to augment ethical problem solving in ophthalmology: GPT-5.1 versus a human ethicist.International ophthalmology · 2026Article
- Ability of Large Language Models to Answer Patients' Questions and Generate Educational Materials for Uncommon Retinal Conditions.Journal of vitreoretinal diseases · 2026Article
- How Often Do Large Language Models Agree with Each Other-And with the Truth? A Consensus- and Complexity-Stratified Analysis of Data Extraction for Neuroimaging AI.Journal of clinical medicine · 2026Article
- Large Language Models in Ophthalmology: A Bibliographic Analysis.Turkish journal of ophthalmology · 2026Article
- Article
- Evaluating the clinical utility of multimodal large language models in rare maculopathy.Scientific reports · 2025Article
- Large Language Models in Neurological Practice: Real-World Study.Journal of medical Internet research · 2025Article
- Evaluation of Responses to Questions About Keratoconus Using ChatGPT-4.0, Google Gemini and Microsoft Copilot: A Comparative Study of Large Language Models on Keratoconus.Eye & contact lens · 2025Article
- Evaluating the competence of large language models in ophthalmology clinical practice: a multi-scenario quantitative study.Frontiers in cell and developmental biology · 2025Article
- Evaluating Large Language Models for Accuracy and Completeness of Vitiligo Patient Education: A Comparative Analysis.Clinical, cosmetic and investigational dermatology · 2025Article
- Evaluation and comparison of large language models' responses to questions related optic neuritis.Frontiers in medicine · 2025Article
- Exploring the Role of ChatGPT-4, BingAI, and Gemini as Virtual Consultants to Educate Families about Retinopathy of Prematurity.Children (Basel, Switzerland) · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
16 authors at 1 institution in 1 country.
Funding
Abstract
backgroundWhile large language models (LLMs) are increasingly used in medicine, their effectiveness compared with human experts remains unclear. This study evaluates the quality and empathy of Expert + AI, human experts, and LLM responses in neuro-ophthalmology.
methodsThis randomized, masked, multicenter cross-sectional study was conducted from June to July 2023. We randomly assigned 21 neuro-ophthalmology questions to 13 experts. Each expert provided an answer and then edited a ChatGPT-4-generated response, timing both tasks. In addition, 5 LLMs (ChatGPT-3.5, ChatGPT-4, Claude 2, Bing, Bard) generated responses. Anonymized and randomized responses from Expert + AI, human experts, and LLMs were evaluated by the remaining 12 experts. The main outcome was the mean score for quality and empathy, rated on a 1-5 scale.
resultsSignificant differences existed between response types for both quality and empathy ( P < 0.0001, P < 0.0001). For quality, Expert + AI (4.16 ± 0.81) performed the best, followed by GPT-4 (4.04 ± 0.92), GPT-3.5 (3.99 ± 0.87), Claude (3.6 ± 1.09), Expert (3.56 ± 1.01), Bard (3.5 ± 1.15), and Bing (3.04 ± 1.12). For empathy, Expert + AI (3.63 ± 0.87) had the highest score, followed by GPT-4 (3.6 ± 0.88), Bard (3.54 ± 0.89), GPT-3.5 (3.5 ± 0.83), Bing (3.27 ± 1.03), Expert (3.26 ± 1.08), and Claude (3.11 ± 0.78). For quality ( P < 0.0001) and empathy ( P = 0.002), Expert + AI performed better than Expert. Time taken for expert-created and expert-edited LLM responses was similar ( P = 0.75).
conclusionsExpert-edited LLM responses had the highest expert-determined ratings of quality and empathy warranting further exploration of their potential benefits in clinical settings.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.