ArticleScientific reports2023
Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology.
Article in Scientific reports, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 53 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
53 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Evaluating Large Language Models in Ophthalmology: Systematic Review.Journal of medical Internet research · 2025Pooled it
- Evaluating the Performance of ChatGPT on Board-Style Examination Questions in Ophthalmology: A Meta-Analysis.Journal of medical systems · 2025Pooled it
- Comprehensive Evaluation of ChatGPT's Diagnostic Accuracy on Image-based Ophthalmic Case Interpretations.Ophthalmology science · 2026Article
- Evaluating how different versions of ChatGPT align with expert opinions on geriatric script concordance tests.European geriatric medicine · 2026Article
- AI-powered Chatbot integration for enhanced accessibility of electronic health records in a pediatric hospital.Scientific reports · 2026Article
- Evolving Minds: Natural Learning vs. Artificial Learning in Ophthalmology Training.Turkish journal of ophthalmology · 2026Observational
- Large Language Models for Psychiatric Diagnosis Based on Multicenter Real-World Clinical Records: Comparative Study.JMIR medical informatics · 2026Article
- Comparative performance of GPT-4, GPT-o3, GPT-5, Gemini-3-Flash, and DeepSeek-R1 in ophthalmology question answering.Frontiers in cell and developmental biology · 2026Article
- LLMs outperform outsourced human coders on complex textual analysis.Scientific reports · 2025Article
- StudyTypeTeller-Large language models to automatically classify research study types for systematic reviews.Research synthesis methods · 2025Article
- Comparative Assessment of Large Language Models in Optics and Refractive Surgery: Performance on Multiple-Choice Questions.Vision (Basel, Switzerland) · 2025Article
- Large language models in ophthalmology: a scoping review on their utility for clinicians, researchers, patients, and educators.Eye (London, England) · 2025Article
- Comparative Analysis of Large Language Model and Physician-Generated Responses in Bariatric Patient Inquiries: Assessing the Accuracy and Patient Satisfaction.Obesity surgery · 2025Article
- Comparative analysis of ChatGPT 3.5 and ChatGPT 4 obstetric and gynecological knowledge.Scientific reports · 2025Article
- Chatbot for the Return of Positive Genetic Screening Results for Hereditary Cancer Syndromes: Prompt Engineering Project.JMIR cancer · 2025Article
- Exploring Detection Methods for Synthetic Medical Datasets Created With a Large Language Model.JAMA ophthalmology · 2025Article
- Enhancing responses from large language models with role-playing prompts: a comparative study on answering frequently asked questions about total knee arthroplasty.BMC medical informatics and decision making · 2025Article
- Quality and reliability of pediatric pneumonia related short videos on mainstream platforms: cross-sectional study.BMC public health · 2025Article
- Assessing the Accuracy and Reliability of Large Language Models in Psychiatry Using Standardized Multiple-Choice Questions: Cross-Sectional Study.Journal of medical Internet research · 2025Article
- Diagnostic Performance of Publicly Available Large Language Models in Corneal Diseases: A Comparison with Human Specialists.Diagnostics (Basel, Switzerland) · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
To compare the performance of humans, GPT-4.0 and GPT-3.5 in answering multiple-choice questions from the American Academy of Ophthalmology (AAO) Basic and Clinical Science Course (BCSC) self-assessment program, available at https://www.aao.org/education/self-assessments . In June 2023, text-based multiple-choice questions were submitted to GPT-4.0 and GPT-3.5. The AAO provides the percentage of humans who selected the correct answer, which was analyzed for comparison. All questions were classified by 10 subspecialties and 3 practice areas (diagnostics/clinics, medical treatment, surgery). Out of 1023 questions, GPT-4.0 achieved the best score (82.4%), followed by humans (75.7%) and GPT-3.5 (65.9%), with significant difference in accuracy rates (always P < 0.0001). Both GPT-4.0 and GPT-3.5 showed the worst results in surgery-related questions (74.6% and 57.0% respectively). For difficult questions (answered incorrectly by > 50% of humans), both GPT models favorably compared to humans, without reaching significancy. The word count for answers provided by GPT-4.0 was significantly lower than those produced by GPT-3.5 (160 ± 56 and 206 ± 77 respectively, P < 0.0001); however, incorrect responses were longer (P < 0.02). GPT-4.0 represented a substantial improvement over GPT-3.5, achieving better performance than humans in an AAO BCSC self-assessment test. However, ChatGPT is still limited by inconsistency across different practice areas, especially when it comes to surgery.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.