ArticlePLOS digital health2023
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models.
Article in PLOS digital health, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1,600 papers, 3 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
1,600 citing papers in PubMed, 3 syntheses or guidelines pooled it.
- Medical Students' Attitudes, Perceptions, and Self-Reported Familiarity With AI in Health Care: Systematic Review and Meta-Analysis.JMIR medical education · 2026Pooled it
- Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.International dental journal · 2026Pooled it
- Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses.BMC medical research methodology · 2026Pooled it
- Effects of Generative AI-Supported Virtual Reality Dental Skill Training on Learning Performance, Visual Behavior, and Cortical Activation Among Dental Students: Stratified Randomized Controlled Trial.Journal of medical Internet research · 2026Trial
- Effectiveness of ChatGPT and DeepSeek in Urology Medical Education: Randomized Controlled Trial.Journal of medical Internet research · 2026Trial
- Preliminary Evaluation of a Large Language Model-Powered Chatbot for Osteoporosis Self-Management Education: Formative Randomized Controlled Trial.JMIR formative research · 2026Trial
- Utilizing generative artificial intelligence to create simulated patient for history taking in gynecology.BMC medical education · 2026Trial
- Agreement, calibration, and failure of three large language models as high-stakes multimodal ospe graders: a comparative psychometric analysis.Medical education online · 2026Article
- Randomized Controlled Trial Comparing AI-Generated, Clinician-Edited to Human-Generated After-Visit Summaries.Mayo Clinic proceedings. Digital health · 2026Article
- The Rise of Small Language Models in Healthcare: A Comprehensive Survey.Computer science review · 2026Article
- How much do language and context affect the accuracy of AI health advice?BMJ health & care informatics · 2026Article
- Artificial intelligence for improving cardiovascular surgery training: A multiyear national exam analysis.Turk gogus kalp damar cerrahisi dergisi · 2026Article
- Automating the triage of rheumatology outpatient referrals: a comparative evaluation of 23 large language models under simple and advanced prompting.Rheumatology international · 2026Article
- Comparison of Two AI Chatbots for Diagnosis and Providing Treatment Suggestions in Retinopathy of Prematurity: Retrospective Study.Journal of medical Internet research · 2026Article
- Article
- Consistency and accuracy of different artificial intelligence models in evaluating longitudinal cracks and fractures of teeth.Odontology · 2026Article
- Entity-centric evaluation of large language model responses for medical question-answering tasks.PLOS digital health · 2026Article
- Large Language Models Versus Multidisciplinary Tumor Board Decisions in Thyroid Cancer.Laryngoscope investigative otolaryngology · 2026Article
- Interpreting antibody identification tests with large language models: Critical role of tabular data formatting.Transfusion · 2026Article
- Large language model-based triage to identify antiretroviral therapy adherence barriers and risk levels in patient messages.JAMIA open · 2026Article
1,540 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.