SynthesisBMC medical education2024
Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis.
Synthesis in BMC medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 47 papers, 6 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
47 citing papers in PubMed, 6 syntheses or guidelines pooled it.
- The performance of ChatGPT and other large language models on multiple-choice questions in biomedical disciplines: A meta-analysis.Anatomical sciences education · 2026Pooled it
- Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.International dental journal · 2026Pooled it
- The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials.JMIR medical education · 2026Pooled it
- Educational Applications of ChatGPT in University-Based Dental Education. A Systematic Review.European journal of dental education : official journal of the Association for Dental Education in Europe · 2026Pooled it
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Performance of large language models in medical licensing examinations: a systematic review and meta-analysis.Journal of educational evaluation for health professions · 2025Pooled it
- Article
- Performance of Vision-Enabled Large Language Models in Image-Based Electrocardiogram Interpretation: Exploratory Evaluation.Journal of medical Internet research · 2026Article
- Article
- Multidimensional evaluation of large language models on the AAP in-service examination: Assessing accuracy, calibration, and citation reliability.PLOS digital health · 2026Article
- ChatGPT's performance on a specialist forensic pathology examination: implications for forensic pathologists and non-specialists.International journal of legal medicine · 2026Article
- Evaluating the clinical decision-making performance of large language models in clinically oriented thoracic anatomy scenarios: a comparative evaluation study.BMC medical education · 2026Article
- Dosage adjustments in renal impairment among medical ward patients: ChatGPT® and DeepSeek® models' effectiveness in assessing those adjustments.Exploratory research in clinical and social pharmacy · 2026Article
- ChatGPT in precision medicine.APL bioengineering · 2026Review
- The effectiveness of large language models in dental specialty questions: a comparative study in the field of prosthodontics.BMC medical education · 2026Article
- Comparing ChatGPT and Dental Students' Performance in an Introduction to Dental Anatomy Examination: A Cross-Sectional Study.European journal of dentistry · 2026Article
- Comparative performance evaluation of ChatGPT-4 Omni and Gemini Advanced in the Turkish Dentistry Specialization Exam.BMC medical education · 2026Article
- Article
- What the performance of multimodal LLMs on a national licensing exam teaches us about occupational therapy education.BMC medical education · 2026Article
- Diagnostic accuracy and repeatability of ChatGPT using textual and radiographic data in reversible pulpitis: a retrospective diagnostic study.Frontiers in bioinformatics · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
Abstract
backgroundChatGPT, a recently developed artificial intelligence (AI) chatbot, has demonstrated improved performance in examinations in the medical field. However, thus far, an overall evaluation of the potential of ChatGPT models (ChatGPT-3.5 and GPT-4) in a variety of national health licensing examinations is lacking. This study aimed to provide a comprehensive assessment of the ChatGPT models' performance in national licensing examinations for medical, pharmacy, dentistry, and nursing research through a meta-analysis.
methodsFollowing the PRISMA protocol, full-text articles from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals were reviewed from the time of ChatGPT's introduction to February 27, 2024. Studies were eligible if they evaluated the performance of a ChatGPT model (ChatGPT-3.5 or GPT-4); related to national licensing examinations in the fields of medicine, pharmacy, dentistry, or nursing; involved multiple-choice questions; and provided data that enabled the calculation of effect size. Two reviewers independently completed data extraction, coding, and quality assessment. The JBI Critical Appraisal Tools were used to assess the quality of the selected articles. Overall effect size and 95% confidence intervals [CIs] were calculated using a random-effects model.
resultsA total of 23 studies were considered for this review, which evaluated the accuracy of four types of national licensing examinations. The selected articles were in the fields of medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). They reported varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4-100% for GPT-4. The overall effect size for the percentage of accuracy was 70.1% (95% CI, 65-74.8%), which was statistically significant (p < 0.001). Subgroup analyses revealed that GPT-4 demonstrated significantly higher accuracy in providing correct responses than its earlier version, ChatGPT-3.5. Additionally, in the context of health licensing examinations, the ChatGPT models exhibited greater proficiency in the following order: pharmacy, medicine, dentistry, and nursing. However, the lack of a broader set of questions, including open-ended and scenario-based questions, and significant heterogeneity were limitations of this meta-analysis.
conclusionsThis study sheds light on the accuracy of ChatGPT models in four national health licensing examinations across various countries and provides a practical basis and theoretical support for future research. Further studies are needed to explore their utilization in medical and health education by including a broader and more diverse range of questions, along with more advanced versions of AI chatbots.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.