ArticleCJEM2025
How well does GPT-4 perform on an emergency medicine board exam? A comparative assessment.
Article in CJEM, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1 paper.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
1 citing paper in PubMed.
- Large language models in emergency medicine education: opportunities, challenges, and implementation pathways.Frontiers in public health · 2026Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundRecent advancements in artificial intelligence have shown promise in enhancing diagnostic precision within healthcare sectors. In emergency departments, artificial intelligence has demonstrated potential for improving triage, guiding the choice of radiologic imaging and crafting individualized medical notes and discharge summaries, including tailored care plans. Advances in generative artificial intelligence have led to the development of sophisticated models such as OpenAI's GPT-4. This study assessed the ability of generative artificial intelligence in diagnosis and management in emergency medicine. Specifically, we compared GPT-4 with the performance of emergency medicine trainees in Canada, as gauged by the Canadian In-Training Examination.
methodsWe compared the performance of emergency medicine residents to GPT-4 on the Canadian in-training exams for the years 2021 and 2022. Each question was entered into a fresh GPT-4 chat and the first response was recorded without any prompting. GPT-4's responses were then assessed using the same marking grid that is employed for evaluating medical trainees. We then compared GPT-4'sscores to the average scores of each post-graduate year (PGY) level of residents across all FRCPC training programs. Ethical approval was obtained, then Canadian In-Training Examination committee provided exam questions and anonymized national results.
resultsThe participants in this study included 389 residents in 2021 and 333 residents in the 2022 exams. In 2021, mean trainee scores increased progressively across the levels, with PGY1 trainees scoring 48.0% (SD 15.6), PGY2 at 56.2% (SD 14.7), PGY3 at 59.8% (SD 16.7), PGY4 at 67.2% (12.3), and PGY5 at 70.1% (SD 12.5), whereas GPT-4 scored 88.7%. In 2022, a similar pattern, with PGY1 scoring 46.3% (SD 14.7), PGY2 at 51.8% (SD 14.7), PGY3 at 58.2% (SD 14.3), PGY4 at 66.2% (SD 15.3), and PGY5 at 64.3% (SD 8.5), while GPT-4 scored 82.0%.
conclusionsGPT-4 has shown impressive capabilities, surpassing the performance of medical trainees at different postgraduate levels in the clinical written exam. These findings highlight the potential of artificial intelligence to serve as a valuable support tool in medical practice. However, it should be used with caution and must not substitute for established, evidence-based medical resources.
Indexed as
Identifiers
40610779What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.