ArticleCanadian Urological Association journal = Journal de l'Association des urologues du Canada2024
Is ChatGPT ready for primetime? Performance of artificial intelligence on a simulated Canadian urology board exam.
Article in Canadian Urological Association journal = Journal de l'Association des urologues du Canada, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 11 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
11 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Artificial intelligence in urological education: a systematic review.BMC medical education · 2026Pooled it
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study.JMIR formative research · 2026Article
- Evaluation of ChatGPT's performance on answering pediatric urology questions based on association guidelines.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2025Article
- Comparative Assessment of Large Language Models in Optics and Refractive Surgery: Performance on Multiple-Choice Questions.Vision (Basel, Switzerland) · 2025Article
- Use of Artificial Intelligence Methods for Improved Diagnosis of Urinary Tract Infections and Urinary Stone Disease.Journal of clinical medicine · 2025Review
- Resident assessment in urology: ChatGPT, take the wheel?Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2025Article
- Use of artificial intelligence-generated multiple-choice questions for the examination of surgical subspecialty residents Report of feasibility and psychometric analysis.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2025Article
- Artificial Intelligence Use in Medical Education: Best Practices and Future Directions.Current urology reports · 2025Review
- Beyond the hype: Unveiling the challenges of large language models in urology.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2024Article
- Artificial Intelligence can Facilitate Application of Risk Stratification Algorithms to Bladder Cancer Patient Case Scenarios.Clinical Medicine Insights. Oncology · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
introductionGenerative artificial intelligence (AI) has proven to be a powerful tool with increasing applications in clinical care and medical education. ChatGPT has performed adequately on many specialty certification and knowledge assessment exams. The objective of this study was to assess the performance of ChatGPT 4 on a multiple-choice exam meant to simulate the Canadian urology board exam.
methodsGraduating urology residents representing all Canadian training programs gather yearly for a mock exam that simulates their upcoming board-certifying exam. The exam consists of written multiple-choice questions (MCQs) and an oral objective structured clinical examination (OSCE). The 2022 exam was taken by 29 graduating residents and was administered to ChatGPT 4.
resultsChatGPT 4 scored 46% on the MCQ exam, whereas the mean and median scores of graduating urology residents were 62.6%, and 62.7%, respectively. This would place ChatGPT's score 1.8 standard deviations from the median. The percentile rank of ChatGPT would be in the sixth percentile. ChatGPT scores on different topics of the exam were as follows: oncology 35%, andrology/benign prostatic hyperplasia 62%, physiology/anatomy 67%, incontinence/female urology 23%, infections 71%, urolithiasis 57%, and trauma/reconstruction 17%, with ChatGPT 4's oncology performance being significantly below that of postgraduate year 5 residents.
conclusionsChatGPT 4 underperforms on an MCQ exam meant to simulate the Canadian board exam. Ongoing assessments of the capability of generative AI is needed as these models evolve and are trained on additional urology content.
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.