ArticleWorld journal of urology2024
How does artificial intelligence master urological board examinations? A comparative analysis of different Large Language Models' accuracy and reliability in the 2022 In-Service Assessment of the European Board of Urology.
Article in World journal of urology, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Automating Clavien-Dindo classification with large language models in percutaneous nephrolithotomy: an exploratory study.World journal of urology · 2026Article
- Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study.JMIR formative research · 2026Article
- [Artificial intelligence in surgical disciplines: Clinical application, advantages, and potential-a Delphi expert consensus].Urologie (Heidelberg, Germany) · 2026Article
- [Large language models as a communication and organizational infrastructure in urology: evidence, limitations, and clinical responsibility].Urologie (Heidelberg, Germany) · 2026Review
- Assessing multiple chatGPT versions on novel content in the Taiwan urology board examination: accuracy, speed, and domain-specific performance.World journal of urology · 2026Article
- From algorithms to operating room: can large language models master China's attending anesthesiology exam? A cross-sectional evaluation.International journal of surgery (London, England) · 2026Article
- [Digital urology : Possible uses for artificial intelligence and digital health applications].Urologie (Heidelberg, Germany) · 2025Review
- Large language models underperform in European general surgery board examinations: a comparative study with experts and surgical residents.BMC medical education · 2025Article
- Comparative analysis of the performance of the large language models DeepSeek-V3, DeepSeek-R1, open AI-O3 mini and open AI-O3 mini high in urology.World journal of urology · 2025Article
- Evaluating interactions of patients with large language models for medical information.BJU international · 2025Article
- Which current chatbot is more competent in urological theoretical knowledge? A comparative analysis by the European board of urology in-service assessment.World journal of urology · 2025Article
- Comparing Patient's Confidence in Clinical Capabilities in Urology: Large Language Models Versus Urologists.European urology open science · 2024Article
- Superhuman performance on urology board questions using an explainable language model enhanced with European Association of Urology guidelines.ESMO real world data and digital oncology · 2024Article
- Artificial intelligence versus human touch: can artificial intelligence accurately generate a literature review on laser technologies?World journal of urology · 2024Article
- Is ChatGPT ready for primetime? Performance of artificial intelligence on a simulated Canadian urology board exam.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2024Article
- Beyond the hype: Unveiling the challenges of large language models in urology.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2024Article
- ChatGPT and generative AI in urology and surgery-A narrative review.BJUI compass · 2024Review
- Performance of ChatGPT-3.5 and ChatGPT-4 on the European Board of Urology (EBU) exams: a comparative analysis.World journal of urology · 2024Article
- Artificial intelligence and master urological board examinations: comment.World journal of urology · 2024Article
Corrections and comments
- Commented on by
- Commented on by
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeThis study is a comparative analysis of three Large Language Models (LLMs) evaluating their rate of correct answers (RoCA) and the reliability of generated answers on a set of urological knowledge-based questions spanning different levels of complexity.
methodsChatGPT-3.5, ChatGPT-4, and Bing AI underwent two testing rounds, with a 48-h gap in between, using the 100 multiple-choice questions from the 2022 European Board of Urology (EBU) In-Service Assessment (ISA). For conflicting responses, an additional consensus round was conducted to establish conclusive answers. RoCA was compared across various question complexities. Ten weeks after the consensus round, a subsequent testing round was conducted to assess potential knowledge gain and improvement in RoCA, respectively.
resultsOver three testing rounds, ChatGPT-3.5 achieved RoCa scores of 58%, 62%, and 59%. In contrast, ChatGPT-4 achieved RoCA scores of 63%, 77%, and 77%, while Bing AI yielded scores of 81%, 73%, and 77%, respectively. Agreement rates between rounds 1 and 2 were 84% (κ = 0.67, p < 0.001) for ChatGPT-3.5, 74% (κ = 0.40, p < 0.001) for ChatGPT-4, and 76% (κ = 0.33, p < 0.001) for BING AI. In the consensus round, ChatGPT-4 and Bing AI significantly outperformed ChatGPT-3.5 (77% and 77% vs. 59%, both p = 0.010). All LLMs demonstrated decreasing RoCA scores with increasing question complexity (p < 0.001). In the fourth round, no significant improvement in RoCA was observed across all three LLMs.
conclusionsThe performance of the tested LLMs in addressing urological specialist inquiries warrants further refinement. Moreover, the deficiency in response reliability contributes to existing challenges related to their current utility for educational purposes.
Indexed as
Identifiers
38197996What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.