ArticleScientific reports2026
Benchmarking large language models on persian surgical subspecialty board examinations: a comparative study of ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash.
Article in Scientific reports, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
This study evaluated the performance of three large language models, including ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash, on 532 Persian multiple-choice questions from the 2025 Iranian surgical subspecialty board examinations. Questions spanned five domains: Pediatric, Cardiovascular, Vascular and Endovascular, Thoracic, and Plastic & Reconstructive Surgery. Using standardized prompts, we assessed overall accuracy, variation across subspecialties and question types, and the effect of question length. Gemini 2.5 Flash and ChatGPT-5 achieved higher accuracy (73.9% and 73.3%) than ChatGPT-4o (68.2%). Agreement with the official key was substantial for Gemini 2.5 Flash (κ = 0.651) and ChatGPT-5 (κ = 0.642), and moderate to substantial for ChatGPT-4o (κ = 0.575). Model performance was stable across subspecialties, but all three showed lower accuracy on surgical technique questions compared with clinical scenarios or basic science items. Question length did not affect ChatGPT-5 or Gemini 2.5 Flash, while longer stems reduced ChatGPT-4o's performance. These findings indicate that newer LLMs provide measurable improvements in surgical question answering, though lower performance on surgical technique items supports cautious integration and further multimodal development.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.