ArticleBMC oral health2026
Performance of five large language models in oral and maxillofacial surgery exam questions: a comparative study.
Article in BMC oral health, 2026. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1 paper.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
1 citing paper in PubMed.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundLarge language models (LLMs) have shown promise in medical education and assessment, but their performance on highly specialized, context-specific exams remains under-explored. This study aimed to evaluate the accuracy of five advanced LLMs – Qwen3, DeepSeek-R1-0528, ChatGPT o3-pro, Gemini 2.5 pro, and Claude 4.0 – in answering oral and maxillofacial surgery multiple-choice questions, and compare their performance.
methodsA total of 110 oral and maxillofacial surgery questions were collected from the 2013–2024 Chinese National Medical Licensing Examination (CNMLE), including 80 specialized knowledge questions and 30 case analysis questions. All questions were presented in Chinese to each LLM, and the correctness of each model’s first answer was recorded. We also attempted to verify the authenticity of any references provided in the models’ answers. Performance differences among models were analyzed with a significance level of P < 0.05.
resultsAll models achieved higher accuracy on knowledge-based questions than on case analysis questions. Qwen3 and DeepSeek showed no significant within-model accuracy difference between question types, whereas ChatGPT, Gemini, and Claude had significantly lower accuracy on case-based questions (P < 0.05). Across questions on tooth extraction, maxillofacial trauma/infection, and maxillofacial tumors, the models performed well (overall accuracy 87.3%–95.5%) with no significant inter-model differences. Qwen3 achieved the highest overall accuracy (95.5%), followed by DeepSeek (94.5%), ChatGPT (89.1%), and Gemini and Claude (87.3% each). However, 37.7%–43.9% of the references provided by the models were not verifiable, with no significant difference among the models.
conclusionLLMs demonstrated high accuracy on oral and maxillofacial surgery exam questions, indicating great potential as assistive tools in medical education and examinations. Nevertheless, their performance in complex case reasoning is notably lower and a substantial portion of their cited references are spurious. These findings highlight the need for further improvements in LLMs’ clinical reasoning capabilities and for reducing reference hallucinations before they can be reliably used in unsupervised clinical decision-making.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.