Artificial Intelligence Performance Under Different Conditions in Answering China's Standardized Training Examination for Resident Physician in Radiology: A Comparative Analysis.

Background: The capabilities of general-purpose large language models (LLMs) on specialized medical examinations have not been systematically compared. To evaluate the performance differences among three LLMs-DeepSeek-R1, ChatGPT-o1, and Gemini-2.0-in answering questions from China's Standardized Training Examination for Resident Physicians in Radiology, and to assess the impact of different questioning conditions (with/without answer choices, and the introduction of doubt) on model accuracy. Methods: A total of 131 questions were analyzed at one time. The LLMs were subjected to two tasks (questions with/without answer choices). Each task included three conditions: no doubt, weak doubt, and strong doubt, with the latter two presented as follow-up questions after the models' initial responses. Subjective evaluation was conducted using the 5-point Likert scale. Results: In both tasks, Gemini-2.0 achieved the highest accuracy (0.763-0.809) and (0.595-0.679). In Task 2, all LLMs' accuracy was lower than in Task 1, but only DeepSeek-R1 showed statistical significance ( Conclusions: LLMs exhibit variable proficiency in tackling radiology resident examination questions, with Gemini-2.0 showing the highest overall accuracy. However, repeated self-examination of LLMs through the introduction of doubt does not consistently or significantly enhance their performance on radiology-related questions.