Performance of Large Language Models on Exam-style Questions and Case Challenges Across Varying Levels of Complexity.

Large language models (LLMs) show significant promise for clinical applications. However, it remains unclear how their performance across structured medical examinations and narrative diagnostic tasks varies across cases of different complexities. This study addresses this gap by evaluating eight popular LLMs using two datasets with distinct complexity levels to determine the impact of case difficulty on examination and diagnostic performance. We tested the LLMs on two datasets: a structured, lower-complexity exam dataset (MedQA-USMLE) and a narrative, higher-complexity clinical dataset (NEJM Case Challenges). Eight representative LLMs were evaluated in our study, including ChatGPT, Copilot, DeepSeek, Doubao, Gemini, Grok, Mistral, and Qwen. We assessed performance using two primary outcomes: overall accuracy and a newly proposed difficulty-adjusted metric, the multi-model comparison score (MMCS). This metric was designed to quantify diagnostic robustness across varying case difficulties. We performed stratified analyses by grouping cases based on a case difficulty index (CDI). On MedQA-USMLE, the mean accuracy was 87.5% (range 78–92%), and the mean MMCS was 5.25. Gemini achieved the highest MMCS (6.88), followed by Grok (6.75) and DeepSeek (6.25). Performance declined sharply on the more complex NEJM Case Challenges dataset, with a mean accuracy of 50.4% (range 41.38–62.07%), and a mean MMCS of 3.64. Grok achieved the highest MMCS (6.50), followed by Gemini (5.75) and ChatGPT (4.75). In stratified analysis, the mean accuracy decreased from 97.63% in the easiest cases (CDI < 0.2) to 5% in the most difficult cases (CDI > 0.8). MMCS further distinguished models with similar overall accuracy by quantifying robustness as case difficulty increased. LLM performance is highly dependent on case complexity. Models excel at structured, low-difficulty tasks but struggle significantly with complex, narrative cases. These findings highlight the need to benchmark LLMs across a full spectrum of clinical difficulty and to incorporate difficulty-adjusted metrics (such as MMCS) before clinical integration.