AI for clinical neurology: Comparing five large language models in neurological diagnosis and lesion localization.

backgroundLarge language models (LLMs) are proposed as decision-support tools in medicine, but their ability to reason through neurological cases—requiring anatomoclinical correlation—remains uncertain.

methodsFive LLMs (Copilot GPT-5, DeepSeek-V3.1, ChatGPT 3.5, ChatGPT 5 Auto, Gemini 2.5) were tested on 86 neurological vignettes. Outputs were graded as Correct, Partial, or Incorrect for diagnosis and lesion site. Accuracy was analyzed using binary (correct vs. not correct), weighted (partial = 0.5), and ordinal (0–1–2 scale) methods. Global comparisons employed Cochran’s Q or Friedman tests; pairwise differences used McNemar or Wilcoxon tests with Holm correction. Interrater reliability was assessed with Fleiss’ κ, Cohen’s κ, and raw percent agreement.

resultsFor diagnosis, ChatGPT 5 Auto, Gemini 2.5, Copilot GPT-5, and DeepSeek-V3.1 each achieved > 90% accuracy, all significantly outperforming ChatGPT 3.5. For lesion localization, ChatGPT 5 Auto achieved the highest diagnostic accuracy, followed closely by Copilot GPT-5 and Gemini, while ChatGPT-3.5 lagged behind and had the highest rate of incorrect responses (13.9%). Weighted and ordinal analyses confirmed the same rank order, clustering the four advanced models together. Agreement was limited for diagnosis (Fleiss’ κ = 0.22) and for localization (Fleiss’ κ = 0.20).

conclusionAdvanced LLMs demonstrate high and comparable performance in neurology, while ChatGPT 3.5 is significantly less accurate. Errors were heterogeneous across models, suggesting potential ensemble benefits but reinforcing the need for human oversight, particularly in the irreplaceable human tasks of history‑taking, examination, and contextual clinical judgment.