ArticleBMJ neurology open2023
Assessment of ChatGPT's performance on neurology written board examination questions.
Article in BMJ neurology open, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 26 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
26 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Evaluating reasoning-tuned large language models for clinical decision-making in spine surgery.Spine deformity · 2026Article
- Modern generative large language models in epilepsy care: a scoping review of current applications, challenges, and future directions.Acta epileptologica · 2026Review
- Humans vs. large language models in neurology board examination: performance, limitations, and reference reliability.Acta neurologica Belgica · 2026Article
- Comparison of the performance of ChatGPT-5, Gemini 3, Copilot, Perplexity, and medical students in answering neurology questions: a cross-sectional study.Scientific reports · 2026Article
- Article
- Artificial intelligence-based chatbots improve the efficiency of course orientation among medical students: a cross-sectional study.BMC medical education · 2025Article
- General AI May Revolutionize Neurology-Or It Might Be Bad.JAMA neurology · 2025Article
- Large Language Models in Neurological Practice: Real-World Study.Journal of medical Internet research · 2025Article
- Article
- Generative pre-trained transformer 4o (GPT-4o) in solving text-based multiple response questions for European Diploma in Radiology (EDiR): a comparative study with radiologists.Insights into imaging · 2025Article
- Assessing ChatGPT's summarization ofAbdominal radiology (New York) · 2025Article
- Exploring the potential of artificial intelligence chatbots in prosthodontics education.BMC medical education · 2025Article
- Article
- ChatGPT and Other Large Language Models in Medical Education - Scoping Literature Review.Medical science educator · 2025Review
- Age against the machine-susceptibility of large language models to cognitive impairment: cross sectional analysis.BMJ (Clinical research ed.) · 2024Article
- Evaluating AI Competence in Specialized Medicine: Comparative Analysis of ChatGPT and Neurologists in a Neurology Specialist Examination in Spain.JMIR medical education · 2024Article
- ChatGPT-4 Omni Performance in USMLE Disciplines and Clinical Skills: Comparative Analysis.JMIR medical education · 2024Article
- Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o.Clinical rheumatology · 2024Article
- Evaluation of responses of the large language model GPT to the neurology question of the week.Neurological sciences : official journal of the Italian Neurological Society and of the Italian Society of Clinical Neurophysiology · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background and objectives: ChatGPT has shown promise in healthcare. To assess the utility of this novel tool in healthcare education, we evaluated ChatGPT's performance in answering neurology board exam questions. Methods: Neurology board-style examination questions were accessed from BoardVitals, a commercial neurology question bank. ChatGPT was provided a full question prompt and multiple answer choices. First attempts and additional attempts up to three tries were given to ChatGPT to select the correct answer. A total of 560 questions (14 blocks of 40 questions) were used, although any image-based questions were disregarded due to ChatGPT's inability to process visual input. The artificial intelligence (AI) answers were then compared with human user data provided by the question bank to gauge its performance. Results: Out of 509 eligible questions over 14 question blocks, ChatGPT correctly answered 335 questions (65.8%) on the first attempt/iteration and 383 (75.3%) over three attempts/iterations, scoring at approximately the 26th and 50th percentiles, respectively. The highest performing subjects were pain (100%), epilepsy & seizures (85%) and genetic (82%) while the lowest performing subjects were imaging/diagnostic studies (27%), critical care (41%) and cranial nerves (48%). Discussion: This study found that ChatGPT performed similarly to its human counterparts. The accuracy of the AI increased with multiple attempts and performance fell within the expected range of neurology resident learners. This study demonstrates ChatGPT's potential in processing specialised medical information. Future studies would better define the scope to which AI would be able to integrate into medical decision making.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.