ArticleClinics and practice2023
Evaluating the Efficacy of ChatGPT in Navigating the Spanish Medical Residency Entrance Examination (MIR): Promising Horizons for AI in Clinical Medicine.
Article in Clinics and practice, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 41 papers, 4 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
41 citing papers in PubMed, 4 syntheses or guidelines pooled it, 72 citations in OpenAlex.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Performance of large language models in medical licensing examinations: a systematic review and meta-analysis.Journal of educational evaluation for health professions · 2025Pooled it
- Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.Journal of medical Internet research · 2024Pooled it
- Evaluating and leveraging large language models in clinical pharmacology and therapeutics assessment: From exam takers to exam shapers.British journal of clinical pharmacology · 2026Article
- Mapping Current Use of Artificial Intelligence in Pharmacology Education via a Scoping Review.Pharmacology research & perspectives · 2026Article
- [Comparative study on the application of large language models in pre-exam learning for the Chinese dental licen-sing examination].Hua xi kou qiang yi xue za zhi = Huaxi kouqiang yixue zazhi = West China journal of stomatology · 2026Article
- A large comprehensive comparison of large language models on ophthalmology board exams.Graefe's archive for clinical and experimental ophthalmology = Albrecht von Graefes Archiv fur klinische und experimentelle Ophthalmologie · 2026Article
- Zero-shot performance of selected large language and multimodal models on the 2023 Brazilian Portuguese medical residency exam.Scientific reports · 2026Article
- Performance Evaluation of Large Language Models in Multilingual Medical Multiple-Choice Questions: Mixed Methods Study.JMIR medical education · 2026Article
- A comparative analysis of embedded chatbot models and ChatGPT-4 for answering orthodontic treatment queries.Scientific reports · 2026Article
- Textbook-level medical knowledge in large language models: comparative evaluation using Japanese National Medical Examination.BMC medical informatics and decision making · 2026Article
- GPT-4o and OpenAI o1 Performance on the 2024 Spanish Competitive Medical Specialty Access Examination: Cross-Sectional Quantitative Evaluation Study.JMIR medical education · 2026Article
- Evaluation of large language models in medical examinations: A scoping review protocol.PloS one · 2026Article
- Performance benchmarking of LLMs on Chinese national medical licensing education: Cross-lingual and question-type effects.PloS one · 2026Article
- Assessing the Efficacy of Ortho GPT: A Comparative Study with Medical Students and General LLMs on Orthopedic Examination Questions.Bioengineering (Basel, Switzerland) · 2025Article
- Artificial Intelligence Chatbots and Their Responses to Most Searched Spanish Cancer Questions.Cancer medicine · 2025Article
- Applications, Challenges, and Prospects of Generative Artificial Intelligence Empowering Medical Education: Scoping Review.JMIR medical education · 2025Article
- Comparative evaluation of large language models performance in medical education using urinary system histology assessment.Scientific reports · 2025Article
- Performance of Open-Source Large Language Models in Psychiatry: Usability Study Through Comparative Analysis of Non-English Records and English Translations.Journal of medical Internet research · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
10 authors at 8 institutions in 6 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
The rapid progress in artificial intelligence, machine learning, and natural language processing has led to increasingly sophisticated large language models (LLMs) for use in healthcare. This study assesses the performance of two LLMs, the GPT-3.5 and GPT-4 models, in passing the MIR medical examination for access to medical specialist training in Spain. Our objectives included gauging the model's overall performance, analyzing discrepancies across different medical specialties, discerning between theoretical and practical questions, estimating error proportions, and assessing the hypothetical severity of errors committed by a physician. MATERIAL AND
methodsWe studied the 2022 Spanish MIR examination results after excluding those questions requiring image evaluations or having acknowledged errors. The remaining 182 questions were presented to the LLM GPT-4 and GPT-3.5 in Spanish and English. Logistic regression models analyzed the relationships between question length, sequence, and performance. We also analyzed the 23 questions with images, using GPT-4's new image analysis capability.
resultsGPT-4 outperformed GPT-3.5, scoring 86.81% in Spanish (
conclusionsGPT-4 performs robustly on the Spanish MIR examination, with varying capabilities to discriminate knowledge across specialties. While the model's high success rate is commendable, understanding the error severity is critical, especially when considering AI's potential role in real-world medical practice and its implications for patient safety.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.