ArticleRevista da Associacao Medica Brasileira (1992)2023
Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation.
Article in Revista da Associacao Medica Brasileira (1992), 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.Journal of medical Internet research · 2024Pooled it
- Comparison of GPT-4 omni and physicians for accurate evaluation of traumatic extremity x-rays: A feasibility study.Medicine · 2026Article
- Textbook-level medical knowledge in large language models: comparative evaluation using Japanese National Medical Examination.BMC medical informatics and decision making · 2026Article
- Review
- Factors Associated with Lower Performance of Artificial Intelligence on Answering Undergraduate Medical Education Multiple-Choice Questions.Medical science educator · 2025Article
- Identification of Online Health Information Using Large Pretrained Language Models: Mixed Methods Study.Journal of medical Internet research · 2025Article
- Article
- Article
- Article
- A framework for human evaluation of large language models in healthcare derived from literature review.NPJ digital medicine · 2024Article
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook.Journal of medical Internet research · 2024Article
- Language discrepancies in the performance of generative artificial intelligence models: an examination of infectious disease queries in English and Arabic.BMC infectious diseases · 2024Article
- Evaluating the performance of ChatGPT-3.5 and ChatGPT-4 on the Taiwan plastic surgery board examination.Heliyon · 2024Article
- Article
- ChatGPT for Tinnitus Information and Support: Response Accuracy and Retest after Three and Six Months.Brain sciences · 2024Article
- Automated machine learning model for fundus image classification by health-care professionals with no coding experience.Scientific reports · 2024Article
- A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence-Based Models in Health Care Education and Practice: Development Study Involving a Literature Review.Interactive journal of medical research · 2024Article
- Performance of ChatGPT-4 in answering questions from the Brazilian National Examination for Medical Degree Revalidation.Revista da Associacao Medica Brasileira (1992) · 2024Article
- Human versus Artificial Intelligence: ChatGPT-4 Outperforming Bing, Bard, ChatGPT-3.5 and Humans in Clinical Chemistry Multiple-Choice Questions.Advances in medical education and practice · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
6 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
objectiveThe aim of this study was to evaluate the performance of ChatGPT-4.0 in answering the 2022 Brazilian National Examination for Medical Degree Revalidation (Revalida) and as a tool to provide feedback on the quality of the examination.
methodsA total of two independent physicians entered all examination questions into ChatGPT-4.0. After comparing the outputs with the test solutions, they classified the large language model answers as adequate, inadequate, or indeterminate. In cases of disagreement, they adjudicated and achieved a consensus decision on the ChatGPT accuracy. The performance across medical themes and nullified questions was compared using chi-square statistical analysis.
resultsIn the Revalida examination, ChatGPT-4.0 answered 71 (87.7%) questions correctly and 10 (12.3%) incorrectly. There was no statistically significant difference in the proportions of correct answers among different medical themes (p=0.4886). The artificial intelligence model had a lower accuracy of 71.4% in nullified questions, with no statistical difference (p=0.241) between non-nullified and nullified groups.
conclusionChatGPT-4.0 showed satisfactory performance for the 2022 Brazilian National Examination for Medical Degree Revalidation. The large language model exhibited worse performance on subjective questions and public healthcare themes. The results of this study suggested that the overall quality of the Revalida examination questions is satisfactory and corroborates the nullified questions.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.