ArticleBMC medical education2023
Assessment of the capacity of ChatGPT as a self-learning tool in medical pharmacology: a study using MCQs.
Article in BMC medical education, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 19 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
19 citing papers in PubMed, 1 synthesis or guideline pooled it, 43 citations in OpenAlex.
- Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.Journal of medical Internet research · 2024Pooled it
- Application of ChatGPT-assisted problem-based learning teaching method in clinical medical education.BMC medical education · 2025Trial
- Mapping Current Use of Artificial Intelligence in Pharmacology Education via a Scoping Review.Pharmacology research & perspectives · 2026Article
- Human vs. artificial intelligence: Physicians outperform ChatGPT in real-world pharmacotherapy counselling.British journal of clinical pharmacology · 2026Article
- Comparative performance of neurosurgery-specific, peer-reviewed versus general AI chatbots in bilingual board examinations: evaluating accuracy, consistency, and error minimization strategies.Acta neurochirurgica · 2025Article
- Performance of ChatGPT-3.5 and ChatGPT-4 in the Taiwan National Pharmacist Licensing Examination: Comparative Evaluation Study.JMIR medical education · 2025Article
- ChatGPT's Attitude, Knowledge, and Clinical Application in Geriatrics Practice and Education: Exploratory Observational Study.JMIR formative research · 2025Observational
- Evaluating ChatGPT-4's performance on oral and maxillofacial queries: Chain of Thought and standard method.Frontiers in oral health · 2025Article
- Qualitative metrics from the biomedical literature for evaluating large language models in clinical decision-making: a narrative review.BMC medical informatics and decision making · 2024Review
- Enhancing puncture skills training with generative AI and digital technologies: a parallel cohort study.BMC medical education · 2024Article
- Assessing the ability of GPT-4o to visually recognize medications and provide patient education.Scientific reports · 2024Article
- A framework for human evaluation of large language models in healthcare derived from literature review.NPJ digital medicine · 2024Article
- Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations.BMC medical education · 2024Article
- Article
- Education and Training Assessment and Artificial Intelligence. A Pragmatic Guide for Educators.British journal of biomedical science · 2024Review
- Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom's Taxonomy.Advances in medical education and practice · 2024Article
- Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance.Frontiers in public health · 2024Article
- The Potential Applications and Challenges of ChatGPT in the Medical Field.International journal of general medicine · 2024Review
- Performance of ChatGPT-4 on the Nepalese Undergraduate Medical Licensing Examination: A Cross-Sectional Study.Journal of medical education and curricular developmentArticle
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
1 author at 1 institution in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundChatGPT is a large language model developed by OpenAI that exhibits a remarkable ability to simulate human speech. This investigation attempts to evaluate the potential of ChatGPT as a standalone self-learning tool, with specific attention on its efficacy in answering multiple-choice questions (MCQs) and providing credible rationale for its responses.
methodsThe study used 78 test items from the Korean Comprehensive Basic Medical Sciences Examination (K-CBMSE) for years 2019 to 2021. 78 test items translated from Korean to English with four lead-in prompts per item resulted in a total of 312 MCQs. The MCQs were submitted to ChatGPT and the responses were analyzed for correctness, consistency, and relevance.
resultsChatGPT responded with an overall accuracy of 76.0%. Compared to its performance on recall and interpretation questions, the model performed poorly on problem-solving questions. ChatGPT offered correct rationales for 77.8% (182/234) of the responses, with errors primarily arising from faulty information and flawed reasoning. In terms of references, ChatGPT provided incorrect citations for 69.7% (191/274) of the responses. While the veracity of reference paragraphs could not be ascertained, 77.0% (47/61) were deemed pertinent and accurate with respect to the answer key.
conclusionThe current version of ChatGPT has limitations in accurately answering MCQs and generating correct and relevant rationales, particularly when it comes to referencing. To avoid possible threats such as spreading inaccuracies and decreasing critical thinking skills, ChatGPT should be used with supervision.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.