ArticlePloS one2023
ChatGPT versus human in generating medical graduate exam multiple choice questions-A multinational prospective study (Hong Kong S.A.R., Singapore, Ireland, and the United Kingdom).
Article in PloS one, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. An erratum has been issued. Cited by 73 papers, 5 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
73 citing papers in PubMed, 5 syntheses or guidelines pooled it.
- Artificial intelligence and academic anxiety? a meta-analysis.BMC psychology · 2026Pooled it
- Pooled it
- Advancements in artificial intelligence transforming medical education: a comprehensive overview.Medical education online · 2025Pooled it
- Evaluating the Potential and Accuracy of ChatGPT-3.5 and 4.0 in Medical Licensing and In-Training Examinations: Systematic Review and Meta-Analysis.JMIR medical education · 2025Pooled it
- Large language models for generating medical examinations: systematic review.BMC medical education · 2024Pooled it
- Psychometric performance and student perceptions of AI- versus student-generated multiple-choice questions: a single-center randomized controlled trial.BMC medical education · 2026Trial
- Artificial intelligence-powered virtual standardized patients in teaching history-taking skills to medical students: a randomized controlled trial.BMC medical education · 2026Trial
- Artificial intelligence in radiology examinations: a psychometric comparison of question generation methods.Diagnostic and interventional radiology (Ankara, Turkey) · 2026Article
- Can AI Write MCQs? A Comparison of ChatGPT-Generated and Educator-Authored Multiple-Choice Questions in Graduate Nursing Education.Nursing reports (Pavia, Italy) · 2026Article
- Automated Generation and Human Evaluation of Neurosurgical Board Examination Self-Assessment Questions.Neurosurgery practice · 2026Article
- LLM-based generation of USMLE-style questions with ASPET/AMSPC knowledge objectives: All RAGs and no riches.British journal of clinical pharmacology · 2026Article
- Examining the Role of Artificial Intelligence in Assessment: A Comparative Study of ChatGPT and Educator-Generated Multiple-Choice Questions in a Dental Exam.European journal of dental education : official journal of the Association for Dental Education in Europe · 2026Article
- Development of vignette-based evaluation with generative artificial intelligence (VEGA) for nursing students: a pilot study.BMC medical education · 2026Article
- Guideline-based clinical reasoning in periodontology education: a comparative study of residents and large language models.BMC medical education · 2026Article
- Opportunities and Challenges of Generative AI in Postgraduate Health Professions Education Assessments From Educator and Learner Perspectives: Qualitative Study.JMIR formative research · 2026Article
- Reimagining pharmacology education.Pharmacological reviews · 2026Review
- Article
- Accuracy of ChatGPT and DeepSeek in answering clinical questions from the 2025 Society for Cardiovascular Angiography & Interventions/Heart Rhythm Society left atrial appendage occlusion guidelines.The Journal of international medical research · 2026Article
- Performance of DeepSeek in the generation of in-training examination questions in radiology resident education.NPJ digital medicine · 2026Article
- Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items.JMIR formative research · 2026Article
13 more citing papers are in PubMed but not listed here.
Corrections and comments
- Erratum issued
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
introductionLarge language models, in particular ChatGPT, have showcased remarkable language processing capabilities. Given the substantial workload of university medical staff, this study aims to assess the quality of multiple-choice questions (MCQs) produced by ChatGPT for use in graduate medical examinations, compared to questions written by university professoriate staffs based on standard medical textbooks.
methods50 MCQs were generated by ChatGPT with reference to two standard undergraduate medical textbooks (Harrison's, and Bailey & Love's). Another 50 MCQs were drafted by two university professoriate staff using the same medical textbooks. All 100 MCQ were individually numbered, randomized and sent to five independent international assessors for MCQ quality assessment using a standardized assessment score on five assessment domains, namely, appropriateness of the question, clarity and specificity, relevance, discriminative power of alternatives, and suitability for medical graduate examination.
resultsThe total time required for ChatGPT to create the 50 questions was 20 minutes 25 seconds, while it took two human examiners a total of 211 minutes 33 seconds to draft the 50 questions. When a comparison of the mean score was made between the questions constructed by A.I. with those drafted by humans, only in the relevance domain that the A.I. was inferior to humans (A.I.: 7.56 +/- 0.94 vs human: 7.88 +/- 0.52; p = 0.04). There was no significant difference in question quality between questions drafted by A.I. versus humans, in the total assessment score as well as in other domains. Questions generated by A.I. yielded a wider range of scores, while those created by humans were consistent and within a narrower range.
conclusionChatGPT has the potential to generate comparable-quality MCQs for medical graduate examinations within a significantly shorter time.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.