ArticleBMC medical education2023
Advantages and pitfalls in utilizing artificial intelligence for crafting medical examinations: a medical education pilot study with GPT-4.
Article in BMC medical education, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 38 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
38 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Large language models for generating medical examinations: systematic review.BMC medical education · 2024Pooled it
- Psychometric performance and student perceptions of AI- versus student-generated multiple-choice questions: a single-center randomized controlled trial.BMC medical education · 2026Trial
- Evaluating AI-generated examination papers in periodontology: a comparative study with human-designed counterparts.BMC medical education · 2025Trial
- Generating multiple-choice questions using reverse engineering techniques.Medical education online · 2026Article
- Examining the Role of Artificial Intelligence in Assessment: A Comparative Study of ChatGPT and Educator-Generated Multiple-Choice Questions in a Dental Exam.European journal of dental education : official journal of the Association for Dental Education in Europe · 2026Article
- Limitations of large language model-generated multiple-choice questions in ophthalmology.Annals of translational medicine · 2026Article
- Effectiveness of Artificial Intelligence-Assisted Peer Teaching in Orthopedic Clinical Education: Historical Cohort Study.JMIR medical education · 2026Article
- Complementary Error Patterns Between Human Evaluators and GPT-4o in Video-Based Cardiopulmonary Resuscitation Skills Assessment: Implications for Artificial Intelligence-Assisted Second Reading.Journal of clinical medicine · 2026Article
- Performance of 5 AI Models on United States Medical Licensing Examination Step 1 Questions: Comparative Observational Study.JMIR AI · 2026Article
- Fine-Tuned Large Language Models for Generating Multiple-Choice Questions in Anesthesiology: Psychometric Comparison With Faculty-Written Items.JMIR formative research · 2026Article
- Artificial intelligence in medical ethics education: a descriptive study of eight models in multiple choice question generation.BMC medical education · 2026Article
- Comparative performance of ChatGPT-5 and DeepSeek on the Chinese ultrasound medicine senior professional title examination.Frontiers in digital health · 2026Article
- Quality of Large Language Model-Generated MCQs Across Three Medical Disciplines: An Expert Rater-Based Comparison of Gemini, GPT-4 and Perplexity Pro.Advances in medical education and practice · 2026Article
- Article
- Assessing LLM-generated vs. expert-created clinical anatomy MCQs: a student perception-based comparative study in medical education.Medical education online · 2025Article
- Evaluating ChatGPT-4 in the development of family medicine residency examinations.PLOS digital health · 2025Article
- Artificial intelligence in undergraduate medical education: an updated scoping review.BMC medical education · 2025Article
- Chatbot To Help Patients Understand Their Health.Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing · 2025Article
- Large Language Models in Randomized Controlled Trials Design: Observational Study.Journal of medical Internet research · 2025Observational
- Exploring Factors Influencing Pre-Service Teachers' Intention to Use GenAI for Instructional Design: A Grounded Theory Study.Behavioral sciences (Basel, Switzerland) · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
11 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThe task of writing multiple choice question examinations for medical students is complex, timely and requires significant efforts from clinical staff and faculty. Applying artificial intelligence algorithms in this field of medical education may be advisable.
methodsDuring March to April 2023, we utilized GPT-4, an OpenAI application, to write a 210 multi choice questions-MCQs examination based on an existing exam template and thoroughly investigated the output by specialist physicians who were blinded to the source of the questions. Algorithm mistakes and inaccuracies, as identified by specialists were classified as stemming from age, gender or geographical insensitivities.
resultsAfter inputting a detailed prompt, GPT-4 produced the test rapidly and effectively. Only 1 question (0.5%) was defined as false; 15% of questions necessitated revisions. Errors in the AI-generated questions included: the use of outdated or inaccurate terminology, age-sensitive inaccuracies, gender-sensitive inaccuracies, and geographically sensitive inaccuracies. Questions that were disqualified due to flawed methodology basis included elimination-based questions and questions that did not include elements of integrating knowledge with clinical reasoning.
conclusionGPT-4 can be used as an adjunctive tool in creating multi-choice question medical examinations yet rigorous inspection by specialist physicians remains pivotal.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.