ArticleChest2025
Quality of Human Expert vs Large Language Model-Generated Multiple-Choice Questions in the Field of Mechanical Ventilation.
Article in Chest, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 2 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
2 citing papers in PubMed.
- Generating Image-Based Multiple-Choice Questions with Multimodal Large Language Models: Expert and Psychometric Evaluation.Journal of imaging informatics in medicine · 2026Article
- Evaluating large language models for diabetic retinopathy multiple-choice question generation in clinical ophthalmic education.Frontiers in medicine · 2026Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
Abstract
backgroundAlthough mechanical ventilation (MV) is a critical competency in critical care training, standardized methods for assessing MV-related knowledge are lacking. Traditional multiple-choice question (MCQ) development is resource intensive, and prior studies have suggested that generative AI tools could streamline question creation. However, the quality of AI-generated MCQs remains unclear. RESEARCH QUESTION: Are MCQs generated by ChatGPT noninferior to human expert (HE)-created questions in terms of quality and relevance for MV education? STUDY DESIGN AND
methodsThree key MV topics were selected: Equation of Motion and Ohm's Law, Tau and Auto-PEEP, and Oxygenation. Fifteen learning objectives were used to generate 15 AI-written MCQs via a standardized prompt with ChatGPT-o1 (preview model; made available September 12, 2024). A group of 31 faculty experts, all of whom regularly teach MV, evaluated both AI- and HE-generated MCQs. Each MCQ was assessed based on its alignment with learning objectives, accuracy of chosen answer, clarity of the question stem, plausibility of distractor options, and difficulty level. The faculty members were blinded to the provenance of the MCQ questions. The noninferiority margin was predefined as 15% of the total possible score (-3.45).
resultsAI-generated MCQs were statistically noninferior to the HE-written MCQs (95% upper CI, [-1.15, ∞]). In additions, respondents were unable to reliably differentiate AI-generated MCQs from HE-written MCQs (P = .32).
interpretationOur results suggest that AI-generated MCQs using ChatGPT-o1 are comparable in quality to those written by HEs. Given the time and resource-intensive nature of human MCQ development, AI-assisted question generation may serve as an efficient and scalable alternative for medical education assessment, even in highly specialized domains such as MV.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.