ArticleMedical education online2023
ChatGPT in medical school: how successful is AI in progress testing?
Article in Medical education online, 2023. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 42 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
42 citing papers in PubMed, 1 synthesis or guideline pooled it, 124 citations in OpenAlex.
- Large language models for generating medical examinations: systematic review.BMC medical education · 2024Pooled it
- The benchmark illusion: on the curious underestimation of AI and the uncertainty of its creators' gold standards.EClinicalMedicine · 2026Review
- Assessing multiple chatGPT versions on novel content in the Taiwan urology board examination: accuracy, speed, and domain-specific performance.World journal of urology · 2026Article
- Enhancing Medical Education: The Role of Artificial Intelligence Tools in the Teaching-Learning Process.Advances in medical education and practice · 2026Review
- Risk-centered benchmarking of large language models for AI-enabled counseling in chronic autoimmune thyroid eye disease.Frontiers in cell and developmental biology · 2026Article
- Benchmarking large language models for congenital cataract parent counseling: safety, readability, and knowledge translation of developmental and genetic information.Frontiers in cell and developmental biology · 2026Article
- Can Artificial Intelligence be Successful as an Anaesthesiology and Reanimation Resident?Turkish journal of anaesthesiology and reanimation · 2025Article
- Performance of DeepSeek-R1, ChatGPT (GPT-o3-mini), and Gemini 2.0 Flash on German Medical Multiple-Choice Questions: Comparative Evaluation.JMIR formative research · 2025Article
- Artificial intelligence in medical education for pulmonary nodule management: a narrative review.Translational lung cancer research · 2025Review
- Assessing the Accuracy and Completeness of AI-Generated Dental Responses: An Evaluation of the Chat-GPT Model.Healthcare (Basel, Switzerland) · 2025Article
- Article
- Article
- Large Language Models in Biochemistry Education: Comparative Evaluation of Performance.JMIR medical education · 2025Article
- Generating learning guides for medical education with LLMs and statistical analysis of test results.BMC medical education · 2025Article
- Detecting Artificial Intelligence-Generated Versus Human-Written Medical Student Essays: Semirandomized Controlled Study.JMIR medical education · 2025Article
- Review
- ChatGPT and Other Large Language Models in Medical Education - Scoping Literature Review.Medical science educator · 2025Review
- Performance of Artificial Intelligence Chatbots on Ultrasound Examinations: Cross-Sectional Comparative Analysis.JMIR medical informatics · 2025Article
- How valuable are the questions and answers generated by large language models in oral and maxillofacial surgery?PloS one · 2025Article
- Beyond the Scalpel: Assessing ChatGPT's potential as an auxiliary intelligent virtual assistant in oral surgery.Computational and structural biotechnology journal · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors at 3 institutions in 1 country.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundAs generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians' performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test.
methodsA total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT's user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question.
resultsOf the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = -0.08, 95% CI [-0.18, 0.02], t(393) = -1.55,
conclusionChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1-3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.