ArticleJMIR medical education2024
Assessing the Ability of a Large Language Model to Score Free-Text Medical Student Clinical Notes: Quantitative Study.
Article in JMIR medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 18 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
18 citing papers in PubMed.
- Evaluating Large Language Model-Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study.JMIR medical education · 2026Article
- Artificial intelligence in reproductive medicine and education: current evidence, challenges, and future directions.Journal of assisted reproduction and genetics · 2026Review
- Comparison of artificial intelligence and lecturers' grading in the evaluation of handwritten open-ended anatomy examinations.Surgical and radiologic anatomy : SRA · 2026Article
- Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation.Journal of microbiology & biology education · 2026Article
- Reliability of a rating rubric to assess quality of preoperative goals of care notes.American journal of surgery · 2026Article
- Benefits of Chain-of-Thought Prompting for Clinical Record Rubric Evaluation in Undergraduate Medical Education: Experimental Evaluation Study With Medical Faculty.JMIR medical education · 2026Article
- Developing a Large Language Model-Based Feedback System for Case Report Writing in Rehabilitation Education: Tutorial.JMIR medical education · 2026Article
- The application of large language models in orthopedic postgraduate education: potentials, challenges, and future prospects.Journal of orthopaedic surgery and research · 2026Review
- Trustworthy AI-Augmented Objective Structured Clinical Examinations in Nursing Education: Taiwan-Japan Viewpoint on 5 AI Roles, Governance, and Cross-Border Implementation.JMIR formative research · 2026Article
- Accuracy of large language model transcription of simulated physician-patient verbal interactions.BMC medical informatics and decision making · 2026Article
- AI- vs Human-Based Assessment of Medical Interview Transcripts in a Generative AI-Simulated Patient System: Cross-Sectional Validation Study.JMIR medical education · 2026Article
- Calibration of AI large language models with human subject matter experts for grading of clinical short-answer responses in dental education.BMC oral health · 2026Article
- A scoping review of the use of generative artificial intelligence tools in health profession education.BMC medical education · 2026Article
- Breaking barriers: Validation of a Spanish oral health knowledge tool to enhance patient-provider communication.PloS one · 2026Article
- Large language model evaluation and feedback of transcribed simulated physician-patient verbal interactions.Frontiers in artificial intelligence · 2026Article
- Artificial intelligence in undergraduate medical education clinical skills curricula: a scoping review of implementations since 2022.Frontiers in digital health · 2026Review
- AI-Based Assessment of Non-Technical Skills in Prehospital Simulations: A Comparative Validation Study.Healthcare (Basel, Switzerland) · 2025Article
- Automated Evaluation of Reflection and Feedback Quality in Workplace-Based Assessments by Using Natural Language Processing: Cross-Sectional Competency-Based Medical Education Study.JMIR medical education · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background: Teaching medical students the skills required to acquire, interpret, apply, and communicate clinical information is an integral part of medical education. A crucial aspect of this process involves providing students with feedback regarding the quality of their free-text clinical notes. Objective: The goal of this study was to assess the ability of ChatGPT 3.5, a large language model, to score medical students' free-text history and physical notes. Methods: This is a single-institution, retrospective study. Standardized patients learned a prespecified clinical case and, acting as the patient, interacted with medical students. Each student wrote a free-text history and physical note of their interaction. The students' notes were scored independently by the standardized patients and ChatGPT using a prespecified scoring rubric that consisted of 85 case elements. The measure of accuracy was percent correct. Results: The study population consisted of 168 first-year medical students. There was a total of 14,280 scores. The ChatGPT incorrect scoring rate was 1.0%, and the standardized patient incorrect scoring rate was 7.2%. The ChatGPT error rate was 86%, lower than the standardized patient error rate. The ChatGPT mean incorrect scoring rate of 12 (SD 11) was significantly lower than the standardized patient mean incorrect scoring rate of 85 (SD 74; P=.002). Conclusions: ChatGPT demonstrated a significantly lower error rate compared to standardized patients. This is the first study to assess the ability of a generative pretrained transformer (GPT) program to score medical students' standardized patient-based free-text clinical notes. It is expected that, in the near future, large language models will provide real-time feedback to practicing physicians regarding their free-text notes. GPT artificial intelligence programs represent an important advance in medical education and medical practice.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.