ArticleBMC medical education2024
Reliability of ChatGPT in automated essay scoring for dental undergraduate examinations.
Article in BMC medical education, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 20 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
20 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Rubric use in dental education: a systematic review of design, implementation, and educational impact.BMC medical education · 2026Pooled it
- Mapping the structure of an emerging field: A scientometric decoding of large language model applications in the dental field.The Japanese dental science review · 2026Article
- Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation.Journal of microbiology & biology education · 2026Article
- Evaluating the Pediatric Behavior Guidance of Students Based on Actual Clinical Transcripts Scored by Faculty and Large Language Models: Pilot Comparative Study.JMIR medical education · 2026Article
- Guideline-based clinical reasoning in periodontology education: a comparative study of residents and large language models.BMC medical education · 2026Article
- Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education?Medical science educator · 2026Article
- Beyond Red Pen: Comparing the machine vs human grading of reflective assignments on clinical reasoning in Dermatology undergraduate students.Pakistan journal of medical sciences · 2026Article
- Implementation of an artificial intelligence-powered training platform in oral medicine and pathology education: a comparative study.BMC medical education · 2026Article
- Calibration of AI large language models with human subject matter experts for grading of clinical short-answer responses in dental education.BMC oral health · 2026Article
- A scoping review of the use of generative artificial intelligence tools in health profession education.BMC medical education · 2026Article
- Operating characteristics of agreement metrics in AI-based scoring: a Monte Carlo simulation.Frontiers in psychology · 2026Article
- Large language models as data-driven engines for benchmarking preventive and clinical knowledge in Chinese dental examinations.Frontiers in oral health · 2026Article
- Feasibility of a Specialized Large Language Model for Postgraduate Medical Examination Preparation: Single-Center Proof-Of-Concept Study.JMIR formative research · 2025Article
- A Scoping Review of Large Language Models in Dental Education: Applications, Challenges, and Prospects.International dental journal · 2025Article
- A research roadmap for AI opportunities in student assessment for medical education.BMC medical education · 2025Review
- Artificial intelligence-based chatbots improve the efficiency of course orientation among medical students: a cross-sectional study.BMC medical education · 2025Article
- Application of large language models in oral health education: a review of the literature.Canadian journal of dental hygiene : CJDH = Journal canadien de l'hygiene dentaire : JCHD · 2025Review
- Large language models underperform in European general surgery board examinations: a comparative study with experts and surgical residents.BMC medical education · 2025Article
- Chat Generative Pre-Trained Transformer (ChatGPT) in Oral and Maxillofacial Surgery: A Narrative Review on Its Research Applications and Limitations.Journal of clinical medicine · 2025Review
- Using ChatGPT for medical education: the technical perspective.BMC medical education · 2025Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
backgroundThis study aimed to answer the research question: How reliable is ChatGPT in automated essay scoring (AES) for oral and maxillofacial surgery (OMS) examinations for dental undergraduate students compared to human assessors?
methodsSixty-nine undergraduate dental students participated in a closed-book examination comprising two essays at the National University of Singapore. Using pre-created assessment rubrics, three assessors independently performed manual essay scoring, while one separate assessor performed AES using ChatGPT (GPT-4). Data analyses were performed using the intraclass correlation coefficient and Cronbach's α to evaluate the reliability and inter-rater agreement of the test scores among all assessors. The mean scores of manual versus automated scoring were evaluated for similarity and correlations.
resultsA strong correlation was observed for Question 1 (r = 0.752-0.848, p < 0.001) and a moderate correlation was observed between AES and all manual scorers for Question 2 (r = 0.527-0.571, p < 0.001). Intraclass correlation coefficients of 0.794-0.858 indicated excellent inter-rater agreement, and Cronbach's α of 0.881-0.932 indicated high reliability. For Question 1, the mean AES scores were similar to those for manual scoring (p > 0.05), and there was a strong correlation between AES and manual scores (r = 0.829, p < 0.001). For Question 2, AES scores were significantly lower than manual scores (p < 0.001), and there was a moderate correlation between AES and manual scores (r = 0.599, p < 0.001).
conclusionThis study shows the potential of ChatGPT for essay marking. However, an appropriate rubric design is essential for optimal reliability. With further validation, the ChatGPT has the potential to aid students in self-assessment or large-scale marking automated processes.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.