ArticleBMC medical education2025
Assessing ChatGPT-4's performance on the US prosthodontic exam: impact of fine-tuning and contextual prompting vs. base knowledge, a cross-sectional study.
Article in BMC medical education, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 16 papers, 1 of them a synthesis that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
16 citing papers in PubMed, 1 synthesis or guideline pooled it.
- Accuracy of Large Language Models in Answering Dental Examination Questions: A Systematic Review and Meta-Analysis.International dental journal · 2026Pooled it
- Bloom's taxonomy-based comparison of artificial intelligence and dental students in restorative dentistry.BMC medical education · 2026Article
- AI-based diagnostic evaluation of GPT-4o for crown-fracture detection on maxillary periapical radiographs: effects of prompt detail and customization.BMC oral health · 2026Article
- Preliminary Evaluation of Large Language Models in Kennedy Classification and Removable Partial Denture Planning: An In Silico Study.Medical science monitor : international medical journal of experimental and clinical research · 2026Article
- The Ethical Boundaries of Chairside Generative Artificial Intelligence in Dental Education.Journal of dental education · 2026Article
- Large Language Models in Medical and Dental Education: A Cross-Sectional Comparison of AI-Generated and Faculty-Authored Prosthodontic Materials.Dentistry journal · 2026Article
- Comparative performance analysis of AI-based large language models in assessing cervical vertebral maturation stages on lateral cephalometric radiographs.BMC oral health · 2026Article
- Factors influencing large language model adoption among dental students: a cross-sectional study.Scientific reports · 2026Article
- Performance of Enhanced Large Language Models on Prosthodontic Multiple-Choice Questions.International dental journal · 2026Article
- Generative Artificial Intelligence and Large Language Models in Paediatric Dentistry: A Scoping Review.International dental journal · 2026Article
- The effectiveness of large language models in dental specialty questions: a comparative study in the field of prosthodontics.BMC medical education · 2026Article
- Performance of GPT-5, DeepSeek, and Claude in dental MCQs for medically compromised patients.Journal of translational medicine · 2026Article
- Mapping Clinical Intent in Internet Hospitals: Identifying and Profiling Patient Demand via Natural Language Processing.Computational and structural biotechnology journal · 2026Article
- Article
- Artificial intelligence-based chatbots improve the efficiency of course orientation among medical students: a cross-sectional study.BMC medical education · 2025Article
- Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
backgroundArtificial intelligence (AI), such as ChatGPT-4 from OpenAI, has the potential to transform medical education and assessment. However, its effectiveness in specialized fields like prosthodontics, especially when comparing base to fine-tuned models, remains underexplored. This study evaluates the performance of ChatGPT-4 on the US National Prosthodontic Resident Mock Exam in its base form and after fine-tuning. The aim is to determine whether fine-tuning improves the AI's accuracy in answering specialized questions.
methodsAn official sample questions from the 2021 US National Prosthodontic Resident Mock Exam was used, obtained from the American College of Prosthodontists. A total of 150 questions were initially considered, and resources were available for 106 questions. Both the base and fine-tuned models of ChatGPT-4 were tested under simulated exam conditions. Performance was assessed by comparing correct and incorrect responses. The Chi-square test was used to analyze accuracy, with significance set at p < 0.05. The Kappa coefficient was calculated to measure agreement between the models' responses.
resultsThe base model of ChatGPT-4 correctly answered 62.7% of the 150 questions. For the 106 questions with resources, the fine-tuned model answered 73.6% correctly. The Chi-square test showed a significant improvement in performance after fine-tuning (p < 0.001). The Kappa coefficient was 0.39, indicating moderate agreement between the models (p < 0.001). Performance varied by topic, with lower accuracy in areas such as Implant Prosthodontics, Removable Prosthodontics, and Occlusion, though the fine-tuned model consistently outperformed the base model.
conclusionsFine-tuning ChatGPT-4 with specific resources significantly enhances its accuracy in answering specialized prosthodontic exam questions. While the base model provides a solid baseline, fine-tuning is essential for improving AI performance in specialized fields. However, certain topics may require more targeted training to achieve higher accuracy.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.