ArticleJournal of hand and microsurgery2025
Impact of prompting on large language model performance: ChatGPT-4 performance on the 2023 hand surgery self-assessment examination.
Article in Journal of hand and microsurgery, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Not yet cited in PubMed.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
0 citing papers in PubMed.
No citing paper in PubMed yet.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
7 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Background: Large language models (LLMs) such as ChatGPT are artificial intelligence programs designed to interpret and respond to text based input These programs can improve output based on prompting and tailored prompt engineering. Multiple studies have assessed the ability of various LLMs to perform on medical exams at different levels of training. The newest version of ChatGPT, GPT-4, allows image recognition which is relevant for many questions on orthopedic surgery exams. Performance of GPT-4, and the potential for LLMs to learn from prior exams remains unclear. The present study analyzed ChatGPT-4 performance on the 2023 hand surgery Maintenance of Certification (MOC) Self-Assessment Examination (SAE) before and after prompting with 5 previous versions of the test. It was hypothesized that GPT-4 would pass the exam and improve performance after prompting. Methods: GPT-4 was tested with all text and image-based questions from the 2023 hand surgery SAE. Video-based questions were excluded. GPT-4 was then provided with questions, answers, and explanations from 5 previous SAEs from 2014 to 2020 and retested on the 2023 SAE text and imaging questions. Responses from GPT-4 on prompted and unprompted tests were recorded and compared. Results: Both prompted and unprompted versions of ChatGPT-4 exceeded SAE exam passing requirement of >50 % correct response rate. GPT-4 answered 67 % of all questions correctly unprompted and 71 % of all questions correctly after prompting (p = 0.51). Sub-analysis demonstrated GPT-4 answered 66 % of image-based questions correctly after prompting, compared to 56 % before prompting (p = 0.25). GPT-4 answered 75 % of text only questions correctly before prompting and 74 % correctly after prompting (p = 1.0). Fischer's exact test on total questions, image only, and text only showed no statistically significant differences between prompted and unprompted versions of GPT-4. Conclusion: GPT-4 demonstrated the ability to analyze orthopedic information, answer specialty-specific questions, and exceed the passing threshold of 50 % on the 2023 Hand Surgery Self-Assessment Exam. However, prompting GPT-4 with previous SAEs did not statistically significantly improve performance. With continued advancements in AI and deep learning, large language models may someday become resources in test simulation and knowledge checks in the realm of hand surgery.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.