Observational studyInternational orthopaedics2024
Effectiveness of AI-powered Chatbots in responding to orthopaedic postgraduate exam questions-an observational study.
Observational study in International orthopaedics, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 21 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
21 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Can Large Language Models Serve as Reliable Tools for Information in Dentistry? A Systematic Review.International dental journal · 2025Pooled it
- Accuracy of Large Language Models When Answering Clinical Research Questions: Systematic Review and Network Meta-Analysis.Journal of medical Internet research · 2025Pooled it
- Effectiveness of ChatGPT-assisted and drawing-based learning in enhancing understanding of intraoral radiographic anatomy among dental students: a comparative study.BMC oral health · 2026Trial
- Performance of ChatGPT, Claude, and AMBOSS on the European Board of Urology In-Service Assessment and Alignment With the European Association of Urology 2025 Guidelines: Comparative Study.JMIR formative research · 2026Article
- Examining the Role of Artificial Intelligence in Assessment: A Comparative Study of ChatGPT and Educator-Generated Multiple-Choice Questions in a Dental Exam.European journal of dental education : official journal of the Association for Dental Education in Europe · 2026Article
- Can artificial intelligence chatbots think like dentists? A comparative analysis based on dental specialty examination questions in restorative dentistry.BMC oral health · 2026Article
- Artificial intelligence and robotic technologies redefining precision and personalization in orthopedic surgery: a narrative review.Frontiers in bioengineering and biotechnology · 2026Review
- Review
- A Comparative Analysis of GPT-3.5, GPT-4, GPT-4 Omni, Gemini Advanced, and Gemini 1.5 in Answering Frequently Asked Questions Regarding High Tibial Osteotomy.Orthopaedic journal of sports medicine · 2025Article
- Comparative Assessment of Large Language Models in Optics and Refractive Surgery: Performance on Multiple-Choice Questions.Vision (Basel, Switzerland) · 2025Article
- Comparison of hand surgery certification exams in Europe and the United States using ChatGPT 4.0.Journal of hand and microsurgery · 2025Article
- Integrating artificial intelligence into orthopedics: Opportunities, challenges, and future directions.Journal of hand and microsurgery · 2025Review
- The crucial role and challenges of medical journal editors in the modern era.Journal of clinical orthopaedics and trauma · 2025Article
- Article
- An Assessment of the Performance of Different Chatbots on Shoulder and Elbow Questions.Journal of clinical medicine · 2025Article
- ChatGPT-3.5 and -4.0 Do Not Reliably Create Readable Patient Education Materials for Common Orthopaedic Upper- and Lower-Extremity Conditions.Arthroscopy, sports medicine, and rehabilitation · 2025Article
- Performance of three artificial intelligence (AI)-based large language models in standardized testing; implications for AI-assisted dental education.Journal of periodontal research · 2025Article
- Testing the knowledge of artificial intelligence chatbots in pharmacology: examples of two groups of drugs.PeerJ. Computer science · 2025Article
- Exploring prospects, hurdles, and road ahead for generative artificial intelligence in orthopedic education and training.BMC medical education · 2024Review
- Accuracy and Repeatability of ChatGPT Based on a Set of Multiple-Choice Questions on Objective Tests of Hearing.Cureus · 2024Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
8 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeThis study analyses the performance and proficiency of the three Artificial Intelligence (AI) generative chatbots (ChatGPT-3.5, ChatGPT-4.0, Bard Google AI®) and in answering the Multiple Choice Questions (MCQs) of postgraduate (PG) level orthopaedic qualifying examinations.
methodsA series of 120 mock Single Best Answer' (SBA) MCQs with four possible options named A, B, C and D as answers on various musculoskeletal (MSK) conditions covering Trauma and Orthopaedic curricula were compiled. A standardised text prompt was used to generate and feed ChatGPT (both 3.5 and 4.0 versions) and Google Bard programs, which were then statistically analysed.
resultsSignificant differences were found between responses from Chat GPT 3.5 with Chat GPT 4.0 (Chi square = 27.2, P < 0.001) and on comparing both Chat GPT 3.5 (Chi square = 63.852, P < 0.001) with Chat GPT 4.0 (Chi square = 44.246, P < 0.001) with. Bard Google AI® had 100% efficiency and was significantly more efficient than both Chat GPT 3.5 with Chat GPT 4.0 (p < 0.0001).
conclusionThe results demonstrate the variable potential of the different AI generative chatbots (Chat GPT 3.5, Chat GPT 4.0 and Bard Google) in their ability to answer the MCQ of PG-level orthopaedic qualifying examinations. Bard Google AI® has shown superior performance than both ChatGPT versions, underlining the potential of such large language processing models in processing and applying orthopaedic subspecialty knowledge at a PG level.
Indexed as
Identifiers
38619565What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.