ArticleEuropean journal of orthopaedic surgery & traumatology : orthopedie traumatologie2024
Evaluating ChatGPT responses in the context of a 53-year-old male with a femoral neck fracture: a qualitative analysis.
Article in European journal of orthopaedic surgery & traumatology : orthopedie traumatologie, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 13 papers, 2 of them syntheses that pooled it.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
13 citing papers in PubMed, 2 syntheses or guidelines pooled it.
- Large Language Models for Chatbot Health Advice Studies: A Systematic Review.JAMA network open · 2025Pooled it
- Examining the Role of Large Language Models in Orthopedics: Systematic Review.Journal of medical Internet research · 2024Pooled it
- Factors Shaping Trust and Satisfaction With AI Medical Chatbots: A Mixed Methods Vignette Survey of Caregivers Seeking Guidance on Pediatric Infectious Diseases.Journal of medical Internet research · 2026Article
- Clinical Implications of ChatGPT-assisted Multimodal Pre-operative Assessment in Elderly Pertrochanteric Fracture Patients: An Exploratory Study.Journal of orthopaedic case reports · 2026Article
- Clinical applications of large language models in medicine and surgery: A scoping review.The Journal of international medical research · 2025Article
- Qualitative metrics from the biomedical literature for evaluating large language models in clinical decision-making: a narrative review.BMC medical informatics and decision making · 2024Review
- Artificial Intelligence in Surgery: A Systematic Review of Use and Validation.Journal of clinical medicine · 2024Review
- A framework for human evaluation of large language models in healthcare derived from literature review.NPJ digital medicine · 2024Article
- Article
- The ability of artificial intelligence tools to formulate orthopaedic clinical decisions in comparison to human clinicians: An analysis of ChatGPT 3.5, ChatGPT 4, and Bard.Journal of orthopaedics · 2024Article
- A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence-Based Models in Health Care Education and Practice: Development Study Involving a Literature Review.Interactive journal of medical research · 2024Article
- Letter to the editor 'Evaluating ChatGPT responses in the context of a 53-year-old male with a femoral neck fracture: a qualitative analysis'.European journal of orthopaedic surgery & traumatology : orthopedie traumatologie · 2024Article
- Exploring the Performance of ChatGPT in an Orthopaedic Setting and Its Potential Use as an Educational Tool.JB & JS open accessReview
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
5 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
purposeThe integration of artificial intelligence (AI) tools, such as ChatGPT, in clinical medicine and medical education has gained significant attention due to their potential to support decision-making and improve patient care. However, there is a need to evaluate the benefits and limitations of these tools in specific clinical scenarios.
methodsThis study used a case study approach within the field of orthopaedic surgery. A clinical case report featuring a 53-year-old male with a femoral neck fracture was used as the basis for evaluation. ChatGPT, a large language model, was asked to respond to clinical questions related to the case. The responses generated by ChatGPT were evaluated qualitatively, considering their relevance, justification, and alignment with the responses of real clinicians. Alternative dialogue protocols were also employed to assess the impact of additional prompts and contextual information on ChatGPT responses.
resultsChatGPT generally provided clinically appropriate responses to the questions posed in the clinical case report. However, the level of justification and explanation varied across the generated responses. Occasionally, clinically inappropriate responses and inconsistencies were observed in the generated responses across different dialogue protocols and on separate days.
conclusionsThe findings of this study highlight both the potential and limitations of using ChatGPT in clinical practice. While ChatGPT demonstrated the ability to provide relevant clinical information, the lack of consistent justification and occasional clinically inappropriate responses raise concerns about its reliability. These results underscore the importance of careful consideration and validation when using AI tools in healthcare. Further research and clinician training are necessary to effectively integrate AI tools like ChatGPT, ensuring their safe and reliable use in clinical decision-making.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.