Evidence map›Paper›PMID 39470681›Full record

ArticleJournal of surgical oncology2025

Assessment of Artificial Intelligence Chatbot Responses to Common Patient Questions on Bone Sarcoma.

Kameel Khabaz, Nicole J Newman-Hung, Jennifer R Kallini, Joseph Kendal, Alexander B Christ, Nicholas M Bernthal, Lauren E Wessel

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Journal of surgical oncology, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 7 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
7citing papers in PubMed, 1 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

7 citing papers in PubMed, 1 synthesis or guideline pooled it.

  1. Pooled it
  2. Article
  3. Article
  4. Article
  5. [AI-enabled clinical decision support systems: challenges and opportunities].Bundesgesundheitsblatt, Gesundheitsforschung, Gesundheitsschutz · 2025
    Review
  6. Article
  7. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

7 authors.

Kameel KhabazDavid Geffen School of Medicine at UCLA, Los Angeles, California, USA.ORCID http://orcid.org/0000-0002-3164-6133
Nicole J Newman-HungDepartment of Orthopaedic Surgery, University of California, Los Angeles, California, USA.
Jennifer R KalliniDepartment of Orthopaedic Surgery, University of California, Los Angeles, California, USA.
Joseph KendalDepartment of Surgery, University of Calgary, Calgary, Alberta, Canada.
Alexander B ChristDepartment of Orthopaedic Surgery, University of California, Los Angeles, California, USA.
Nicholas M BernthalDepartment of Orthopaedic Surgery, University of California, Los Angeles, California, USA.ORCID http://orcid.org/0000-0003-3338-5878
Lauren E WesselDepartment of Orthopaedic Surgery, University of California, Los Angeles, California, USA.ORCID https://orcid.org/0000-0002-6240-5981

Funding

Regenerative Musculoskeletal Medicine Training ProgramT32AR059033 · NIAMS · UNIVERSITY OF CALIFORNIA LOS ANGELES · PI John S Adams, Nicholas M. Bernthal · 2011 to 2026
$4.8M
NIAMS NIH HHS T32 AR059033The authors received no specific funding for this work.
6 · The paper itself

Abstract

BACKGROUND AND

objectivesThe potential impacts of artificial intelligence (AI) chatbots on care for patients with bone sarcoma is poorly understood. Elucidating potential risks and benefits would allow surgeons to define appropriate roles for these tools in clinical care.

methodsEleven questions on bone sarcoma diagnosis, treatment, and recovery were inputted into three AI chatbots. Answers were assessed on a 5-point Likert scale for five clinical accuracy metrics: relevance to the question, balance and lack of bias, basis on established data, factual accuracy, and completeness in scope. Responses were quantitatively assessed for empathy and readability. The Patient Education Materials Assessment Tool (PEMAT) was assessed for understandability and actionability.

resultsChatbots scored highly on relevance (4.24) and balance/lack of bias (4.09) but lower on basing responses on established data (3.77), completeness (3.68), and factual accuracy (3.66). Responses generally scored well on understandability (84.30%), while actionability scores were low for questions on treatment (64.58%) and recovery (60.64%). GPT-4 exhibited the highest empathy (4.12). Readability scores averaged between 10.28 for diagnosis questions to 11.65 for recovery questions.

conclusionsWhile AI chatbots are promising tools, current limitations in factual accuracy and completeness, as well as concerns of inaccessibility to populations with lower health literacy, may significantly limit their clinical utility.

Indexed as

Artificial IntelligenceBone NeoplasmsOsteosarcomaPatient Education as TopicSarcomaComprehensionGenerative Artificial IntelligenceHumansSurveys and Questionnairesartificial intelligencebone sarcomaschatbotschondrosarcomaEwing sarcomaosteosarcoma

Identifiers

PMID39470681
PMCPMC12065442

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.