Evidence map›Paper›PMID 39509585›Full record

ArticleUrology practice2025

Evaluation of ChatGPT as a Reliable Source of Medical Information on Prostate Cancer for Patients: Global Comparative Survey of Medical Oncologists and Urologists.

Arnulf Stenzl, Andrew J Armstrong, Eamonn Rogers, Dany Habr, Jochen Walz, Martin Gleave, Andrea Sboner, Jennifer Ghith, Lucile Serfass, Kristine W Schuler and 4 more

Abstract readComparative Study
PubMed Full text at PMC Publisher
In one paragraph

Article in Urology practice, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 1 paper.

0numbers the graph read from it
0cells of the map it votes in
1citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

1 citing paper in PubMed.

  1. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

14 authors.

Arnulf StenzlDepartment of Urology, University of Tübingen, Tübingen, Germany.
Andrew J ArmstrongDivision of Medical Oncology, Department of Medicine, Duke Cancer Institute Center for Prostate and Urologic Cancer, Durham, North Carolina.
Eamonn RogersDepartment of Urology, University College Hospital, Galway, Ireland.
Dany HabrPfizer Oncology, Pfizer Inc, New York, New York.
Jochen WalzDepartment of Urology, Institut Paoli-Calmettes Cancer Center, Marseilles, France.
Martin GleaveDepartment of Urologic Sciences, The Vancouver Prostate Centre, The University of British Columbia, Vancouver, British Columbia, Canada.
Andrea SbonerEnglander Institute for Precision Medicine, Department of Pathology and Laboratory Medicine, Institute for Computational Biomedicine, Meyer Cancer Center, Weill Cornell Medicine, New York, New York.
Jennifer GhithPfizer Oncology, Pfizer Inc, New York, New York.
Lucile SerfassPfizer Oncology, Pfizer Inc, Paris, France.
Kristine W SchulerPfizer Oncology, Pfizer Inc, New York, New York.
Sam GarasPfizer Oncology, Pfizer Inc, New York, New York.
Dheepa ChariPfizer Oncology, Pfizer Inc, New York, New York.
Ken TrumanInsights & Connections, MedThink Inc, Cary, North Carolina.
Cora N SternbergEnglander Institute for Precision Medicine, Meyer Cancer Center, Weill Cornell Medicine, New York Presbyterian Center, New York, New York.

Funding

Insulin Stimulated Production Of Nitric Oxide In Vascular EndotheliumZ01AT000001 · NCCIH · NATIONAL CENTER FOR COMPLEMENTARY & INTEGRATIVE HEALTH · PI QUON, MICHAEL J. · 2002 to 2008
$2.4M
Intramural NIH HHS Z01 AT000001
6 · The paper itself

Abstract

introductionNo consensus exists on performance standards for evaluation of generative artificial intelligence (AI) to generate medical responses. The purpose of this study was the assessment of Chat Generative Pre-trained Transformer (ChatGPT) to address medical questions in prostate cancer.

methodsA global online survey was conducted from April to June 2023 among > 700 medical oncologists or urologists who treat patients with prostate cancer. Participants were unaware that this was a survey evaluating AI. In component 1, responses to 9 questions were written independently by medical writers (MWs; from medical websites) and ChatGPT 4.0 (AI-generated from publicly available information). Respondents were randomly exposed and blinded to both AI-generated and MW-curated responses; evaluation criteria and overall preference were recorded. Exploratory component 2 evaluated AI-generated responses to 5 complex questions with nuanced answers in the medical literature. Responses were evaluated on a 5-point Likert scale. Statistical significance was denoted by

resultsIn component 1, respondents (N = 602) consistently preferred the clarity of AI-generated responses over MW-curated responses in 7 of 9 questions (

conclusionsThese findings provide insight into how clinicians rate AI-generated and MW-curated responses with evaluation criteria that can be used in future AI validation studies.

Indexed as

Artificial IntelligenceOncologistsProstatic NeoplasmsUrologistsGenerative Artificial IntelligenceHumansMaleMedical OncologySurveys and Questionnairesartificial intelligencemedical oncologyproof of concept studysurvey and questionnairesurology

Identifiers

PMID39509585
PMCPMC12708041

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.