Evidence map›Paper›PMID 40130001›Full record

ArticleMayo Clinic proceedings. Digital health2025

Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses From Closed-Domain Large Language Models Versus Clinical Teams.

Yuexing Hao, Jason Holmes, Jared Hobson, Alexandra Bennett, Elizabeth L McKone, Daniel K Ebner, David M Routman, Satomi Shiraishi, Samir H Patel, Nathan Y Yu and 4 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in Mayo Clinic proceedings. Digital health, 2025. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 15 papers, 1 of them a synthesis that pooled it.

0numbers the graph read from it
0cells of the map it votes in
15citing papers in PubMed, 1 pooled it
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

15 citing papers in PubMed, 1 synthesis or guideline pooled it.

  1. Pooled it
  2. Article
  3. Article
  4. Review
  5. Article
  6. Large language models for bioinformatics.Quantitative biology (Beijing, China) · 2026
    Review
  7. Article
  8. Review
  9. Article
  10. Article
  11. Article
  12. Article
  13. Article
  14. Article
  15. Article
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

14 authors.

Yuexing HaoDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).ORCID 0000-0003-4375-7655
Jason HolmesDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Jared HobsonDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Alexandra BennettDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Elizabeth L McKoneDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Daniel K EbnerDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
David M RoutmanDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Satomi ShiraishiDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Samir H PatelDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Nathan Y YuDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Chris L HallemeierDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Brooke E BallDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Mark WaddleDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).
Wei LiuDepartment of Radiation Oncology, Mayo Clinic, Phoenix, AZ (Y.H., J.H., S.H.P., N.Y.Y., W.L.); Cornell University, Ithaca, NY (Y.H.); Department of Electric Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA (Y.H.); and Department of Radiation Oncology, Mayo Clinic, Rochester, MN (A.B., E.L.M., D.K.E., D.M.R., S.S., C.L.H., B.E.B., M.W.).ORCID 0000-0003-0935-3999

Funding

Paul Calabresi Program in Clinical/Translational Research at Mayo ClinicK12CA090628 · NCI · MAYO CLINIC ROCHESTER · PI WEROHA, SARAVUT · 2001 to 2025
$19.7M
Dose Linear Energy Transfer Volume Histogram and Dosimetric Seed Spot Analysis in Spot Scanning Proton TherapyR01CA280134 · NCI · MAYO CLINIC ARIZONA · PI Wei Liu · 2024 to 2026
$1.7M
NCI NIH HHS K12 CA090628NCI NIH HHS R01 CA280134
6 · The paper itself

Abstract

Objective: To evaluate the effectiveness of RadOnc-generative pretrained transformer (GPT), a GPT-4 based large language model, in assisting with in-basket message response generation for prostate cancer treatment, with the goal of reducing the workload and time on clinical care teams while maintaining response quality. Patients and Methods: RadOnc-GPT was integrated with electronic health records from both Mayo Clinic-wide databases and a radiation-oncology-specific database. The model was evaluated on 158 previously recorded in-basket message interactions, selected from 90 patients with nonmetastatic prostate cancer from the Mayo Clinic Department of Radiation Oncology in-basket message database in the calendar years 2022-2024. Quantitative natural language processing analysis and 2 grading studies, conducted by 5 clinicians and 4 nurses, were used to assess RadOnc-GPT's responses. Three primary clinicians independently graded all messages, whereas a fourth senior clinician reviewed 41 responses with relevant discrepancies, and a fifth senior clinician evaluated 2 additional responses. The grading focused on 5 key areas: completeness, correctness, clarity, empathy, and editing time. The grading study was performed from July 20, 2024 to December 15, 2024. Results: The RadOnc-GPT slightly outperformed the clinical care team in empathy, whereas achieving comparable scores with the clinical care team in completeness, correctness, and clarity. Five clinician graders identified key limitations in RadOnc-GPT's responses, such as lack of context, insufficient domain-specific knowledge, inability to perform essential meta-tasks, and hallucination. It was estimated that RadOnc-GPT could save an average of 5.2 minutes per message for nurses and 2.4 minutes for clinicians, from reading the inquiry to sending the response. Conclusion: RadOnc-GPT has the potential to considerably reduce the workload of clinical care teams by generating high-quality, timely responses for in-basket message interactions. This could lead to improved efficiency in health care workflows and reduced costs while maintaining or enhancing the quality of communication between patients and health care providers.Abbreviations and AcronymsAI; artificial intelligence; LLM; large language model; NLP; natural language processing; RadOnc-GPT; radiation oncology generative pretrained transformer.

Identifiers

PMID40130001
PMCPMC11932704

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.