Evidence map›Paper›PMID 39706895›Full record

ArticleNPJ digital medicine2024

Human AI collaboration for unsupervised categorization of live surgical feedback.

Rafal Kocielnik, Cherine H Yang, Runzhuo Ma, Steven Y Cen, Elyssa Y Wong, Timothy N Chu, J Everett Knudsen, Peter Wager, John Heard, Umar Ghaffar and 2 more

Abstract read
PubMed Full text at PMC Publisher
In one paragraph

Article in NPJ digital medicine, 2024. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 4 papers.

0numbers the graph read from it
0cells of the map it votes in
4citing papers in PubMed
–field-weighted citation impact
1 · What the graph read from it

What it found

Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.

The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.

2 · The registry

The trial behind it

Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.

Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.

3 · Its place in the literature

Who cites it

4 citing papers in PubMed.

  1. Review
  2. Artificial intelligence in urology training Enhancing annotation, feedback, and evaluation in robotic, laparoscopic, and endoscopic surgery.Canadian Urological Association journal = Journal de l'Association des urologues du Canada · 2026
    Review
  3. The ACPGBI AI taskforce report: A mixed-methods roadmap for AI in colorectal surgery.Colorectal disease : the official journal of the Association of Coloproctology of Great Britain and Ireland · 2025
    Article
  4. Review
4 · The record

Corrections and comments

PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.

5 · Who and what money

Authors and funding

12 authors.

Rafal KocielnikComputing+Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA.
Cherine H YangDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
Runzhuo MaDepartment of Urology, New York Presbyterian Hospital, Weill Cornell Medicine, New York, NY, USA.
Steven Y CenKeck School of Medicine, University of Southern California, Los Angeles, CA, USA.
Elyssa Y WongDepartment of Urology, University of Texas Southwestern Medical Center, Dallas, TX, USA.
Timothy N ChuDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
J Everett KnudsenDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
Peter WagerDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
John HeardDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
Umar GhaffarDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA.
Anima AnandkumarComputing+Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA.ORCID http://orcid.org/0000-0002-6974-6797
Andrew J HungDepartment of Urology, Cedars-Sinai Medical Center, Los Angeles, CA, USA. ajhung@gmail.com.ORCID http://orcid.org/0000-0002-7201-6736

Funding

Preserving Erectile Function by Quantifying the Nerve-Sparing step of the Robotic ProstatectomyR01CA273031 · NCI · UNIVERSITY OF SOUTHERN CALIFORNIA · PI Jim Hu, Andrew Hung · 2022 to 2026
$3.6M
Automated Assessment for Robotic Suturing Utilizing Deep Learning AlgorithmsR01CA251579 · NCI · UNIVERSITY OF SOUTHERN CALIFORNIA · PI HUNG, ANDREW · 2021 to 2025
$3.0M
NCI NIH HHS R01 CA251579NCI NIH HHS R01 CA273031U.S. Department of Health & Human Services | NIH | National Cancer Institute (NCI) R01CA251579U.S. Department of Health & Human Services | NIH | National Cancer Institute (NCI) R01CA273031
6 · The paper itself

Abstract

Formative verbal feedback during live surgery is essential for adjusting trainee behavior and accelerating skill acquisition. Despite its importance, understanding optimal feedback is challenging due to the difficulty of capturing and categorizing feedback at scale. We propose a Human-AI Collaborative Refinement Process that uses unsupervised machine learning (Topic Modeling) with human refinement to discover feedback categories from surgical transcripts. Our discovered categories are rated highly for clinical clarity and are relevant to practice, including topics like "Handling and Positioning of (tissue)" and "(Tissue) Layer Depth Assessment and Correction [during tissue dissection]." These AI-generated topics significantly enhance predictions of trainee behavioral change, providing insights beyond traditional manual categorization. For example, feedback on "Handling Bleeding" is linked to improved behavioral change. This work demonstrates the potential of AI to analyze surgical feedback at scale, informing better training guidelines and paving the way for automated feedback and cueing systems in surgery.

Identifiers

PMID39706895
PMCPMC11662073

What OpenQuestion holds

Textmetadata
LicenceCC BY-NC-ND
Read underepoch 390

Registered trials

None linked

Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.