ArticleAmerican journal of surgery2021
Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.
Article in American journal of surgery, 2021. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 12 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
12 citing papers in PubMed.
- Leveraging Large Language Models to Summarize Data and Free-Text Comments From Resident Assessments.Journal of graduate medical education · 2026Article
- Tackling Administrative Burden in Graduate Medical Education: Consider AI-Powered Tools.Journal of graduate medical education · 2026Article
- Construction of a Feedback Comment Analysis Model for Evaluation of Endoscopic Surgical Skill.Annals of gastroenterological surgery · 2026Article
- Using Artificial Intelligence in Clerkship Learner Assessment: A CERA Study.Family medicine · 2025Article
- EMYWAY Workplace-Based Entrustable Professional Activities Assessments in Otolaryngology Residency Training: A Nationwide Experience.Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery · 2025Article
- The Role of Natural Language Processing in Graduate Medical Education: A Scoping Review.Cureus · 2025Review
- Artificial Intelligence Applications for Thoracic Surgeons: "The Phenomenal Cosmic Powers of the Magic Lamp".Journal of clinical medicine · 2024Article
- Large language models in physical therapy: time to adapt and adept.Frontiers in public health · 2024Article
- Automated Analysis of Preceptor Comments: A Pilot Study Using Sentiment Analysis to Identify Potential Student Issues in Experiential Education.American journal of pharmaceutical education · 2023Article
- Utilizing Natural Language Processing of Narrative Feedback to Develop a Predictive Model of Pre-Clerkship Performance: Lessons Learned.Perspectives on medical education · 2023Article
- Multi-disciplinary assessment of the entrustable professional activities of surgery residents.Global surgical education : journal of the Association for Surgical Education · 2022Article
- Artificial Intelligence Surgery: How Do We Get to Autonomous Actions in Surgery?Sensors (Basel, Switzerland) · 2021Review
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
9 authors.
Funding
Abstract
backgroundEntrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice.
methodsAll text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters.
resultsOver 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics).
conclusionsLDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.