Feasibility of Tailoring Artificial Intelligence-Assisted Ambient Scribes for Intensive Care Unit Rounds: Algorithm Development and Validation.

Background: The increasing documentation burden on physicians is a significant contributor to burnout and decreases in care quality. Artificial intelligence (AI) has been proposed as a solution to reduce documentation burden in clinical care, but there are very limited data on its use in the inpatient and intensive care unit (ICU) environments. Objective: This pilot study aimed to explore the feasibility of using AI-assisted ambient scribes to capture interprofessional ICU rounds and synthesize a singular document to improve documentation efficiency and clinician satisfaction during ICU rounds. In this paper, we showcase our findings from customizing prompts for large language models (LLMs) to generate and evaluate daily progress notes from transcripts of simulated ICU cases. Methods: This project is divided into 2 phases. In the first phase, a randomly selected transcript of an audio recording of a simulated ICU rounds case was used to iteratively evaluate and improve the prompts for the LLMs. Multiple models (n=5) were used in phase 1, and the best-performing model (M1, based on the highest accuracy) was selected for the next phase. In the subsequent phase, 5 cases were selected and evaluated using the refined prompt and 2 models: M1 from phase 1 and M6, a technological upgrade of M1. Accuracy and error percentages were used as primary metrics. Additionally, error severity and usability were assessed using the Harm scale (adapted for potential harm risk) from the Agency for Healthcare Research and Quality and the 9-item Physician Documentation Quality Instrument, respectively. Results: Iterative improvements to the prompt increased accuracy and reduced errors during phase 1. In phase 2, M1 and M6 achieved accuracies of 69% and 80%, respectively (P=.04). Overall, errors of omission were most common (mean 15.5%, SEM 2.7%), followed by partial errors (mean 7.2%, SEM 0.92%) and then errors of commission (mean 2.6%, SEM 0.7%). The error severity of both models was low (µ=0.61 vs 0.53; P=.10), with most errors categorized as having potential for no harm to low harm. Both models performed well on the 9-item Physician Documentation Quality Instrument assessment, with the M6 model outperforming the M1 (35.8 vs 38.3; P=.06). Conclusions: Our findings demonstrate the feasibility of integrating AI-assisted scribes for ICU documentation. Both prompt improvements and technological advancements in LLMs are noted to be helpful. This study lays the groundwork for future research into AI applications in ICU settings, paving the way for broader improvements in health care documentation.