Physician-revised AI-generated drafts are associated with higher ratings of written explanations in end-of-life care in the intensive care unit: a scenario-based single-center cross-sectional study.

backgroundIn end-of-life care (EOL) in the intensive care unit (ICU), intensivists are expected to provide medically appropriate and empathetic communication to support shared decision-making with patients and their families. Large language models (LLMs) have shown potential to generate medical responses that are perceived as informative and empathetic, and collaborative workflows involving physician review of LLM-generated drafts have been evaluated, particularly for ambulatory patient portal messaging. However, it remains unclear whether this collaborative approach is associated with higher ratings of communication quality in ICU EOL care. In this study, we used clinical scenarios simulating ICU EOL situations to compare the quality of written explanations generated by physicians alone, an LLM-based chatbot alone, and physicians who revised LLM-generated drafts.

methodsIn this preliminary, scenario-based study, a total of 45 participants were enrolled, including healthcare professionals and nonhealthcare professionals. Three clinical scenarios simulating ICU EOL care and questions from patients or family members were prepared. For each question, three types of responses were prepared: physician-only, AI-only, and physician-AI collaborative responses. The three prespecified primary outcomes were medical validity and empathy rated by healthcare professionals from diverse specialties and clinical settings, and empathy rated by nonhealthcare professionals, using 5-point Likert scales.

resultsIn the primary analyses, physician-AI collaborative responses received significantly higher ratings than both physician-only and AI-only responses across all three prespecified primary outcomes. The observed differences between physician-AI collaborative responses and both other response groups were more pronounced for empathy than for medical validity. Physician-only and AI-only responses did not differ significantly for any primary outcome in the primary analyses. Sensitivity and exploratory analyses supported the same overall pattern of higher ratings for physician-AI collaborative responses. By contrast, comparisons between physician-only and AI-only responses varied across analytic approaches.

conclusionsIn this scenario-based study of written explanations in ICU EOL care, physician revision of LLM-generated drafts was associated with higher ratings, particularly for empathy. However, the clinical importance of these differences remains uncertain, and the contribution of the LLM-generated draft could not be distinguished from physician revision or the two-stage drafting-and-revision process. Further multicenter, process-matched prospective studies are warranted.