Real-World Implementation of Large Language Models for Writing Clinical Discharge Summaries Within a Secure Data Environment: Development and Expert Evaluation Study.

backgroundA discharge summary should be a clinical report that documents a patient's hospital stay, including test results, diagnoses, management, and follow-up. Currently, discharge summaries are written by clinicians who manually locate pertinent information across the electronic health record, of which approximately 80% is free text. This process is time-consuming and may be suitable for automation using large language models.

objectiveThis study developed a template-based prompting system that can produce clinically acceptable discharge summaries, specifically the "clinical summary" and "plan and requested actions" sections, from routinely collected electronic patient records.

methodsThis study used electronic health record data from Imperial College Healthcare National Health Service Trust, a network of 5 hospitals in northwest London. It was conducted within the Imperial Secure Data Environment under Data Access and Ethics Committee approval. In total, 52 inpatient encounters were selected by the clinical team to ensure diversity in clinical specialty, reason for admission, complexity, length of stay, and sociodemographic characteristics; 83% (n=43) of the cases were allocated to the development dataset, and 17% (n=9) comprised the test dataset. The system synthesized clinical notes related to an inpatient hospital encounter and used structured template prompts with OpenAI's generative pretrained transformer-4 to generate a discharge summary. The prompt was co-designed across 3 iterations. Resident physicians completed an evaluation form to assess the clinical acceptability of the generated summaries, including the primary outcome (global confidence rating) and secondary outcomes (accuracy, completeness, readability, formatting, sociodemographic bias, and potential clinical harm). Sensitivity analyses assessed the effect of length of stay and admission type (emergency department vs other and surgical vs other) on the primary outcome.

resultsA total of 52 patients (n=32, 62% female) were included, with a mean age of 44.8 (SD 27.1) years and an average length of stay of 15.2 (SD 21.1) days. In the test dataset, 89% (8/9) of generative pretrained transformer-4-generated summaries received a positive global confidence rating ("yes" or "yes, with minor changes"). Secondary outcomes were positive for the "clinical summary" section (8/9, 89% complete and 7/9, 78% accurate) and the "plan and requested actions" section (7/9, 78% complete and 7/9, 78% accurate). Readability, formatting, sociodemographic bias, and potential clinical harm also showed positive results in the test dataset. Sensitivity analyses showed no statistically significant variation in the primary outcome across length of stay or admission type (length of stay: P=.29; surgical admission: P=.99; emergency department admission: P=.15).

conclusionsOur results demonstrate the feasibility of the pipeline, but rigorous statistical evaluation in a larger, adequately powered sample is needed.