Automated Calculation of the Pediatric Emergency Care Applied Research Network (PECARN) Intra-Abdominal Injury Rule from Unstructured Emergency Department Notes Using a Large Language Model.

Objective: The Pediatric Emergency Care Applied Research Network (PECARN) intra-abdominal injury clinical prediction rule identifies children at very low risk for clinically significant intra-abdominal injury after blunt abdominal trauma. However, retrospective evaluation of the rule's use in practice requires resource-intensive chart review. We investigated whether large language models (LLMs) can automatically calculate PECARN intra-abdominal injury risk scores from real-world unstructured emergency department (ED) notes. Methods: We conducted a retrospective secondary analysis of children after blunt abdominal trauma presenting to 2 urban pediatric EDs. The LLM, OpenAI o1, was compared with clinician-derived standards for PECARN risk scores and predictor variable extraction from unstructured physician ED notes. Performance was assessed using standard diagnostic test characteristics. Results: Among 196 children with complete ED notes, the LLM calculated binary PECARN risk scores with 95.4% accuracy (95% confidence interval [CI] 92.5 to 98.3), 94.4% sensitivity (95% CI 90.6 to 98.2), 98.2% specificity (95% CI 94.6 to 100), and an F1 score of 0.97 (95% CI 0.90 to 0.98). Agreement by weighted Cohen's kappa between multicategorical LLM-derived risk scores and clinician-derived risk scores was high, 0.96 (95% CI 0.92 to 1.00). Predictor variable extraction demonstrated strong performance across all PECARN variables, with accuracy greater than 94%. Conclusion: The LLM had comparable performance to clinicians in calculating binary and multicategorical PECARN intra-abdominal injury risk scores and extracting predictor variables from unstructured ED notes. LLMs may enable automated calculation of clinical prediction rules from clinical documentation.