Development and Validation of a Large Language Model Case Identification Strategy for Eosinophilic Esophagitis.

Background and Aims: Epidemiologic research in eosinophilic esophagitis (EoE) is limited by the accuracy and efficiency of case identification algorithms. We aimed to evaluate rule-based natural language processing (RB-NLP) and large language model-based natural language processing (LLM-NLP) pipelines for identifying EoE diagnoses and features from unstructured text. Methods: We identified gastrointestinal pathology reports with any mention of "eosinophil" paired with gastroenterology clinic notes. Three hundred randomly selected patients were divided into training (n = 200, 56 with EoE) and testing (n = 100, 36 with EoE) sets. Manual chart review was the reference standard. RB-NLP used spaCy with medspaCy's clinical components; LLM-NLP prompts were developed through iterative human-in-the-loop refinement. In the validation set, we compared International Classification of Diseases (ICD) codes, RB-NLP, and LLM-NLP against the reference standard using sensitivity (recall), positive predictive value (precision), and F1 score. Results: In the validation set, ICD codes alone had a sensitivity 0.86 (95% confidence interval [CI]: 0.75-0.97), a positive predictive value of 0.97 (95% CI: 0.91-1.0), and an F1 value of 0.91 (95% CI: 0.84-1.0). Combining ICD and LLM-assigned diagnosis yielded a 3-point improvement in F1 score (95% CI: -0.01 to 0.07; Conclusion: Combining LLM-NLP with a single ICD code reduced false negatives and modestly improved the F1 score compared to either method alone. This may represent a scalable approach to enhance EoE case identification in real-world data.