Development and validation of a machine learning model to identify individuals at high risk for psychotic disorders using medical record data.

backgroundReducing the duration of untreated psychosis among individuals with early psychosis is associated with improved clinical outcomes and decreased long-term impairment. However, timely identification of individuals at high risk for psychotic disorders in routine clinical practice is challenging, and many individuals are only identified several years following psychotic-symptom onset. This study aimed to leverage comprehensive electronic medical records to develop and validate a machine learning model to identify individuals at high risk of conversion to a psychotic-spectrum disorder (PSD).

methodsThis was a cross-sectional, retrospective analysis of electronic health record (EHR) data consisting of clinician free-text documentation and structured data (i.e., age, sex, race/ethnicity, psychiatric diagnoses, encounter modality, and department) among 406,268 Kaiser Permanente Northern California members aged 15–29 years with ≥ 1 primary-care encounter between 2017 and 2019 (~ 1,694,531 encounters). Patients with a new-onset PSD were distinguished from those without a diagnosis if they had ≥ 1 PSD diagnosis within 12 months following the index primary care encounter. The prediction models were developed using cross-validation with the gradient boosting and elastic net algorithms on features extracted from notes, and validated in a random test set.

resultsA gradient-boosting model including text features model yielded the highest area under the curve (AUC 0.827 [95% CI: 0.799 to 0.853]), outperforming an elastic-net model (AUC 0.791 [95% CI 0.760 to 0.821]) and a gradient-boosting model that incorporated only discrete variables (AUC 0.610 [95% CI 0.595 to 0.626]). Model performance was similar across subgroups by sex, age, and race/ethnicity. However, all models exhibited suboptimal calibration, with predicted probabilities systematically underestimating observed PSD risk. Increasing the ratio of PSD cases to non-cases improved discrimination, but worsened calibration. Further, predicted probabilities of developing a PSD compressed with imbalance, causing abrupt metric drops at higher thresholds.

conclusionsThis study suggests that individuals at elevated risk of developing a PSD may be identified from a general clinical population using a machine-learning model trained on routine clinical documentation and structured EHR data. However, the low incidence of PSDs led to suboptimal calibration. Future studies may restrict prediction to populations with higher PSD incidence, such as mental health clinics, to improve model calibration. CLINICAL TRIAL NUMBER: Not applicable.

trial registrationNot applicable.