Predicting Remission in Schizophrenia Using Machine Learning-Assessing the Impact of Sample Size and Predictor Overinclusion.

introductionMachine learning studies sometimes include a high number of predictors relative to the number of training cases. This increases the risk of overfitting and poor generalizability. A recent study hypothesized that between-trial heterogeneity precluded generalizable outcome prediction in schizophrenia from being achieved. However, an alternative explanation is that predictor overinclusion might explain the low generalizability in that analysis.

methodsPositive and Negative Syndrome Scale (PANSS) item-data, age, sex, and treatment allocation (antipsychotic/placebo) from 18 placebo-controlled trials of risperidone and paliperidone, in schizophrenia or schizoaffective disorder, were used as predictors for training five supervised learning models to predict symptom remission after 4 weeks of treatment. Sensitivity analyses varying the number of training cases and including simulated uninformative predictors were conducted to assess model performance, as were analyses on simulated data.

resultsBetter-than-chance predictions could be achieved for all models using as few as 384 training cases (BAC 0.60, SD 0.035 for an ensemble model). Model performance increased with the number of training cases (n = 4384, BAC 0.63, SD 0.041) and was higher when validated on a set of unseen trials without placebo controls (n = 1508, BAC 0.68, SD 0.013). Predictive performance was substantially decreased by including simulated uninformative predictors. Analyses of simulated data suggest that considerably larger sample sizes than commonly used might be required to effectively separate weakly informative from uninformative predictors.

conclusionSupervised learning models can generate better-than-chance predictions in schizophrenia from small datasets, but this requires that not too many uninformative predictors are included. Since highly predictive models have not yet been established for schizophrenia-and since strong linear predictors are easy to identify-commonly collected clinical trial data likely do not contain predictors with strong linear relations to clinically relevant outcomes. If correct, future machine learning analyses should focus on maximizing the probability of identifying weakly predictive features.