A general sample size framework for developing or updating a predictive algorithm: with application to clinical prediction models.

backgroundWhen developing or updating a predictive algorithm to make predictions (e.g., risk estimates) in individuals, the sample size of the development (training) dataset is an important consideration. Small datasets reduce a model's predictive performance, generalisability and fairness, and may lead to harm. To help identify (minimum) sample sizes required for model development or updating, various sample size calculations exist but their underlying theory is based on standard (unpenalised) regression and focus on minimising overfitting. To address this, we propose a more general approach, which allows extension to any machine learning or AI-based modelling method, and to any performance metric (estimand) of interest.

methodsThe proposed approach draws samples from anticipated posterior distributions to examine the impact on degradation in a model's predictive performance compared to a reference model. It requires researchers to provide candidate predictors (features), a reference model (e.g., based on mean outcome incidence, predictor weights and c-statistics of previous models), and a (existing, synthetic or pilot) dataset reflecting the joint distribution of candidate predictors in the target population. A fully simulation-based approach then generates thousands of models conditional on a chosen sample size and model development strategy, to produce posterior distributions of individual predictions and model performance (degradation) metrics, to inform required sample sizes. To improve computational speed for penalised regression, we also propose a one-sample Bayesian approximation that combines shrinkage priors with a likelihood decomposed into sample size and Fisher's unit information.

resultsThe approaches are illustrated when developing models for pre-eclampsia using logistic regression (unpenalised, uniform shrinkage, lasso or ridge) and random forests; and are shown to encompass existing sample size calculation criteria whilst also providing model assurance probabilities, instability metrics, and degradation statistics about calibration, discrimination, clinical utility, prediction error and fairness. Uncertainty in the reference model can also be accounted for in the calculations.

conclusionsOur approach provides a general framework for (minimum) sample size calculations for developing or updating a prediction model, and is applicable for any statistical, machine learning or AI-based development method for supervised learning. Crucially, the recommended (minimum) sample size will depend on users' chosen estimands and model development (or updating) approach. Example R and Stata code is provided.