Comparing meta-analysis and linear mixed model-based approaches for the analysis of continuous outcomes from batched stepped wedge trials.

backgroundStepped wedge designs are a specific type of cluster randomised trial design that requires all clusters to start and end the study at the same time. Batched stepped wedge designs are a variant that allows clusters to commence the study in batches instead of all at once. Since different batches of clusters may have different characteristics, it is possible that the treatment effect differs across batches. Here, we explore different approaches for accommodating this treatment effect heterogeneity when analysing data from a batched stepped wedge design where each participant provides a single outcome measurement: linear mixed models with a random coefficient for the intervention for each batch of clusters with estimation via restricted maximum likelihood; and meta-analysis approaches that pool the batch-specific treatment effect estimates using fixed- or random-effects meta-analysis approaches. In many batched stepped wedge designs, the number of batches will be low (e.g. 2); it is unclear what impact the small number of batches may have on bias and confidence interval coverage of these approaches.

methodsWe simulated continuous outcome data from a range of batched stepped wedge designs with two and five batches, different within-cluster correlation structures (exchangeable, block-exchangeable), with and without period effects shared across batches, and with and without treatment effect heterogeneity across batches. Using the 'lme4' and 'meta' packages in R 4.4.1, we analysed each simulated data set using both linear mixed effects models (with and without treatment effect heterogeneity) and a range of meta-analysis approaches. We assessed the relative performance of all methods to estimate the true treatment effect using rates of convergence, bias, confidence interval coverage, and empirical standard error.

resultsTreatment effect estimates from all meta-analysis approaches were unbiased. When period effects were incorrectly specified, treatment effects from linear mixed effects models were biased; when correctly specified, treatment effect estimates from these models were unbiased. Confidence interval coverage was similar for all random-effects meta-analysis approaches and close to the nominal level. For fixed-effect meta-analysis and linear mixed models, coverage depended on whether treatment effect heterogeneity was present.

conclusionWhen batched stepped wedge trials include a small number of batches (two and five batches), linear mixed models provide unbiased estimates of treatment effects when correctly specified, but exhibit under-coverage of confidence intervals when treatment effect heterogeneity is present. Random-effects meta-analysis approaches provide unbiased estimates and nominal confidence interval coverage whether or not period or treatment effects vary across batches, but can yield excessively wide confidence intervals. When batched stepped wedge trials include small numbers of batches, we recommend the use of random-effects meta-analysis approaches for the analysis of continuous outcomes.