How to quantify between-study heterogeneity in single-arm evidence synthesis?-It depends!

backgroundRandom-effects meta-analysis models account for between-study heterogeneity by estimating and incorporating the heterogeneity variance parameter

methodsWe compared seven heterogeneity variance estimators for random-effects meta-analysis. The estimators were selected on the basis of methodological diversity and availability and were evaluated both empirically and in a simulation study. We simulated typical meta-analysis scenarios for continuous and binary outcomes in a single-arm meta-analysis setting. Through a non-systematic literature review, we assessed which heterogeneity variance estimators are currently used in high-ranked journals, and evaluated their reporting quality.

resultsOur simulation study showed that all evaluated heterogeneity estimators were imprecise and often failed to estimate the true amount of heterogeneity. The estimation is particularly imprecise in situations where the meta-analysis contained few studies or when the binary outcomes included rare events. Moreover, we discovered that most heterogeneity variance estimators produce zero heterogeneity estimates under all simulated conditions, even though heterogeneity was present. The estimated overall effect was found to be relatively robust to different estimators in the empirical application and in our simulation study. However, the prediction intervals for the overall effect vary depending on the estimator chosen.

conclusionsAlthough different heterogeneity variance estimators produce substantially different heterogeneity variance estimates, too little attention is paid to selecting a suitable heterogeneity variance estimator in single-arm evidence synthesis. Based on our literature review, we conclude that the awareness of different heterogeneity variance estimators and their properties needs to be strengthened in practice. Given that it is rarely appropriate to rely on a single heterogeneity variance estimator, we suggest careful consideration and evaluation of a wider range of plausible estimators in a sensitivity analysis before drawing a final conclusion about the meta-analysis results.