Using cluster-based permutation tests to estimate MEG/EEG onsets: How bad is it?

Localising effects in space, time and other dimensions is a fundamental goal of magneto- and electroencephalography (EEG) research. A popular exploratory approach applies mass-univariate statistics followed by cluster-sum inferences, an effective way to correct for multiple comparisons while preserving high statistical power by pooling together neighbouring effects. Yet, these cluster-based methods have an important limitation: each cluster is associated with a unique p-value, such that there is no error control at individual timepoints, and one must be cautious about interpreting when and where effects start and end. Sassenhagen and Draschkow (2019) provided an important reminder of this limitation. They also reported results from a simulation, suggesting that onsets estimated from EEG data are both positively biased and very variable. However, the simulation lacked comparisons to other methods. Here, I report such comparisons in a new simulation, replicating the positive bias of the cluster-sum method, but also demonstrating that it performs relatively well, in terms of bias and variability, compared to other methods that provide pointwise p-values: two methods that control the false discovery rate and two methods that control the familywise error rate (cluster-depth and maximum statistic methods). I also present several strategies to reduce estimation bias, including group calibration, group comparison and using binary segmentation, a simple change point detection algorithm that outperformed mass-univariate methods in simulations. Finally, I demonstrate how to generate onset hierarchical bootstrap confidence intervals that integrate variability over trials and participants, a substantial improvement over standard group approaches that ignore measurement uncertainty.