How Robust Are fMRI- and EEG-Based Representational Similarity Analysis?

In EEG and fMRI analysis, researchers choose from a combinatorially large set of theoretically indistinguishable options while building a data processing pipeline based on individual beliefs and other factors. However, not all pipelines are reliable, although the same might not be evident while performing the analysis. Thus, re-analyzing the data through various alternate pipeline configurations presents an opportunity to determine the reliability of the reported results. Results that are replicated across a larger number of pipeline specifications may be considered reliable. In this paper, we adapt a technique recently developed in the psychology literature, specification curve analysis (SCA), to quantitatively assess the robustness of noninvasive neuromodeling results drawn from fMRI and EEG data. Our empirical results, based on a reanalysis of the THINGS dataset, show that the conclusions drawn from EEG-based RSA are fairly robust to alternative specifications, but not fMRI-based RSA. We present a decision tree-based approach to identifying the most robust specification given a set of alternative specifications and dataset for any analysis. However, even the most robust set of specifications for fMRI-based analysis still yield fairly epistemically fragile conclusions. We conclude, based on these results, that SCA could and should be applied without loss of generality to nearly all event-related fMRI data analysis protocols, with suitable modifications to the set of alternative specifications we used in our work.