Improving the Interpretability of fMRI Decoding using Deep Neural Networks and Adversarial Robustness.

While deep neural networks (DNNs) are being increasingly used to make predictions from high-dimensional, complex data, they are widely seen as uninterpretable "black boxes", as it can be difficult to discover what input information is used to make predictions. This ability is particularly important for applications in cognitive neuroscience and neuroinformatics. A saliency map is a common approach for producing interpretable visualizations of the relative importance of input features in prediction. However, many methods for creating these maps fail due to focusing too much on the input or being extremely sensitive to small input noise. It is also challenging to evaluate how well saliency maps correspond to the truly relevant input information, given that ground truth is not always available. In this paper, we briefly review a variety of methods for producing gradient-based saliency maps, and present a new adversarial training method we developed to make DNNs robust to input noise with the goal of improving interpretability. We introduce two quantitative evaluation procedures for saliency map methods in functional magnetic resonance imaging (fMRI), applicable whenever a model is being trained to decode some information from imaging data. We describe the rationale for the procedures using a synthetic dataset where the complex activation structure is known. We then use them to evaluate saliency maps produced for linear and DNN models performing task decoding in the Human Connectome Project (HCP) dataset. Our key finding is that saliency maps produced with different methods vary widely in interpretability, as measured by our evaluation procedures in synthetic and HCP data. Strikingly, even when linear and DNN models decode at comparable levels of performance, gradient-based saliency maps from the DNN score higher on interpretability than maps derived from the linear model (via weights or gradient). Finally, saliency maps produced with our adversarial training method outperform those produced with alternative methods.