Noisy models of the ventral stream reveal the impact of recurrence and learned representations on information processing timescales.

In vision neuroscience, the temporal dynamics of the sensory stream and of its neural representations are thought to be deeply linked to the function of the hierarchy of cortical areas that deal with object recognition, known as the visual ventral stream. Neural representations that are invariant under identity-preserving object transformations, and therefore allow for efficient learning of object identity, are theorized to emerge from a self-supervised learning process that attempts to extract "temporally stable" features from the sensory input. Conversely, invariance increases along the hierarchy, putatively implying progressively slower neural codes in higher-level areas. Recent neurophysiological evidence shows that indeed, as one moves along this cortical hierarchy, neural representations of dynamic stimuli become slower, and additionally the temporal scales of the within-trial fluctuations of these representations (called "intrinsic timescales") increase starkly. However, while these timescale hierarchies have been reproduced in biologically grounded recurrent models, their network determinants have remained largely unexplored in image-computable models of the ventral stream. Here we investigate the temporal structure of the neural codes in a noisy, recurrent and adaptive model of the ventral visual stream. We show that, surprisingly, the organization of the representation timescales is set by the broad architectural features of the network, regardless of training, while the ordering of the intrinsic timescales across layers is sensitive to the details of the functions implemented by each layer. Our work underscores the importance of the temporal structure of the neural code as a probe for the link between structure and function in models of the vertebrate visual system.