Estimating the motor exploration in reinforcement learning.

What exploration strategies do animals use to learn motor skills? Reinforcement learning (RL) theory is a powerful framework to study motor learning, but provides no guidance for estimating motor exploration - the behavioral component aimed at discovering better strategies. We address this gap by taking inspiration from the brain's modular organization and postulating a latent learner that explores by injecting an additive source of ideal randomness into behavior. Assuming the learner is ignorant of other motor components, evolutionary fitness argues that these should display mainly non-ideal variability. We verify this behavioral decomposition in songbirds undergoing vocal pitch conditioning. The estimated vocal explorations account for the motor contribution of a cortico-basal ganglia pathway, while other components capture birds' suboptimal learning trajectories. Latent RL therefore provides a normative improvement over classical RL, making exploration explicit and suggesting that evolutionary pressure favors the randomness of exploration over strict behavioral optimality.