ArticlePLoS computational biology2011
An imperfect dopaminergic error signal can drive temporal-difference learning.
Article in PLoS computational biology, 2011. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 22 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
22 citing papers in PubMed.
- Unsupervised Learning and Clustered Connectivity Enhance Reinforcement Learning in Spiking Neural Networks.Frontiers in computational neuroscience · 2021Article
- A Closed-Loop Toolchain for Neural Network Simulations of Learning Autonomous Agents.Frontiers in computational neuroscience · 2019Article
- Neuromodulated Synaptic Plasticity on the SpiNNaker Neuromorphic System.Frontiers in neuroscience · 2018Article
- Reinforcement learning with Marr.Current opinion in behavioral sciences · 2016Article
- Goal-Directed Decision Making with Spiking Neurons.The Journal of neuroscience : the official journal of the Society for Neuroscience · 2016Article
- Closed Loop Interactions between Spiking Neural Network and Robotic Simulators Based on MUSIC and ROS.Frontiers in neuroinformatics · 2016Article
- Functional Relevance of Different Basal Ganglia Pathways Investigated in a Spiking Model with Reward Dependent Plasticity.Frontiers in neural circuits · 2016Article
- Opening bottlenecks on weighted networks by local adaptation to cascade failures.Journal of complex networks · 2015Article
- Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning.The European journal of neuroscience · 2015Article
- How attention can create synaptic tags for the learning of working memories in sequential tasks.PLoS computational biology · 2015Article
- Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules.Frontiers in neural circuits · 2015Review
- Liquid computing on and off the edge of chaos with a striatal microcircuit.Frontiers in computational neuroscience · 2014Article
- Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits.Frontiers in neural circuits · 2014Article
- A framework for plasticity implementation on the SpiNNaker neural architecture.Frontiers in neuroscience · 2014Article
- Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior.The Journal of neuroscience : the official journal of the Society for Neuroscience · 2013Article
- Reinforcement learning using a continuous time actor-critic framework with spiking neurons.PLoS computational biology · 2013Article
- Reward-based learning under hardware constraints-using a RISC processor embedded in a neuromorphic substrate.Frontiers in neuroscience · 2013Article
- Rare neural correlations implement robotic conditioning with delayed rewards and disturbances.Frontiers in neurorobotics · 2013Article
- A reafferent and feed-forward model of song syntax generation in the Bengalese finch.Journal of computational neuroscience · 2011Article
- Compositionality of arm movements can be realized by propagating synchrony.Journal of computational neuroscience · 2011Article
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors.
Funding
No grant is acknowledged in the PubMed record.
Abstract
An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.