ArticlePLoS computational biology2013
Reinforcement learning using a continuous time actor-critic framework with spiking neurons.
Article in PLoS computational biology, 2013. The graph could read no effect estimate from its abstract, so it casts no vote on the map. Cited by 62 papers.
What it found
Each row is one number read from the abstract, on the scale the paper reported it, with its interval. Left of the dashed line favours the treatment, right favours the comparator. Under each row is the sentence it came from. New to these charts? A ten-minute tutorial.
The abstract states no effect estimate the extractor could read, or names no intervention and outcome on the map, so this paper lights no cell and moves no belief. It is still indexed, cited and linked below.
The trial behind it
Trials whose registry record cites this paper, or whose number appears in the abstract. A trial that started after this paper was published is citing it as background, not reporting it.
Neither the registry nor the abstract names a trial number. If this is a trial report, that itself is worth knowing.
Who cites it
62 citing papers in PubMed, 211 citations in OpenAlex.
- Neuromorphic Devices and Computing for Sensing, Memory, and Control.Advanced science (Weinheim, Baden-Wurttemberg, Germany) · 2026Review
- DeltaQ: Value-Guided Hebbian Learning in Spiking Neuronal Networks for Multi-Goal Navigation.bioRxiv : the preprint server for biology · 2026Article
- Three-factor learning in spiking neural networks: An overview of methods and trends from a machine learning perspective.Patterns (New York, N.Y.) · 2025Review
- A theory of cerebellar learning as spike-based reinforcement learning in continuous time and space.PNAS nexus · 2025Article
- Practice reshapes the geometry and dynamics of task-tailored representations.Cerebral cortex (New York, N.Y. : 1991) · 2025Article
- An accurate and fast learning approach in the biologically spiking neural network.Scientific reports · 2025Article
- Actor-critic networks with analogue memristors mimicking reward-based learning.Nature machine intelligence · 2025Article
- Brain-inspired learning rules for spiking neural network-based control: a tutorial.Biomedical engineering letters · 2025Review
- Exploring spiking neural networks for deep reinforcement learning in robotic tasks.Scientific reports · 2024Article
- Review
- Practice Reshapes the Geometry and Dynamics of Task-tailored Representations.bioRxiv : the preprint server for biology · 2024Article
- Astrocytes enhance plasticity response during reversal learning.Communications biology · 2024Article
- Towards biologically plausible model-based reinforcement learning in recurrent spiking networks by dreaming new experiences.Scientific reports · 2024Article
- [Spatial navigation method based on the entorhinal-hippocampal-prefrontal information transmission circuit of rat's brain].Sheng wu yi xue gong cheng xue za zhi = Journal of biomedical engineering = Shengwu yixue gongchengxue zazhi · 2024Article
- A Review of Brain-Inspired Cognition and Navigation Technology for Mobile Robots.Cyborg and bionic systems (Washington, D.C.) · 2024Review
- Global remapping emerges as the mechanism for renewal of context-dependent behavior in a reinforcement learning model.Frontiers in computational neuroscience · 2024Article
- A spiking neural network for active efficient coding.Frontiers in robotics and AI · 2024Article
- Serotonergic Modulation of the Excitation/Inhibition Balance in the Visual Cortex.International journal of molecular sciences · 2023Article
- A Navigation Path Search and Optimization Method for Mobile Robots Based on the Rat Brain's Cognitive Mechanism.Biomimetics (Basel, Switzerland) · 2023Article
- A self-adaptive hardware with resistive switching synapses for experience-based neurocomputing.Nature communications · 2023Article
2 more citing papers are in PubMed but not listed here.
Corrections and comments
PubMed lists nothing against this paper. Absence here is not a guarantee, only a check that was made.
Authors and funding
3 authors at 2 institutions in 2 countries.
Funding
No grant is acknowledged in the PubMed record.
Abstract
Animals repeat rewarded behaviors, but the physiological basis of reward-based learning has only been partially elucidated. On one hand, experimental evidence shows that the neuromodulator dopamine carries information about rewards and affects synaptic plasticity. On the other hand, the theory of reinforcement learning provides a framework for reward-based learning. Recent models of reward-modulated spike-timing-dependent plasticity have made first steps towards bridging the gap between the two approaches, but faced two problems. First, reinforcement learning is typically formulated in a discrete framework, ill-adapted to the description of natural situations. Second, biologically plausible models of reward-modulated spike-timing-dependent plasticity require precise calculation of the reward prediction error, yet it remains to be shown how this can be computed by neurons. Here we propose a solution to these problems by extending the continuous temporal difference (TD) learning of Doya (2000) to the case of spiking neurons in an actor-critic network operating in continuous time, and with continuous state and action representations. In our model, the critic learns to predict expected future rewards in real time. Its activity, together with actual rewards, conditions the delivery of a neuromodulatory TD signal to itself and to the actor, which is responsible for action choice. In simulations, we show that such an architecture can solve a Morris water-maze-like navigation task, in a number of trials consistent with reported animal performance. We also use our model to solve the acrobot and the cartpole problems, two complex motor control tasks. Our model provides a plausible way of computing reward prediction error in the brain. Moreover, the analytically derived learning rule is consistent with experimental evidence for dopamine-modulated spike-timing-dependent plasticity.
Indexed as
Identifiers
What OpenQuestion holds
Registered trials
Read under generation 80e0d062 · epoch 390. Bibliography from PubMed, PubMed Central and OpenAlex; grants from NIH RePORTER; trial links from ClinicalTrials.gov; estimates, votes and beliefs from the OpenQuestion graph.