Recurrent policy gradients

Daan Wierstra, Alexander Förster, Jan Peters, Jürgen Schmidhuber

Research output: Contribution to journalArticlepeer-review

52 Scopus citations

Abstract

Reinforcement learning for partially observable Markov decision problems (POMDPs) is a challenge as it requires policies with an internal state. Traditional approaches suffer significantly from this shortcoming and usually make strong assumptions on the problem domain such as perfect system models, state-estimators and a Markovian hidden system. Recurrent neural networks (RNNs) offer a natural framework for dealing with policy learning using hidden state and require only few limiting assumptions. As they can be trained well using gradient descent, they are suited for policy gradient approaches. In this paper, we present a policy gradient method, the Recurrent Policy Gradient which constitutes a model-free reinforcement learning method. It is aimed at training limited-memory stochastic policies on problems which require long-term memories of past observations. The approach involves approximating a policy gradient for a recurrent neural network by backpropagating return-weighted characteristic eligibilities through time. Using a "Long Short-Term Memory" RNN architecture, we are able to outperform previous RL methods on three important benchmark tasks. Furthermore, we show that using history-dependent baselines helps reducing estimation variance significantly, thus enabling our approach to tackle more challenging, highly stochastic environments. © The Author 2009. Published by Oxford University Press. All rights reserved.
Original languageEnglish (US)
Pages (from-to)620-634
Number of pages15
JournalLogic Journal of the IGPL
Volume18
Issue number5
DOIs
StatePublished - Sep 9 2009
Externally publishedYes

ASJC Scopus subject areas

  • Philosophy

Fingerprint

Dive into the research topics of 'Recurrent policy gradients'. Together they form a unique fingerprint.

Cite this