Solving deep memory POMDPs with Recurrent Policy gradients

Daan Wierstra; Alexander Foerster; Jan Peters; Jürgen Schmidhuber

doi:10.1007/978-3-540-74690-4_71

Solving deep memory POMDPs with Recurrent Policy gradients

Daan Wierstra, Alexander Foerster, Jan Peters, Jürgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

95 Scopus citations

Abstract

This paper presents Recurrent Policy Gradients, a model-free reinforcement learning (RL) method creating limited-memory stochastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations. The approach involves approximating a policy gradient for a Recurrent Neural Network (RNN) by backpropagating return-weighted characteristic eligibilities through time. Using a "Long Short-Term Memory" architecture, we are able to outperform other RL methods on two important benchmark tasks. Furthermore, we show promising results on a complex car driving simulation task. © Springer-Verlag Berlin Heidelberg 2007.

Original language	English (US)
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Publisher	Springer Verlag
Pages	697-706
Number of pages	10
ISBN (Print)	9783540746898
DOIs	https://doi.org/10.1007/978-3-540-74690-4_71
State	Published - Jan 1 2007
Externally published	Yes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2022-09-14

Access to Document

10.1007/978-3-540-74690-4_71

Cite this

@inproceedings{f0332d54f87a47fe9a5027676876b644,

title = "Solving deep memory POMDPs with Recurrent Policy gradients",

abstract = "This paper presents Recurrent Policy Gradients, a model-free reinforcement learning (RL) method creating limited-memory stochastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations. The approach involves approximating a policy gradient for a Recurrent Neural Network (RNN) by backpropagating return-weighted characteristic eligibilities through time. Using a {"}Long Short-Term Memory{"} architecture, we are able to outperform other RL methods on two important benchmark tasks. Furthermore, we show promising results on a complex car driving simulation task. {\textcopyright} Springer-Verlag Berlin Heidelberg 2007.",

author = "Daan Wierstra and Alexander Foerster and Jan Peters and J{\"u}rgen Schmidhuber",

note = "Generated from Scopus record by KAUST IRTS on 2022-09-14",

year = "2007",

month = jan,

day = "1",

doi = "10.1007/978-3-540-74690-4_71",

language = "English (US)",

isbn = "9783540746898",

pages = "697--706",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

address = "Germany",

}

Solving deep memory POMDPs with Recurrent Policy gradients. / Wierstra, Daan; Foerster, Alexander; Peters, Jan et al.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Springer Verlag, 2007. p. 697-706.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Solving deep memory POMDPs with Recurrent Policy gradients

AU - Wierstra, Daan

AU - Foerster, Alexander

AU - Peters, Jan

AU - Schmidhuber, Jürgen

N1 - Generated from Scopus record by KAUST IRTS on 2022-09-14

PY - 2007/1/1

Y1 - 2007/1/1

N2 - This paper presents Recurrent Policy Gradients, a model-free reinforcement learning (RL) method creating limited-memory stochastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations. The approach involves approximating a policy gradient for a Recurrent Neural Network (RNN) by backpropagating return-weighted characteristic eligibilities through time. Using a "Long Short-Term Memory" architecture, we are able to outperform other RL methods on two important benchmark tasks. Furthermore, we show promising results on a complex car driving simulation task. © Springer-Verlag Berlin Heidelberg 2007.

AB - This paper presents Recurrent Policy Gradients, a model-free reinforcement learning (RL) method creating limited-memory stochastic policies for partially observable Markov decision problems (POMDPs) that require long-term memories of past observations. The approach involves approximating a policy gradient for a Recurrent Neural Network (RNN) by backpropagating return-weighted characteristic eligibilities through time. Using a "Long Short-Term Memory" architecture, we are able to outperform other RL methods on two important benchmark tasks. Furthermore, we show promising results on a complex car driving simulation task. © Springer-Verlag Berlin Heidelberg 2007.

UR - http://link.springer.com/10.1007/978-3-540-74690-4_71

UR - http://www.scopus.com/inward/record.url?scp=38149018611&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-74690-4_71

DO - 10.1007/978-3-540-74690-4_71

M3 - Conference contribution

SN - 9783540746898

SP - 697

EP - 706

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

PB - Springer Verlag

ER -

Solving deep memory POMDPs with Recurrent Policy gradients

Abstract

Bibliographical note

Access to Document

Other files and links

Fingerprint

Cite this