Policy gradients with parameter-based exploration for control

Frank Sehnke; Christian Osendorfer; Thomas Rückstieß; Alex Graves; Jan Peters; Jürgen Schmidhuber

doi:10.1007/978-3-540-87536-9_40

Policy gradients with parameter-based exploration for control

Frank Sehnke, Christian Osendorfer, Thomas Rückstieß, Alex Graves, Jan Peters, Jürgen Schmidhuber

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

47 Scopus citations

Abstract

We present a model-free reinforcement learning method for partially observable Markov decision problems. Our method estimates a likelihood gradient by sampling directly in parameter space, which leads to lower variance gradient estimates than those obtained by policy gradient methods such as REINFORCE. For several complex control tasks, including robust standing with a humanoid robot, we show that our method outperforms well-known algorithms from the fields of policy gradients, finite difference methods and population based heuristics. We also provide a detailed analysis of the differences between our method and the other algorithms. © Springer-Verlag Berlin Heidelberg 2008.

Original language	English (US)
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Pages	387-396
Number of pages	10
DOIs	https://doi.org/10.1007/978-3-540-87536-9_40
State	Published - Dec 1 2008
Externally published	Yes

Bibliographical note

Generated from Scopus record by KAUST IRTS on 2022-09-14

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-540-87536-9_40

Cite this

@inproceedings{45a1cf6cd5744df1ad56cc30c19d65e0,

title = "Policy gradients with parameter-based exploration for control",

abstract = "We present a model-free reinforcement learning method for partially observable Markov decision problems. Our method estimates a likelihood gradient by sampling directly in parameter space, which leads to lower variance gradient estimates than those obtained by policy gradient methods such as REINFORCE. For several complex control tasks, including robust standing with a humanoid robot, we show that our method outperforms well-known algorithms from the fields of policy gradients, finite difference methods and population based heuristics. We also provide a detailed analysis of the differences between our method and the other algorithms. {\textcopyright} Springer-Verlag Berlin Heidelberg 2008.",

author = "Frank Sehnke and Christian Osendorfer and Thomas R{\"u}ckstie{\ss} and Alex Graves and Jan Peters and J{\"u}rgen Schmidhuber",

note = "Generated from Scopus record by KAUST IRTS on 2022-09-14",

year = "2008",

month = dec,

day = "1",

doi = "10.1007/978-3-540-87536-9_40",

language = "English (US)",

isbn = "3540875352",

pages = "387--396",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

Policy gradients with parameter-based exploration for control. / Sehnke, Frank; Osendorfer, Christian; Rückstieß, Thomas et al.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2008. p. 387-396.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Policy gradients with parameter-based exploration for control

AU - Sehnke, Frank

AU - Osendorfer, Christian

AU - Rückstieß, Thomas

AU - Graves, Alex

AU - Peters, Jan

AU - Schmidhuber, Jürgen

N1 - Generated from Scopus record by KAUST IRTS on 2022-09-14

PY - 2008/12/1

Y1 - 2008/12/1

N2 - We present a model-free reinforcement learning method for partially observable Markov decision problems. Our method estimates a likelihood gradient by sampling directly in parameter space, which leads to lower variance gradient estimates than those obtained by policy gradient methods such as REINFORCE. For several complex control tasks, including robust standing with a humanoid robot, we show that our method outperforms well-known algorithms from the fields of policy gradients, finite difference methods and population based heuristics. We also provide a detailed analysis of the differences between our method and the other algorithms. © Springer-Verlag Berlin Heidelberg 2008.

AB - We present a model-free reinforcement learning method for partially observable Markov decision problems. Our method estimates a likelihood gradient by sampling directly in parameter space, which leads to lower variance gradient estimates than those obtained by policy gradient methods such as REINFORCE. For several complex control tasks, including robust standing with a humanoid robot, we show that our method outperforms well-known algorithms from the fields of policy gradients, finite difference methods and population based heuristics. We also provide a detailed analysis of the differences between our method and the other algorithms. © Springer-Verlag Berlin Heidelberg 2008.

UR - http://link.springer.com/10.1007/978-3-540-87536-9_40

UR - http://www.scopus.com/inward/record.url?scp=58849128358&partnerID=8YFLogxK

U2 - 10.1007/978-3-540-87536-9_40

DO - 10.1007/978-3-540-87536-9_40

M3 - Conference contribution

SN - 3540875352

SP - 387

EP - 396

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

ER -

Policy gradients with parameter-based exploration for control

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this