Reward-Weighted Regression Converges to a Global Optimum

Miroslav Štrupl, Francesco Faccio, Dylan R. Ashley, Rupesh Kumar Srivastava, Juergen Schmidhuber

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations


Reward-Weighted Regression (RWR) belongs to a family of widely known iterative Reinforcement Learning algorithms based on the Expectation-Maximization framework. In this family, learning at each iteration consists of sampling a batch of trajectories using the current policy and fitting a new policy to maximize a return-weighted log-likelihood of actions. Although RWR is known to yield monotonic improvement of the policy under certain circumstances, whether and under which conditions RWR converges to the optimal policy have remained open questions. In this paper, we provide for the first time a proof that RWR converges to a global optimum when no function approximation is used, in a general compact setting. Furthermore, for the simpler case with finite state and action spaces we prove R-linear convergence of the state-value function to the optimum.
Original languageEnglish (US)
Title of host publicationProceedings of the AAAI Conference on Artificial Intelligence
Number of pages9
StatePublished - Feb 23 2022

Bibliographical note

KAUST Repository Item: Exported on 2022-12-21
Acknowledgements: We would like to thank Sjoerd van Steenkiste and Frantisek ˇ Zˇ ak for their insightful comments. This work was supported ´ by the European Research Council (ERC, Advanced Grant Number 742870), the Swiss National Supercomputing Centre (CSCS, Project s1090), and by the Swiss National Science Foundation (Grant Number 200021 192356, Project NEUSYM). We also thank both the NVIDIA Corporation for donating a DGX-1 as part of the Pioneers of AI Research Award and IBM for donating a Minsky machine.


Dive into the research topics of 'Reward-Weighted Regression Converges to a Global Optimum'. Together they form a unique fingerprint.

Cite this