Alleviating the estimation bias of deep deterministic policy gradient via co-regularization

Yao Li; Yu Hui Wang; Yao Zhong Gan; Xiao Yang Tan

doi:10.1016/j.patcog.2022.108872

Alleviating the estimation bias of deep deterministic policy gradient via co-regularization

Yao Li, Yu Hui Wang, Yao Zhong Gan, Xiao Yang Tan^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

The overestimation in Deep Deterministic Policy Gradients (DDPG) caused by value approximation error may result in unstable policy training. Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses the overestimation but suffers from the underestimation. In this paper, we propose a Co-Regularization based Deep Deterministic (CoD2) policy gradient method to mitigate the estimation bias. Two learners characterized by overestimated and underestimated biases are trained with Co-regularization to achieve this goal. The overestimated and underestimated values are updated conservatively in CoD2 for policy evaluation. Experimental results show that our method achieves comparable performance compared with other methods.

Original language	English (US)
Article number	108872
Journal	Pattern Recognition
Volume	131
DOIs	https://doi.org/10.1016/j.patcog.2022.108872
State	Published - Nov 2022

Bibliographical note

Funding Information:
This work is partially supported by National Key R&D program of China (2021ZD0113203), National Science Foundation of China ( 61976115 , 61732006 ), AI+ Project of NUAA (NZ2020012, 56XZA18009), and research project (50912040302).

Publisher Copyright:
© 2022 Elsevier Ltd

Keywords

Co-training
Deterministic policy gradient
Overestimation
Reinforcement learning
Underestimation

ASJC Scopus subject areas

Software
Signal Processing
Computer Vision and Pattern Recognition
Artificial Intelligence

Access to Document

10.1016/j.patcog.2022.108872

Cite this

@article{6e545a2fdbe14a7686216fd912b2d6cd,

title = "Alleviating the estimation bias of deep deterministic policy gradient via co-regularization",

abstract = "The overestimation in Deep Deterministic Policy Gradients (DDPG) caused by value approximation error may result in unstable policy training. Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses the overestimation but suffers from the underestimation. In this paper, we propose a Co-Regularization based Deep Deterministic (CoD2) policy gradient method to mitigate the estimation bias. Two learners characterized by overestimated and underestimated biases are trained with Co-regularization to achieve this goal. The overestimated and underestimated values are updated conservatively in CoD2 for policy evaluation. Experimental results show that our method achieves comparable performance compared with other methods.",

keywords = "Co-training, Deterministic policy gradient, Overestimation, Reinforcement learning, Underestimation",

author = "Yao Li and Wang, {Yu Hui} and Gan, {Yao Zhong} and Tan, {Xiao Yang}",

note = "Funding Information: This work is partially supported by National Key R&D program of China (2021ZD0113203), National Science Foundation of China ( 61976115 , 61732006 ), AI+ Project of NUAA (NZ2020012, 56XZA18009), and research project (50912040302). Publisher Copyright: {\textcopyright} 2022 Elsevier Ltd",

year = "2022",

month = nov,

doi = "10.1016/j.patcog.2022.108872",

language = "English (US)",

volume = "131",

journal = "Pattern Recognition",

issn = "0031-3203",

publisher = "Elsevier",

}

TY - JOUR

T1 - Alleviating the estimation bias of deep deterministic policy gradient via co-regularization

AU - Li, Yao

AU - Wang, Yu Hui

AU - Gan, Yao Zhong

AU - Tan, Xiao Yang

N1 - Funding Information: This work is partially supported by National Key R&D program of China (2021ZD0113203), National Science Foundation of China ( 61976115 , 61732006 ), AI+ Project of NUAA (NZ2020012, 56XZA18009), and research project (50912040302). Publisher Copyright: © 2022 Elsevier Ltd

PY - 2022/11

Y1 - 2022/11

N2 - The overestimation in Deep Deterministic Policy Gradients (DDPG) caused by value approximation error may result in unstable policy training. Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses the overestimation but suffers from the underestimation. In this paper, we propose a Co-Regularization based Deep Deterministic (CoD2) policy gradient method to mitigate the estimation bias. Two learners characterized by overestimated and underestimated biases are trained with Co-regularization to achieve this goal. The overestimated and underestimated values are updated conservatively in CoD2 for policy evaluation. Experimental results show that our method achieves comparable performance compared with other methods.

AB - The overestimation in Deep Deterministic Policy Gradients (DDPG) caused by value approximation error may result in unstable policy training. Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses the overestimation but suffers from the underestimation. In this paper, we propose a Co-Regularization based Deep Deterministic (CoD2) policy gradient method to mitigate the estimation bias. Two learners characterized by overestimated and underestimated biases are trained with Co-regularization to achieve this goal. The overestimated and underestimated values are updated conservatively in CoD2 for policy evaluation. Experimental results show that our method achieves comparable performance compared with other methods.

KW - Co-training

KW - Deterministic policy gradient

KW - Overestimation

KW - Reinforcement learning

KW - Underestimation

UR - http://www.scopus.com/inward/record.url?scp=85133777680&partnerID=8YFLogxK

U2 - 10.1016/j.patcog.2022.108872

DO - 10.1016/j.patcog.2022.108872

M3 - Article

AN - SCOPUS:85133777680

SN - 0031-3203

VL - 131

JO - Pattern Recognition

JF - Pattern Recognition

M1 - 108872

ER -

Alleviating the estimation bias of deep deterministic policy gradient via co-regularization

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this