Code for value function clipping about code-for-paper HOT 6 CLOSED

Hi all,

Thanks for raising this issue! Indeed, the min in the paper is a typo and should be the max, and the pow(2) should be applied first. Both of these are fixed in our official code release, found here: https://github.com/MadryLab/implementation-matters

from code-for-paper.

Same question here

from code-for-paper.

Another question here.Why use value function clip before pow(2)?

from code-for-paper.

Another question here.Why use value function clip before pow(2)?

Personal opinion, I guess it's the same reason with clip pi, to limit the update step length of critic.

from code-for-paper.

Another question here.Why use value function clip before pow(2)?

Personal opinion, I guess it's the same reason with clip pi, to limit the update step length of critic.

Thank you!But in the OpenAI baselines ppo code,the value clipping is:

OpenAI use square(pow(2)) before the max.But in this code,it use max before pow(2):

I am confused about it.

from code-for-paper.

@yuanzhi0515
I guess that pow is more proper to use before max, since max may be affected by the neg/pos.

from code-for-paper.