Issues with the RewardFilter about code-for-paper HOT 2 OPEN

hi @vwxyzjn,
To my understanding, the state_filter and reward_filter always keep the inside RunningStats object alive, and will never reset, which means keep track of running stats all over the episodes rather than per episode.(I am a little bit of concerned about its reasonability).

the gamma in reward_filter and the advantage calculation have different meaning and purpose. the former usage is to keep running sum of rewards(discount past returns and then update with momentum), and the latter one is to discount future rewards.

from code-for-paper.

Hi all, I'm an author of the corresponding paper for this repo. Since this was an anonymized submission we were unable to comment/change the code during submission. There is now an update repository with better hyper parameters, where we also switched to a system where we reset the reward filter: https://github.com/MadryLab/implementation-matters

from code-for-paper.