implementation-matters / code-for-paper Goto Github PK

Can you update the version of the libraries in the requirements.txt? Thank you!

Hi,

I would like to use the code for my research. Is it possible, that a license e.g. MIT is added to the repository?
Thanks.

In the paper (sec 2) and description of value function clipping, the PPO-like objective is the minimum of clipped and unclipped. However, the code applies the maximum of them, does it have different effects from the paper said?

The original implementation first squares both the clipped and unclipped losses then takes the maximum. Any comment as to why this is handled differently here?

Hi, thanks for the great work. I have three questions if you don't mind.

1. In

   code-for-paper/src/policy_gradients/torch_utils.py

   Line 358 in 094994f

   class RewardFilter:

   ,
   the comments suggest it uses Incorrect reward normalization. I was wondering if you could elaborate. Does that mean we should avoid using RewardFilter because of the incorrect normalization and try to use Zfilter instead for the reward normalization?

Another concern I have is with the reset() call of the RewardFilter. It seems that in your customized envs,

    def reset(self):
        # Reset the state, and the running total reward
        start_state = self.env.reset()
        self.total_true_reward = 0.0
        self.counter = 0.0
        self.state_filter.reset()
        return self.state_filter(start_state, reset=True)

2. It seems the reward_filter will never reset. However, the reward_filter always multiply the existing returns by gamma. Could this be a bug?

3. The reward_filter is already using the gamma as part of its inputs, but do you still calculate the advantage using the gamma again or is this somehow omitted?

Thanks.

When I run the PPO-M with the default parms in MuJoCo.json,the average mean_reward of the last train step over 10 agents is much smaller than the result in paper.In Walker2d-v2,my result of PPO-M is about 600-900.And in Hopper-v2,the result is about 300. In Humanoid-2,it always fail to run using the default ppo_lr_adam=1e-4