Rewards Not Zero after Done about brax HOT 6 CLOSED

When using the gym.vector.VectorEnv API (which is what you're using in this case since batch_size=2048), you don't need to reset the individual envs because they already auto-reset when the episode in any given env is done.

Also just FYI:

• when you have done=True for the environment at a given index env_idx, then obs[env_idx] is the first observation of the next episode, not the final observation of the previous episode.
• Likewise, the reward for env env_idx when done[env_idx]==True is the reward associated with the last action you sent. You will have non-zero rewards at that index after that step: since those will be the rewards of the next episode!

Hope this helps :)

from brax.

We have some new reset logic plumbing that should resolve this issue in the next couple of days. Originally, we found that we didn't really need to reset during rollouts--we'd just run a rollout for a fixed episode length and then mask out frames after the point at which a done was triggered. Admittedly, this is not what folks are used to, so we'll introduce a wrapper that does that actual resetting logic as you'd usually expect.

from brax.

OK! This should be addressed. Envs by default now reset after done=True. You can still get the old behavior if you wish to control auto-resetting yourself, by calling envs.create(..., auto_reset=False)

from brax.

@lebrice Thank you very much for the help!
But if they're on an auto-reset mode, done[env_idx] of next episode should be False.
I have checked the next value of the same env_idx but it was still True. That's a bug?

from brax.

done[env_idx] of next episode should be False.

No! (Edit: maybe I'm misunderstanding your problem though)

Are you saying that you get done[env_idx]==True multiple steps in a row for the same env_idx?

from brax.

Are you saying that you get done[env_idx]==True multiple steps in a row for the same env_idx?

@lebrice Yes.

from brax.