cgel / drl Goto Github PK

In your code RM.add(state, action, reward, terminal) in BaseAgent, timestep of self.game_state[:, :, :, -1] self.game_action self.game_reward seem not consistent. your self.game_reward is not the reward when you take action self.game_action on self.game_state[:,:,;-1], however, it is the reward you got from last step.

Hey @cgel,

I have been studying your code since you posted in dennybritz repo and it has been really helpful!!

I have one comment regarding the Replay Memory Class. If I am not wrong, you use the following line to avoid sampling terminal states from the RM:

if self.terminals[(index - self.buff_size -1):index].any():
     continue

But I would say that the conditional should be more like:

if self.terminals[(index - self.buff_size +1):index+1].any():

So that you verify that none of the last buff_size frames are terminal. Ideally, however, there should be another logic block taking care of verifying that index is not smaller than buff_size, and if it is, check also the last entries of the RM (similar to what it's done in the function get_state).

I hope I was clear XD. Thanks again for sharing the code, it helped me a lot.