Replayloader doesn't work for Atari about drqv2 HOT 9 CLOSED

Hi, I think before starting sampling from the replay buffer you need to have at least N episodes to be stored in the replay buffer, where N is the number of workers. Otherwise some of the workers end up with no data to sample from and throw this error.

So make sure to modify your code in such a way that you already have enough data collected.

from drqv2.

Curiously, I tried that, even set the number of seed frames to num_replay_workers*episode_length, but it still kept throwing that error.

from drqv2.

Oh, looks like I found the problem. It was something unrelated.

from drqv2.

Sorry if this is a lot of trouble to answer, but I've implemented a version of DrQ with DQN, including double Q learning and Duel Q Networks like in the original DrQ paper. Same hyperparams, the only difference is no terminal states since I'm using this code's replay buffer (although I guess it wouldn't be too hard to add them). The only thing is, I'm not able to reproduce the reported Atari results in that paper... Are there any other additions I should consider besides the double/duel Q learning and matching up hyperparams? I store the "episodes" in 100-frame increments for the replay buffer, but the actual training proceeds normally until the episode is really completed. Also, exploration and intensity augmentation are same as in DrQ paper, too. I even borrowed from DrQv2 and made the action sampling "noisy" via an increasing softmax temperature and categorical sampling for training instead of just taking the max.

from drqv2.

Are you using the exact Atari wrapper than described in Rainbow? For example, stick actions, terminate on life loss, etc.? Those are very important to get right. Also for Atari we used additional data augmentation in a form of noise.

It is hard to pin point exactly what is the issue with your code, but I'm happy to take a look at it and see if I can spot anything. Please email me at [email protected] if you want me to take a look at your code.

from drqv2.

I'm using a slightly different Atari wrapper. If it's alright, I'll send you the code, because I'm having trouble reproducing the results. Any chance you have a script handy that I can use to compile results output by this repo? Currently, I've just been manually looking at the eval CSVs. Just checking, would save me some time having to code one up from scratch.

from drqv2.

Here is a sample script that you can use to plot csvs: https://github.com/denisyarats/pytorch_sac/blob/master/data/sac.ipynb

Ok, send me our your code and I can take a look.

from drqv2.

Just sent over the code. I'm pretty shell-shocked by how low the performance turned out:

5 seeds, 99996 frames:
alien: 377.0 ± 126.4
amidar: 42.2 ± 15.7
assault: 371.7 ± 33.7
asterix: 309.0 ± 183.0
bankheist: 29.6 ± 17.6
battlezone: 2980.0 ± 1144.4
boxing: -12.5 ± 5.8
breakout: 3.4 ± 1.5
choppercommand: 518.0 ± 314.3
crazyclimber: 120.0 ± 240.0
demonattack: 656.3 ± 174.3
freeway: 24.7 ± 1.1
frostbite: 158.6 ± 15.3
gopher: 103.2 ± 53.9
hero: 2261.0 ± 1096.3
jamesbond: 11.0 ± 22.0
kangaroo: 192.0 ± 170.5
krull: 186.8 ± 223.4
kungfumaster: 270.0 ± 256.7
mspacman: 397.4 ± 99.8
pong: -21.0 ± 0.0
privateeye: 52.0 ± 43.5
qbert: 325.0 ± 15.8
roadrunner: 1270.0 ± 361.6
seaquest: 128.8 ± 34.1
upndown: 834.8 ± 466.6
Mean Human Normalized: -0.04271625231291354
Median Human Normalized: 0.015681259862356113

from drqv2.

Hi, did you ever get a chance to look at this code?

from drqv2.