Comments (9)
Hi, I think before starting sampling from the replay buffer you need to have at least N
episodes to be stored in the replay buffer, where N
is the number of workers. Otherwise some of the workers end up with no data to sample from and throw this error.
So make sure to modify your code in such a way that you already have enough data collected.
from drqv2.
Curiously, I tried that, even set the number of seed frames to num_replay_workers*episode_length
, but it still kept throwing that error.
from drqv2.
Oh, looks like I found the problem. It was something unrelated.
from drqv2.
Sorry if this is a lot of trouble to answer, but I've implemented a version of DrQ with DQN, including double Q learning and Duel Q Networks like in the original DrQ paper. Same hyperparams, the only difference is no terminal states since I'm using this code's replay buffer (although I guess it wouldn't be too hard to add them). The only thing is, I'm not able to reproduce the reported Atari results in that paper... Are there any other additions I should consider besides the double/duel Q learning and matching up hyperparams? I store the "episodes" in 100-frame increments for the replay buffer, but the actual training proceeds normally until the episode is really completed. Also, exploration and intensity augmentation are same as in DrQ paper, too. I even borrowed from DrQv2 and made the action sampling "noisy" via an increasing softmax temperature and categorical sampling for training instead of just taking the max.
from drqv2.
Are you using the exact Atari wrapper than described in Rainbow? For example, stick actions, terminate on life loss, etc.? Those are very important to get right. Also for Atari we used additional data augmentation in a form of noise.
It is hard to pin point exactly what is the issue with your code, but I'm happy to take a look at it and see if I can spot anything. Please email me at [email protected] if you want me to take a look at your code.
from drqv2.
I'm using a slightly different Atari wrapper. If it's alright, I'll send you the code, because I'm having trouble reproducing the results. Any chance you have a script handy that I can use to compile results output by this repo? Currently, I've just been manually looking at the eval CSVs. Just checking, would save me some time having to code one up from scratch.
from drqv2.
Here is a sample script that you can use to plot csvs: https://github.com/denisyarats/pytorch_sac/blob/master/data/sac.ipynb
Ok, send me our your code and I can take a look.
from drqv2.
Just sent over the code. I'm pretty shell-shocked by how low the performance turned out:
5 seeds, 99996 frames:
alien: 377.0 ± 126.4
amidar: 42.2 ± 15.7
assault: 371.7 ± 33.7
asterix: 309.0 ± 183.0
bankheist: 29.6 ± 17.6
battlezone: 2980.0 ± 1144.4
boxing: -12.5 ± 5.8
breakout: 3.4 ± 1.5
choppercommand: 518.0 ± 314.3
crazyclimber: 120.0 ± 240.0
demonattack: 656.3 ± 174.3
freeway: 24.7 ± 1.1
frostbite: 158.6 ± 15.3
gopher: 103.2 ± 53.9
hero: 2261.0 ± 1096.3
jamesbond: 11.0 ± 22.0
kangaroo: 192.0 ± 170.5
krull: 186.8 ± 223.4
kungfumaster: 270.0 ± 256.7
mspacman: 397.4 ± 99.8
pong: -21.0 ± 0.0
privateeye: 52.0 ± 43.5
qbert: 325.0 ± 15.8
roadrunner: 1270.0 ± 361.6
seaquest: 128.8 ± 34.1
upndown: 834.8 ± 466.6
Mean Human Normalized: -0.04271625231291354
Median Human Normalized: 0.015681259862356113
from drqv2.
Hi, did you ever get a chance to look at this code?
from drqv2.
Related Issues (20)
- Get to 96 frame throughput? HOT 8
- How can i start with window 10. HOT 1
- Truncation for exploration? HOT 2
- Evaluation video overwritten HOT 2
- Not meant as an issue, but a little perplexed by feature_dim... HOT 5
- Reported frame-time question... HOT 3
- One more question, sorry... HOT 3
- custom env HOT 1
- Is it possible to do 1-step learning without done signal?
- DRQ-v2's Performance on Manipulator tasks (bring ball, insert ball, bring peg, insert peg)
- a certain chance to fail
- A question about num_train_frames
- Hyperparameters optimization
- Dreamerv2 learning curve
- Error when Running Training Command HOT 6
- Does drqv2 support multi-processing for the env interaction?
- Why using 'F.grid_sample()' in class RandomShiftsAug?
- Training Manipulation Tasks
- Memory Leak
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drqv2.