Comments (13)
Additionally, I use the test bash script to evaluate the model checkpoint saved in my running 1. The result is as below
[2022-04-06 13:47:41,957][test][INFO][main.py><module>] ==> Test Mean Score: 9603.125 (max: 15300.0, min: 4100.0)
[2022-04-06 13:47:41,958][test][INFO][main.py><module>] ==> Test Std Score: 3000.5712346776572
from efficientzero.
Hello,
First of all, thank you for releasing this much-needed open-source MuZero implementation :).
Strengthening the relevance of this reproducibility issue
Here are my performance results on CrazyClimber, 4 seeds:
I confirm that running the provided training scripts does not lead to results (around 10K mean perf) comparable to those obtained in the paper (around 84k mean perf).
Note that I performed the following modifications to fit my setup: I used "--object_store_memory 100000000000" and "--num_cpus 80", which should not impact performance (right ?).
Potential reasons
One potential reason for this could be that the current train.sh and Atari/Muzero config files differ slightly from those used in the paper. @authors, could it be the case ?
Or maybe it is the version of PyTorch/Python that @yueyang130 and I are using that causes these differences ?
On my side I ran experiments with PyTorch 1.11.0 and Python 3.7.13.
It could also be bad luck, but now that we have more data points it seems unlikely. A similar reproducibility issue has also been raised for Freeway (#23).
Any ideas ? (I'm willing to perform additional experiments if needed)
Hoping to help :)
Best,
Rémy
from efficientzero.
I would love to be connected.
I got these results for the crazy climber by training with 5 MCTS simulations
`[2022-06-14 16:57:12,911][train_test][INFO][log.py>_log] ==> #0 Test Mean Score of CrazyClimberNoFrameskip-v4: 568.75 (max: 1000.0 , min:300.0 , std: 142.38482187368146)
[2022-06-14 18:17:05,152][train_test][INFO][log.py>_log] ==> #10003 Test Mean Score of CrazyClimberNoFrameskip-v4: 2765.625 (max: 3500.0 , min:1900.0 , std: 485.8043426884943)
[2022-06-14 19:20:19,497][train_test][INFO][log.py>_log] ==> #20001 Test Mean Score of CrazyClimberNoFrameskip-v4: 1968.75 (max: 2300.0 , min:1400.0 , std: 189.4688298903015)
[2022-06-14 20:20:17,917][train_test][INFO][log.py>_log] ==> #30009 Test Mean Score of CrazyClimberNoFrameskip-v4: 2121.875 (max: 3000.0 , min:800.0 , std: 548.1413908609712)
[2022-06-14 21:20:22,081][train_test][INFO][log.py>_log] ==> #40034 Test Mean Score of CrazyClimberNoFrameskip-v4: 2400.0 (max: 3300.0 , min:900.0 , std: 765.6696415556777)
[2022-06-14 22:20:22,987][train_test][INFO][log.py>_log] ==> #50046 Test Mean Score of CrazyClimberNoFrameskip-v4: 17384.375 (max: 26200.0 , min:10300.0 , std: 3134.197402745239)
[2022-06-14 23:19:36,779][train_test][INFO][log.py>_log] ==> #60039 Test Mean Score of CrazyClimberNoFrameskip-v4: 21493.75 (max: 29000.0 , min:17100.0 , std: 4227.952925175492)
[2022-06-15 00:18:15,167][train_test][INFO][log.py>_log] ==> #70050 Test Mean Score of CrazyClimberNoFrameskip-v4: 44531.25 (max: 70900.0 , min:11400.0 , std: 14378.900286096292)
[2022-06-15 01:15:28,391][train_test][INFO][log.py>_log] ==> #80031 Test Mean Score of CrazyClimberNoFrameskip-v4: 62137.5 (max: 75800.0 , min:29100.0 , std: 11427.946830030318)
[2022-06-15 02:11:17,348][train_test][INFO][log.py>_log] ==> #90055 Test Mean Score of CrazyClimberNoFrameskip-v4: 36909.375 (max: 55400.0 , min:19900.0 , std: 7831.122180720653)
[2022-06-15 03:06:37,899][train_test][INFO][log.py>_log] ==> #100033 Test Mean Score of CrazyClimberNoFrameskip-v4: 61681.25 (max: 75800.0 , min:27200.0 , std: 14602.106815028439)
[2022-06-15 04:02:12,701][train_test][INFO][log.py>_log] ==> #110069 Test Mean Score of CrazyClimberNoFrameskip-v4: 58618.75 (max: 73500.0 , min:37800.0 , std: 13647.560164274784)
[2022-06-15 04:53:40,941][train_test][INFO][main.py>] ==> #120000 Test Mean Score of CrazyClimberNoFrameskip-v4: 83828.125 (max: 118100.0 , min:21100.0 , std: 29290.103217373184)`
from efficientzero.
Hello @yix081,
The results you are showing are similar to those reported by authors (well played!). Did you obtain them using the current version of the codebase, without additional modifications ? If yes could you share additional details, e.g. your python package versions ('conda list' if you use conda) please ?
On my side since I could not reach these scores, I contacted the authors who kindly shared a previous version of their codebase (before the clean up and open source release). With this "raw" code I was able to reproduce their results. By slowly updating this raw version to the current open-source release, I found that there is probably some bug in the current reanalyze.py, although I can't find it.
from efficientzero.
@rPortelas I changed mcts_num_simulations, so technically I didn't reproduce the original results.
I can help to debug with "raw" code. I will contact the authors today for the code, and @rPortelas do you want to work together to debug reanalyze.py? Why do you think it is due to reanalyze.py?
from efficientzero.
@yix081 Could you post your modifications here? Thank you!
from efficientzero.
@szrlee
The only modification is mcts_num_simulations, and it doesn't work for other games (e.g. will hurt performance). It is very game-specific.
This is an accidental finding while I try different mcts_num_simulations. I don't encourage you to try that. I think the "raw" code @rPortelas mentioned is a better way to go.
from efficientzero.
Yes I would be happy to share my modifications, and collaborate on improving it :)
I contacted the authors by email to get their permission to share it (since they are based on their raw codebase). @YeWR
Meanwhile, I will also do additional experiments to see if I can reproduce results from other environments than CrazyClimber.
from efficientzero.
@rPortelas @yix081 Thank you! Very interested in the reproducibility issue. Is there anything to share about the debugging in the current reanalyze.py?
from efficientzero.
I contacted the authors: they would like to look at this script by themselves (ASAP) to find the exact bug instead of releasing a half-baked reanalyze script.
Which means I cannot share my modifications on this issue, sorry.
from efficientzero.
Hello @szrlee @yix081 @rPortelas ,
Can you reproduce the results of Up N Down with the code in Github? I only get 3297.8, which is far away from 22717.5 written in the paper. Actually, I also cannot get as good results as the paper in some other games, but the gap is not big as CrazyClimber and UpNDown. (Seed=0)
Game Paper Real Run
Qbert 15458.1 11515.6
Assault 1436.3 1101.1
MsPacman 1465.0 839.4
Pong 20.6 14.2
Breakout 432.8 384.2
Asterix 18421.9 3237.5
UpNDown 22717.5 3297.8
CreazyClimber 98640.2 13584.3
from efficientzero.
@dw-Zhao Confirmed with @rPortelas. We both can get 5k on average for UpNDown.
One thing we found is that you need to be careful with kornia version and maybe the OpenCV version too. Although we don't know the exactly right version to pick. Still investigating.
@dw-Zhao I suggest you and other people open new issues to get potential support from authors.
from efficientzero.
Any update on this issue? Thanks
from efficientzero.
Related Issues (20)
- EfficientZero high memory consumption / keeps increasing after replay buffer is full HOT 7
- The first selfplay worker uses the same seed for all parallel environments HOT 2
- reproduce results for other environment
- Clarification on the atari environment?
- Envs seem not to work in parallel
- Cannot reproduce Breakout results
- Question about the transform between true reward and value prefix HOT 1
- Question about the index of pad_child_visits_lst in selfplay_worker.py HOT 2
- Question about the effect of state encoding indentity connection in dynamics network HOT 1
- Question about the effect of state encoding indentity connection in dynamics network
- Question about getting zero test score when I try to run EfficientZero on BabyAI grid environment HOT 2
- In reanalyze_worker GPU worker, why prepare policy targets and value targets separately? HOT 2
- Question about whether need to train multiple agents for different games HOT 1
- WSL2 NVIDIA 3090 or M1 MBP correct environment
- How to use with SLURM
- Question about the effect of discount factor and done mask when calculating the target value?
- Question about the test phase not always running fully HOT 1
- ray warning HOT 1
- Code for continuous action space HOT 1
- EfficientZero V2 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from efficientzero.