reproduce the result of CrazyClimber about efficientzero HOT 13 OPEN

Additionally, I use the test bash script to evaluate the model checkpoint saved in my running 1. The result is as below

[2022-04-06 13:47:41,957][test][INFO][main.py><module>] ==> Test Mean Score: 9603.125 (max: 15300.0, min: 4100.0)
[2022-04-06 13:47:41,958][test][INFO][main.py><module>] ==> Test Std Score: 3000.5712346776572

from efficientzero.

Hello,

First of all, thank you for releasing this much-needed open-source MuZero implementation :).

Strengthening the relevance of this reproducibility issue

Here are my performance results on CrazyClimber, 4 seeds:

I confirm that running the provided training scripts does not lead to results (around 10K mean perf) comparable to those obtained in the paper (around 84k mean perf).
Note that I performed the following modifications to fit my setup: I used "--object_store_memory 100000000000" and "--num_cpus 80", which should not impact performance (right ?).

Potential reasons

One potential reason for this could be that the current train.sh and Atari/Muzero config files differ slightly from those used in the paper. @authors, could it be the case ?

Or maybe it is the version of PyTorch/Python that @yueyang130 and I are using that causes these differences ?
On my side I ran experiments with PyTorch 1.11.0 and Python 3.7.13.

It could also be bad luck, but now that we have more data points it seems unlikely. A similar reproducibility issue has also been raised for Freeway (#23).

Any ideas ? (I'm willing to perform additional experiments if needed)

Hoping to help :)

Best,
Rémy

from efficientzero.

I would love to be connected.

I got these results for the crazy climber by training with 5 MCTS simulations

`[2022-06-14 16:57:12,911][train_test][INFO][log.py>_log] ==> #0 Test Mean Score of CrazyClimberNoFrameskip-v4: 568.75 (max: 1000.0 , min:300.0 , std: 142.38482187368146)

[2022-06-14 18:17:05,152][train_test][INFO][log.py>_log] ==> #10003 Test Mean Score of CrazyClimberNoFrameskip-v4: 2765.625 (max: 3500.0 , min:1900.0 , std: 485.8043426884943)

[2022-06-14 19:20:19,497][train_test][INFO][log.py>_log] ==> #20001 Test Mean Score of CrazyClimberNoFrameskip-v4: 1968.75 (max: 2300.0 , min:1400.0 , std: 189.4688298903015)

[2022-06-14 20:20:17,917][train_test][INFO][log.py>_log] ==> #30009 Test Mean Score of CrazyClimberNoFrameskip-v4: 2121.875 (max: 3000.0 , min:800.0 , std: 548.1413908609712)

[2022-06-14 21:20:22,081][train_test][INFO][log.py>_log] ==> #40034 Test Mean Score of CrazyClimberNoFrameskip-v4: 2400.0 (max: 3300.0 , min:900.0 , std: 765.6696415556777)

[2022-06-14 22:20:22,987][train_test][INFO][log.py>_log] ==> #50046 Test Mean Score of CrazyClimberNoFrameskip-v4: 17384.375 (max: 26200.0 , min:10300.0 , std: 3134.197402745239)

[2022-06-14 23:19:36,779][train_test][INFO][log.py>_log] ==> #60039 Test Mean Score of CrazyClimberNoFrameskip-v4: 21493.75 (max: 29000.0 , min:17100.0 , std: 4227.952925175492)

[2022-06-15 00:18:15,167][train_test][INFO][log.py>_log] ==> #70050 Test Mean Score of CrazyClimberNoFrameskip-v4: 44531.25 (max: 70900.0 , min:11400.0 , std: 14378.900286096292)

[2022-06-15 01:15:28,391][train_test][INFO][log.py>_log] ==> #80031 Test Mean Score of CrazyClimberNoFrameskip-v4: 62137.5 (max: 75800.0 , min:29100.0 , std: 11427.946830030318)

[2022-06-15 02:11:17,348][train_test][INFO][log.py>_log] ==> #90055 Test Mean Score of CrazyClimberNoFrameskip-v4: 36909.375 (max: 55400.0 , min:19900.0 , std: 7831.122180720653)

[2022-06-15 03:06:37,899][train_test][INFO][log.py>_log] ==> #100033 Test Mean Score of CrazyClimberNoFrameskip-v4: 61681.25 (max: 75800.0 , min:27200.0 , std: 14602.106815028439)

[2022-06-15 04:02:12,701][train_test][INFO][log.py>_log] ==> #110069 Test Mean Score of CrazyClimberNoFrameskip-v4: 58618.75 (max: 73500.0 , min:37800.0 , std: 13647.560164274784)

[2022-06-15 04:53:40,941][train_test][INFO][main.py>] ==> #120000 Test Mean Score of CrazyClimberNoFrameskip-v4: 83828.125 (max: 118100.0 , min:21100.0 , std: 29290.103217373184)`

from efficientzero.

Hello @yix081,

The results you are showing are similar to those reported by authors (well played!). Did you obtain them using the current version of the codebase, without additional modifications ? If yes could you share additional details, e.g. your python package versions ('conda list' if you use conda) please ?

On my side since I could not reach these scores, I contacted the authors who kindly shared a previous version of their codebase (before the clean up and open source release). With this "raw" code I was able to reproduce their results. By slowly updating this raw version to the current open-source release, I found that there is probably some bug in the current reanalyze.py, although I can't find it.

from efficientzero.

@rPortelas I changed mcts_num_simulations, so technically I didn't reproduce the original results.

I can help to debug with "raw" code. I will contact the authors today for the code, and @rPortelas do you want to work together to debug reanalyze.py? Why do you think it is due to reanalyze.py?

from efficientzero.

@yix081 Could you post your modifications here? Thank you!

from efficientzero.

@szrlee
The only modification is mcts_num_simulations, and it doesn't work for other games (e.g. will hurt performance). It is very game-specific.

This is an accidental finding while I try different mcts_num_simulations. I don't encourage you to try that. I think the "raw" code @rPortelas mentioned is a better way to go.

from efficientzero.

Hello @szrlee @yix081 ,

Yes I would be happy to share my modifications, and collaborate on improving it :)
I contacted the authors by email to get their permission to share it (since they are based on their raw codebase). @YeWR

Meanwhile, I will also do additional experiments to see if I can reproduce results from other environments than CrazyClimber.

from efficientzero.

@rPortelas @yix081 Thank you! Very interested in the reproducibility issue. Is there anything to share about the debugging in the current reanalyze.py?

from efficientzero.

Hello @szrlee @yix081,

I contacted the authors: they would like to look at this script by themselves (ASAP) to find the exact bug instead of releasing a half-baked reanalyze script.

Which means I cannot share my modifications on this issue, sorry.

from efficientzero.

Hello @szrlee @yix081 @rPortelas ,
Can you reproduce the results of Up N Down with the code in Github? I only get 3297.8, which is far away from 22717.5 written in the paper. Actually, I also cannot get as good results as the paper in some other games, but the gap is not big as CrazyClimber and UpNDown. (Seed=0)
Game Paper Real Run
Qbert 15458.1 11515.6
Assault 1436.3 1101.1
MsPacman 1465.0 839.4
Pong 20.6 14.2
Breakout 432.8 384.2
Asterix 18421.9 3237.5
UpNDown 22717.5 3297.8
CreazyClimber 98640.2 13584.3

from efficientzero.

@dw-Zhao Confirmed with @rPortelas. We both can get 5k on average for UpNDown.

One thing we found is that you need to be careful with kornia version and maybe the OpenCV version too. Although we don't know the exactly right version to pick. Still investigating.

@dw-Zhao I suggest you and other people open new issues to get potential support from authors.

from efficientzero.

Any update on this issue? Thanks

from efficientzero.