Human-expert normalized scores about rainbow HOT 4 OPEN

Although they apparently got the human rewards from the original paper, you can check this paper for human rewards and evaluation.

from rainbow.

Looks like I came up with a script in my Atari repo that can do this, but I can't remember where I got the details (must have scoured through lots of DQN papers). I'm not going to do it myself, but if you want to submit a PR that adds the computation and plotting of this score to test.py then I'd be happy to accept it.

from rainbow.

The scores for some games are different from the ones from the DQN paper:

beam_rider:

• DQN paper's human score: 7456
• your code's human score: 5774.70

Enduro

• DQN paper's human score: 368
• your code's human score: 309.60

Qbert

• DQN paper's human score: 18900
• your code's human score: 13455.00

Pong

• DQN paper's human score: -3
• your code's human score: 9.30

Space invaders:

• DQN paper's human score: 3690
• your code's human score: 1652.30

Do you remember which papers you got these numbers from?

from rainbow.

Unfortunately not. Maybe you can email one of the authors of Rainbow to see if they can give you a list of the human rewards and also confirm the score calculation?

from rainbow.