Atari Reinforcement Learning Leaderboard

Any scores out of date? Make a Pull Request.

This is a leaderboard comparing world record human performance to start of the art machine performance in the Arcade Learning Environment (ALE).

• N NTSC, no emulator results available
• J Score from jvgs.net
• E Game is so easy there's no world record category
• 1B Game 1, Difficulty B
• * Game 6, Difficulty B
• + Game 7, Difficulty B
• ** Game 1, Points
• ++ Game 2, Difficulty A

What the point of this?

I decided to put this together after noticing two trends in reinforcement learning papers:

• Not comparing to state of art.
• Comparing an algorithm with 1000s of hours playtime to a human that played for a few hours.

Respectively, these make it hard to see the relative progress of the field from paper to paper, and the absolute progress compared to human level game playing.

Though RL papers routinely quote >100% normalized human performance, the reality is that machine learning algorithms just barely beat humans on only 5 out of 49 games here, and humans have a substantial lead in the rest. We have a long way to go.

Performance Among Machines

When we exclude human scores, per-algorithm win count are as follows (two way ties friendly, three or more unfriendly):

Methodology

Human Scores

Since the ALE uses the stella Atari emulator, the Top Human Score is the top human score on an emulator. Atari (and other game) releases tend to vary across region, so this is the only way to ensure that both human and machine have, for example, equal access to game breaking bugs.

If possible, scores are taken from Twin Galaxies, which is the Guiness source for game world records, otherwise links are provided to score sources.

Machine Scores

A valid machine score is one achieved by a reinforcement learning algorithm trained directly on pixels and raw rewards, such as one that can be trained against common ALE wrappers / forks, like gym or xitari. This means that algorithms like this one which use hand-engineered intermediate rewards do not qualify.

Reference papers vary in:

• Start type (no-op, random-op, human-op)
• Number of test trials (from 30-200)

I take the approach here of favouring no-op starts over random ones (they usually have higher scores anyway), and treating all sample sizes equally.

References

• Human
• A2C
• A3C
• ACER
• Distribution DQN
• Dueling DDQN
• HyperNeat, also checked against original paper
• Rainbow
• Prioritized DDQN
• Proximal Policy Optimization
• Vanilla Evolution Strategies
• Vanilla Genetic Algorithm