This repo contains the code that was used in the paper Tactical Decision-Making in Autonomous Driving by Reinforcement Learning with Uncertainty Estimation. If this code is used, please cite the paper and the repo.

Videos of the resulting agents are shown below.

This section shows how the trained ensemble RPF and DQN agents perform in a few testing episodes within the training distribution, and in situations that are outside of the training distribution.

Ensemble RPF agent:

Displayed in x4 real time.

DQN agent:

Ensemble RPF agent:

Displayed in x1 real time.

DQN agent:

Ensemble RPF agent:

Displayed in x1 real time.

Python and Simulation of Urban Mobility (SUMO), version 1.8 (something have changed in the interface in later versions, and I have not had the time to update the code).

Install SUMO from source (tested for ubuntu 18.04):

$ apt-get update
$ apt-get install -y git cmake python3.7 python3-pip g++ libxerces-c-dev libfox-1.6-dev libgdal-dev libproj-dev libgl2ps-dev
$ git clone --depth 1 --branch v1_8_0 https://github.com/eclipse/sumo
$ mkdir sumo/build/cmake-build
$ cd sumo/build/cmake-build
$ cmake ../..
$ make -j$(nproc)
$ make install
$ cd ../../..
$ rm -r sumo
$ export SUMO_HOME="/usr/local/share/sumo"

Install dependencies:

$ pip3 install -r requirements.txt

Using Docker makes it even easier to get started. From the top folder, simply build with Dockerfile:

$ docker build -t <name> -f Dockerfile .

The code is relatively well documented. The following steps help to get you started.

To start the training of the agent, from src/, run:

$ python3 train_agent.py <log_name>

The performance of the agent is regularly tested on a set of test episodes. Logs are saved in logs/, where among other things the following are saved:

• Rewards and number of steps of the test episodes
• Weights of the neural networks for the test episodes
• A tensorboard log

The training process can be followed in tensorboard.

The parameters of the agent are set in parameters.py and the parameters of the simulation environment are set in parameters_simulation.py.

The performance of the trained agent can be observed in SUMO's GUI, both on the test episodes and on the special cases that are mentioned in the paper.

In run_trained_agent.py, set filepath and agent_name to the agent you would like to test. Then, from src/, run:

$ python3 run_trained_agent.py

An example of the output is shown below.

The rows below the road show the estimated Q-values of the different ensemble members for all the actions, followed by the mean Q-values and the coefficient of variation for all the actions.

To make sure that the code is working and all dependencies are fulfilled, run the tests from tests/ by

$ python3 -m unittest discover .

This code is released under the MIT license, see LICENSE.md for details.

Just send an email if you have any questions, [email protected].