This software tool provides evaluation capabilities for autonomous driving solutions using simulation. We provide a series of quantitative metrics for the evaluation of autonomous driving solutions with support for two simulators, CARLA (main supported simulator) and gazebo (partial support). Currently supported tasks include:

• Follow-lane
• Driving in traffic
• Navigation

Each task comes with its custom evaluation metrics that can help compare autonomous driving solutions. The main component of the ego vehicle is the controller (brain), which receives sensor data, manipulates it and generates robot control commands based on it. The inner part of the brain can be controlled by an end-to-end deep learning model, written in Tensorflow or PyTorch, a reinforcement learning policy, or even an explicitly programmed policy.

The software provides two main pipelines, a graphical user interface (GUI) and a headless mode (scripted). The first one is intended for testing one brain+model (controller) at a time and debugging it visually while the headless mode is intended for running lots of experiments at the same time for comparison of a batch of brain+models (controllers) in different scenarios.

For more information about the project and how to install it, you can consult the website of Behavior Metrics.

We provide examples for the follow-lane task using CARLA:

• For an example of a robot brain using a Tensorflow model for control with GUI pipeline, run:

python3 driver_carla.py -c configs/CARLA/default_carla_tensorflow.yml -g

• For an example of a robot brain using a PyTorch model for control with GUI pipeline, run:

python3 driver_carla.py -c configs/CARLA/default_carla_torch.yml -g

• For an example of an explicitly programmed robot brain with GUI pipeline, run:

python3 driver_carla.py -c configs/CARLA/default_carla.yml -g

• For an example of the headless pipeline, run:

python3 driver_carla.py -c configs/CARLA/default_carla_multiple.yml -s

If you want to contribute, please first check out CONTRIBUTING.md section.

Behavior Metrics provides two different evaluation modes, GUI evaluation and headless.

In this mode, activated with flag -g, the simulator and software application are displayed.

• Video:

• Scheme:

In this mode, activated with flag -s, the evaluation is conducted without graphical interface.

• Video:

• Scheme:

The robot controller (brain folder) is the main controller of the ego vehicle.

Behavior Metrics uses a publish/subscribe design to communicate with the simulator