Version information of this project

This repository is for Deep Reinforcement Learning Based Self Driving Car Control project in ML Jeju Camp 2017

There are 2 main goals for this project.

• Making vehicle simulator with Unity.

• Control self driving car in the simulator with some safety systems.

  As a self driving car engineer, I used lots of vehicle sensors(e.g. RADAR, LIDAR, ...) to perceive environments around host vehicle. Also, There are a lot of Advanced Driver Assistant Systems (ADAS) which are already commercialized. I wanted to combine these things with my deep reinforcement learning algorithms to control self driving car.

Simple overview of my project is as follows.

I will use sensor data and camera image as inputs of DRL algorithm. DRL algorithm decides action according to the inputs. If the action may cause dangerous situation, ADAS controls the vehicle to avoid collision.

Software

• Windows7 (64bit)
• Python 3.5.2
• Anaconda 4.2.0
• Tensorflow 1.0.1

Hardware

• CPU: Intel(R) Core(TM) i7-4790K CPU @ 4.00GHZ

• GPU: GeForce GTX 1080

• Memory: 8GB

  ​

• DQN.py: This is basic DQN model for the simulation.
• Duel_DQN.py: Dueling architecture DQN model for the simulation.
• Final_Model.py: The Proposed DQN model (Double + Prioritized Experience Replay + Dueling) for the simulation

I also upload the other DQN codes which I tested with the games that I made. Check out my DRL github repo

This is my PPT file of final presentation

Also, this are the links for my Driving Simulators.

ADAS Version - Windows

ADAS Version - Mac

ADAS Version - Linux

No ADAS Version - Windows

No ADAS Version - Mac

No ADAS Version - Linux

Specific explanation of my simulator and model is as follows.

I made this simulator to test my DRL algorithms. Also, to test my algorithms, I need sensor data and Camera images as inputs, but there was no driving simulators which provides both sensor data and camera images. Therefore, I tried to make one by myself.

​ The simulator is made by Unity which is widely used for making games. There were so many errors which I had to fix for making this simulator. 😩 It still has some minor issues and one major issue. The major issue is that sometimes connection between DRL code and simulator is disconnected. I am not sure about the reason, but I will keep fixing those errors.

Deep reinforcement learning code is made by Python, I connected this code and unity game by SocketIO. For using SocketIO, I referred to the unity project of Driving simulator (Udacity).

As, I mentioned simulator provides 3 inputs to DRL algorithm. Forward camera, Backward camera, Sensor data. The example of those inputs are as follows.

Front Camera Image	Rear Camera Image	Sensor data Plotting

Also, vehicles of this simulator have some safety functions. This functions are applied to the other vehicles and host vehicle of ADAS version. The sensor overview is as follows.

The safety functions are as follows.

• Forward warning
  • Control the velocity of host vehicle equal to velocity of the vehicle at the front.
  • If distance between two vehicles is too close, rapidly drop the velocity to the lowest velocity
• Side warning: No lane change
• Lane keeping: If vehicle is not in the center of the lane, move vehicle to the center of the lane.

As a result, the action of the vehicle is as follows.

• Do nothing
• Acceleration
• Deceleration
• Lane change to left lane
• Lane change to right lane

For this project, I read papers as follows.

1. Human-level Control Through Deep Reinforcement Learning

2. Deep Reinforcement Learning with Double Q-Learning

3. Prioritized Experience Replay

4. Dueling Network Architecture for Deep Reinforcement Learning

5. Deep Recurrent Q-Learning for Partially Observable MDPs

6. Deep Attention Recurrent Q-Network

7. Playing FPS Games with Deep Reinforcement Learning

I wrote the codes 1 ~ 5. I checked performance of 1 ~ 4 and still trying to check the performance of 5. Lastly, I am studying 6 and 7. As I mentioned you can find the code of those algorithms at my DRL github. (I should write markdown of this repository...)

Therefore, I applied algorithms 1 ~ 4 to my DRL model. The network model is as follows.

Also, I used Prioritized Experience Replay when I choose mini batch and I used Double Q Learning technique when I calculate target value.

This is graph of step - average reward (1000 steps)

The average reward increases! (2.4 ~ 4.2)

After training, host vehicle drives mush faster (almost at the maximum speed!!!) with little lane change!! Yeah! :happy: