Deep Reinforcement Learning based autonomous navigation for quadcopters using PPO algorithm.

License: GNU Affero General Public License v3.0

Python 100.00%

ppo stable-baselines3 gym airsim unreal-engine deep-reinforcement-learning

ppo-based-autonomous-navigation-for-quadcopters's Introduction

PPO-based Autonomous Navigation for Quadcopters

This repository contains an implementation of Proximal Policy Optimization (PPO) for autonomous navigation in a corridor environment with a quadcopter. There are blocks having circular opening for the drone to go through for each 4 meters. The expectation is that the agent navigates through these openings without colliding with blocks. This project currently runs only on Windows since Unreal environments were packaged for Windows.

🛠️ Libraries & Tools

Overview

The training environment has 9 sections with different textures and hole positions. The agent starts at these sections randomly. The starting point of the agent is also random within a specific region in the yz-plane.

Observation Space

State is in the form of a RGB image taken by the front camera of the agent.
Image shape: 50 x 50 x 3

Action Space

There are 9 discrete actions.

Environment setup to run the codes

#️⃣ 1. Clone the repository

git clone https://github.com/bilalkabas/PPO-based-Autonomous-Navigation-for-Quadcopters

#️⃣ 2. From Anaconda command prompt, create a new conda environment

I recommend you to use Anaconda or Miniconda to create a virtual environment.

conda create -n ppo_drone python==3.8

#️⃣ 3. Install required libraries

Inside the main directory of the repo

conda activate ppo_drone
pip install -r requirements.txt

#️⃣ 4. (Optional) Install Pytorch for GPU

You must have a CUDA supported NVIDIA GPU.

Details for installation

For this project, I used CUDA 11.0 and the following conda installation command to install Pytorch:

conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=11.0 -c pytorch

#️⃣ 4. Edit settings.json

Content of the settings.json should be as below:

The setting.json file is located at Documents\AirSim folder.

{
    "SettingsVersion": 1.2,
    "LocalHostIp": "127.0.0.1",
    "SimMode": "Multirotor",
    "ClockSpeed": 20,
    "ViewMode": "SpringArmChase",
    "Vehicles": {
        "drone0": {
            "VehicleType": "SimpleFlight",
            "X": 0.0,
            "Y": 0.0,
            "Z": 0.0,
            "Yaw": 0.0
        }
    },
    "CameraDefaults": {
        "CaptureSettings": [
            {
                "ImageType": 0,
                "Width": 50,
                "Height": 50,
                "FOV_Degrees": 120
            }
        ]
    }
  }

How to run the training?

Make sure you followed the instructions above to setup the environment.

#️⃣ 1. Download the training environment

Go to the releases and download TrainEnv.zip. After downloading completed, extract it.

#️⃣ 2. Now, you can open up environment's executable file and start the training

So, inside the repository

python main.py

How to run the pretrained model?

Make sure you followed the instructions above to setup the environment. To speed up the training, the simulation runs at 20x speed. You may consider to change the "ClockSpeed" parameter in settings.json to 1.

#️⃣ 1. Download the test environment

Go to the releases and download TestEnv.zip. After downloading completed, extract it.

#️⃣ 2. Now, you can open up environment's executable file and run the trained model

So, inside the repository

python policy_run.py

Training results

The trained model in saved_policy folder was trained for 280k steps.

Test results

The test environment has different textures and hole positions than that of the training environment. For 100 episodes, the trained model is able to travel 17.5 m on average and passes through 4 holes on average without any collision. The agent can pass through at most 9 holes in test environment without any collision.

Author

Bilal Kabas

License

This project is licensed under the GNU Affero General Public License.

ppo-based-autonomous-navigation-for-quadcopters's People

Contributors

Stargazers

Watchers

Forkers

drone-3-team lsl001006 ramseyous0109 ralami1859 sigmalex mychoi97 tk1363704 abiaoaaaaa nninept

ppo-based-autonomous-navigation-for-quadcopters's Issues

env page not found

hello i want to learn this model and env. but, your env release page wasn't found. can i get some env file?

Unreal Engine Environment

Thank you very much for providing the UE environment, but when we downloaded and opened it, it turned out to be a packaged UE engine. May I ask if you can upload an unpackaged environment (i.e. a folder with .uproject files)? We would like to know the coordinate design of the hole, which is a crucial issue. Thank you for your work and time!

A warning I met during I perform "python policy_run.py"

I have followed each step as suggested by the readme. However, I encounter the problem as follow:

WARNING:tornado.general:Connect error on fd 336: WSAECONNREFUSED
WARNING:tornado.general:Connect error on fd 336: WSAECONNREFUSED
WARNING:tornado.general:Connect error on fd 336: WSAECONNREFUSED
WARNING:tornado.general:Connect error on fd 336: WSAECONNREFUSED
WARNING:tornado.general:Connect error on fd 336: WSAECONNREFUSED
Traceback (most recent call last):
File "policy_run.py", line 14, in
env = DummyVecEnv([lambda: Monitor(
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py", line 25, in init
self.envs = [fn() for fn in env_fns]
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py", line 25, in
self.envs = [fn() for fn in env_fns]
File "policy_run.py", line 15, in
gym.make(
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\gym\envs\registration.py", line 235, in make
return registry.make(id, **kwargs)
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\gym\envs\registration.py", line 129, in make
env = spec.make(kwargs)
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\gym\envs\registration.py", line 90, in make
env = cls(_kwargs)
File "E:\Project\PPO_based_ANfQ\PPO-based-Autonomous-Navigation-for-Quadcopters\scripts\airsim_env.py", line 169, in init
super(TestEnv, self).init(ip_address, image_shape, env_config)
File "E:\Project\PPO_based_ANfQ\PPO-based-Autonomous-Navigation-for-Quadcopters\scripts\airsim_env.py", line 19, in init
self.setup_flight()
File "E:\Project\PPO_based_ANfQ\PPO-based-Autonomous-Navigation-for-Quadcopters\scripts\airsim_env.py", line 174, in setup_flight
super(TestEnv, self).setup_flight()
File "E:\Project\PPO_based_ANfQ\PPO-based-Autonomous-Navigation-for-Quadcopters\scripts\airsim_env.py", line 36, in setup_flight
self.drone.reset()
File "E:\Project\PPO_based_ANfQ\PPO-based-Autonomous-Navigation-for-Quadcopters\scripts\airsim\client.py", line 26, in reset
self.client.call('reset')
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\msgpackrpc\session.py", line 41, in call
return self.send_request(method, args).get()
File "E:\Anaconda\envs\PPO_drone\lib\site-packages\msgpackrpc\future.py", line 43, in get
raise self._error
msgpackrpc.error.TransportError: Retry connection over the limit

I would be grateful if anyone could tell me how to fix this.

TestEnv

Do we need to rewrite the env_config ["TrainEnV"] in the policy run. py file we run during testing to env_config ["TestEnV"]? I didn't see TestEnv in config.yml, do I need to redefine it myself? Do I need to correspond to the holes in the UE environment in TestEnv.zip?

bilalkabas / ppo-based-autonomous-navigation-for-quadcopters Goto Github PK