liu-qingzhen / minitaur-pybullet-td3-reinforcement-learning Goto Github PK

This is the project of NUS ME5406

If this documentation is not clear enough, then you can watch the 5 videos in the

VIDEOS/READMEVIDEOS folder. If it can't play, you can watch them on Youtube,too.

https://www.youtube.com/playlist?list=PLnCTl6QmZINAu1SZ4EpS-6Tk7wW9zoPKU

I recorded in the project folder that introduces how to check my project. Thanks!

The first part would be the introduction of files in my project folder

Then there will be preparations before running code

Then we can run the code

Finally, we can check our results with tensorboard

In this folder, you may see:

1 SOURCE_CODE Folder: in which there are

5 .py files.
	5406TD3.py is the file for training agent with TD3 algorithm.

	test_NetworkI.py 
		This is the file for testing the result of the training with network structure I in my report. Trained model I,II,II,IV are based on this structure.

	test_NetworkI.py 
		This is the file for testing the result of the training with network structure II in my report. Trained model V is based on this structure.

	ppo_stablelines.py 
		This is to test PPO's performance with Stable Baselines

	minitaur_gym_env.py 
		This is the modified environment file
		(we should replace the environment file in the pybullet with this one
		the location of the original pybullet env file is:
		'your conda env nave'/pybullet_envs/bullet/minitaur_gym_env.py)


3 .txt files
	pip_requirements.txt it's the dependencies for running the project
	(use pip install -r pip_requirements.txt to install the packages;
	there seems to be some problem with the requirement list generated by conda)

	conda_requirements.txt
		requirements generated by conda

	Training_Records_DDPG.txt it's a unsuccessful training with DDPG


6 Folders
	Trained_model_I 
	Trained_model_II 
	Trained_model_III 
	Trained_model_IV
	Trained_model_V
		each has 6 saved parameters file for each network + one folder of tensorboard
		trainig records + the environment file for this training
	runs
		tensorboard will save its files here

2 VIDEOS: in which there are

1. final_video.mp4
	This is the training video of my model 
	
1. raibert_controller folder
	in which are the videos of raibert controller of different speed


1. README_VIDEOS
	in which are the videos showing how to test and train my model in case the program doesn't work

3 ME5406_PrjII_LiuQingzhen.pdf This is my report

Preparations:

1. open terminal and cd to the SOURCE_CODE directory

2. create an conda environment: conda create -n 'env name' python =3.7

3. install packages: pip install -r pip_requirements.txt

4. It's recommended to replace the minitaur_gym_env.py in pybullet with the environment because the environment will view the robot as fallen based on its location. If it is too troublesome, then just run with the default environment. However, the score can be low for many episodes and there is no rendering too.

   minitaur_gym_env.py is located in the SOURCE_CODE folder this .py file is my modified environment (we should replace the environment file in the pybullet with this one the location of the original pybullet env file is: anaconda3/'your conda env name'/pybullet_envs/bullet/minitaur_gym_env.py)

Then:

1. How to train your model?

   😊(It's recommended to use cd in terminal to this directory first, or the os.getcwd() may return your current directory, so the model will not be saved in the project directory.)😊 MacOS seems to have no such problem, but Linux has.

   next, type: python Train_TD3.py in terminal OR run with pycharm or jupyter notebook

   (Not neccesary) you can choose to use GPU and change the folder name to save model by:

    python --GPU True  (by default is with cpu) 
    	 		--name "the name you like"  ( byd efault is 5406_Model)
   

   if there is still someting wrong with the path, it is recommended to replace the folder path with your absolute path.

   then it will start to train. By default, the environment will not render If you didn't change the environment file and you'd like to see the rendered environment You should go to the pybullet_envs/bullet/MinitaurBulletEnv to change IsRender=True

2. How to test the trained model?

   open a terminal: python test_networkI.py to use trained model from I-IV to see the training result change the load_path to see result of different trained model load_path = os.path.join(os.getcwd(),'Trained_model_I','actor_td3') load_path = os.path.join(os.getcwd(),'Trained_model_II','actor_td3') load_path = os.path.join(os.getcwd(),'Trained_model_III','actor_td3') load_path = os.path.join(os.getcwd(),'Trained_model_IV','actor_td3')

   python test_networkII.py to use trained model V.

   it should work replace the load_path with the absolute path of the file if there is something wrong with the file directory

3. How to check the training records in tensorboard?

   after running the training code, then it should generate a file named 'runs' then open the terminal tensorboard --logdir="the path of the training folder, not the file" then open the browser at localhost:6006

After training, there should be: a new folder with your trained model (by default the name should be '5406'); a new folder in runs that saved your tensorboard file (Linux may generate the file in HOME folder)