cpgeier / santosnet Goto Github PK

View Code? Open in Web Editor NEW

77.0 8.0 23.0 12.23 MB

A simple self driving car in GTAV that uses the Xception deep neural network model with DeepGTAV

Python 100.00%

self-driving-car keras-tensorflow computer-vision gtav

santosnet's Introduction

SantosNet

Simple tools that I used to generate and run a self-driving car model with VPilot and DeepGTAV.

Requirements

DeepGTAV must be installed and VPilot must be in the same folder.
Keras, Tensorflow, Numpy, h5py
A GPU with 6 or more gigabytes of VRAM. A 980ti is barely able to simultaneously run the model and the game.

Files

dataset.py - Uses in-game AI to drive car. Captures screen and saves driving variables in a pickle file
drive_categorical.py - Excecutes a trained model on captured frames
load_and_train.py - Continues training a model on a dataset
model_xception.py - Main training program. Generates a .h5 model file that can be loaded for predictions
pickling.py - A simple program that displays frames from a pickled file
preprocessing.py - Loads batches of frames from a pickled file to be used in training
sample.gif - Demonstration of the ability of a model generated using these programs

Download a sample model here

Model

The Xception V1 model is used to classify different images to different positions of the steering wheel. The steering position is collected as a float between -1 and 1, so I converted it to a positive integer between 0 and 999. This was then simply converted to a one-hot vector for input into the model. This preserves the overall structure of the Xception V1 model because the Imagenet dataset that it was designed for included 1000 classes of images. I tried to use the Xception model for regression, but did not have as good of results. During driving, the model is sent the images and returns a vector which is converted into a value between -1 and 1 to be sent to DeepGTAV for input into the game. To clarify, the model only controls the steering of the car while I control the throttle on the keyboard.

Training

The dataset I generated was around 66 gigabytes after running for 10 hours, so I would advise having plenty of space on a hard drive to store data. Also, I training the model on an AWS p2xlarge instance for around 12 hours, during that time I was only able to go through 1 epoch of the data. However, the model still yeilded great results in running and is able to avoid obstacles and stay between road lines most of the time.

Improvements

If you would like to contribute I have some ideas on how this model could be improved:

Dataset balancing - Most of the steering values will be around 0, with few around -1 and 1, so balancing the dataset would probably improve performance
Fine tune with ImageNet initilization - I initialized with random weights, but its probably best to fine tune the model with pre-trained weights
Better preprocessing

santosnet's People

Contributors

Stargazers

Watchers

santosnet's Issues

Git LFS: (0 of 1 files) 0 B / 262.27 MB .batch response: This repository is over its data quota. Purchase more data packs to restore access.

Hi:
I'd like to git clone the folder and I met the problem as following:

D:\Git LFS\1>git lfs clone https://github.com/cpgeier/SantosNet.git
WARNING: 'git lfs clone' is deprecated and will not be updated
with new flags from 'git clone'
'git clone' has been updated in upstream Git to have comparable
speeds to 'git lfs clone'.
Cloning into 'SantosNet'...
remote: Enumerating objects: 79, done.
remote: Total 79 (delta 0), reused 0 (delta 0), pack-reused 79
Unpacking objects: 100% (79/79), done.
Git LFS: (0 of 1 files) 0 B / 262.27 MB
batch response: This repository is over its data quota. Purchase more data packs to restore access.
error: failed to fetch some objects from 'https://github.com/cpgeier/SantosNet.git/info/lfs'

would you like to purchase more data packs and restore access, thanks very much!

Excepted as: Error when checking input: expected input_1 to have shape (80, 320, 3) but got array with shape (160, 320, 3)

Hey I get this error:
Excepted as: Error when checking input: expected input_1 to have shape (80, 320, 3) but got array with shape (160, 320, 3) do you know why ?

training fails with "Resource exhausted"

Hi,

I've tried getting this to work but got stuck in the training step. No matter how few samples in the dataset, I run out of memory even though this runs on a GTX 1070 with 8GB VRAM.
Was there a change causing it to use more memory? Is there something to tweak?
I know this here is pretty much provided 'as-is' but would be happy about any pointers.

Here is the full error:

2017-10-13 00:34:32.073550: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 149082112 totalling 142.18MiB
2017-10-13 00:34:32.073739: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:696] 5 Chunks of size 198066176 totalling 944.45MiB
2017-10-13 00:34:32.073941: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 198066432 totalling 188.89MiB
2017-10-13 00:34:32.074106: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:696] 1 Chunks of size 396130304 totalling 377.78MiB
2017-10-13 00:34:32.074279: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:700] Sum Total of in-use chunks: 6.35GiB
2017-10-13 00:34:32.074451: I c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:702] Stats:
Limit:                  6814913823
InUse:                  6814913024
MaxInUse:               6814913536
NumAllocs:                    2329
MaxAllocSize:           1624768512

2017-10-13 00:34:32.074716: W c:\l\work\tensorflow-1.1.0\tensorflow\core\common_runtime\bfc_allocator.cc:277] **************************************xx***************************x********************************
2017-10-13 00:34:32.074799: W c:\l\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Resource exhausted: OOM when allocating tensor with shape[728]
Excepted with OOM when allocating tensor with shape[728]
         [[Node: block12_sepconv1_bn/moments/sufficient_statistics/mean_ss = Sum[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false, _device="/job:localhost/replica:0/task:0/gpu:0"](block12_sepconv1_bn/moments/sufficient_statistics/Sub, block12_sepconv1_bn/moments/sufficient_statistics/mean_ss/reduction_indices)]]
         [[Node: block14_sepconv2_bn/moments/sufficient_statistics/Gather/_187 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_9496_block14_sepconv2_bn/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op 'block12_sepconv1_bn/moments/sufficient_statistics/mean_ss', defined at:
  File "model_xception.py", line 226, in <module>
    model = Xception(include_top=True,weights=None)
  File "model_xception.py", line 166, in Xception
    x = BatchNormalization(name=prefix + '_sepconv1_bn')(x)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\keras\engine\topology.py", line 596, in __call__
    output = self.call(inputs, **kwargs)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\keras\layers\normalization.py", line 177, in call
    epsilon=self.epsilon)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\keras\backend\tensorflow_backend.py", line 1650, in normalize_batch_in_training
    shift=None, name=None, keep_dims=False)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\ops\nn_impl.py", line 642, in moments
    y, axes, shift=shift, keep_dims=keep_dims, name=name)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\ops\nn_impl.py", line 564, in sufficient_statistics
    m_ss = math_ops.reduce_sum(m_ss, axes, keep_dims=keep_dims, name="mean_ss")
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1236, in reduce_sum
    name=name)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 2656, in _sum
    keep_dims=keep_dims, name=name)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\windo\AppData\Local\conda\conda\envs\santosnet\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[728]
         [[Node: block12_sepconv1_bn/moments/sufficient_statistics/mean_ss = Sum[T=DT_FLOAT, Tidx=DT_INT32, keep_dims=false, _device="/job:localhost/replica:0/task:0/gpu:0"](block12_sepconv1_bn/moments/sufficient_statistics/Sub, block12_sepconv1_bn/moments/sufficient_statistics/mean_ss/reduction_indices)]]
         [[Node: block14_sepconv2_bn/moments/sufficient_statistics/Gather/_187 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_9496_block14_sepconv2_bn/moments/sufficient_statistics/Gather", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Trouble in receiving throttle, brake and steering

Hi, I use VPilot to receive the messages to from DeepGTAV as you said, but it seems some problems happen when I receive the message. I can get the correct data such as 'frame', 'speed', 'reward' and so on, but the data of 'throttle', 'brake' and 'steering' do not reflect correctly, which is always keeping as 0 or 1. Does the game version or driving mode influence this?

i tried training model on GTX 1050 but it gives error

Excepted with OOM when allocating tensor with shape[32,128,77,157] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[{{node training/Adadelta/gradients/AddN_120-0-TransposeNHWCToNCHW-LayoutOptimizer}} = Transpose[T=DT_FLOAT, Tperm=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](training/Adadelta/gradients/block2_sepconv2_bn/cond/FusedBatchNorm/Switch_grad/cond_grad, PermConstNHWCToNCHW-LayoutOptimizer)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[{{node loss/mul/_1543}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_16371_loss/mul", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

i dont know why that error is coming