dubezoniner / deep-qlearning-demo-csharp Goto Github PK

This is the neural network:

int quantityShootingInputs = 8; int quantityShootingActions = 4; int networkShootingSize = quantityMovementInputs * temporalWindow + quantityMovementActions * temporalWindow + quantityMovementInputs; List<LayerDefinition> shootingLayers = new List<LayerDefinition>(); shootingLayers.Add(new LayerDefinition { type = "input", out_sx = 1, out_sy = 1, out_depth = networkShootingSize }); shootingLayers.Add(new LayerDefinition { type = "fc", num_neurons = 70, activation = "relu" }); shootingLayers.Add(new LayerDefinition { type = "fc", num_neurons = 70, activation = "relu" }); shootingLayers.Add(new LayerDefinition { type = "fc", num_neurons = 70, activation = "relu" }); shootingLayers.Add(new LayerDefinition { type = "regression", num_neurons = quantityShootingActions }); // training configuration TrainingOptions trainingOptions = new TrainingOptions(); trainingOptions.temporal_window = temporalWindow; trainingOptions.experience_size = 30000; trainingOptions.start_learn_threshold = 1000; trainingOptions.gamma = 0.7; trainingOptions.learning_steps_total = 100000; trainingOptions.learning_steps_burnin = 3000; trainingOptions.epsilon_min = 0.05; trainingOptions.epsilon_test_time = 0.00; trainingOptions.layer_defs = shootingLayers; trainingOptions.options = new Options { method = "adadelta", l2_decay = 0.001, batch_size = 10 }; neuralNetwork = new DeepQLearn(quantityInputs, quantityActions, trainingOptions);

and this is the error:

IndexOutOfRangeException: Array index is out of range. ConvnetSharp.FullyConnectedLayer.forward (ConvnetSharp.Volume V, Boolean is_training) (at Assets/Scripts/Player/NeuralEngines/BySensors/Layers/FullyConnectedLayer.cs:47) ConvnetSharp.Net.forward (ConvnetSharp.Volume V, Boolean is_training) (at Assets/Scripts/Player/NeuralEngines/BySensors/Utilities/Net.cs:130) DeepQLearning.DRLAgent.DeepQLearn.policy (System.Double[] s) (at Assets/Scripts/Player/NeuralEngines/BySensors/Q-Learning/DeepQLearn.cs:241) DeepQLearning.DRLAgent.DeepQLearn.backward (Double reward) (at Assets/Scripts/Player/NeuralEngines/BySensors/Q-Learning/DeepQLearn.cs:381) SensorAgent.Backward (Double reward) (at Assets/Scripts/Player/NeuralEngines/BySensors/SensorAgent.cs:79) NeuralBySensorPlayer.Update () (at Assets/Scripts/Player/NeuralBySensorPlayer.cs:115)

The crash there is when start the learning and if I decrease the temporal window at 1 it works!

I trained the agent overnight on fast computer, resulted in these stats:
experience replay size: 30000
exploration epsilon: 0
age: 3662244
average Q-learning loss: 0.0158955232113
smooth-ish reward: 0.870696559286462

The result is the bot is spinning in place in the wall corner.
This is not working as it should.
Not sure if it's the bot of the DQN itself :(