Option hiddenLayers. Question about readme about brain.js HOT 21 CLOSED

And one example more. Just for history.

I have training data with 292 different columns, and by default hiddenLayers equal to [145]. And come out very large nn (as json) and very slow training.
But if I set custom hiddenLayer like as [3, 3] that's make nn better and less (~5Mb vs 50Kb).

So strange =)

from brain.js.

The first citation you reference is from:

if (!hiddenSizes) {
      sizes.push(Math.max(3, Math.floor(inputSize / 2)));
    } else {
      hiddenSizes.forEach(size => {
        sizes.push(size);
      });
    }

This context would be when hiddenSizes aren't defined, as you mentioned a default value, otherwise it uses whatever you give it for hiddenSizes.

The question: "what size should my hidden layers be?!?!" is always one I see tossed around a lot, and it seems there is a lot of debate on what makes it a better size. I tend to usually use defaults, and try and tweak but have never really found an optimum set of sizes.

I do know that it is easier for the net to go deeper (ie deep learning which is 3 or more) hidden layers than wider based off of research, and that sounds like it would make sense, as there are more layers to abstract patterns into, rather than the other than flattening them immensely. If you were to go even deeper, say [3, 3, 3] or [145, 50, 3] the same rule applies. Deeper is better than wider. What does your output look like?

from brain.js.

For reference and credit: http://stats.stackexchange.com/questions/181/how-to-choose-the-number-of-hidden-layers-and-nodes-in-a-feedforward-neural-netw

from brain.js.

After when I wote this, i divided training data to train set and test set (90% - 170 samples and 10% - 20 samples). And what I got:
For hiddenLayers [200] I got accuracy 65% on test data set and 0.002 training error.
For hiddenLayers [50] I got acuracy 60% on test data set and 0.001 training error.

Deeper is better than wider.

Okey, I will testing with 2- and 3- count of hiddenLayers array.
In some sources I readed what neuro networks with more that 3 hidden layers not effective.

Some info about my data, if it interesting for you.
I have some commands and their effecienty as from 0 to 0.5 is bad effective and from 0.5 to 1.0 is good effective.
For example, my data look as:

{
    input:  {
        "m_id_58374": 1, // leader id, he is main participiant, always 1
        "e_id_254123": 0.5, // member of group, he involved in project 50%
        "e_id_914260": 0.25,
        "e_id_535412": 0.2,
        "e_id_530421": 0.04,
        "e_id_228592": 0.01
    },
    output: [0.29] // points of result for this command, mean bad result as 42% ((0.5 - 0.29)*2*100)
}

And nn config:

var net = new brain.NeuralNetwork({
	hiddenLayers: [200] // here I past different values for experiments
});
var trainResult = net.train(trainingData, {
	errorThresh: 0.001,
	iterations: 50000,
	learningRate: 0.2,
	
	callbackPeriod: 1000,
	callback: function(state) {
		// run on testing data set
	}
});

from brain.js.

And about errorThresh. In process of train():
err: 0.087; accuracy on test data: 79%;
err: 0.004; accuracy on test data: 67%;
err: 0.003; accuracy on test data: 65%;
err: 0.002; accuracy on test data: 62%;
err: 0.001; accuracy on test data: 59%;

@robertleeplummerjr wat? NN very strong adapted for training data?
Is it overfitting?

from brain.js.

Can you ask that question a different way? I apologize but I don't understand.

from brain.js.

Yep, my english very good :) Sorry!

In config of net I set errorThresh: 0.001 and see train log from callback:

state.error: 0.087; accuracy on test data: 79%
state.error: 0.004; accuracy on test data: 67%
state.error: 0.003; accuracy on test data: 65%
state.error: 0.002; accuracy on test data: 62%
state.error: 0.001; accuracy on test data: 59%

As you can see with a decrease in error is reduced and accuracy in the test sample.
And my question - is it overfitting? Do I need increase errorThreshold in this case?

P.S. At this moment I set 0.0025 and this look better.

from brain.js.

How long are you training for?

from brain.js.

How long are you training for?

For example,
for hiddenLayers [20,20,5] 73 sec. and 2800 iterations
for hiddenLayers [120,20,5] 840 sec. and 5400 iterations

And yes, I use cycle for get better hiddenLayers param =)

for (var iLayer_3 = 5; iLayer_3 <= 30; iLayer_3+=5) {
	for (var iLayer_2 = 10; iLayer_2 <= 100; iLayer_2+=10) {
		for (var iLayer_1 = 20; iLayer_1 <= 200; iLayer_1+=20) {

from brain.js.

On the issue

Option hiddenLayers. Ask about readme

I wanted to write "question" instead "ask".
And my question about readme. It is not written about dividing input data size and it's confusing. Is not it?

from brain.js.

How much data are you training with?

from brain.js.

167 samples same as I wrote
19 samples for test

from brain.js.

Any way you could increase the data into a few thousand samples?

from brain.js.

I don't think. It's real data from database.
Maximum what I can – to attempt get error about 70..75% on test data. It's norm for my issue.

from brain.js.

Results of my experiment with double layer.
Net trained to 0.0025 error on each param hiddenLayers.

I wrote just best results with accuracy on test data over 66%:
(columns - count neurons in 1 layer, row - count neurons in 2 layer)

I think what you right about [145, 50, 5], I will continue experiments around these numbers. Thanks!

from brain.js.

This is really fascinating! It would be nice to arrive at some formula to understand the optimum size to go with. Too, keep in mind, it will take much longer to train, but you can go deeper than 3 layers.

from brain.js.

This is really fascinating! It would be nice to arrive at some formula to understand the optimum size to go with.

I think, these parameters depend havily on type train dataset.

Too, keep in mind, it will take much longer to train, but you can go deeper than 3 layers.

Could not get a good result as at 4-layers network. And it's really slow.
At this moment I have best result with hiddenLayers = [180,30,5] and 70.13% of accuracy.

I'll keep you informed :)

from brain.js.

It would be nice to arrive at some formula to understand the optimum size to go with.

About optimization, what I found.
I use callback for check accuracy on test data and... training data, and what I got – on state.error = 0.01 accuracy on training data is around 90%, it's enough result for my task and it does not make sense to continue. It's saves a lot of time for train(). Because usual I set errorThresh in range 0.001..0.0025, now I set errorThresh to 0.01.
Success story! :)

from brain.js.

what did your layers look like in the end?

from brain.js.

While in the process.
Now I experimenting with hidden Layer [X, Y, 5].
And I think what in a future set errorThresh even more... because in two direction - accuracy on test data and training data train() make these cross.
For example train() for some net:

I think what exist some point when these two directions cross and it will be most better result for my dataset.

P.S. I found point of intersection these directions. For my dataset this point is 80% and require errorThreshold is 0.05. It's mean that when error of training is 0.05 accuracy on training and test data become around 80%. And if I will continue training for less error than accuracy on training data will grow, but accuracy on test data will less.

from brain.js.

fantastic tracking of results!

from brain.js.