SELU weight init about deep-learning-with-cats HOT 3 OPEN

Thanks for raising this point! You're right, it's said in the paper on page 6 (https://arxiv.org/pdf/1706.02515.pdf) that one should use var = 1/n_weights. It's still an attracting point so not initializing correctly can still work but yes this should be modified.

If someone can try it and see if things still work well (using the 1/n_weights first, then .50/n_weights second if that doesn't work well), I'll accept a pull request. Otherwise, I'll try it eventually when I have the time and I'll change it.

from deep-learning-with-cats.

I'm about to try it now. You're right that it says to use Var = 1/n_weights but in the official TF implementation they released they use Var = 1/n_neurons_in : https://github.com/bioinf-jku/SNNs/blob/master/selu.py#L31 , not sure which one will work better but hopefully will have some results to find out 🤔

from deep-learning-with-cats.

Just saw this earlier today: http://cs231n.github.io/neural-networks-2/#init. It shows why use fan_in.

from deep-learning-with-cats.