Likely typo error in the algorithm as shown in the paper about sgd-feedback HOT 8 CLOSED

I reran the CIFAR100 experiment ONCE for each optimizer (cf. https://github.com/tdeboissiere/DeepLearningImplementations/tree/master/Eve) and this time, Eve outperforms the other 2.

from sgd-feedback.

Yup! You're right - it's a typo in the paper. Thank you very much for pointing this out - we'll rectify this as soon as possible.

from sgd-feedback.

By the way, what were the best parameters (learning rate, decay) for Eve on CIFAR 100 ?

I am unable to reproduce your results on this dataset.

from sgd-feedback.

Hi @tdeboissiere: for that experiment the best parameters for Eve are learning rate 0.0001 and decay 0.0001. What results did you get?

from sgd-feedback.

Eve actually did worse than Adam and SGD. My best guess is that without a lower learning rate + annealing, training is quite hard. I also had to switch on BatchNorm to get any optimizer to work.

I tried both you implementation and mine with much the same results.

What were the best parameters for SGD and Adam ?

from sgd-feedback.

Did you try the learning rates I mentioned?

The best parameters for SGD, Adam:
SGD Nesterov: lr 0.01, decay 0.0001
Adam: lr 0.0001, decay - turns out to not have been saved because of a previous bug in Keras. But I would imagine it's the same as for Eve: 0.0001. I will try to run the cross-validation again when I can

from sgd-feedback.

I am going to run the experiments on CIFAR100 with the best parameters you found. Did you use BatchNorm in the end ?

from sgd-feedback.

No, I didn't. The exact model is in src/models.py (get_big_cnn_model)

from sgd-feedback.