training encounter with NAN about confusion HOT 9 CLOSED

Hey,

Can you try lowering the value of lambda? Since we experimented mostly with problems of up to 500 classes, lambda was never higher than 50. Perhaps for 3k classes it doesn't need such a high value of lambda. This in turn would probably lead to very large gradients somewhere in the training (after cross-entropy is sufficiently small) that moves the outputs to nan.

I'd say start with lambda=10 and then increase/decrease to see how performance varies. What network are you using and which codebase (caffe or pytorch), and with which optimizer (adam or SGD)?

from confusion.

ok. I would try to decay lambda.
I finetune resnet-50 on pretrained model in caffe.

from confusion.

I increase/decay lambda. Both go into NAN finally after different iterations. I try to add gradient clipping but cause a terrible accuracy.

from confusion.

Can you share your prototxt?

from confusion.

Train_val prototxt is as follows:
train_val.txt

from confusion.

Thanks. Can you also share the prototxt you generate while using confusion loss? Also, does this go to NaN even without the loss?

from confusion.

I have updated the train_val prototxt with confusion loss(the last uploaded prototxt is a mistake ). Without confusion loss, the training process acts normally.

from confusion.

Just took a look at your prototxt. Since I don't have this dataset, I can't test it myself, but I'd give 2 suggestions - i) try with a larger batch-size (32, 64 or 128 if possible), and ii) try with \lambda = 0.01, 0.1, 1 and so on.

from confusion.

For limited gpu memory, I set batch-size to 24 and set iter_size to 4(so batch size is 24 * 4 actually). That lambda is less than 1 is what I have not tried. Thanks!

from confusion.