ValueError: Input contains NaN about sl-dml HOT 6 CLOSED

Hi, I'm the author of PyTorch Metric Learning. I wandered in here after I noticed this repo in my dependencies list.

Not sure if this is the cause, but FYI if you have pytorch-metric-learning >= v0.9.90 and pytorch < 1.6.0, the loss will be NaN. This is due to a bug in pytorch that was only fixed in 1.6.0. So if you can't use pytorch 1.6.0, you should use pytorch-metric-learning <= v0.9.89.

from sl-dml.

Hi, Kevin. Thanks for your kindness and a great package!

Since I use Pytorch ==1.5.0, I downgrade pytorch-metric-learning to v.0.9.89 and it works well.
Thanks for your help!

from sl-dml.

Great to hear, that you managed to start training now. Just to make sure. You are using the most recent version from github?
What exact command did you use? I try to reproduce the error then.
Further, the most valuable information from the error is cut. I just see that train.py throws an error. I assume that the loss is overloading for a reason.

You can try this command, to see if something happens to the metric loss:

python train.py dataset=ntu_swap_axis loss=only_classifier

You could also check if the error happens with one of the other datasets , looks like you are using the NTU 120 one shot experiments. E.g. with the Simitate experiment as described here.

from sl-dml.

Yes, I downloaded the newest version from GitHub.
I created env with requirements.txt and cudatoolkit==10.1.
I tried python train.py dataset=ntu_swap_axis and encountered the error.

I tried python train.py dataset=ntu_swap_axis loss=only_classifier with one epoch and it works well. (total_loss is calculated.)

from sl-dml.

@KevinMusgrave, first thanks for actively joining this issue. I will missuse the issue to also thank you for the great pytorch-metric-learning library, which I think is a great contribution towards more reproducible and comparable metric learning approaches. I also recently read your "A metric learning reality check" paper [1] which pointed me out to a great set of hyperparameters and flaws in the training and evaluation strategy of the first preprint. I tried my best to follow your suggestions where applicable to which an updated preprint will follow.

[1]: Musgrave, Kevin, Serge Belongie, and Ser-Nam Lim. "A metric learning reality check." arXiv preprint arXiv:2003.08505 (2020).

from sl-dml.

Thanks for the kind words 😄

from sl-dml.