Finish validation without running model.train() about vsepp HOT 5 CLOSED

Based on the reruns, the behaviour is affected noticeably by enabling the training mode. I reverted the change but added a flag to enable training mode. To make the default behaviour reproducing the results, by default the flag is disabled.

Regarding why the results are affected, we only have guesses. Results for VGG19 that only has dropout is affected more compared to Resnet with only batchnorm. In general, dropout is not necessarily needed for all models. Especially, in learning joint embeddings it might make the optimization more difficult. Maybe the amount of data matters. In any case, we will investigate further.

Thanks again for reporting.

from vsepp.

Revisiting this issue, the test loss seems to be lower for when the training mode is reset correctly compared to when it's only reset at the beginning of the epoch. There is a difference in the performance as measured by R@K but it's not a better or worse solution considering all K's. The master branch stays on the PyTorch 0.2 compatible code to keep the results reproducible. PyTorch >0.4 forces us to fix this issue, so there is a new branch for compatibility with PyTorch 0.4.

from vsepp.

Good catch. Thank you. I'm checking if it has any effect on the results but I don't expect it to have much. The image models used are pre-trained and kept fixed unless finetuning is enabled. So batch norm should be mostly fine. But dropout could help by regularizing the model.

from vsepp.

Hi,
I also found this problem and I found that in my experiment, fixing this bug will cause the performance drop when finetuning CNN is enabled, which is strange....

from vsepp.

Thanks @xh-liu for reporting. I'm reopening this issue as I'm observing non-negligible changes in the results.

from vsepp.