Loss stuck, not decreasing about vsepp HOT 2 CLOSED

Thanks for reporting.
Generally, we observed that MAX loss can be harder to optimize. There are ways to reduce the difficulty:

• Train with the SUM loss and make sure it achieves a reasonable performance. This is rather a sanity check for the rest of your code.
• Stage-wise optimization. First, start from pretrained encoders and train only the embedding, then switch to fine-tuning and train end-to-end.
• Tune the batch size. I expect the optimal batch size for MAX loss to be dataset-dependent. If the batch size is too large hard negatives can be outliers. If they are too small, it can take longer to train or the best we end up with is bounded by the SUM loss.
• We have unpublished studies on the effect of a separate negative set. On MSCOCO, it happended that the optimal negative set size was the same as the batch size used (128). Depending on your setting, a large negative set could be cheaper. You can trade-off speed for a larger negative set. Choose a large negative set, compute all embeddings, get rid of activations for memory saving, match your mini-batch to a single example in the negative set, then do another forward pass only for selected pairs.
  Here is a plot for varying the negative set size on MSCOCO.

from vsepp.

Thank you very much for these hints. Actually, I think that the stage-wise optimization is the way to go. If I first optimize using SUM loss and then I resume, after 10 epochs, using MAX loss, the problem disappears and the validation metrics keep increasing smoothly.

However, I will pay attention also to the batch size, as you suggested.
Thanks again

from vsepp.