FP 16 Training for mt and bt steps. about xlm HOT 1 CLOSED

Hi,

Yes so the problem is that PyTorch distributed doesn't let you perform updates on a module if only some of the parameters are updated. Now if for instance you have a model M that contains the encoder E and the decoder D, and that your optimizer contains the parameters of M, then you cannot perform a MLM update on the encoder only, as the decoder parameters are not going to be updated.

So what I did is that I used an optimizer for E, and one for D. So now if I train on MLM I update on E only, and if I train on MT I call both the E and D optimizers.

The problem if you have fp16 on top of that is that the apex library requires to have a single optimizer. Or at least I couldn't find a simple way to use fp16 with 2 optimizers. Maybe this could be useful though: https://github.com/NVIDIA/apex/tree/master/apex/amp#multiple-optimizers-or-backward-passes

So currently, what I suggest is either

1. Modify the code a bit to have a single optimizer for both E and D. It is probably the simplest solution, but then you will not be able to train the encoder if you don't train the decoder, so you will need to have MT / BT steps only, no language modeling training steps on the encoder.
   or
2. Adapt the code to support 2 optimizers with apex. Probably not that hard, I would have to check.

from xlm.