The result becomes 0 at the end of second epoch when I pretrain a model with the MLM objective for Mongolian and Chinese about xlm HOT 4 CLOSED

Mmm the model diverged, I guess this may be because of your optimizer. Maybe try something like this to use a learning rate linear warmup + decay (it's usually good for transformer training):

optimizer = adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001,weight_decay=0

Basically the learning rate was too big in your experiment. attention_dropout = 0.2 is big, maybe try just 0 or 0.1 instead.
Also, batch size 16 is small, the bigger the more stable the model will be. In practice we use up to 32 GPU to have very large batch size.

from xlm.

Thank you for these details. And I will try again based on your suggestion. The batch size 16 is the biggest value that fits in my computer‘s ’memory.

from xlm.

@glample
Which part stands for warmup?

--optimizer adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001,weight_decay=0

from xlm.

Nothing in this command. It will use default warmup here. Otherwise you can use:

--optimizer adam_inverse_sqrt,lr=0.00020,warmup_updates=30000,beta1=0.9,beta2=0.999,weight_decay=0.01,eps=0.000001

and warmup_updates is the number of warmup steps.

from xlm.