couldn't match SOTA performance on wmt14 EnDe about xlm HOT 3 CLOSED

Hi,

Yes, unfortunately I also tried, and I have never been able to reproduce fairseq results with XLM on the supervised tasks, there was always a difference of 1 or 2 BLEU. This is a bit annoying, because probably if we could match the supervised results we would also be better in unsupervised / semi-supervised, etc.

I really don't think that the differences in terms of architecture (we have one extra layer norm after the embeddings I believe, and when I compared I didn't use GELU) could explain the difference in BLEU. There are a couple of things that we don't have that fairseq has, such as "smoothed softmax", "average checkpointing", etc. and I think these are more the features we are missing in order to get SOTA results in supervised MT. If you see other features that may explain the differences I can try to implement them and retry on the supervised task.

from xlm.

Thank you for your clarification. I'll keep looking into it, and let you know when I find anything.

from xlm.

Hi @yilinyang7 I wanted to pretrain a language model with MLM objective and use it to train on supervised MT for En-De translation. I am however unable to do so with some error If you did supervised machine translation using the pretrained language model can you please elaborate on the steps to follow to do so. It would be great if you could share the commands with which you tried. Kindly share the progress.
Thanks in advance .

from xlm.