How is XLM different from Multilingual BERT? about xlm HOT 4 CLOSED

Hi @fallingstar621, as mentioned in the paper there are several differences between our implementation and the one of BERT like the fact that we subsample the outputs, we use streams of text, we use BPE instead of sentencepiece etc. For the multilingual part, the multilingual BERT is MLM-only. Our XLM-MLM is thus very similar to the multilingual BERT. For XNLI, we use TLM in addition to MLM which further boosts the accuracy. Hope this helps! Thanks

from xlm.

@yilinyang7 yes, only when the TLM objective is involved. Also, we never use the "next sentence prediction" objective.

from xlm.

Thank you guys for sharing more insights, it is very helpful!

from xlm.

I believe the main difference is that XLM is additionally trained on the parallel corpus.

from xlm.