Error when using multi-GPU for training MT only about xlm HOT 5 CLOSED

Ah yeah. Sorry about that, I definitely need to deal with the distributed error because this is not convenient. For now can you try to put: strict=False in the load_state_dict parameters? It is an option to only reload the available parameters, i.e. everything apart from the pred_layer projection.

from xlm.

I believe this error happens on recent version of PyTorch Distributed when some parameters (but not all) are updated. Can you try to set with_output=False in the encoder construction here? https://github.com/facebookresearch/XLM/blob/master/src/model/__init__.py#L127 When you do MT and not language modeling, the encoder does not need an output layer.

from xlm.

If I set with_output=False then such an error occurs:

from xlm.

I also met this problem as I upgraded PyTorch to 1.0.1.post2 version. It is regular when i use 0.4.1.

from xlm.

Sorry for no respond. When I change strict to False it's working :)

from xlm.