Embeddings for each subword in a sentence about xlm HOT 2 CLOSED

Hi,

thanks for your message! :)
It would be indeed very useful to get the full set of parameters in the "forward" of the notebook. We'll work on this. We also plan to incorporate some visualizations of the attentions and the parameters in the notebook. In the meantime, a hacky way to get these embeddings is to modify this part of the code:

https://github.com/facebookresearch/XLM/blob/master/src/model/transformer.py#L368-L404

so that it returns "self.embeddings(x)" after line 368.

Hope this helps! :)
Alexis

from xlm.

Yes, copying this part of the code is probably the easiest way: https://github.com/facebookresearch/XLM/blob/master/src/model/transformer.py#L367-L374

Just adding a new cell into the notebook with:

# embeddings
tensor = model.embeddings(word_ids)
tensor = tensor + model.position_embeddings(positions).expand_as(tensor)
if langs is not None:
    tensor = tensor + model.lang_embeddings(langs)
tensor = model.layer_norm_emb(tensor)
tensor = F.dropout(tensor, p=model.dropout, training=model.training)
tensor *= mask.unsqueeze(-1).to(tensor.dtype)

should do the trick.

from xlm.