On the calculation of Xt partial loss in Zt about diffuseq HOT 6 CLOSED

Hi,
It is true that the $x_t$ part is 0, referring to Eq.(2) in the paper. There is still $y_t$ part for the mse loss when training.

from diffuseq.

Hi, It is true that the xt part is 0, referring to Eq.(2) in the paper. There is still yt part for the mse loss when training.

1. Yes, what you mainly train is the loss in yt, but the X part entered into the mean_flat function is 0, and the function will also count the number of X, resulting in a smaller loss. the constant Xt shares part of Yt loss. For example, Zt = [0,0,0,2,8], where 0 is the part of MSE_X, it actually calculates the average of (2+8)/5 (including the number of X's and Y's) instead of calculating the average of (2+8)/2 (the number of Y's).
2. What does the Embedding part of Transformer learn in training? learn how to embed text(X) in the tail of T = 2000？
3. Is the diffusion model in NLP suitable only for large data, but not for small data?
4. You are the first to use the seq2seq task for the diffusion model, I would like to ask if Transformer and training methods are similar to DiffLM?

Thank you for such beautiful code！

from diffuseq.

Hi,

1. You're right, it is more rigorous. For the whole sequence, the total length is fixed to 128, and the length of x would not vary much, so I think this loss is bearable.
2. The embedding learns the word embedding of each token, just the same as other pretrain language models. The word embedding here also corresponds to the semantic meaning of the token, e.g. similar words have closer vector cosine scores.
3. There is no doubt that larger data helps better language modeling capacity, but I am not sure how DiffuSeq works on a small set of data.
4. The transformer architecture (layers, hidden dims, etc) and the training methods (vlb loss, optimizer, learning rate annealing) are similar to Diffusion-LM.

from diffuseq.

Hi,

1. You're right, it is more rigorous. For the whole sequence, the total length is fixed to 128, and the length of x would not vary much, so I think this loss is bearable.
2. The embedding learns the word embedding of each token, just the same as other pretrain language models. The word embedding here also corresponds to the semantic meaning of the token, e.g. similar words have closer vector cosine scores.
3. There is no doubt that larger data helps better language modeling capacity, but I am not sure how DiffuSeq works on a small set of data.
4. The transformer architecture (layers, hidden dims, etc) and the training methods (vlb loss, optimizer, learning rate annealing) are similar to Diffusion-LM.

Thank you for your answer. For the second Embedding section. There is a Embedding(EMB) for mapping W to X0 by consulting the source code, but there is also a word_embedding in the Bert model itself, which shares the weight with lm_head. What's the difference? By referring to your code you will map X to Xt using word_embedding in Bert to implement the sample when sampling. So what role does EMB(W) play in the sampling process during training?

from diffuseq.

Hi,
I guess your "Embedding(W)" refers to randomly initialized:

In fact, we didn't use it in the end, both for training and sampling. This code is the legacy of the previous work and we leave it as a possible interface that you can initialize word embedding using your own weights (e.g. Glove embedding). We only use model.word_embedding.

from diffuseq.

Hi, I guess your "Embedding(W)" refers to randomly initialized:

DiffuSeq/basic_utils.py

Line 73 in bdc8f0a

model = torch.nn.Embedding(tokenizer.vocab_size, args.hidden_dim)

In fact, we didn't use it in the end, both for training and sampling. This code is the legacy of the previous work and we leave it as a possible interface that you can initialize word embedding using your own weights (e.g. Glove embedding). We only use model.word_embedding.

Thank you for your patience,
Good luck!

from diffuseq.