Thank you very much for coming out with the follow-up work V2, which brings a very important role to diffusion language modeling !I'd like to ask you some questions. Thank you. about diffuseq HOT 3 CLOSED

Hi,

1. Usually the larger the batch_size, the better.
2. You can decide it based on the performance on the validation set.
3. Two aspects to improve training speed: (1) mixed precision training (use --fp16) (2) the learned absorbing state makes the training converge faster.

from diffuseq.

Hi,

1. Usually the larger the batch_size, the better.
2. You can decide it based on the performance on the validation set.
3. Two aspects to improve training speed: (1) mixed precision training (use --fp16) (2) the learned absorbing state makes the training converge faster.

Thank you!

from diffuseq.

Hi,

1. Usually the larger the batch_size, the better.
2. You can decide it based on the performance on the validation set.
3. Two aspects to improve training speed: (1) mixed precision training (use --fp16) (2) the learned absorbing state makes the training converge faster.

Hi author, read your article and code. I have some new doubts which I hope you can answer:

1. You said bigger batch_size is better, why did you just set 425 during your training, A100 80G can allow bigger batch_size

2. Is the loss convergence of the validation set equivalent to the metric convergence of the validation set, e.g., BELU?

3. Add the soft absorption state ,My understanding is that you parameterize the clamp() operation with the trainable parameter mean_embed, and it in your code as follows:
   x_t = ( _extract_into_tensor(self.sqrt_alphas_cumprod, t, x_start.shape) * (x_start - mean_embed[None, None]) + _extract_into_tensor(self.sqrt_one_minus_alphas_cumprod, t, x_start.shape) * noise + mean_embed[None, None] * mask )

   I don't understand how this formula is derived, can you explain?

4. In your paper on the table of BELUs obtained based on training time, I would like to ask if the BELUs are calculated directly using the X0 predicted by the model -> x0 = model(xt,t) , or based on the X0 obtained from the T-step REVERSE sampling ?

Thanks！

from diffuseq.