About loss in training_losses_seq2seq() when time step t=0 about diffuseq HOT 5 OPEN

Hi,
the q_sample() can work for t=0, which returns x_start. The model learns the mse loss between $x_0$ and $Emb(w^x)$ here.

from diffuseq.

Sorry, but I still don't understand why q_sample() returns x_start when t=0.

The param x_start here equals to x_start_mean + self.sqrt_one_minus_alphas_cumprod[0] * noise (according to self._get_x_start()). So it stands for $x_0 = Emb(w) + \beta_0\times noise$ in your paper.

Now dig into the q_sample() function.
The return variable x_t equals to self.sqrt_alphas_cumprod[0] * x_start + self.sqrt_one_minus_alphas_cumprod[0] * noise. So it stands for $x_t = \alpha_0 x_0 + \beta_0\times noise$. I don't know what the return value x_t stands for.

from diffuseq.

Hi,
Sorry for the ambiguity, let me elucidate. For canonical noise scheduler, $\beta_0 \rightarrow 0$, so it returns $x_0$. However, we use the sqrt noise scheduler, where $\beta_0=0.121$ when $T=2000$. The model learns the mse loss between $\alpha_0 x_0 + \beta_0 * noise$ and $Emb(w^x) + \beta_0 * noise$, so the statement above still holds.

from diffuseq.

Sorry, but I cannot deduce your conclusion from your code.

The model learns the mse loss between α0x0+β0∗noise and Emb(wx)+β0∗noise

Please give me a hint based on some code links. Again, thank you for your patience!

from diffuseq.

Hi! I got a similar question regarding the quoted lines.

It seems to me that model_out_x_start=model_output since we are not predicting the noise term. In addition, we are not using pred_xprev. In this case, why we bother to use the helper function _x0_helper?

Thanks!

from diffuseq.