NLL for q0-q2 is 0 but for q3 is >2 about diffuseq HOT 5 OPEN

Hi,
I didn't encounter such a situation before. Which datasets did you use? It seems that $x_t$ with a larger sampled $t$ didn't get sufficient training.

from diffuseq.

Thanks! I am using a protein sequence dataset with 100K training sequences and 6K diffusion steps however the same change of metrics happen when I use 2K.
Do you mean with $x_t$ with a larger sampled $t$ didn't get sufficient training that the last 25% of the denoising steps are not yet trained properly however the previous denoising steps are already "done"?
If yes, can I influence that without changing the previous denoising steps because I also noticed that the q2 nll goes up again after 1K iterations, whereas the q0/1 nll stays the same and the q3 is still gradually going down.

from diffuseq.

Do you mean with $x_t$ with a larger sampled $t$ didn't get sufficient training_ that the last 25% of the denoising steps are not yet trained properly however the previous denoising steps are already "done"?

Yes. This pattern is not observed in the text sequence, not sure about the protein sequence. Maybe you can edit the time sampler to add more weight on q2 steps.

from diffuseq.

Thanks again :)
Two follow ups to your answer:

sampler to add more weight on q2 steps.

1. Do you mean to adjust the fixed sampler so that t's that are in q2 are weighted higher?

2. What is the expected behavior of the the different metrics for different qn's? I am currently running DiffuSeq on a reduced Conversation dataset and there, all metrics of a type are in the same ballpark for each qn. Is that also what you have observed?

from diffuseq.

Respond to 1: Yes
Respond to 2: Yes, the specific number of q_n is computed after re-weighting.

from diffuseq.