Hi @andrew-cr

I tried to reproduce the training on cifar-1o and obtained the following losses.

It seems to me a little weird that the neg_elbo term is sooo large compared to the NLL terms. Am I wrong? FYI, I changed the batch size to 40 to save memory consumption. Do you guys also obtain similar values in the initial training stage?

NLL: 8.108, neg_elbo: 12571671.000

Best,
Zhangzhi

Hello,

I am currently exploring the implementation details of the tauLDR model and have a question regarding the detailed balance condition as implemented in the code. Specifically, I am referring to the line in models.py:

rate[i, j] = rate[j, i] * np.exp(- ( (j+1)**2 - (i+1)**2 + S*(i+1) - S*(j+1) ) / (2 * self.Q_sigma**2) )

My question pertains to the addition of +1 in each term of the numerator within the exponential function. Upon reviewing Appendix E of the associated paper, I did not find a mention or requirement for adding 1 to the indices i and j in the detailed balance condition.

Could you please provide some insight into the rationale behind this addition? Is it related to a zero-indexing adjustment for Python, or is there another theoretical justification or correction applied here that might not be immediately apparent from the paper's description?

Understanding the reasoning behind this discrepancy will greatly aid in my comprehension of the model's implementation and ensure alignment with the theoretical foundations laid out in the paper.

Thank you for your time and for your contributions to this fascinating project.

Hi,

Experimenting myself with the PC scheme, I was wondering if there is any rational explanation behind the choice of the corrector_step_size_multiplier parameter ?

Do have any tips to tune it or have you simply swept over different values for each experiment ?

Best regards,

Antoine

Hi Andrew!

Thank you so much for this great work! We are trying to better understand the continuous time discrete diffusion framework and the evaluations reported in the paper. In Table 1, an ELBO of -3.59 is reported on the test set in bits per dimension for CIFAR10. However, as mentioned in #4, the neg_elbo on CIFAR is on the scale of 1e7 due to many dropped constants, and even after scaling by the image dimensions 32 x 32 x 3 is not at the right scale. The nll term does look to be on the right scale. We are wondering whether the reported ELBO in the paper is actually just the nll term averaged over the test set (hence over all time steps), or there is an implementation of the actual ELBO somewhere?

Thank you so much!

Best,
Bear

Thank you very much for the great work and being advocate of open source! Would you please provide the license?

Hi,
Thank you for the great work!

Any guidelines for training/testing on a different dataset -aside CIFAR and piano dataset?

Training on a different dataset would most likely involve chaning the data path in config file,
any other adjustments to also keep in mind while also making these changes?

Thank you!

Hi,

Great work and nice implementation !

I have a question on the way you choose the dimension of transition during training (code below) :

    rate_vals_square = rate[
        torch.arange(B, device=device).repeat_interleave(D),
        x_t.long().flatten(),
        :
    ] # (B*D, S)

    rate_vals_square[
        torch.arange(B*D, device=device),
        x_t.long().flatten()
    ] = 0.0 # 0 the diagonals

    rate_vals_square = rate_vals_square.view(B, D, S)

    rate_vals_square_dimsum = torch.sum(rate_vals_square, dim=2).view(B, D)

    square_dimcat = torch.distributions.categorical.Categorical(
        rate_vals_square_dimsum
    )

    square_dims = square_dimcat.sample() # (B,) taking values in [0, D)

What I understand is that you sample the dimension of transition from the distribution of the total outgoing rates of x components. Even though it seems pretty intuitive to me, I don't see how this is justified from a theoretical perspective, and can't find any clear explanation in your paper.

With that in mind, could you please elaborate on how this implementation can be related to your model ?

Thanks,

Antoine