Wrong normalisation dimension about denoising-diffusion-pytorch HOT 13 CLOSED

@inspirit seems like i forgot a scaling factor too (alpha)

from denoising-diffusion-pytorch.

you are actually doing it different few lines above in training block:

you pack weight before applying normalisation and then bring it back

from denoising-diffusion-pytorch.

yup looks good to me :)

from denoising-diffusion-pytorch.

@inspirit one uncertainty I had was whether the values in the cosine sim attention is also pixel normed, just putting that out there in case you have an opinion

from denoising-diffusion-pytorch.

I'm adopting it to a slightly different use: generator model in adversarial learning
will see ig i can make it learn :)

from denoising-diffusion-pytorch.

@inspirit yea for sure.. improvement to a unet should float all applications, from biomedical imaging to diffusion policies, etc. best of luck

from denoising-diffusion-pytorch.

yup, thats exactly my thoughts

from denoising-diffusion-pytorch.

👋

yes, that's how it is in Algorithm 1. if I am not mistakened?

from denoising-diffusion-pytorch.

yes, it is different:
you are using 'F.normalize(t, dim = dim, eps = eps)' with default dim = -1

while in example you show above they flatten array starting dim 1 and use this last dimension to compute normalisation

from denoising-diffusion-pytorch.

@inspirit 🤦 oh yes, thank you

could you double check the latest commit?

from denoising-diffusion-pytorch.

@inspirit ok, do let me know how the Karras unet fairs! Tero Karras is in a league of his own in AI research, so highly optimistic on his magnitude preserving layers

from denoising-diffusion-pytorch.

Hi! getting back to the weight normalisation part :)
as far as i understand in this line we multiply by 'fan_in' scale factor

thats basically reverting back so whats the idea?

from denoising-diffusion-pytorch.

i see from the paper this is intentional for gradient updates in the optimizer

from denoising-diffusion-pytorch.