denoising_diffusion_pytorch.py doesn't use eval mode for testing about denoising-diffusion-pytorch HOT 4 CLOSED

@lucidrains I've just finished looking over the EMA library. So, if I got it right, this is what happens:

• During Trainer.__init__, you initialize the EMA. This creates a deep copy of the model which is then passed to the FidEvaluator
• At testing time, you copy the parameters to the model stored in the EMA
• You call .eval() for the FidEvaluator model (which is actually the EMA model), which means that the original model isn't affected and remains in training mode

If this is correct, I'll go ahead and close the issue, If there isn't any downside, I'll also probably edit the script so that it calls .eval() and .train() manually (I need to compute some other metrics besides the FID).

Out of curiosity, what is the rationale for keeping a deep-copied EMA model and manually updating the parameters? Is it for the sake of thread safety / accelerator black magic?

from denoising-diffusion-pytorch.

@samuelemarro hey Samuele, thanks for raising this issue

i think this is taken care of here? but i just realized it may not set it back to train mode, which may be necessary given the dropouts just added

let me quickly do that

from denoising-diffusion-pytorch.

@samuelemarro oh actually, it is fine, since the FidEvaluator is working off the EMA model

from denoising-diffusion-pytorch.

@samuelemarro i'm not sure where the practice of keeping an EMA of generator came from, but i first encountered it in StyleGAN2 from Tero Karras years ago. ever since, many papers use this technique and it has become mainstream

the rationale is that the smoothed parameter updates yield a model with better quality

from denoising-diffusion-pytorch.