NaNs during CycleGAN training about pytorch-cyclegan-and-pix2pix HOT 4 CLOSED

Hi revilokeb,

I believe it is due to repeatedly applying normalization with images of low variance.

When applying normalization like InstanceNorm, the gradients tend to blow up fast if the image has low variance. This becomes even worse because we are going through multiple normalizations in the deep network. This problem is more frequent in the CycleGAN architecture because we use InstanceNorm and deep network with many normalization layers.

I personally ran into this issue when one image in the dataset was uniformly black due to corrupt image. I think this problem can be alleviated by increasing the value of epsilon, or removing the few images that cause the problem.

from pytorch-cyclegan-and-pix2pix.

I am running into the same problem.

So are you saying that images with low-variance may lead to exploding gradients? Could some type of gradient clipping be used to avoid that?

I've been trying to debug this, but it is hard to isolate the problem =/

from pytorch-cyclegan-and-pix2pix.

The problem was solved by upgrading the version of torch.

from pytorch-cyclegan-and-pix2pix.

which version of torch should I choose? The default is torch1.4

from pytorch-cyclegan-and-pix2pix.