AdaptiveInstanceNorm2D issues... about funit HOT 12 CLOSED

oh this is unobvious behavior of pytorch, this is why I prefer tensorflow :D
thanks for your time.

from funit.

also I checked running_mean and running_var of AdaptiveInstanceNorm2d during training
they are always static, running_mean is always zero and running_var is always one.

so instead of

out = F.batch_norm(
            x_reshaped, running_mean, running_var, self.weight, self.bias,
            True, self.momentum, self.eps)

you are actually training

x * weight + bias

without any normalization at all!

from funit.

I think more logically first normalize the input and then apply style coded gamma and beta.

from funit.

quote from paper:

For each sample, AdaIN first normalizes the activations of a sample in each channel to have a zeromean and unit variance.

but it does not!

from funit.

@iperov
You missed

x_reshaped = x.contiguous().view(1, b * c, *x.size()[2:])

before

out = F.batch_norm(
x_reshaped, running_mean, running_var, self.weight, self.bias,
True, self.momentum, self.eps)

from funit.

Thanks for the reply.

Nothing is missed.

1. Concatenating batch and channels together you compute mean and var across all batch, it's called batch normalization.
   But in instance normalization, mean and var are computed across channels per each batch.

2. currently running_mean, running_var that you pass in batch_norm are always zero and one. Therefore your adain layer works like x * weight + bias.

3. I made batchnorm works with momentum 0.1 in my keras port. It crashes the model immediately.

from funit.

D acc: 0.5641    G acc: 0.4028
Elapsed time in update: 21.891000
Iteration: 00000006/00100000
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> running_mean
tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:0')
>>> running_var
tensor([1., 1., 1.,  ..., 1., 1., 1.], device='cuda:0')

but

x * weight + bias.

also works.
May be not so good as with normalization :)

from funit.

@iperov

batch_norm computes a mean value and a deviation value per channel. By doing b x c, we are treating each sample's each channel as a channel. There are b*c channel now. Now, the effect of batch norm is equivalent to instance norm.

from funit.

Yes, my mistake, you're right.

But what about

>>> running_mean
tensor([0., 0., 0.,  ..., 0., 0., 0.], device='cuda:0')
>>> running_var
tensor([1., 1., 1.,  ..., 1., 1., 1.], device='cuda:0')

?

from funit.

Adding zero and then multiplying one is equal to doing nothing.

from funit.

F.batch_norm should do

(x - running_mean) / sqrt(running_var+eps)

then compute current mean and var
and add to running_mean and running_var with momentum 0.1
So I guess there is no normalization applied.
am I wrong?

from funit.

torch.nn.functional.batch_norm(input, running_mean, running_var, weight=None, bias=None, training=False, momentum=0.1, eps=1e-05)

Above is from the Pytorch doc. Since we set the training mode to true, running_mean and running var are not used.

out = F.batch_norm(
x_reshaped, running_mean, running_var, self.weight, self.bias,
True, self.momentum, self.eps)

from funit.