tanh 150 constant - why? about fast-neural-style HOT 6 OPEN

https://github.com/gordicaleksa/pytorch-nst-feedforward I've implemented the paper without any tanh activation and it works like a charm!

Could you maybe link this repo as a PyTorch implementation of Johnson's original paper? I've documented the differences that it has from the original paper in transformer_net.py file.

from fast-neural-style.

I would think it is connected to the practice back then of using 0..255 pixel range directly in models. Eg. the original neural-style used a VGG model like that.

This issue seems to support my guess:
#112

from fast-neural-style.

I'm aware that Caffe used BGR, 0..255 images mean ([123.675, 116.28, 103.53]) normalized using ImageNet's mean to train VGG models.
Whereas PyTorch which I'm using uses RGB, 0..1 range images also normalized using both std ([0.229, 0.224, 0.225]) and mean ([0.485, 0.456, 0.406]) to train VGG nets. Those are the same numbers just depending whether you use 0..1 or 0..255 range.

But that still doesn't explain the magic 150 number, 127.5 would make some sense then you could just shift [-127.5, 127.5] from the model output to [0, 255].

What I think happened here is that Johnson tried to emulate (using this 150*tanh output activation) what would happen when you used [0, 255] image and subtracted ImageNet's mean from it. That way you can directly feed the output of the transformer net into the perceptual net (VGG).

Namely, if you take 0..255 image and subtract [123.675, 116.28, 103.53] the biggest number will be 151.47 and the smallest will be -123.675 (assuming that we have 255 and 0s in the right channels). So that's kinda -150, 150...

from fast-neural-style.

That way you can directly feed the output of the transformer net into the perceptual net (VGG).

That's about what I was thinking too. But also that 150 may just have been a value that appeared to work. Note that it is in fact a command line option.

from fast-neural-style.

Do we have a way of pinging Johnson for this thing? It'd be nice to explain the meaning of that magic number and whether our hypothesis is true. @jcjohnson

from fast-neural-style.

Seems to me that he's not been active here for a long time.

The logical place to start is to consult the paper https://cs.stanford.edu/people/jcjohns/papers/eccv16/JohnsonECCV16.pdf .

From section 3.1

"All nonresidual convolutional layers are followed by batch normalization [50] and ReLU
nonlinearities with the exception of the output layer, which instead uses a scaled
tanh to ensure that the output has pixels in the range [0, 255]."

from fast-neural-style.