sunshineatnoon / linearstyletransfer Goto Github PK

Hi, thanks for your great work! I want to konw why the encoder and decoder are trained in different datasets?

May I know what is the issue?
My configuration is: cuda9.1 (I don't have cuda8.0), pytorch 0.4.1, python 3.6.8

== Starting run at Tue Sep  3 22:20:58 SGT 2019
== Job ID: 157471
== Node list: node19
== Submit dir. : /home/user/repository/image_enhancement/LinearStyleTransfer
/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:177: UserWarning: nn.UpsamplingNearest2d is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.UpsamplingNearest2d is deprecated. Use nn.functional.interpolate instead.")
/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:122: UserWarning: nn.Upsampling is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.Upsampling is deprecated. Use nn.functional.interpolate instead.")
/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/functional.py:1961: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))
----------------- Options ---------------
                batchSize: 1                             
              contentPath: data/photo_real/content/images/
           contentSegPath: data/photo_real/contentSeg/   
                     cuda: True                          
              decoder_dir: models/dec_r41.pth            
                 fineSize: 512                           
                    layer: r41                           
               matrixPath: models/r41.pth                
                     outf: PhotoReal/                    
                  spn_dir: models/r41_spn.pth            
                stylePath: data/photo_real/style/images/ 
             styleSegPath: data/photo_real/styleSeg/     
                  vgg_dir: models/vgg_r41.pth            
----------------- End -------------------
<module 'libs.pytorch_spn._ext.gaterecurrent2dnoind' from '/home/user/repository/image_enhancement/LinearStyleTransfer/libs/pytorch_spn/_ext/gaterecurrent2dnoind/__init__.py'>
--------------
Traceback (most recent call last):
  File "TestPhotoReal.py", line 105, in <module>
    filtered = spn(transfer,whitenV)
  File "/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/repository/image_enhancement/LinearStyleTransfer/libs/SPN.py", line 138, in forward
    out1 = self.left_right(X,G[0],G[1],G[2])
  File "/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/repository/image_enhancement/LinearStyleTransfer/libs/SPN.py", line 29, in forward
    return self.propagator(x,G1,G2,G3)
  File "/home/user/miniconda3/envs/linearST/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/repository/image_enhancement/LinearStyleTransfer/libs/pytorch_spn/modules/gaterecurrent2dnoind.py", line 12, in forward
    return GateRecurrent2dnoindFunction(self.horizontal, self.reverse)(X, G1, G2, G3)
  File "/home/user/repository/image_enhancement/LinearStyleTransfer/libs/pytorch_spn/functions/gaterecurrent2dnoind.py", line 22, in forward
    gaterecurrent2d.gaterecurrent2dnoind_forward_cuda(self.horizontal, self.reverse, X, G1, G2, G3, output)
AttributeError: module 'libs.pytorch_spn._ext.gaterecurrent2dnoind' has no attribute 'gaterecurrent2dnoind_forward_cuda'

Thanks for your great work, i want to konw the normalized affinity formula for a vectorized feature in section 3.3

Are there any references about the Eq(4)?

I download the pretrained models and a style image:

after running the test command python TestArtistic.py, I got the following result

The performance is inferior to the release demo. The demo result seems more sharper and more beautiful. Where am I wrong?

The style weight and content weight in other style transfer model usually is the former is larger than latter.
But in your experiment is inverse, the former is lower than the latter , is there any reason for this, please?

Hi , i want to konw the style image for the training is only ones in the data folder？

Hi, thank you for your great work!
I find that your vgg19's layers and parameters are both different from what i use .I wanna know how you get your vgg19 and if it came from other people ,how can i get whole layer's parameters of vgg19 with the format of ".pth"? For example,your pretrained vgg19 model only has "vgg_r31.pth","vgg_r41.pth","vgg_r51.pth",and what if i want to use relu_5_2 in vggnet?
here is part of your vggnet:

here is part of my vggnet:

Hi, is there any possible for you to provide the code for the heat map which is mentioned in paper?

I have tried segmenting a very simple architecture image as seen below:
(All images are in .jpg format)

Content image and its segmented image:

Style image and its segmented image:

Output:

All the images I have tried, with relatively similar segmentation classes, received the same output where the image appears to be grayish and blurred.
Could you please explain if there are additional criteria i need to take into consideration when styling images with segmentation? Or any extra parameters I need to tweak?

About the gram matrix in code:
return G.div_(c*h*w)
Why here use the div function and use the chw , shouldn't we use MSE directly to the matrix's elements?

An error occurred when I ran the SPN.py file in the libs folder. Error is as follows:

torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
torch.Size([1, 3, 256, 256])
Traceback (most recent call last):
File "SPN.py", line 155, in
output = spn(x,rgb)
File "/home/cgl/miniconda3/envs/SPN/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "SPN.py", line 138, in forward
out1 = self.left_right(X,G[0],G[1],G[2])
File "/home/cgl/miniconda3/envs/SPN/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "SPN.py", line 17, in forward
sum_abs = G1.abs() + G2.abs() + G3.abs()
RuntimeError: CUDA error: out of memory

Hi, this is a nice work and the code is easy to reuse, do you have any scheduler to update your code for a new pytorch verison (like 1.0+)? I am really aware of this reposity is based on pytorch-spn, which is implemented in pytorch 0.4, but pytorch 0.4 shows poor compatibility with later version.

As the fig.2 show that you get the CxC matrix by the fc layer, why did you say the CNNs so many times? And the CNNs in Fig2 is just to compress as you said in session 5.1.

Hello, how about style migration (without GAN), in the training process, can you use multiple pictures to train multiple pictures? The following is an example:

Suppose there are 3 pictures in category A and 3 pictures in category B. In the network training process, the first batch: A1 A2 corresponds to B1 training; the second batch A1 A2 corresponds to B2 training; the third batch A2 A3 corresponds to B3 Training, is this ok?

I checked your code and can only train multiple pictures for one picture. Maybe I understand the wrong code. Please feel free to answer my questions. Thank you very much.

您好，请问风格迁移(不使用GAN)，在训练过程中，可以使用多张图片对应多张图片训练么？如下是一个例子：

假设A类有3张图片，B类也有3张图片，在网络训练过程中，第一批次：A1 A2 对应B1训练；第二批次A1 A2对应B2训练；第三批次A2 A3对应B3训练，这样可以么？

我查阅了您的代码，只能多张图片对应一张图片进行训练，也许是我理解错了您的代码，请您在闲暇之余，能解答一下我的疑问，十分感谢。

Hi
I had read your paper "Learning Linear Transformations for Fast Image and Video Style Transfer". And it was a great inspiration to me.
Now I want to use encoder and decoder module pretrained in my own dataset. I hope you can provide the code of the pretraining process. It will be very helpful to me. Thank you for taking the time.

HI, when i run the TestPhotoReal.py for photorealistic style transfer, there are RuntimeError,
such as:
RuntimeError: index_copy_(): Source/destination tensor must have same slice shapes. Destination slice shape: [512] at dimension 1 and source slice shape: [843] at dimension 0.

The images i used are your released ones.
How to solve it?

When I run 'real-time-demo.py', system tells me there is an error: AttributeError: 'Namespace' object has no attribute 'dec_dir'. So I think if we should change the 59th line in file 'real-time-demo.py' to: dec.load_state_dict(torch.load(opt.decoder_dir)). Thanks!

There are so many bugs if trying to compile pytorch-spn in another computer with different cuda installation path. your build.py has set the cuda path as '/usr/local/cuda'， which can not easily be spotted. There are other bugs due to the link of library and the installation path of CUDA, can you upload the already compiled library?
I've compiled the pytorch-spn either on windows10+cuda10.0 and ubuntu16.04+cuda9.0, all in vain. I would rather reading your paper for 10hrs rather than debugging your codes for 10+hr. But the reality is I read your paper for 2hrs and get stuck in your codes for 10+hrs just to compile the damed pytorch-spn......

Hi,

I tried to run with torch 0.4.1, but torchvision cannot be found when I run:

 python TestPhotoReal.py

When I tried to install torchvision, the pytorch is automatically updated to 1.2, which will raise another error:

ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.

Did anyone meet similar problems ?