jiahuiyu / wdsr_ntire2018 Goto Github PK

Hello, Thanks for your sharing!
I followed the steps to train WDSR-b model with --resblock 8 and --n_feats 64 , and got 34.755 PSNR on DIV2K 801~900 photos.
My inference time is 1.1 seconds in 1080Ti for 1020×678 to 2040×1356, I would like to know if there is any way to reduce the inference time when network' s parameters are constant.
I am looking forward for you reply, thanks a lot!

Hi,thanks for your code.I want test one single image of my own with MDSR(x4),and I did something like this:

1. Add in option.py:
   # Option for WDSR parser.add_argument('--r_mean', type=float, default=0.4488, help='Mean of R Channel') parser.add_argument('--g_mean', type=float, default=0.4371, help='Mean of G channel') parser.add_argument('--b_mean', type=float, default=0.4040, help='Mean of B channel') parser.add_argument('--block_feats', type=int, default=128, help='Block feats')

2. move wdsr_a.py,wdsr_b.py to model

3. use the lines in demo.sh like this and comment others:
   python main.py --data_test Demo --scale 4 --pre_train ../experiment/model/MDSR_baseline.pt --test_only --save_results

4. run command 'sh demo.sh'

Last I got an image of all gray with nothing,no color,no cotent, I need some help to know where the issue is.

Hi, Jiahui.
Could you tell me how many epoches the L1 loss convergence or PSNR reaches the optimal value when you train WSDR_B?

Hi Jiahui, thanks for your work and I have some conceptual questions (maybe very basic).

In section 3.1 of the paper, in the vanilla residual network, why is the number of parameter 2 * W1^2 * k^2? I don't get where W1^2 comes from.
Why is the final HR image representation S^2 * 3?

In the other issue (wdsr_b increase channel problem), I noticed this "48->48x6(1x1)->48x0.8->48" and "32->192(1x1)->25(1x1)->32". Do you mind explaining this? How do you determine how large the third number is (48*0.8 and 25)?

In section 3.2, what does low-rank convolution mean here? The paper says it factorizes a large convolution kernels to two low-rank convolution kernels. I think the two low-rank kernels are the 1x1 and 3x3 kernel, but what is the "large convolution kernel" refering to?

Is there a reason why these scale numbers (2-4,6-9) are chosen? Have you tried what happens if r is very large, such as 20?
Also, what makes you come up with this 1x1 convolution to solve this?

How do you determine the number of residual box?

Thanks for reading! Sorry if some of them are too basic. I am really interested in the paper, but I am stuck by these questions.

Hi, Jiahui,

I am trying to implement wdsr_a with the scale 4. I would like to compare its performance with EDSR( # of residual blocks = 32, n_feats = 256). In this case, what would be your recommendation to set n_feats and block_feats?
Currently I am using wdsr_a with n_feats 128 and block_feats 512. However I am not sure if it is the intended approach while constructing the network.

hi jiahui. I want to reproduce your WDSRA Baseline x2 result 34.77 dB. I used EDSR-Pytorch framework to train and set n_res_blocks=16, n_feats=32, n_block_feats=128, lr = 1e-3, output patch_size=96, batch size=16， epochs=300,lr_decay at 200 and 250 with decay gamma=0.1. But only got about 34.62~34.65 dB for each attempt. This result is not very satisfactory. Can you give me some train advice?

Thanks for the great work. Really inspiring.
Though, an issue was found when I am playing with the real data. for images destroyed by jpeg compression, the improvement is usually minor. I tried re-train the neural nets with jpeg images. However, it does not help much, usually giving a psnr value close to 31.7db, significantly lower than its bicubic part. Is there any solution? Thanks in advance.

I use the pre-model trained by X2, how to adjust the parameters to train the X4 model.

How to train models with other datasets? Which part of the program do I need to modify?

Hi,

Thank you for providing the code.

The wide-activated network is very effective. Do you have idea why it works? Thanks.

you make a increase channel unit in wdsr_b 32->326->320.8->32
I have a problem, does 32*6 is for scale 2 times, how about x3 and x4, should I change 6 to other number.
you paper said
It enables even wider activation (6� to 9�) without additional parameters or computation.

Hello,

torch.nn.utils.weight_norm does the crucial weight re-parametrization during training, but does weight normalization(re-parametrization) also happen during testing? If yes should not they be removed before proceeding for testing? Or is it alright that weight normalization happen during testing also?

Thankyou

When the number of feature maps increase above a certain level, the author will set residual scaling with 0.1 to ensure the training procedure stable in EDSR. I don't find the illustration about res_scale in your paper while I find that in your code. Then I want to know how do you set res_scale base on the value of n_feats and block_feats?
Best~

Thank you for your sharing.
When I try to train WDSRa on DIV2k by setting the number of wide residual blocks as 16, it works well with the loss of the first 100 iterations being about 15. However, when the number of wide residual blocks increases to 128, the loss of the first 100 iterations is about 2e8 and the PSNR evaluated on Set5 (x2) is about 18dB . I did not find the reason. So ask you, when the number of blocks increases, how to set the relevant hyperparameters? Thank you.

Hi Jiahui,
Do you have pytorch pretrained model for WDSR ? Could you please share?

Thank you.

Can you send me the result of the DIV2K verification?E-mail: 1970915834 @qq.com

Hi, when I train with lr=1e-4, batch_size=16, out_patch_size=192, my loss was normal, after about 15,000 steps (1 epoch = 30,000 steps), the loss suddenly turned into nan, my data was clean, and I used L1Loss. How should this problem be solved? thanks

WDSR-A
x/2->2x->x/2
computation:
3x3x2/2+3x3x2/2=18
act:
2
3x3Conv:
100%

WDSR-B:
x->6x->0.8x->x
computation:
6+6x0.8+3x3x0.8=18
act:
6
3x3Conv:
40%
(I think you use the second 1x1 Conv just to align the computation with one EDSR block and WDSR-A block)

WDSR-B-modified:
x/2->3x->x/2
computation:
3/2+3/2x3x3=15
act:
3
3x3Conv:
90%
So you can compare the performance: WDSR-A-B5 vs. WDSR-B-modified-B6.

Hello, @JiahuiYu !

I'm trying to implement your network.

When I checked your light models for WDSR-B, I found the difference between your results and my results. I saying WDSR-B with n resblocks as WDSR-B_nblks, you assert that the numbers of parameters of WDSR-A_nblks and WDSR-B_nblks are equal. ( e.g. the number of parameters of WDSR-A_1blks and WDSR-B_1blks is 0.08M )

According to my computation, the numbers of parameters of WDSR-A_nblks are almost equal, but those of WDSR-B_nblks are quite different. ( more precisely, my results are less than yours )

How to compute the number of parameters?
I use this formula: kernel_size^2 * ch_in * ch_out (weight) + ch_out (bias) for each convolution.

Thanks in advance!

Hobin

What's your WDSR Baseline(in table 1)'s option?
n_res_blocks?
n_feats?
WDSR-A or WDSR-B?
if the option is not the same, I don't konw what causes the better performence...

Also, in table 2, I don't know the better performence is caused by the difference of blocks or by the global residual pathway and upsampler.

Hi,
Could you please explain if and how can I run your network on images from track3 of the challenge?
I am interested in the test\inference stage (not the training).
Would this work also on any image I choose at random or is it limited in some sense?

Thank you

In #40, you said you are doing data normaliztion. but i have no idea where 127.5 come from.

as far as i know, data normalization is something like y = (x - μ)/σ, where μ is mean, and σ is the standard viarence. I guess [r, g, b]_means you used here may be the avg value for DIV2K_train_LR_bicubic:

R - 0.4488,
G - 0.4371,
B - 0.4040,

However, in forward, you used x = (x - self.rgb_mean.cuda()*255)/127.5
so i am confused where the σ = 127.5 come from? it should be [r, g, b]_σ but here is only one σ

And another question is which dataset to use when calculate mean RGB, HR or LRx2, x3, x4. You said in the Experimental Results of your paper that you subtracted the mean RGB of DIV2K training set. I have tested that the mean value is different between HR and LRx2,x3,x4.

Hi, Jiahui,
When I run wdsr_a, there is a error:

Making model...
Traceback (most recent call last):
File "D:/project/pytorch/SR/EDSR-m/src/main.py", line 27, in
main()
File "D:/project/pytorch/SR/EDSR-m/src/main.py", line 17, in main
model = model.Model(args, checkpoint)
File "D:\project\pytorch\SR\EDSR-m\src\model_init_.py", line 24, in init
self.model = module.make_model(args).to(self.device)
File "D:\project\pytorch\SR\EDSR-m\src\model\wdsr_a.py", line 42, in init
[args.r_mean, args.g_mean, args.b_mean])).view([1, 3, 1, 1])
AttributeError: 'Namespace' object has no attribute 'r_mean'

Then How can I to get args.r_mean, args.g_mean and args.b_mean?

Thanks!

Hi! I have a question about weight normalization.
Did you put weight normalization to where the batch normalization is in the residual block?
Thanks!

Hi, I have a doubt about nn.PixelShuffle and nn.ConvTranspose2d. In the upsampling layer, some use nn.PixelShuffle while others use nn.ConvTranspose2d. The number of filters belong to nn.PixelShuffle is scale^2 times that of number of filters belong to nn.ConvTranspose2d. However, I don't find any authors describing the difference between the two methods in their paper.

Have you thought about this and experimented? Will these two methods have an impact on the training results?

Best~

hi jiahui. I want to reproduce your WDSRA x2 result 34.541 dB. I used EDSR-Pytorch framework to train and set --patch_size 96 --n_resblocks 8 --n_feats 32 --block_feats 128 --res_scale 1 .
All results are lower than yours, can you tell me the settings of other hyperparameters for your training?Thanks.

What would the terminal command be to evaluate the WDSR models with EDSR as backbone?

I added the following to the option.py

parser.add_argument('--r_mean', type=float, default=0.4488, help='Mean of R Channel') parser.add_argument('--g_mean', type=float, default=0.4371, help='Mean of G channel') parser.add_argument('--b_mean', type=float, default=0.4040, help='Mean of B channel') parser.add_argument('--block_feats', type=int, default=128, help='Block feats')

And added the wdsr_a and wdsr_b models to src/model/.

What would the corresponding command for the WDSR be something like this:

python main.py --data_test Set5 --data_range 801-900 --scale 4 --n_resblocks 32 --n_feats 256 --res_scale 0.1 --pre_train download --test_only --self_ensemble

Any help appreciated!

Dear author:
When using my own image dataset (image size is 600 *600) to train WDSR with the command '--scale 3 --patch size 144 or -- scale 4 --patch size 192', there exited an error:

Traceback (most recent call last):
File "main.py", line 25, in
t.train()
File "/home/zhangshu/WDSR/src/trainer.py", line 41, in train
for batch, (lr, hr, _, idx_scale) in enumerate(self.loader_train):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 286, in next
return self._process_next_batch(batch)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 307, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
File "/home/zhangshu/WDSR/src/dataloader.py", line 47, in _ms_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/zhangshu/WDSR/src/dataloader.py", line 47, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 81, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/home/zhangshu/WDSR/src/data/srdata.py", line 149, in getitem
pair_t = common.np2Tensor(*pair, rgb_range=self.args.rgb_range)
File "/home/zhangshu/WDSR/src/data/common.py", line 57, in np2Tensor
return [_np2Tensor(a) for a in args]
File "/home/zhangshu/WDSR/src/data/common.py", line 57, in
return [_np2Tensor(a) for a in args]
File "/home/zhangshu/WDSR/src/data/common.py", line 52, in _np2Tensor
tensor = torch.from_numpy(np_transpose).float()
RuntimeError: Trying to resize storage that is not resizable at /pytorch/aten/src/TH/generic/THStorage.c:183

When I train the model with DIV2K dataset ,everything is ok .However, when transferring to my own dataset ,the error above exited with scale 3 or 4. I don't know that whether my images are too small (600 *600) compared with DIV2K dataset?
Hoping for your reply!
Thank you very much!

Hi, Thank you for your sharing. Do you have the pre_trained model of Pytorch, because I just want to test.

hello, Jiahui. I want to reproduce the results of table2. But I found that the amount of parameters in your table seems to be 10 times larger. I think maybe this is a writing mistake, is that right? For example，I set n_resblocks = 1，n_feats = 64，scale = [2]， rgb_range = 255，res_scale = 1，n_colors =3 for EDSR，and total parameters is 262,043, not 2.6M， is this correct?

Hi, thank you very much for sharing your code.
I ran your model(WDSR_b ) based on EDSR-Pytorch as backbone training framework, but it reported an error as follows:

AttributeError: 'Namespace' object has no attribute 'r_mean'

The relevant code is located in wdsr-b.py:
self.rgb_mean = torch.autograd.Variable(torch.FloatTensor(
[args.r_mean, args.g_mean, args.b_mean])).view([1, 3, 1, 1])

And I can't find the values of r_mean,g_mean and b_mean in option.py, so could you tell how to set these values?

Hi, Jiahui,
Thanks for your sharing.

When you compare EDSR and WDSR, are the the values of n_feats the same and whether both values are set to 64？

Thanks.

When I used WDSR_A, after adjusting the parameters, I found that the Set5 PSNR value was 43.975 using the Set5 test set test results. I don't know if there will be an abnormal situation in this result.

In the paper, it said that linear low-rank convolution factorizes a large convolution kernel to two low-rank convolution kernels.
but i did not see the detailed, and i am confused about it. i would like to know which part of the code is for this.
i am new to this.
Thank you!

Hello,thank you very much for sharing your code.
I ran your model(WDSR_B) based on EDSR-Pytorch as backbone training framework, but when I ran here:

self.rgb_mean = torch.autograd.Variable(torch.FloatTensor(
[args.r_mean, args.g_mean, args.b_mean])).view([1, 3, 1, 1])

I found the following errors:
AttributeError: 'Namespace' object has no attribute 'r_mean'

and I can't find the values of r_mean,g_mean,b_mean in the option.py, could you tell me how to set the values of r_mean,g_mean,b_mean?

Hi, I read through the paper, and find the discussion of weight normalization as follow:

It is also noteworthy that naively introducing weight normalization in training image restoration networks may not help that much. We find empirically that weight normalization allows higher learning rate (i.e. 10×), with which the loss of training normal networks explodes. The advantages of weight normalization are shown in Figure 3.

It seems naively introducing the weight normalization would not help much towards super-resolution task, but in Fig. 3 we definitely observe performance gain brought by weight normalization. So I was wondering how about non-naive way to introduce wn in the paper.

Hi, Jiahui,

Thank you for sharing the code.

Did you compare the performance of WDSR-A and WDSR-B when the number of resiudal blocks is larger than 8？

It seems that the WDSR-B is more effective than WDSR-A when the resiudal blocks are very few.

Thanks

Hi, thanks for making your great work available to everyone.
Is it possible to train the model with my own custom dataset ?
I would like to train the model with microscopy images.
Could you kindly give me some clues on which lines of codes need to be changed to make this possible?

Thank you very much

Hello, I check your paper, seems you do preprocess for best psnr.
is it Data alignment for LR/HR pair. if true, could you share how to do that.

Hi, Jiahui,

Thank you for sharing.
I am trying to implement the wdsr_a x4 and I would like to know whether the number of block_feats is 4 time of the n_feats and is it necessary to train a wdsr_a x2 model and set it as the pre_train of the wdsr_a x4?

Thanks

Hi,I trained WDSR_A model on DIV2K_wild_x4 training set,but the final test PSNR is about 19.423.Is the result worse than yours?Here's some of my settings,and could you please help me check whether my setting is correct.
1.trainging dataset：3200 wild LR images(I mixed 800x4 wild LR images),test dataset:100 wild LR images
2.parameters:
--model wdsr_a --scale 4 --batch_size=64 --patch_size 192 --data_range '1-3200/3201-3300'
--n_feats 128 --block_feats 512 --lr 1e-4 --reset

Hi,when i run main.py ,there is an AttributeError: module 'model.wdsr_a' has no attribute 'make_model'