greatlog / dan Goto Github PK

Many thanks for your work.
When I tried to test your code by using the default settings, there are some errors. It seems that the mismatch of the output size and the gt size.

On x4
Traceback (most recent call last):
File "test.py", line 119, in
psnr = util.calculate_psnr(cropped_sr_img * 255, cropped_gt_img * 255)
File "../../utils/util.py", line 698, in calculate_psnr
mse = np.mean((img1 - img2) ** 2)
ValueError: operands could not be broadcast together with shapes (1348,2032,3) (1144,2032,3)

On x2
Traceback (most recent call last):
File "test.py", line 119, in
psnr = util.calculate_psnr(cropped_sr_img * 255, cropped_gt_img * 255)
File "../../utils/util.py", line 698, in calculate_psnr
mse = np.mean((img1 - img2) ** 2)
ValueError: operands could not be broadcast together with shapes (1352,2036,3) (1148,2036,3)

你好，我使用你给出的训练好的model进行test没有问题，但我加载它进行fine-tuning的时候，会出现如下问题，请问是为什么？
Traceback (most recent call last):
File "train.py", line 337, in
main()
File "train.py", line 202, in main
model = create_model(opt) # load pretrained model of SFTMD
File "/media/xbm/data/xbm/Self-ImageSR-Extend/DAN-master/codes/config/DANv1/models/init.py", line 17, in create_model
m = M(opt)
File "/media/xbm/data/xbm/Self-ImageSR-Extend/DAN-master/codes/config/DANv1/models/blind_model.py", line 37, in init
self.load()
File "/media/xbm/data/xbm/Self-ImageSR-Extend/DAN-master/codes/config/DANv1/models/blind_model.py", line 234, in load
self.load_network(load_path_G, self.netG, self.opt["path"]["strict_load"])
File "/media/xbm/data/xbm/Self-ImageSR-Extend/DAN-master/codes/config/DANv1/models/base_model.py", line 102, in load_network
network.load_state_dict(load_net_clean, strict=strict)
File "/home/xbm/anaconda3/envs/pytorch1.8/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict
raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for DAN:
Missing key(s) in state_dict: "init_kernel", "init_ker_map".

非常感谢您的工作，我受益匪浅！关于论文，我想请教您是如何设计这个结构的。
在https://www.bilibili.com/video/BV1Zg411c7zn 这个talk第13:35秒，您提到这种迭代求解k和SR的方法在传统方法中很常见，请问您能给一些参考文献吗？

Hello. Can your code be used to predict the blur kernel of an input test image, along with the SR output?
Thanks!

Hi, it's definitely a fantastic work.
About testing on real image, I am confused.
According to your paper, the SR model inferencing on real image is trained with AWGN. However, in codes/config/DANv1/inference.py, SR model is the same with that of setting1.
I also tested it (with inference.py) on image captured by my own device. The result is quite blur.
Is there any misunderstanding.
Thanks a lot.

It seems pca_aniso_matrix_x4.pth and pca_matrix.pth are identical, while one is anisotropic and the other isotropic. Please check

Hello, and thanks for making the code available!
I am curious as to what the SR results are when not using the blur kernel during the SR process (for ex., only inputting a delta kernel as the conditional input for the Restorer during the training process). Of course, in this case it won't be possible to estimate the blur kernel.
If you have checked the SR results when not using the blur kernel, it would be great if you could share them!

请问在setting1下训练不同分辨率模型使用的x2HR.lmdb、x3HR.lmdb、x4HR.lmdb 是相同的么，如果不相同，各自是怎么得到的呢，感谢解答！

In Class 2 Line 4 PSNR 25.43 SSIM 0.6907
But when I use RealSRhttps://github.com/Tencent/Real-SR pre-model DF2K.pth to test DIV2KRK, get the result PSNR 28.044 SSIM 0.779

Hi, thanks for providing detailed codes and it's a great job!

I'm testing the pretrained model in local on self generated Gaussian 8 testset with kernel size = 21x21.
And I got the performance as below.

Though the local test perf is also excellent, it's different from the reported one (quote from paper).
I'm a little bit confused... Not sure whether there is a mistake made during testing...
Expect for your reply!

I'm interested in the great work. But I found the size of blur kernel is not consistent in DANv1 and v2 for setting2 scale 4.
In DANv1 and kernelGAN, the kernel size is 11.
In DANv2, it is 31.
The quantitative results in the paper of DANv2 is the same as those in DANv1 paper except the additional DANv2, which means testing of DANv2 is the same as DANv1. So you train DANv2 with data blurred by kernel 31 and test it with data blurred by kernel 11. Is there something wrong in my understanding?

I am having the following error;

~/DAN/codes/config/DANv1$ python3 test.py -opt=options/setting1/test_setting1_x4.yml
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    import options as option
  File "/home/alper/DAN/codes/config/DANv1/options.py", line 14, in <module>
    Loader, Dumper = OrderedYaml()
NameError: name 'OrderedYaml' is not defined

File "/home/lct/L/DAN/codes/config/DANv2/train.py", line 203, in main
model = create_model(opt) # load pretrained model of SFTMD
File "/home/lct/L/DAN/codes/config/DANv2/models/init.py", line 17, in create_model
m = M(opt)
File "/home/lct/L/DAN/codes/config/DANv2/models/blind_model.py", line 90, in init
lr_scheduler.MultiStepLR_Restart(
File "/home/lct/L/DAN/codes/config/DANv2/models/lr_scheduler.py", line 27, in init
super(MultiStepLR_Restart, self).init(optimizer, last_epoch)
File "/home/lct/anaconda3/envs/DAN/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 78, in init
self.step()
File "/home/lct/anaconda3/envs/DAN/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 141, in step
values = self.get_lr()
File "/home/lct/L/DAN/codes/config/DANv2/models/lr_scheduler.py", line 34, in get_lr
return [
File "/home/lct/L/DAN/codes/config/DANv2/models/lr_scheduler.py", line 35, in
group["initial_lr"] * weight for group in self.optimizer.param_groups
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'
(DAN) lct@251:~/L/DAN/codes/config/DANv2$
训练DANv1时并没有报错，能正常训练，但容易Loss发散；训练DANv2时会出现以上的报错信息，我应该如何修改才能正常训练呢？万分感谢！

Hi, I noticed that due to some bugs in the older version of IKC, I can't get the same SSIM results in Table 1. Could you please provide the codes about calculating SSIM results in Table 1? I can't find a version of IKC which provides the same SSIM results as in their paper. Thank you!

Hi, thank you for this excellent work and thank you for sharing the code.

I have the following problem when runing the testing code as :
python test_single_img.py -opt=test_option.yml -input_dir=D:/2020/ReferenceCode/DAN-master/our_testimages/ -output_dir=D:/2020/ReferenceCode/DAN-master/our_results/

The environment is: python 3.7 / torch 1.6

Could you please give me some suggestions to slove this problem? Thanks

G:\Anaconda\envs\py3.7to1.6\python.exe D:/2020/ReferenceCode/DAN-master/codes/config/DAN/test_single_img.py
export CUDA_VISIBLE_DEVICES=0
OrderedDict([('name', 'DIV2KRK'), ('mode', 'LQGTker'), ('dataroot_GT', '/data/DIV2KRK_public/HRblur.lmdb'), ('dataroot_LQ', '/data/DIV2KRK_public/x2LRblur.lmdb')])
Traceback (most recent call last):
File "D:/2020/ReferenceCode/DAN-master/codes/config/DAN/test_single_img.py", line 32, in
opt = option.parse(args.opt, is_train=False)
File "D:\2020\ReferenceCode\DAN-master\codes\config\DAN\options.py", line 56, in parse
config_dir = path.split("/")[-3]
IndexError: list index out of range

Process finished with exit code 1

Thank you for sharing. I learned a lot from your paper, but I still have a question about the kernel settings. Does the kernel width and its standard deviation mean the same thing? In your paper you mentioned that "During training, the kernel width is uniformly sampled in [0.2, 4.0], [0.2, 3.0] and [0.2, 2.0] for scale factors 4 , 3 and 2 respectively", but I found in your code that the kernels sigma are all set to [0.2, 4.0] for different upsampling rates, so do I need to change this part when retraining if I want to keep the same settings as in your original paper? Thank you very much!

Hi,DAN is very cool and thanks you for releasing the code!
When I run the test.py in DANv1,I didn't run the generate_mod_blur_LR_bic.py and create_lmdb.py,just set the LR and HR images in options/setting1/xx.yml,such as in setting1_x2.yml:

 test1:
    name: Set5
    mode: LQGT
    dataroot_GT: /datasets/Set5/LR/x2
    dataroot_LQ: /datasets/Set5/HR/x2

But the result of PSNR/SSIM is 35.59dB/0.938,is lower than the results in your paper."37.34/0.9526"
May I ask is that the different setting of the .yml file contribute to the non-ideal results？

RuntimeError: ../../../checkpoints/DANx4.pth is a zip archive (did you mean to use torch.jit.load()?)

/DAN-master/codes/config/DAN/models/base_model.py", line 96, in load_network
load_net = torch.load(load_path)

Hello,

Thank you for this implementation. I would like to train DAN on my own dataset which is in greyscale. So I created a new yml file in options and I set color option to grey. However weight are still of the size [nbkernel, 3, kernelsize, kernelsize] instead of [nbkernel, 1, kernelsize, kernelsize].
Can't we change the number of channel of the weights ? and How ?

Thank you a lot.

抱歉，因为我的英文可能表达的不太准确，我在test Set5数据集x2时，PSNR=28.85，我还是没能找到问题的所在，我可以把我操作的过程说明一下吗？
Step1：设置generate_mod_blur_LR_bic.py

up_scale = 2
mod_scale = 2
sourcedir = '/datasets/Set5'
savedir = '/datasets/DAN/Set5'
'rate_iso' : 1.0

其他的都没有更改,点击Run，生成的四个文件夹，HR和LRblur都是40张图片，不同的siga。
Step2:之后将这两个数据集放入create_lmdb.py

ima_folder = '/datasets/DAN/Set5/HR/x2/*'#../LRblur/x2/*
lmdb_save_path = '/datasets/DAN_lmdb/Set5/HR.lmdb'#../LRblur.lmdb
meta_info = {"name":"Set5"}

其他内容也没有更改，点击Run，HR.lmdb和LRblur.lmdb都有三个文件.
Sept3:分别放入option/test/test_setting1_x2.yml文件中

suffix : x2
scale: 2
pca_matrix_path:'/DAN/pca_matrix/DANv2/pca_aniso_matrix_x2.pth'
datasets:
    test1:
    dataroot_GT: /datasets/DAN_lmdb/Set5/HR.lmdb
    dataroot_LQ: /datasets/DAN_lmdb/Set5/LRblur.lmdb
pretain_model_G: ../../danv2_x2_setting1.pth

结果报了错：

RuntimeError: Expected tensor to have size 144 at dimension1, but got size 121 for argument #2 'batch2'(while checking argument for bmm)

如果将pca_matrix_path改为pca_matrix.pth就可以正常跑程序，但是结果PSNR=28.648437dB，SSIM=0.817695
这个问题我一直没能解决，不知道那里操作有误导致结果和论文中的结果不相同，还请您指正！

24-01-20 17:47:31.710 - INFO: img:baby - PSNR: 4.304825 dB; SSIM: 0.010210; PSNR_Y: 5.539371 dB; SSIM_Y: 0.173895.
24-01-20 17:47:32.200 - INFO: img:bird - PSNR: 9.119347 dB; SSIM: 0.052580; PSNR_Y: 9.932940 dB; SSIM_Y: 0.291013.
24-01-20 17:47:32.561 - INFO: img:butterfly - PSNR: 5.385315 dB; SSIM: 0.000441; PSNR_Y: 6.387867 dB; SSIM_Y: 0.097870.
24-01-20 17:47:33.031 - INFO: img:head - PSNR: 8.389810 dB; SSIM: 0.031692; PSNR_Y: 10.075234 dB; SSIM_Y: 0.338463.
24-01-20 17:47:33.388 - INFO: img:woman - PSNR: 4.811639 dB; SSIM: 0.010355; PSNR_Y: 6.219289 dB; SSIM_Y: 0.218729.

作者你好，谢谢你非常不错的工作，
在我训练时大约在60k个iter会出现loss都变成NAN的情况并且按照 issue#8 的解决办法选取checkpoint继续train 但是一段时间后还是会出现loss变为NAN的情况（如下图），似乎无法解决，想请教一下您怎么train让模型收敛到最终的结果的？

Traceback (most recent call last): File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\train.py", line 349, in <module> main() File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\train.py", line 207, in main model = create_model(opt) # load pretrained model of SFTMD File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\models\__init__.py", line 17, in create_model m = M(opt) File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\models\blind_model.py", line 90, in __init__ lr_scheduler.MultiStepLR_Restart( File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\models\lr_scheduler.py", line 27, in __init__ super(MultiStepLR_Restart, self).__init__(optimizer, last_epoch) File "C:\dev\PycharmInterpreters\PyTorchStar\lib\site-packages\torch\optim\lr_scheduler.py", line 77, in __init__ self.step() File "C:\dev\PycharmInterpreters\PyTorchStar\lib\site-packages\torch\optim\lr_scheduler.py", line 154, in step values = self.get_lr() File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\models\lr_scheduler.py", line 34, in get_lr return [ File "C:\dev\PycharmProjects\DAN\codes\config\DANv2\models\lr_scheduler.py", line 35, in <listcomp> group["initial_lr"] * weight for group in self.optimizer.param_groups TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

PyTorch 1.11 and 1.5. tried both.

Hi all, has anyone solved the following error message when trying to train DANv2?

group["initial_lr"] * weight for group in self.optimizer.param_groups TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

It does not appear with DANv1, and I've seen others report the same issue, but there are no solutions to be found.

Traceback (most recent call last):
File "test_single_img.py", line 36, in
model = create_model(opt)
File "/workspace/Visual_Media_Restore/DAN/codes/config/DAN/models/init.py", line 17, in create_model
m = M(opt)
File "/workspace/Visual_Media_Restore/DAN/codes/config/DAN/models/blind_model.py", line 27, in init
self.netG = networks.define_G(opt).to(self.device)
File "/workspace/Visual_Media_Restore/DAN/codes/config/DAN/models/networks.py", line 43, in define_G
pca_matrix_path=opt["pca_matrix_path"],
File "/workspace/Visual_Media_Restore/DAN/codes/config/DAN/models/modules/dan_arch.py", line 173, in init
self.init_kernel.view(1, 1, self.ksize ** 2).matmul(self.encoder)[:, 0],
RuntimeError: Expected tensor to have size 441 at dimension 1, but got size 121 for argument #2 'batch2' (while checking arguments for bmm)

I have generated the test dataset according to the test setting1 and test the provided pre-trained model on it. The PNSR results are roughly the same as you have mentioned in Table 1. However, the SSIM is much lower. Could tell me how to solve this issue.

你好，谢谢你很有意义的工作和代码。但是我有一个问题就是提供的代码中，yaml的训练数据的模式是LQGTKer，但是训练中却是将GT模糊下采样后得到LR_img, ker_map并用他们训练。dataloader中的LQ数据并没有被使用且没有ker_map的lmdb。这样做是为了增加模糊核的随机性么？将LQ GT 和模糊核都做成lmdb而不是在训练的时候生成会不会降低性能，减少训练时间？

Hi, I would like to ask a question about the dataset preparation. I find that in 'train.py', the 'blur_lr_imgs' are generated online, that is to say, they are generated during training process. For those 'blur_lr_img' generated offline, they are only used in validating process. Does that mean I should only run 'python3 codes/scripts/generate_mod_blur_LR_bic.py' for validation dataset not for train dataset? I think it's not clarified very well in this repo.

@greatlog Hello, thanks for your impressive work.
Could you please provide the checkpoint for setting 1?

你好，想问下setting2中train和test的yaml配置中input_pare=15,而在setting1中用10？

您好，冒昧打扰了。

I suppose the params of the network structures need to be modified? Could you pls provide an x2 version yml file? Thanks!
What I have changed is:
_name: DANx2
suffix: x2
scale: 2
pca_matrix_path: ../../../pca_aniso_matrix_x2.pth
upscale: 2
kernel_size: 11
pca_matrix_path: ../../../pca_aniso_matrix_x2.pth
path:
pretrain_model_G: ../../../checkpoints/DANx2.pth

name: DANx2
suffix: _x2  # add suffix to saved images
model: blind
distortion: sr
scale: 2
crop_border: ~  # crop border when evaluation. If None(~), crop the scale pixels
gpu_ids: [0]
pca_matrix_path: ../../../pca_aniso_matrix_x2.pth # ../../../pca_aniso_matrix_2.pth when scale=2

datasets:
  test0:
    name: DIV2KRK
    mode: LQGT
    dataroot_GT: /data/DIV2KRK_public/HRblur.lmdb
    dataroot_LQ: /data/DIV2KRK_public/x4LRblur.lmdb


#### network structures
network_G:
  which_model_G: DAN
  setting:
    nf: 64
    nb: 40
    input_para: 10
    loop: 4
    upscale: 2
    kernel_size: 11
    pca_matrix_path: ../../../pca_aniso_matrix_x2.pth 

#### path
path:
  pretrain_model_G: ../../../checkpoints/DANx2.pth

Hello, DAN is an amazing framework, but I couldn't save the test results when I was testing the real image, I don't know what went wrong, hope to get a reply. thank you.

请问作者您有遇到这种情况吗？一开始报错找不到pca_matrix.pth路径，我在generate_mod_blur_LR_bic.py里修改路径（../../pca_matrix/DANv1/pca_matrix.pth）之后出现如图的报错。

Thanks for your work so much.
Like the title, I meet the error that loss value is NAN at 22800 iterations (Second time: 29200) suddenly.
Did you have ever met this kind of error?

您好，对于X2的SR，请问怎么生成Gaussian8的kernel，Set5的每张测试图片都需要生成8张不同的LR图片吗

Hi.

Is there any way to run this if I don't have an external GPU?

(I just have an integrated graphics card.)

Thanks.

Hi, it' a wonderful job.
Why two pretrained weights ("danv1_x4_setting1.pth" and "danv1_x4_setting2.pth") are the same?

想知道这方法与esrgan相比会不会有观感上的提升

Thanks for your nice work!
I am a beginner in blind sr, so I do not understand why methods like EDSR are trained under the bicubic downsampling setting while tested under the multiple-degradation setting.
Looking forward to your reply！

hello,
I am confused about the generation process of 'kernel_code'.
When calculating the PCA matrix, the code do 'X = X - X_mean.expand_as(X)' before get the PCA matrix.
Then, in 'PCAEncoder', It seems direct projection by 'torch.bmm(
batch_kernel.view((B, 1, H * W)), self.weight.expand((B,) + self.size)
).view((B, -1))' without '- X_mean'.
Did we need '- X_mean'?

Thank you!

Thanks a lot for the code.

I have some questions on this part of the code that appears both in v1 and v2:

It seems that your loss is the combination of SR loss (contained in d_sr) + kernel loss (contained in d_ker) but only for the last step of the Restorer and Estimator since the lines "total_loss += d_ker" and "total_loss += d_sr" are outside the loop.
It leads me to two questions:

• Why are you keeping track of the d_sr and d_ker for the other steps.
• Did you try to use the d_sr and d_ker for each step and is it giving bad results?

Thanks a lot,
Charles

How can I only get output from a custom image?