jingyunliang / rvrt Goto Github PK

Hello RVRT Team 👋 Thank you again for your great code.

I have followed your paper's instructions to train my own RVRT model. However, my network's quality does not match the network you share online, nor the numbers you report.

Maybe I have missed something on my end, but I want to know if you have a training script so I can inspect all the details myself?

Thank you.

Denoising algorithm not working even on the algorithm mentioned in git repo (DAVIS-test). Problem of tensor mismatch for number of channels exists.

Hi, I am using pretrained model for video denoising. Checking the code, the model split the frames into overlapped patches to process. However, for the result, the first few patches seems to have a great result, but for the latter one it seems that the noise are not been removed and having visible edges between different patches.

Here is the 2 frames of processed results:

here is my command, I did not modify any code in main_test_rvrt.py. Is that the way I test was wrong?

python main_test_rvrt.py --task 006_RVRT_videodenoising_DAVIS_16frames \
    --sigma 1 \
    --folder_lq my_video_path \
    --folder_gt my_video_path  \
    --tile 0 256 256 --tile_overlap 2 20 20 \
    --save_result

Thanks

When I tested it, everything seemed to work fine at first. But later the GPU-Util of the GPU is 0, and at this time the video memory of the GPU is occupied. The program seems to be stuck and does not continue to execute.

Do you have any good suggestions?

4. Visualization, I get the error: IndexError: list index out of range

Hi! Thank you for releasing this wonderful work. I have a question on how RFR operates:

Since the dependencies of F_t^i are the items to its LEFT and TOP (like this drawing), and we get shallow features for all clips (t=0,1,...T/N -1) at layer i=0, how do we calculate features for clip t=0 at all layers (i=1,2,...L-1) in order to start the recurrency for later clips?
For example, to calculate F(i=1, t=1), it depends on F(i=0, t=0), F(i=0, t=1) (these two I understand are the shallow features to begin with), and F(i=1, t=0) (which I'm not sure where it comes from).

Thank you in advance! I'd appreciate your help in explaining!

Has anyone encountered this error?

Is there certain video card family or GPU memory requirements we need to run this?

There seems to be a dependency issue of cuda kernel while using deform_attn while training if the current dependencies were installed.

RuntimeError: Error building extension 'deform_attn': [1/2] c++ -MMD -MF deform_attn_cuda_pt110.o.d -DTORCH_EXTENSION_NAME=deform_attn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /opt/conda/lib/python3.7/site-packages/torch/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/jeyamariajose/projects/RVRT/KAIR/models/op/deform_attn_cuda_pt110.cpp -o deform_attn_cuda_pt110.o 
FAILED: deform_attn_cuda_pt110.o 
c++ -MMD -MF deform_attn_cuda_pt110.o.d -DTORCH_EXTENSION_NAME=deform_attn -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /opt/conda/lib/python3.7/site-packages/torch/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -isystem /opt/conda/lib/python3.7/site-packages/torch/include/TH -isystem /opt/conda/lib/python3.7/site-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /opt/conda/include/python3.7m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -c /home/jeyamariajose/projects/RVRT/KAIR/models/op/deform_attn_cuda_pt110.cpp -o deform_attn_cuda_pt110.o 
/home/jeyamariajose/projects/RVRT/KAIR/models/op/deform_attn_cuda_pt110.cpp: In function ‘void deform_attn_cuda_backward(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, int, int, int, int, int, int, int, int, int, int, int)’:
/home/jeyamariajose/projects/RVRT/KAIR/models/op/deform_attn_cuda_pt110.cpp:187:68: error: invalid initialization of reference of type ‘const at::Tensor&’ from expression of type ‘const c10::ScalarType’
     grad_attns = at::_softmax_backward_data(grad_attns, attns, -1, dtype);
                                                                    ^~~~~
In file included from /opt/conda/lib/python3.7/site-packages/torch/include/ATen/ATen.h:15,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/types.h:3,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader_options.h:4,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/base.h:3,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader/stateful.h:3,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data/dataloader.h:3,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/data.h:3,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/csrc/api/include/torch/all.h:8,
                 from /opt/conda/lib/python3.7/site-packages/torch/include/torch/extension.h:4,
                 from /home/jeyamariajose/projects/RVRT/KAIR/models/op/deform_attn_cuda_pt110.cpp:4:
/opt/conda/lib/python3.7/site-packages/torch/include/ATen/Functions.h:5243:29: note: in passing argument 4 of ‘at::Tensor at::_softmax_backward_data(const at::Tensor&, const at::Tensor&, int64_t, const at::Tensor&)’
 TORCH_API inline at::Tensor _softmax_backward_data(const at::Tensor & grad_output, const at::Tensor & output, int64_t dim, const at::Tensor & self) {
                             ^~~~~~~~~~~~~~~~~~~~~~
ninja: build stopped: subcommand failed.

Hi, I read your work and run with commind: python main_test_rvrt.py --task 001_RVRT_videosr_bi_REDS_30frames --folder_lq testsets/ --tile 100 128 128 --tile_overlap 2 20 20 --save_result
My test data have 180 LR image with 854*480 , the infer time took 1,115s which is much longer than expected time.
Why this and how to solve it ? thx.

Hi, thank you for all the great work with video restoration models.

I notice that you are also an author of the ReViD paper (https://arxiv.org/pdf/2208.11803v2.pdf), which contains a link to a github repo that appears to have been taken down. Is there any chance the source code is available somewhere else?

I'm also curious, would you recommend ReViD or RVRT to use for video denoising? Does RVRT run as fast as ReViD?

Thank you!

Carson

Hello,

Thanks for your great work!

I have a question about candidates of GDA. I think candidates=9 means one position in Q should calculate correlation with 9 positions in K and V like what illustrated in Figure 3. Does it means the dimension of K/V should expand 9 times?

Looking forward to your reply, thanks!

Hi! I find your work very interesting, but I ran into all types of problems trying to run the code on my Windows 11, even though I have all the requirements installed and followed all the steps. Could not run it on Collab as well, it downloads the reds dataset instead of using the one I uploaded, and ends up giving an error. Can you please check if there's anything wrong? It could be me lol, I am learning how to use command line by myself.

Run with bellow commod:
python main_test_rvrt.py --task 001_RVRT_videosr_bi_REDS_30frames --folder_lq testsets/your/own --tile 0 0 0 --tile_overlap 2 20 20

Error detail infomation:
File "/opt/conda/envs/env_rvrt/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 613, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/opt/conda/envs/env_rvrt/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 608, in _conv_forward
return F.conv3d(
RuntimeError: Given groups=1, weight of size [192, 4, 1, 3, 3], expected input[1, 3, 38, 256, 256] to have 4 channels, but got 3 channels instead

Hi, how to run 2x upscale for SR ?
thanks!

Hei, Can it be applied to real-time video enhancement?

I run for this task: "python main_test_rvrt.py --task 002_RVRT_videosr_bi_Vimeo_14frames --folder_lq testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences --folder_gt testsets/vimeo90k/vimeo_septuplet/sequences --tile 0 0 0 --tile_overlap 0 20 200". However, I cannot download Vimeo-90k and when I run this program, it raises error like this:
/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1678402374358/work/aten/src/ATen/native/TensorShape.cpp:3483.)
return VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
loading model from ./model_zoo/rvrt/model_zoo/rvrt/002_RVRT_videosr_bi_Vimeo_14frames.pth
Vimeo dataset is not at testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences! Please refer to #training of Readme.md to download it.
4
5
6
7
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0266/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0268/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0275/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0278/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0285/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0287/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0291/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0619/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0622/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0625/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0627/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0628/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0629/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0632/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0786/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0646/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0791/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0794/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0797/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0783/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0790/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0800/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0643/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0804/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0807/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0808/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0636/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0805/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0799/im1.png'): can't open/read file: check file path/integrity
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0812/im1.png'): can't open/read file: check file path/integrity
Traceback (most recent call last):
File "/home/dell/RVRT-main(2)/main_test_rvrt.py", line 362, in
[ WARN:[email protected]] global loadsave.cpp:244 findDecoder imread_('testsets/vimeo90k/vimeo_septuplet_matlabLRx4/sequences/00001/0809/im1.png'): can't open/read file: check file path/integrity
main()
File "/home/dell/RVRT-main(2)/main_test_rvrt.py", line 89, in main
for idx, batch in enumerate(test_loader):
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 634, in next
data = self._next_data()
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1346, in _next_data
return self._process_data(data)
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1372, in _process_data
data.reraise()
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/_utils.py", line 644, in reraise
raise exception
AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
data = fetcher.fetch(index)
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/dell/anaconda3/envs/test/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 51, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/dell/RVRT-main(2)/data/dataset_video_test.py", line 278, in getitem
imgs_lq = utils_video.read_img_seq(lq_path)
File "/home/dell/RVRT-main(2)/utils/utils_video.py", line 75, in read_img_seq
imgs = [cv2.imread(v).astype(np.float32) / 255. for v in img_paths]
File "/home/dell/RVRT-main(2)/utils/utils_video.py", line 75, in
imgs = [cv2.imread(v).astype(np.float32) / 255. for v in img_paths]
AttributeError: 'NoneType' object has no attribute 'astype'

How can I solve this issue? I really appreciate someone's reply!

Hi there, it would be nice to have "FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation" too:

git: https://github.com/MCG-NKU/FMNet

Thanks in advance and hope that inspires !

Hi, dear author,
I'd like to train an RVRT model on my dataset. Now I try to train successfully on your training dataset, however, there also requires a test set called Set8 which is unavailable by your shared link.
Could you please share the available link with me? Hope for your quick prompt reply! Thank you very much~

Hello 👋 Thank you for your awesome code. I want to understand how you train a network using 14 frames when Vimeo90K only provides 7 frame sequences.

I notice many sequences continue consecutively with subfolder ID (i.e. 00001/0003 and 00001/0004 seem from the same sequence). However, many sequences do not (i.e. 00001/0001 and 00001/0002 are from the different sequences).

Did you use a script to identify only sequences with 14 frames for grouping? Or did you group pairs of adjacent 7-frame sequences together using an odd+even ID value? Or maybe another method I've not thought of?

Thank you for your thoughts.

i give a .mp4 video as a input file , but code gaives me some error shown below:

RuntimeError: Given groups=1, weight of size [192, 4, 1, 3, 3], expected input[1, 3, 100, 64, 64] to have 4 channels, but got 3 channels instead

sr model runs well with images input , denosing model should input a video? is it right
what is the forth channel with video?

大佬好，啥时候能开源代码呢？论文有些地方没明白，想结合代码来看

python main_test_rvrt.py --task 006_RVRT_videodenoising_DAVIS_16frames --sigma 50 --folder_lq C:/RVRT/testsets/Set8/ --folder_gt C:/RVRT/testsets/Set8/ --tile 0 256 256 --tile_overlap 2 20 20
Traceback (most recent call last):
File "C:\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1740, in _run_ninja_build
subprocess.run(
File "C:\Python310\lib\subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\RVRT\main_test_rvrt.py", line 18, in
from models.network_rvrt import RVRT as net
File "C:\RVRT\models\network_rvrt.py", line 21, in
from .op.deform_attn import deform_attn, DeformAttnPack
File "C:\RVRT\models\op\deform_attn.py", line 19, in
deform_attn_ext = load(
File "C:\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1144, in load
return jit_compile(
File "C:\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1357, in jit_compile
write_ninja_file_and_build_library(
File "C:\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1469, in write_ninja_file_and_build_library
run_ninja_build(
File "C:\Python310\lib\site-packages\torch\utils\cpp_extension.py", line 1756, in run_ninja_build
raise RuntimeError(message) from e
RuntimeError: Error building extension 'deform_attn': [1/2] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\bin\nvcc --generate-dependencies-with-compile --dependency-output deform_attn_cuda_kernel.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=deform_attn -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Python310\lib\site-packages\torch\include -IC:\Python310\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Python310\lib\site-packages\torch\include\TH -IC:\Python310\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\include" -IC:\Python310\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c C:\RVRT\models\op\deform_attn_cuda_kernel.cu -o deform_attn_cuda_kernel.cuda.o
FAILED: deform_attn_cuda_kernel.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\bin\nvcc --generate-dependencies-with-compile --dependency-output deform_attn_cuda_kernel.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=deform_attn -DTORCH_API_INCLUDE_EXTENSION_H -IC:\Python310\lib\site-packages\torch\include -IC:\Python310\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Python310\lib\site-packages\torch\include\TH -IC:\Python310\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\include" -IC:\Python310\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c C:\RVRT\models\op\deform_attn_cuda_kernel.cu -o deform_attn_cuda_kernel.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.3\include\crt/host_config.h(160): fatal error C1189: #error: -- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2019 (inclusive) are supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.
deform_attn_cuda_kernel.cu
ninja: build stopped: subcommand failed.

Hi i treid to run the colab on custom dataset but i get the following error:

/usr/local/lib/python3.9/dist-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3483.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
loading model from ./model_zoo/rvrt/model_zoo/rvrt/001_RVRT_videosr_bi_REDS_30frames.pth
using dataset from testsets/uploaded
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py", line 1133, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.9/multiprocessing/queues.py", line 113, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 424, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.9/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.9/selectors.py", line 416, in select
    fd_event_list = self._selector.poll(timeout)
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 17693) is killed by signal: Killed. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/content/RVRT/main_test_rvrt.py", line 335, in <module>
    main()
  File "/content/RVRT/main_test_rvrt.py", line 68, in main
    for idx, batch in enumerate(test_loader):
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py", line 634, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py", line 1329, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py", line 1295, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.9/dist-packages/torch/utils/data/dataloader.py", line 1146, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
RuntimeError: DataLoader worker (pid(s) 17693) exited unexpectedly

Any idea?
Thanks :)

code : offset1 = offset1 + flows[0].flip(2).repeat(1, 1, offset1.size(2) // 2, 1, 1)
why flows need flip ?

Could you please give a brief explanation how --tile and --tile_overlap arguments affect quality and speed so that enthusiasts don't have to read the paper to use the code?

P.S. I noticed that the tile option should be at least 'x 64 64', otherwise the interpolate doesn't work, and that with only 4GBs of VRAM, I couldn't use the 'x 128 128' option for inference on REDS. Would be worth mentioning in the README.

Thanks!

The RAM increases sharply during the running of the main_test_rvrt.py, causing the process to be killed. I tried to lower the tile value, but it still didn't work. I ran this model on a Tesla P100. I would like to ask you how to solve this problem.

I encountered a problem where using the RVRT pre trained model to test 2k and 4k images for deblurring tasks is far less enhance than the VRT model, and the most important point is that the same processed frames are more than ten times larger than the original image, such as 1.jpg (1.3M), 1_enhanced. jpg (18M). Have any friends encountered the same problem?

Hello,

MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

• Video Upscalers Benchmark: Quality Enhancement determines the best upscaling methods for increasing video resolution and improving visual quality.
• Super-Resolution for Video Compression benchmark aims to test Super-Resolution methods on compressed videos and select the best model for each video codec standard.

If you are interested in participating, you can add your algorithm following the submission steps:

• Submit for Video Upscalers Benchmark: Quality Enhancement
• Submit for Super-Resolution for Video Compression benchmark

We would be grateful for your feedback on our work!