replicating benchmarks about eccv2022-rife HOT 11 CLOSED

I add a Vimeo90K evaluation script and you will get 35.73PSNR as reported on our arXiv paper.
Please see README to check the reproducibility.

from eccv2022-rife.

1. Please don't feel pressured by me pointing this out, there is no need to rush updating the arXiv paper unless you want to. I have a paper with an inaccurate table as well (see sniklaus/3d-ken-burns#34) and have yet to update the arXiv version. Granted, the paper has already successfully been peer reviewed. ¯\_(ツ)_/¯
2. When comparing runtimes, it is in my opinion crucial to run all methods on the same hardware using the same software configuration. Take the 5 FPS for SepConv for 480p footage for example, which is what we stated in the Middlebury benchmark. Since the release of SepConv, GPUs have become much faster and so have libraries like cuDNN (they use better implementations of convolutions now). Therefore, I would expect SepConv to achieve much more than 5 FPS on 480p footage if it had been run nowadays using a Titan X (Pascal) and the newest cuDNN version.

I am happy to run benchmarks of your model(s) on my end as well if you want. If so, feel free notify me by posting in this thread or by sending me an email. My current benchmark suite computes PSNR, SSIM, and LPIPS for Middlebury (the samples with know ground truth), Vimeo-90k, UCF101 (the selection from Ziwei, which reminds me of something I will outline in the next paragraph), and Xiph (the selection from the SoftSplat paper) as shown in Table 5 of: https://arxiv.org/abs/2011.01280

The paper would probably be stronger if it were to compare to "Video Frame Synthesis using Deep Voxel Flow" in the related work. In essence, voxel flow is intermediate flow as well and it was the first paper that I am aware of that performed video frame interpolation by having a neural network estimate optical flow from the perspective of the frame that is ought to be synthesized.

from eccv2022-rife.

You can find the SoftSplat results here such that you can compute any metrics you like: sniklaus/softmax-splatting#24

The referenced Middlebury results are the "other" category. As for the public ones, it may be good to send Daniel Scharstein an email if you haven't heard back from your submission. He is (understandably) pretty busy though.

from eccv2022-rife.

Wow, I have seen a lot of your works. I will find a way to give benchmark code within today. This will take some time because our data is stored in a special format on our OSS system, so I need to try to switch to a public data reading method.

from eccv2022-rife.

I know that the calculation method of SSIM is not consistent with your paper. We use the matlab algorithm, which is consistent with the DAIN paper.

from eccv2022-rife.

Thank you for being willing to look into this and no rush!

from eccv2022-rife.

Thanks for sharing the script! I am a little confused, where is the quantization? It is common practice to calculate the error metrics with the interpolation result converted to uint8 (instead of doing it in float32 like in your script). In fact, all interpolation methods that I am aware of perform this quantization step in their evaluation. Not doing the quantization leads to better error metrics though. For example, the PSNR on Vimeo-90k for SoftSplat improves from 36.10 with the typical quantization to 36.14 without quantization. So it seems like the quantitative results in Table 2 are biased in favor of RIFE.

I am also curios as to how the computational runtime for the methods shown in Figure 1 is obtained. I am asking because it lists SepConv at about 5 FPS and RIFE at about 48 FPS, so roughly an order of magnitude difference. However, I just did a quick benchmark on my laptop (which has a GeForce GTX 1650) and it takes SepConv 0.57 seconds to interpolate a 480p frame whereas it takes RIFE 0.22 seconds, so about a factor of 2.6 and not an order of magnitude as shown in the paper. I was using the following code for SepConv and RIFE to benchmark the interpolation time, what am I doing wrong? Would the ratio between the two methods change significantly if I had used a Titan X (Pascal) like in the paper?

torch.set_grad_enabled(False)
torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark = True
a = torch.rand(1, 3, 480, 640).cuda()
b = torch.rand(1, 3, 480, 640).cuda()
for i in range(100000):
	before = time.time()
	c = model.inference(a, b) # for sepconv: c = netNetwork(a, b)
	torch.cuda.synchronize()
	after = time.time()
	print(after-before)

Thanks again for sharing your code, it is a great contribution to the area of video frame interpolation! It may also be a good idea to release your RAFT-Large model since it is what future interpolation papers that do not focus on computational efficiency should compare to.

from eccv2022-rife.

1. Thank you very much for letting me know that. I will retest all the PSNR reported in paper and update the arXiv in 3 days according to your advice. I believe it will not destroy the qualitative results. We have updated the evaluation script. If you don't mind, can you re-test SoftSplat's SSIM using our script? It will be very helpful to us.
2. For running speed, I think there is no problem with your script. I repeatedly test RIFE on TITAN X (Pascal) using your script and believe our results are accurate. And we quote the results of SepConv from DAIN(https://arxiv.org/pdf/1904.00830.pdf). I will try to retest Sepconv's speed using your repo if you think the results we quote are not reliable enough.

Reproducibility is the direction I am working on now. I will add the test scripts for other benchmarks and release more models. I think it is very important to get your approval for this paper, I hope you will continue to pay attention.

Thanks again.

from eccv2022-rife.

1. I have confirmed our Vimeo90K evaluation script and hope you can test SoftSplat's SSIM. For Middlebury, I used the OTHER dataset with GT because there is no feedback on the results I submitted on the official website. If you can provide the result of SoftSplat running on the OTHER set, we will add it to the paper.
2. I haven't read DVF paper, I will read it and add it to our paper.

from eccv2022-rife.

Any update on when we get to verify the RIFE-Large results from the paper?

I also just looked at the benchmark script again and the SSIM is still computed pre-quanitfication which puts the other methods at a disadvantage.

https://github.com/hzwer/arXiv2020-RIFE/blob/857e8b7842e18c6e30a08e1c868eb38baedc391b/benchmark/Vimeo90K.py#L34

from eccv2022-rife.

@sniklaus I added RIFE-Large just. I also re-tested SSIM using pre-quanitfication and found very slightly difference.

from eccv2022-rife.