The performance on LIVE-VQA dataset. about dvqa HOT 6 CLOSED

As far as I know, the paper of this repository is C3DVQA: LINK.

I don't know if I'm right.

Thank you and hoping for replying.

from dvqa.

During training, I used 2 K80 GPU cards for 300 epochs in 38 hours, and the other parameters in opt.py are reserved not changed.

from dvqa.

@zhixuanli Thanks for your interest on this repo. I will try to address your concerns

During training, I used 2 K80 GPU cards for 300 epochs in 38 hours, and the other parameters in opt.py are reserved not changed.

The parameters in opt.py was optimized to train on the VideoSet dataset. If you really want to train the model from scratch, I would recommend to train on CSIQ first and adjust the parameters accordingly.
There is no guarantee the parameters in opt.py work well on different datasets, especially when the model is trained on different datasets independently.

As far as I know, the paper of this repository is C3DVQA: LINK.

I don't know if I'm right.

Thank you and hoping for replying.

Yes. The link points to the right paper.

Hi, I have trained the C3DVQA model on LIVE-VQA dataset, and the performance is very strange.

I find that in this line:

DVQA/dataset/LIVE/prep_live_score.py

Line 62 in 2172733

tst_mos.append(100.0 - float(mos[0]))

the mos should not be subtracted by 100.

Like what's shown in the following table:

change SROCC PLCC
in paper \ 92.61 91.22
origin 100-mos 30.46 29.3
bug fixed mos 31.43 42.2
And the performance is not as well as those in paper C3DVQA.

Can you help me? I'm new to this field. Thanks a lot!

Would you give some information that supports your claim ? the mos should not be subtracted by 100..
As far as I know, it is a common practice [1] [2] [3], to linearly rescale MOS such that a larger number indicates better perceptual quality.

The results, i.e. SROCC 0.2930 or 0.4220 are not reasonable. It is not straightforward to get good results when trained on scratch. Even so, the results should be much better, at least 0.87 for most epochs if the model is well trained.

Please read more paper if you want more background knowledge in the VQA field.

[1] Su, Shaolin, Qingsen Yan, Yu Zhu, Cheng Zhang, Xin Ge, Jinqiu Sun, and Yanning Zhang. "Blindly Assess Image Quality in the Wild Guided by a Self-Adaptive Hyper Network." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3667-3676. 2020.
[2] Yang, Sheng, Qiuping Jiang, Weisi Lin, and Yongtao Wang. "SGDNet: An End-to-End Saliency-Guided Deep Neural Network for No-Reference Image Quality Assessment." In Proceedings of the 27th ACM International Conference on Multimedia, pp. 1383-1391. 2019.
[3] Kim, Woojae, Jongyoo Kim, Sewoong Ahn, Jinwoo Kim, and Sanghoon Lee. "Deep video quality assessor: From spatio-temporal visual sensitivity to a convolutional neural aggregation network." In Proceedings of the European Conference on Computer Vision (ECCV), pp. 219-234. 2018.

from dvqa.

Really thank you for replying, and it's very helpful!

Actually, I'm not very sure about the mos should not be subtracted by 100, and I think your advice is right.

In the next, I'll train on CSIQ first and then adjust on LIVE-VQA.

Could you please provide the training hyperparameter （learning rate and so on) for CSIQ and LIVE-VQA?

Thank you!
:)

from dvqa.

Really thank you for replying, and it's very helpful!

Actually, I'm not very sure about the mos should not be subtracted by 100, and I think your advice is right.

In the next, I'll train on CSIQ first and then adjust on LIVE-VQA.

Could you please provide the training hyperparameter （learning rate and so on) for CSIQ and LIVE-VQA?

Thank you!
:)

Unfortunately I have left tencent for months. I don't have any extra data besides this repo.

from dvqa.

Haha that's so sad.

I'll try to contact the first author by email.

Thank you again for your patiently answering.

I have mailed you with my we-chat id.

If possible, hoping for further communication. :)

from dvqa.