DCVC-FM model is only applicable to datasets converted using the BT.709 standard under RGB test conditions? about dcvc HOT 5 OPEN

Thanks for the comments. We are focusing on YUV420 color space as most of traditional video codecs. We find that the same model could also work for RGB content (at least using BT.709 conversion). We did not test other RGB content. Would you please check whether the following two lines using the correct conversion matrix as expected (if not using BT.709 matrix)?

https://github.com/microsoft/DCVC/blob/4df94295c8dbe0a26456582d1a0eddb3465f1597/DCVC-FM/src/utils/test_helper.py#L88C9-L88C37
https://github.com/microsoft/DCVC/blob/4df94295c8dbe0a26456582d1a0eddb3465f1597/DCVC-FM/src/utils/test_helper.py#L123C1-L123C72

from dcvc.

Yes, I used the right matrix. My test pipeline is [1. read PNG files into RGB]->[2. convert RGB into YcBcR using function "rgb_to_ycbcr444"(src.transforms.functional)]->[3. compress the YcBcR frames with DCVC-FM model]->[4.convert YcBcR back into RGB using function "ycbcr444_to_rgb"(src.transforms.functional) ].

I surmise that the probable cause of the issue is the amplification of information loss during the transitions from RGB to YCbCr444 and back to RGB in steps 2 and 4, respectively. This loss is likely exacerbated for source frames or datasets that have not undergone conversion using the BT.709 standard.

But, traditional codecs/previous RGB-space neural codecs appear to be more robust against variations in the color space conversion methods used for source frames.

Could you please fine-tune a model for RGB input/output (I mean a model specifically for RGB frames, without YUV conversion) and release this model? I think just a few steps are enough to get this model. But since I don't have a training strategy, I might need to trouble you to go through this fine-tuning process. This will help identify the problem. Thanks! !

from dcvc.

As you mentioned, rgb_to_ycbcr444 and ycbcr444_to_rgb in src.transforms.functional were used to for color space conversion. However, BT.709 is assumed in these two functions.
If you RGB is not converted from BT.709, I would suggest modifying rgb_to_ycbcr444 and ycbcr444 (or add new functions) to use the correct conversion matrix.

from dcvc.

Thanks, I will try that.

from dcvc.

As you mentioned, rgb_to_ycbcr444 and ycbcr444_to_rgb in src.transforms.functional were used to for color space conversion. However, BT.709 is assumed in these two functions. If you RGB is not converted from BT.709, I would suggest modifying rgb_to_ycbcr444 and ycbcr444 (or add new functions) to use the correct conversion matrix.

Could you release the traing codes of DCVC-FM，since we want to do the fine tuning in our dataset. Thanks.

[email protected]

from dcvc.