Why is important to change the order of content['q'] and <video>? about video-chatgpt HOT 3 CLOSED

Hi @shufangxun,

Thank You for your interest in our work. The comment is referring to the importance of adding <video> token to the annotation which is going to be replaced with the actual video content during training. It is not referring to the order of content['q'] and <video>. However, during training this alternate order could be considered as form of regularization.

Please let me know if it clarifies the confusion. Thank you

from video-chatgpt.

Thanks for your reply! by the way, how much is the performance difference between these two ways？

from video-chatgpt.

Hi @shufangxun,

In our training, in half of the data samples we kept the text prompt after the video tokens and in other half before the video tokens. However, we do not conduct any ablation to find which method is more effective. The performance difference should be negligible as we are not updating the LLMs' parameters.

Please let me know if you have any questions or insights from your experiments.

from video-chatgpt.