I have some doubts about the framework about text-to-video-finetuning HOT 4 CLOSED

Thank you again for your explanation～

from text-to-video-finetuning.

Hi, @ExponentialML

This is a really useful repo, but I have a question to ask. In the video fusion paper, it seems that they decouple the denoising process into base noise and residual noise. However, I couldn't find this in the pipeline of diffusers, which confuses me. Is this a completely new version?

Hey! In all honesty, I cannot verify that the Pipeline used from ModelScope's repository is verbatim with the paper. I only implemented this repository after the initial release of the ModelScope video diffusion model (referencing showlab's Tune-A-Video's repository for training).

Also, I havn't yet referenced any of the paper's implementations, but loosely follow others in same field. When I was asked as to which paper this was referencing, I found that the paper (assuming you found it on paperswithcode) and linked it as the closest candidate.

In terms of base noise along with a residual, I don't think that this would be too difficult to implement. I'll give it a go as a side experiment (not on my todo list at the moment), but in the mean time if you would like to, it would be great to open a PR and attempt to implement it.

from text-to-video-finetuning.

So for this repo, to achieve video generation, it just added a temporal layer in the unet, right? (Perhaps this is not a rigorous way to put it, but that's roughly what I see from the code). The reason I raised this issue is just to confirm if I missed any code details.

Regarding the implementation of basic noise and residual noise denoising you mentioned, I'd be happy to submit a PR for it. I'll implement it soon.

from text-to-video-finetuning.

That's the correct assumption. Loosely, there are temporal layers in the form of:

(b h w) f c batch, height, width, frames, channels

Then, double self attention is used such that it replaces the cross attention layer (text input in the majority cases for CrossAttention) before the feed forward. 3D temporal convolution layers are also added.

Thanks for willing to submit the PR! Looking forward to it.

from text-to-video-finetuning.