Implementation about classifier free guidance about arldm HOT 8 CLOSED

If you just discard some frames, and remain the other frames in the training, it's like MAE, let model learn how to generate the next frame based on the corrupted previous frames (let's say if we are generating the frame 3, and your free idx goes to the frame 1, that means the stable diffusion is trying to learn how to diffuse from (ClIP[3]+BLIP[0, 2]) context. But this method doesn't align how you do sampling.

from arldm.

Hi @skywalker00001, thank you for your comment. Actually, you can treat our batch size as B * V, which means we generate every single frame according to the previous frames in a story. We do drop all clip and blip conditions at the classifier_free_id frames. And for other frames, the conditons are not corrupted, as you can see we save a copy in

And the code

you mentioned is actually following the implementation of Diffusers

https://github.com/huggingface/diffusers/blob/b9b891621e8ed5729761cc6a31b23072315d2df0/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py#L686

actually the $scale=1+w$, the two formula is the same

from arldm.

Hi, xichenpan, thank you for your answer!
I see. Does that mean you actually trained 4 extra unconditional models (let's say our task is continuation ) for each frames?
unconditional model 1: [null clip+ null blip frame 0]
unconditional model 2: [null clip+ null blip frame0 + null blip frame1]
unconditional model 3: [null clip+ null blip frame0 + null blip frame1 + null blip frame2]
unconditional model 4: [null clip+ null blip frame0 + null blip frame1 + null blip frame2 + null blip frame 3]
And use attention mask = 1 to mask the future frames in each model.
Is my interpretation right?

And for the second question, yeah, you are right, the two formulas are the same.
Thanks again for your patience!

from arldm.

Besides, I encountered another problem.
How do you calculate the FID score?
I debugged and output the shape of "original_images" and "images"

original_images has the shape: # (4, 3, 128, 128), but don't we need to permute it before turn it to PIL image?
Because the "original_images" now has the PIL image shape 128 * 3 but "images" has the PIL image shape of 512 * 512 (the stable diffusion output size).

I know in the code

And I use [transforms.ToPILImage()(self.fid_augment(im)).save("faked_pics/ori_{:02}.png".format(idx)) for idx, im in enumerate(images)], the generated images are all corrupted for "original_images". But for the generated "images", there will not be error. the "transforms.ToTensor()(images[0])" has the shape (3, 512, 512) and PIL save the correct image.

Therefore, I suppose should it be

original_images = original_images.permute(0, 2, 3, 1).cpu().numpy().astype('uint8')

in the line 316?

If that is the case, maybe the FID scores will be changed too...

from arldm.

And I think the multi-GPU inference is supported by Pytorch Lightning.

I only add "strategy="ddp"," in the Trainer, and set "args.gpu_ids" to [0, 1, 2, 3] and comment your code in line

It succeed. So I think it may help others.

for

from arldm.

@skywalker00001 Hi, for the frist issue, it is one single uncond model (with varied length), because all params are shared.
For the second issue, thanks for pointing that out, I found in our original implementation, we do not permute the original_image, and it has a shape of BHWC

from arldm.

Got it! Thanks again.

from arldm.

from arldm.