Insert the attention mechanism in the codes to the stylegan2-ada but fail about gigagan-pytorch HOT 7 CLOSED

@lucidrains Thanks for your kindness!
The results changed after a few modifications, and I will keep training to see what happens.
By the way, I am curious why you dont't let the style codes participate in the attention process? for example, let the style codes modulate the weights of q&k on self-attention.
Tks.

from gigagan-pytorch.

tks! You are so nice. Waiting for your continuing work.

from gigagan-pytorch.

@landian60 i can do a quick review of your integration code if you have a public repo or a gist

from gigagan-pytorch.

i'm actually not too sure if they also used the l2 distance attention for cross attention as well as the extra transformer blocks they appended to the clip text encoder

maybe i should make it an option to fallback to regular attention

from gigagan-pytorch.

glad to hear it is working!

the style codes already participate in modulating the convolutional kernels. as for the text conditioning, that is cross attended to by each image feature map token

from gigagan-pytorch.

Hello Phil,
I have a question about training the model with cross attention. I only used clip contrastive loss and did not add any other losses. And the generative results always became the same color and the G loss was very big even to NaN. I think the reason is the model collapse. Have you encountered similar issues when training the model? And have you noticed that some losses or well-designed discriminator containing some vital structures could help to stabilize the training process with the attention mechanism?
Tks.

from gigagan-pytorch.

@lucidrains

from gigagan-pytorch.