How to disable flash_attention about mobillama HOT 6 OPEN

@OmkarThawakar Thanks, it's worked!
But i got another error like this "
attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: The size of tensor a (10) must match the size of tensor b (19) at non-singleton dimension 1 "
May the shape of q, k,v not matched!

from mobillama.

I also encounter the same problem

from mobillama.

Hi @lqniunjunlper and @Sunwood-ai-labs ,

Thanks for the interest in our work.

In place of flash_attention you can use default PyTorch attention.

Just replace line "from flash_attn import flash_attn_func" with "torch.nn.functional.scaled_dot_product_attention" .

from mobillama.

same

from mobillama.

@OmkarThawakar Thanks, it's worked! But i got another error like this " attn_output = torch.nn.functional.scaled_dot_product_attention( RuntimeError: The size of tensor a (10) must match the size of tensor b (19) at non-singleton dimension 1 " May the shape of q, k,v not matched!

Do you solve the problem?

from mobillama.

+1, @OmkarThawakar did u solve it?
thanks

from mobillama.