Adapter: v1 vs v2 implementation about llama-adapter HOT 2 CLOSED

You are totally right! The two implementations are mathematically equivalent: Given that $Q, K_1, K_2, V_1, V_2$ are matrices of queries, keys and values with shapes seq_len x dimension, we can find that

$$ [ Softmax(QK_1^T) , \alpha \cdot Softmax(QK_2^T) ][ V_1^T, V_2^T ]^T = Softmax(QK_1^T) V_1 + \alpha \cdot Softmax(QK_2^T) V_2 $$

with the left hand side being the v1 implementation and the right hand side being v2.

from llama-adapter.

Cool! Thanks @linziyi96 for such a detailed response 👍

from llama-adapter.