SVD Implementation in loftQ Algorithm about loftq HOT 3 CLOSED

Thanks for your inserts of our work. Setting either full_matrices=True or full_matrices=False will not affect the results. For square matrices, they are the same. For non-square matrices, reduced SVD is also the same as full SVD but removes the zero singular vectors. In our work, we are actually using truncated SVD, which only obtain the first r singular vectors.

from loftq.

Thank you for your kind response. I have some additional questions regarding the LoftQ algorithm.

I am struggling to intuitively understand how repeatedly performing quantization and SVD approximation leads to progressively better initialization at adapter weight.

If we rewrite LoftQ Algorithm 1 with an added Error Term, it looks as follows: ($\epsilon$ is a error term)

As in Equation 3, when we approximate the difference between $W_{FP}$ and the $Q_t + A_tB_t$ using SVD, let's say the SVD approximation error is $\epsilon^{svd}_t$.
​
I have personally measured how this error term ($\epsilon^{svd}_t$) changes across layers with each iteration.
The results show that in all layers, this svd error term decreases as the number of iteration steps increases.

In summary, as the steps in the LoftQ algorithm increase, the SVD approximation becomes more accurate, effectively minimizing the main objective, as stated in eq.6 of the paper. However, I am not entirely clear on why this error minimizes through the repetition of these two steps (1)Quantization 2)SVD). Could you please explain this once more?

I conceptually understand how the initialization of quantized weight and adapter weight are jointly optimized but it is not clear to me why this process minimizes $W_{FP} - Q - AB^T$ analytically. (maybe I miss something)
I would greatly appreciate additional clarification on this, as it would help me deeply understand the core idea of this excellent paper.

from loftq.

Hi @MarsJacobs, the error decreasing as the step increasing is not guaranteed. This algorithm is heuristic. For some models, like some layers in DeBERTa-v3-base, the error fluctuates as steps increases.

from loftq.