Is there any way for using LoftQ to GPTQ or AWQ model? about loftq HOT 2 OPEN

Hi @LameloBally, thanks for your interest of our work.

Unfortunately the answer is no. LoftQ aims to provide a data-free initialization for all downstream task finetuning, so quantization with data calibration is out of our scope. However, we welcome you to explore more possibilities of our method: you can obtain a quantized backbone (Q) from GPTQ or AWQ and then initialize LoRA adapters, A and B, by SVD(W-Q), where W is the full-precision pre-trained weight.

Let me know if you have further questions.

from loftq.

@yxli2123 Thanks for your kind answer. It was very helpful for me!!
But, I have another question. How Can I initialize LoRA Adapter by SVD(W-Q) s.t. Q is GPTQ model & W is full precision.
I thought LoftQ is very innovative idea so I want to apply it to other cases to develop it further.
It will be very helpful for me if you let me know How can I find any guide or the way I can do it.

Thanks!

from loftq.