How to execute uniform quantization instead of NF4 quantization? about loftq HOT 4 OPEN

Hi, thank you for the interest of our work. LoftQ supports any existing quantization function in theory, but GPTQ implementation AutoGPTQ doesn't support dequantization, which is required in LoftQ (see Section 2.2 in LoftQ paper).

If you can find GPTQ implementation that has the dequantization method, please let me know. I'm glad to add it to LoftQ :)

from loftq.

Plus, we do have the experimental uniform quantization method at https://github.com/yxli2123/LoftQ/blob/main/glue/utils.py#L103. However, it's not the same uniform quantization used in GPTQ.

from loftq.

Hi, thank you for the interest of our work. LoftQ supports any existing quantization function in theory, but GPTQ implementation AutoGPTQ doesn't support dequantization, which is required in LoftQ (see Section 2.2 in LoftQ paper).

If you can find GPTQ implementation that has the dequantization method, please let me know. I'm glad to add it to LoftQ :)

Do you mean that vecquant4matmul is not a seperate dequantization function (dequantization + matmul) ?

from loftq.

@yxli2123 Thank you for providing experimental details. And congratulations to LoftQ for being accepted as a oral at ICLR 2024! It's sure that AutoGPTQ uses group-wise quantization and bit compression. Maybe LoftQ requires a custom dequantization function if it have to integrate into PEFT.

I found some related discussions about Pytorch-like Dequatization function:

Faster Pytorch dequantize() + matmul for quantized models

hqq_aten.cpp

A dequantization function seems to be implemented by offical pytorch:
FUNCTION AT::_WEIGHT_INT4PACK_MM

I hope the above information will help you.

from loftq.