Why are base weights on HF LoftQ models in 16-bit? about loftq HOT 2 OPEN

Hi, @RonanKMcGovern . In the old version of bitsandbytes, it is not allowed to save nf4 format weight. We avoid this issue by saving it in 16 bits in the disk and transforming it into 4 bits when loading it into GPU. However, as the bitsandbytes gets updated recently, it is possible to save it in nf4 format. We will update the code soon.

from loftq.

Ok, but are you even running the nf4 quantization then?

Or are you just directly saving the bf16 weights? If you're doing that, there is going to be error when reloading the model because the saved bf16 should be the dequantized weights, not the original...

Seems to me something is off because doing even one iteration of loftQ should improve results, but I see worsening results for 1 iteration and more (see this vid), as does kaitchup.substack.com

from loftq.