What magnitude of avg loss indicates a relatively good result for a quantization model about autogptq HOT 6 OPEN

My rule of thumb is if your losses are > 1.0 for early [1-3] layers, calibration data is off or tokenizer is not properly configured. Each module in each layer has it's own loss trend in my experience. Some modules just are harder to quantize. MOE models are the worst-case for gptq due the gating/router layer.

• use running quant avg loss as guide to usable quant
• run ppl after quant for test 1
• human eval test for test 2

from autogptq.

My rule of thumb is if your losses are > 1.0 for early [1-3] layers, calibration data is off or tokenizer is not properly configured. Each module in each layer has it's own loss trend in my experience. Some modules just are harder to quantize. MOE models are the worst-case for gptq due the gating/router layer.

• use running quant avg loss as guide to usable quant
• run ppl after quant for test 1
• human eval test for test 2

Thanks for your quick reply! @Qubitium
My losses are lower than 0.05 in the first three layers as you mentioned above, but will eventually turns to above 10.0 in the last 40 layers, is it normal in your experience?
(ps: my model is a finetuned verison of Qwen-72b-chat, which has 80 layers in total.)
I'll test the 2 tests you mentioned above after i finish quantizing my model.

from autogptq.