Static quantization issue about optimum HOT 4 CLOSED

We have recently enabled the pytorch fx graph mode for quantization aware training and post-training quantization, so no need to add QuantStub and DeQuantStub in the model anymore. If you are interested on using the pytorch eager mode for post-training quantization and on how the model was modified by the Intel Neural Compressor team, you can take a look at : https://github.com/intel/neural-compressor/blob/master/examples/pytorch/eager/language_translation/ptq/transformers/modeling_bert.py

from optimum.

Thanks a lot @echarlaix. Works like a charm with torch FX graph mode. However, when running the test (https://github.com/huggingface/optimum/blob/main/tests/intel/test_lpot.py) for static quantization and changing the batch size, I always run in an error if the number of samples divided by the batch size has a remainder and the last batch then has less samples than the actual batch size (e.g. with batch size 16):

[ERROR] Unexpected exception RuntimeError("shape '[16, 128, 12, 64]' is invalid for input of size 786432") happened during tuning.

Is there some way to circumvent this?

from optimum.

Currently our tracing of the model with torch fx does not support dynamic inputs shapes and we are currently working towards it. In the meantime, a simple fix could be to set dataloader_drop_last of the TrainingArguments to True.

from optimum.

Would definitely be a good solution for training. Since I do not want to loose any samples at inference, I just duplicated the last sample to fill up the batch and later on removed it again :).

I'll close the issue since my initial problem is fixed with torch fx. Thanks for your help!

from optimum.