你好，我注意到代码中支持int8下的推理。请问有关于_quantize_int_activation_函数使用的例子吗

I just run the code with any changed and I got the different accuracy like:
0.7599 as vit_small_patch32_224 (float)
PTQ4VIT: 0.7541 as vit_small_patch32_224 (w8,a8) BasePTQ: 0.74856 vit_small_patch32_224 (w8,a8)
PTQ4VIT: 0.7174 as vit_small_patch32_224 (w6,a6) BasePTQ: 0.64272 vit_small_patch32_224 (w6,a6)

and other models have the same problem
I didnt change any in original code
Is it because we're using different libraries? Versions of my library:
Python==3.10.6
timm==0.6.13
torch==1.12.1

请问量化后，分类准确率衰减与其它下游任务如目标检测、语义分割等，效果会基本一致吗？或者作者您对目前大多数的量化工作，只在分类上验证效果，有什么见解吗？

Hi, thank you for your great work and excellent code. I would like to know if it is possible to add extra input token after calibration, which has fixed the padding parameters.

I got the weight file after successful quantization, but I found that the weight file after quantization is inconsistent with that before quantization, how can I read the int8 weight file correctly.

I am wondering why the taskloss in the code is kl_div(log_softmax(quant_out),softmax(fp32_out)), which seems to be misaligned with the formula $$\frac{\partial L}{\partial O^l}$$ in paper

Hi, thanks for sharing the work!

I meet a problem when trying to load the "vit_base_patch16_224.pth". The shape of 'blocks.0.attn.qkv' in pth file is torch.Size([3, 1, 2304, 768]). However, the shape of 'blocks.0.attn.qkv.weight' in model should be torch.Size([2304, 768]). What does the first and second dimension in torch.Size([3, 1, 2304, 768]) mean? I think it should be torch.Size([2304, 768]).

In the batching_quant_calib() function of the HessianQuantCalibrator class in quant_calib.py,
if self.sequential is set to False (the default value in the provided code), the quantization interval is not reflected in the calibration step of the next module after calibrating the current module.
In this case, when calculating loss = KL_div(y, y_hat), instead of computing it between the output y of the full-precision (FP) model and the output y_hat of the quantized model, it would be calculated between two outputs of the FP model (i.e., KL_div(y, y)).

So, should self.sequential be set to True?
However, when setting self.sequential=True, the accuracy is reported to be very low.
Could you please clarify this issue and provide guidance on the appropriate setting for self.sequential?

There are traditional convolution and TRANSFORMER in my model, which are applied to image generation. Can this method be used for quantification?

First of all, thank you for the great work and the official code.

I have one question.

Where is the code implementation for constraining the scaling factors for post-softmax and post-gelu, i.e., ∆R2 = 2m∆R1, where m is an unsigned integer, in order for an efficient process.

I really appreciate for providing the code once again.

Hi, may I ask for more details on what this function does?

   def _initialize_calib_parameters(self):
        """ 
        set parameters for feeding calibration data
        """
        self.calib_size = int(self.raw_input.shape[0])
        self.calib_batch_size = int(self.raw_input.shape[0])
        while True:
            numel = (2*(self.raw_input.numel()+self.raw_out.numel()) /
                     self.calib_size*self.calib_batch_size)  # number of parameters on GPU
            self.parallel_eq_n = int((15*1024*1024*1024/4)//numel)
            if self.parallel_eq_n <= 1:
                self.calib_need_batching = True
                self.calib_batch_size //= 2
            else:
                break

I am adapting your code for 1DConvolution and the self.parallel_eq_n is multiplying by the output channels of the weights and causing memory issues. If you could provide further details it would be really helpful. Thankyou. Ed

请问该方法产生的8 bit量化模型，理想实现下在运行时能减少多少显存占用呢

Dear author, thank you for your outstanding contribution. But I'm having some problems running the program:
I used the dataset Imagenet2012 in your program and did not change any parameters (I found the α of PTQ4ViT =0.01 in the program), but there is a little difference between the result of several attempts and the original result. For example, in the result of PTQ4ViT, your ViT-B/224 accuracy rate of w8a8 is 84.25, but mine is 84.148. Your ViT-B/224 accuracy rate of w6a6 is 81.65, but mine is 81.844, and so on. I use an Nvidia RTX 3090-24G.
Did I do something wrong? We look forward to your reply. Thanks again!

I cannot find your most import code of Searches for the optimal scaling factors of each layer.
Could you mind release it, thanks

I download one quantized model "vit_small_patch16_224.pth", but when I tried the following code:

pthfile = "vit_small_patch16_224.pth"
net.load_state_dict(torch.load(pthfile))

I got this error:

RuntimeError: Error(s) in loading state_dict for VisionTransformer:
Missing key(s) in state_dict: "cls_token", "pos_embed", "patch_embed.proj.weight", "patch_embed.proj.bias", "blocks.0.norm1.weight", "blocks.0.norm1.bias", "blocks.0.attn.qkv.weight", "blocks.0.attn.qkv.bias", "blocks.0.attn.proj.weight", "blocks.0.attn.proj.bias",...

I guess this may be caused by not saving the bias? I was also wondering the correct way to load that model.
Really thank you for your help!

你好，请问在推理过程中选用的是Imagenet 验证集吗？我看代码中datasets是 train 和 val 两部分，不需要用到测试集吗？谢谢

Excellent work! After quantization, how to save the quantized model?

Thank you for your excellent work. I have a question about the reasoning process.
Regarding the reasoning process:
y =x * w + b
Quantitative reasoning process:
y = (x_int * _int) *( w_interval * a_interval )
This can achieve the x * w process of int8 type.

But in the source code, I noticed that the quantization linear layer such as MinMaxQuantLinear() has the following implementation:
The quant_weight_bias (self) and quant_input (self, x) methods:
w_sim=w.mul_(self.w_interval)
x_sim.mul_(self.a_interval)
This means that after quantifying the activation values and weights, they are immediately inverted to float32, which means that in actual operations, it is still float32. This kind of reasoning seems to have no acceleration effect. Is my understanding correct?

If I directly run the test_all.py, will I save the quantized parameters such as weights and bias automatically in somewhere?

Does this framework support SwinTransformer3D quantification?

Hi,
Are the quantized models differentiable(i.e. can we get gradients using backprop using these) or is it not possible due to actual INT8 quantization? Pl reply ASAP

Hi!
Thanks for your great work! There was a problem when I was trying to load the quantized model 'vit_base_patch16_384.pth' (you've provided) into the net created by timm.create_model. The error is as follows.

RuntimeError: Error(s) in loading state_dict for VisionTransformer:
Missing key(s) in state_dict: "cls_token", "pos_embed", "patch_embed.proj.weight", "patch_embed.proj.bias", "blocks.0.norm1.weight", "blocks.0.norm1.bias", "blocks.0.attn.qkv.weight", "blocks.0.attn.qkv.bias", "blocks.0.attn.proj.weight", "blocks.0.attn.proj.bias", "blocks.0.norm2.weight", "blocks.0.norm2.bias", "blocks.0.mlp.fc1.weight", "blocks.0.mlp.fc1.bias", "blocks.0.mlp.fc2.weight", "blocks.0.mlp.fc2.bias".......

Would you please provide the correct method to load your quantized model? Thank you a lot~