[FR]: Whisper int4 support about olive HOT 14 CLOSED

Hey it works with a really great result. Whisper-int4

from olive.

A simple experiment:
model: whisper-medium
experiment: 15s audio running 10 times.

in conclusion:

Running int4 on the CPU seems to be limited by the CPU processing speed, resulting in 40% slower than int8 and 1G less memory.
Int4 on the GPU runs 7 to 8 times faster than int8. At the same time, the computing resource usage is close to 100%. The memory has increased by 1G. Int8 cannot fully utilize the performance of 4090.
Overall, Int4 should be faster but requires more computing resources.

from olive.

Hey. I made rud test
Driver Description: NVIDIA GeForce GTX 1660 Ti with Max-Q Design
Video Memory: 6144 MBytes of GDDR6 SDRAM [Micron]

1272-141231-0002.mp3
File duration : 13s 428ms

whisper-large-v2-int8
Transcribe duration avg : 16s 713ms
realtime ratio 0.803458437

whisper-large-v2-int4
Transcribe duration avg : 2s 963ms

faster in 5.640482619 times
realtime ratio 4.531893351

I tested in C# demo app

from olive.

"blockwise_quant_int4":{
"type": "OnnxMatMul4Quantizer",
"disable_search": true
},

"bnb_quantization": {
"type": "OnnxBnb4Quantization",
"config": {
"quant_type": "fp4",
"save_as_external_data": true,
"all_tensors_to_one_file": true
}
},

"bnb_quantization": {
"type": "OnnxBnb4Quantization",
"config": {
"quant_type": "nf4",
"save_as_external_data": true,
"all_tensors_to_one_file": true
}
},

bnb_quantization fp4 has better performance than nf4 and blockwise_quant_int4. Compared with blockwise_quant_int4, the performance is improved by 40%.

15s audio running 50 times on nvidia 4090.

from olive.

I did not compare the latency with fp32/fp16, but the int4 should work for olive whisper example, you can just change the int8 dynamic quantization as above config. Like:

1. run whisper prepare config to generate a few configs fp32/fp16/int8 and etc.
2. pick one config to add int4 config.
3. run it with olive as the whisper readme shows.

from olive.

@trajepl Thanks I will try it later

from olive.

Hey it works with a really great result. Whisper-int4

Do you mind paste some perf numbers? Would love to see the exciting results. :)

We recently are adding other int4 supports which may help if you met any performance issues.

from olive.

Wow, great to see this big improvements!

from olive.

I ran the inference of the original model (GPU) through the following command. The speed of audio inference for 5 minutes is similar to that of int4 onnx (GPU), and the memory usage is about twice as much. Is there any problem with the int8 onnx model (GPU) that prevents full use of gpu resources?
gpu inference (4090 utilization 90%):
time whisper 1.wav --language zh --model medium

real 0m28.003s
user 0m40.866s
sys 0m3.852s

from olive.

Thanks for the comparison! Currently, the int8 is not supported very well in onnxruntime gpu. But fortunately, the supporting is coming.

I do not know the detailed reason whey int8 does not fully utilize the gpu memory. But I suppose it might be caused by some operations(quantization) are not optimized in gpu.

Also, we have similar experiences before, that leads following convention:

1. For gpu, int4, fp16 are better, and avoid apply int8 quantization as of now.
2. For cpu, avoid fp16. For int4 there are several passes to get the int4 model, we need to search for better performance.

from olive.

Excellent! Will try to add the official example in Olive later.

from olive.

The whisper variants I currently use include faster-whisper, whisper.cpp, and olive-whisper. If sorted by inference speed (gpu nvidia 4090), these three are almost the same. If sorted by memory usage (from small to large), faster-whisper->whisper.cpp->olive-whisper.
The memory footprint of whisper.cpp or faster-whisper is half that of olive-whisper.
@trajepl Is there a way to optimize the memory footprint of olive-whisper?

I prefer using the onnx model and don't want to give up the olive whisper model.

from olive.

I have no idea about how to optimize memory footprint.
It might not be a fair comparison? The factors like input data size, batch size, or memory cache may affect the memory footprint.

from olive.

Closing issue since int4 quantization is already supported.

Memory footprint is out of scope for Olive and handled my onnxruntime https://github.com/microsoft/onnxruntime.
There have been multiple int4 improvements made in ORT so it might have already been addressed by them.

from olive.