can quantized onnx t5 run on GPU about fastt5 HOT 3 CLOSED

unfortunately no. quantized onnx models can only be run on CPU & onnxruntime-gpu does not support quantization.
if you want more details on this question you should create an issue in onnxruntime.

from fastt5.

@Ki6an HI, do we support onnxruntime-gpu mode with V100 or T4 if dont use the quantization?

i see with onnxruntime only we can have x2 speedup ?

from fastt5.

@mblank5 no, this library uses onnxruntime, and to support GPUs you need to have onnxruntime-GPU installed.

BUT you can uninstall onnxruntime after the fastt5 library is installed, and install onnxruntime-gpu and try running the model but not sure you'll get speed up. for more info refer to this issue.

with onnxruntime, you'll get speed up if you are using modern CPUs and more CPU cores. refer benchmark section of README.

from fastt5.