Support whisper multi-lingual model? about olive HOT 5 CLOSED

Hi @jambayk
Thanks for the response & hard work.
I will test whisper multilingual model in a few days.
And will report the result.
thanks again for your hard work.

from olive.

Thanks for bringing this up! Our whisper example currently doesn't support multi-language support since the changes related to it were introduced in a newer unreleased version of onnxruntime. However, we have plans to add support for ort > 1.15 (nightly versions) and multi-language.

I will link this issue when the corresponding PR is made.

from olive.

Hi. @jambayk
Thanks for the response.
I hope that the work will go smoothly.

from olive.

@gyujin-kim The linked PR above added multilingual support. Please refer to the updated Whisper example readme for details on how to use it.

from olive.

Hi @jambayk

I have succeeded in creating a whisper onnx model and testing it.

• olive for convert : commit e559816
• onnxruntime for convert & inference : ort-nightly 1.16.0.dev20230625001
• inference test : use test_transcription.py

And I succeed to convert the other whisper model(ex. whisper-small) but recognized text quality is poor than torch model and onnx encoder-decoder model.
(https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_onnxruntime.ipynb)
Testing several languages by tweaking 'decoder_input_ids' in the 'whisper_dataset.py' file, the quality loss was worse in non-English languages.
I can't figure out what could be the problem.
Any suggesions to check this problem?

Edit1:
I have test with int8 model. I will test another precision and re-checking.
Edit2:
I have test with fp32 model and got better result.
But still worse than original model output.
I did't test lot of input, I will be update the status later after more test.
thanks.

from olive.