any advice to solve Pretrained PWGVocoder is currently not supported?
thanks

Thanks for creating this and simplifying onnx export.

1. Do you have a timeline for the transducer ASR model support?

from espnet2.bin.asr_inference import Speech2Text
import os
import yaml
from espnet_onnx.export import ModelExport
from pathlib import Path

m = ModelExport()

os.chdir("egs2/librispeech_100/asr1/")


transducer_conf = yaml.safe_load(Path('conf/decode_rnnt_conformer.yaml').read_text())

speech2text = Speech2Text(asr_train_config="exp/asr_train_rnnt_conformer_raw_en_bpe5000_sp/config.yaml",
                          asr_model_file="exp/asr_train_rnnt_conformer_raw_en_bpe5000_sp/latest.pth",
                          transducer_conf=transducer_conf["transducer_conf"],
                          lm_weight=0.0)

m.export(speech2text, 'onnx_export', quantize=True)

^ I was trying to export a sample transducer checkpoint like this and was running into attribute errors at espnet_onnx/export/asr/export_asr.py" _create_config function.

2. Can you comment if the above snippet is correct? If I just provide None to attributes that dont exist (Eg: model.beam_search), would the conversion still work ?

Hi,

First, thanks a lot for your work on onnx conversion of espnet models!

I am trying to convert a streaming Conformer-Transformer model (encoder: contextual_block_conformer, decoder: transformer) from espnet2 to onnx format.
It is not a pretrained espnet_zoo model but I trained it on my own data.

My call is this:

import sys
from espnet_onnx.export import ASRModelExport

model="./asr_train_asr_streaming_conformer7_n_fft256_hop_length128_conv2d_n_mels40_medium_raw_de_bpe5000_sp_valid.acc.ave.zip"
tag_name="streaming_conformer-transformer"

m = ASRModelExport()
m.export_from_zip(
  model,
  tag_name,
  optimize=True,
  quantize=True
)

I am getting the following error:

  m.export_from_zip(
  File "/home/espnetUser/scm/external/espnet_onnx/espnet_onnx/export/asr/export_asr.py", line 191, in export_from_zip
    self.export(model, tag_name, quantize, optimize)
  File "/home/espnetUser/scm/external/espnet_onnx/espnet_onnx/export/asr/export_asr.py", line 92, in export
    model_config.update(encoder=enc_model.get_model_config(
  File "/home/espnetUser/scm/external/espnet_onnx/espnet_onnx/export/asr/models/encoders/contextual_block_xformer.py", line 188, in get_model_config
    frontend=self.get_frontend_config(asr_model.frontend),
  File "/home/espnetUser/scm/external/espnet/tools/anaconda/envs/espnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1185, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'ContextualBlockXformerEncoder' object has no attribute 'get_frontend_config'

I also tried converting two pretrained espnet models:

Non-streaming model:

kamo-naoyuki/librispeech_asr_train_asr_conformer6_n_fft512_hop_length256_raw_en_bpe5000_scheduler_confwarmup_steps40000_optim_conflr0.0025_sp_valid.acc.ave.zip

This worked fine for me.

Streaming model:

Emiru_Tsunoo/aishell_asr_train_asr_streaming_transformer_raw_zh_char_sp_valid.acc.ave.zip

Here I am getting the same AttributeError: 'ContextualBlockXformerEncoder' object has no attribute 'get_frontend_config' error as for my own streaming model.

Any suggestions what is going wrong? Do I need a special espnet/python etc. version for the streaming models?

Thanks!

the pretrained model got from
https://github.com/espnet/espnet/tree/master/egs2/librispeech/asr1
Conformer-RNN Transducer
Environments
date: Wed Apr 27 09:30:57 EDT 2022
python version: 3.8.5 (default, Sep 4 2020, 07:30:14) [GCC 7.3.0]
espnet version: espnet 0.10.7a1
pytorch version: pytorch 1.8.1+cu102
Git hash: 21d19be00089678ca27f7fce474ef8d787689512
Commit date: Wed Mar 16 08:06:52 2022 -0400
ASR config: conf/tuning/transducer/train_conformer-rnn_transducer.yaml
Decode config: conf/tuning/transducer/decode.yaml
Pretrained model: https://huggingface.co/espnet/chai_librispeech_asr_train_conformer-rnn_transducer_raw_en_bpe5000_sp

export onnx script:
from espnet2.bin.asr_inference import Speech2Text
import os
import yaml
from espnet_onnx.export import ASRModelExport
from pathlib import Path
m = ASRModelExport()
m.set_export_config(max_seq_len=5000)
transducer_conf = yaml.safe_load(Path('espnet/egs2/librispeech/asr1/conf/decode_rnnt_conformer.yaml').read_text())
speech2text = Speech2Text(asr_train_config="test/espnet_onnx2/rnnt/exp/asr_train_rnnt_conformer_ngpu4_raw_en_bpe5000_sp/config.yaml",
asr_model_file="espnet_onnx2/rnnt/exp/asr_train_rnnt_conformer_ngpu4_raw_en_bpe5000_sp/valid.loss.ave_10best.pth",
transducer_conf=transducer_conf["transducer_conf"],
lm_train_config = "test/espnet_onnx2/rnnt/lm_config/config.yaml",
lm_file="test/espnet_onnx2/rnnt/lm_config/17epoch.pth",
lm_weight=0.0
)

m.export(speech2text, 'onnx_export', quantize=False)

And I got 4 onnx files

the infer script is:
import librosa
from espnet_onnx import Speech2Text
#from pyacl.acl_infer import init_acl, release_acl
from tqdm import tqdm
import os,re

def findAllFile(base):
for root, ds, fs in os.walk(base):
for f in fs:
if re.match(r'.\d.', f) and f.endswith("flac"):
fullname = os.path.join(root, f)
yield fullname

#init_acl(0)
#speech2text = Speech2Text(tag_name='')
speech2text = Speech2Text(model_dir='/root/.cache/espnet_onnx/onnx_export/')
path = "espnet/egs2/librispeech/asr1/downloads/LibriSpeech/test-clean/"
j = 0
with open("test2.txt", 'w') as fout:
for i in findAllFile(path):
y, sr = librosa.load(i, sr=16000)
nbest = speech2text(y)
res = ""
res = "".join(nbest[0][1])
fout.write('{} {}\n'.format(i.split('/')[-1].split('.')[0], res))

the ERROR is:
root@ubuntu:/home/test/espnet_onnx2/rnnt# python3 infer.py
Traceback (most recent call last):
File "infer.py", line 24, in
nbest = speech2text(y)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/espnet_onnx/asr/asr_model.py", line 84, in call
nbest_hyps = self.beam_search(enc[0])[:1]
File "/usr/local/python3.7.5/lib/python3.7/site-packages/espnet_onnx/asr/beam_search/beam_search_transducer.py", line 111, in call
nbest_hyps = self.search_algorithm(enc_out)
File "/usr/local/python3.7.5/lib/python3.7/site-packages/espnet_onnx/asr/beam_search/beam_search_transducer.py", line 238, in default_beam_search
lm_tokens, max_hyp.lm_state, None
File "/usr/local/python3.7.5/lib/python3.7/site-packages/espnet_onnx/asr/model/lms/transformer_lm.py", line 62, in score
k: v for k, v in zip(self.enc_in_cache_names, state)
TypeError: zip argument #2 must support iteration
I found the script run function score in espnet_onnx/asr/model/lms/transformer_lm.py, but the state is None, that may be the reason for this ERROR

Hi, thanks for you share of the espnet_onnx system!

I met two problems when I tried to inference thorough your codes. My acoustic model is trained by myself on our own dataset. The AM architecture is the typical Conformer. I downloaded this code on June.

First, the decoding speed is too slow by it. When using torch to decode, the RTF is around 2.32; however it becomes around 20 when using the transformed onnx.

Second, the CER calculated in the torch version is 7.8% while for the onnx, it becomes 10.6%. I think it is probably wrong.

I'm giving some configs here:

export.py

import sys
sys.path.append('espnet-master')
sys.path.append('espnet-master/espnet_tts_frontend-master')
sys.path.append('espnet_onnx-master/espnet_onnx/export/asr')
import torch

from export_asr import ModelExport
from espnet2.bin.asr_inference import Speech2Text

if __name__ == '__main__':
    m = ModelExport(cache_dir = sys.argv[5])

    # export from trained model
    speech2text=Speech2Text(
            asr_train_config = sys.argv[1],
            asr_model_file=sys.argv[2],
            lm_train_config=sys.argv[3],
            lm_file=sys.argv[4],
            )

    m.export(model = speech2text, tag_name = 'speech2text', quantize=True)

And I get an onnx dir structured like:

asr/onnx/speech2text/
      config.yaml
      feats_stats.npz
      full/
      quantize/

The test wav is a filelist, structured as:

bigfar_001_000001 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000001.wav
bigfar_001_000002 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000002.wav
bigfar_001_000003 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000003.wav
bigfar_001_000004 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000004.wav
bigfar_001_000005 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000005.wav
bigfar_001_000006 /home/dangfeng/exp_xiandao/for_xiandao/onnx_enh/output_0703/enh/bigfar_001_000006.wav
...

The decoding process is:

decode.py

import sys
sys.path.append('espnet_onnx-master/espnet_onnx/asr')

import time
import threading
import librosa
import os
from tqdm import tqdm
from asr_model import Speech2Text

if __name__ == '__main__':
    """ step1: load onnx file """
        speech2text = Speech2Text(tag_name = 'speech2text', model_dir=sys.argv[3],)

        """ step2: ASR """
        f = open(sys.argv[1])
        lines = f.readlines()
        for line in tqdm(lines):
            with open(os.path.join(sys.argv[2], 'hyp_flush_1process.trn'),'a') as fout:
                wav_name = line.split(' ')[0].strip()
                processing_wav = line.split(' ')[1].strip()

                start = time.time()
                y, sr = librosa.load(processing_wav, sr=16000)
                nbest = speech2text(y)
                asr_result = nbest[0][0]
                end = time.time()

                for j in range (len(asr_result)):
                    fout.write(asr_result[j])
                    if j != len(asr_result) - 1:
                        fout.write(' ')
                fout.write('\t')
                fout.write('(')
                fout.write(wav_name)
                fout.write('-')
                fout.write(wav_name)
                fout.write(')')
                fout.write('\n')

                print('processing:  ', processing_wav)
                print('Result:         ', asr_result)
                print('Time:           ', end-start, 's')

Furthermore, I noticed that you have mentioned there may be some problems for Conformer AM considering ASR in latest issue, has it been fixed?

Looking forward for your reply!

Hello, this project looks very useful to me

Is it possible to use this project as a registered model on huggingface ( ASR model- VGGRNN) ?
If so, how should I use tag name or model_dir?

I have pth file trained on Espnet, can I convert pth model to onnx by using this project?

@Masao-Someki Hi Masao，if the model has torch.istft() operation, how to export it to onnx format, do you have relevant experience ?

when i use https://github.com/espnet/espnet/tree/v.202205/egs2/librispeech/asr1
the Conformer-RNN Transducer model
date: Fri Mar 25 04:35:42 EDT 2022
python version: 3.9.5 (default, Jun 4 2021, 12:28:51) [GCC 7.5.0]
espnet version: espnet 0.10.7a1
pytorch version: pytorch 1.8.1+cu111
Git hash: 21d19be00089678ca27f7fce474ef8d787689512
Commit date: Wed Mar 16 08:06:52 2022 -0400
ASR config: conf/train_rnnt_conformer.yaml
Pretrained model: https://huggingface.co/espnet/chai_librispeech_asr_train_rnnt_conformer_raw_en_bpe5000_sp

to export onnx, i met the following error:
Traceback (most recent call last):
File "export.py", line 30, in
m.export(speech2text, 'onnx_export', quantize=False)
File "/home/p00510131/espnet_onnx/espnet_onnx/export/asr/export_asr.py", line 110, in export
ctc_model = CTC(model.asr_model.ctc.ctc_lo)
AttributeError: 'NoneType' object has no attribute 'ctc_lo'

the export script is :
from espnet2.bin.asr_inference import Speech2Text
import os
import yaml
from espnet_onnx.export import ASRModelExport

m = ASRModelExport()
transducer_conf = yaml.safe_load(Path('espnet/egs2/librispeech/asr1/conf/decode_rnnt_conformer.yaml').read_text())
speech2text = Speech2Text(asr_train_config="exp/asr_train_rnnt_conformer_ngpu4_raw_en_bpe5000_sp/config.yaml",
asr_model_file="exp/asr_train_rnnt_conformer_ngpu4_raw_en_bpe5000_sp/valid.loss.ave_10best.pth",
transducer_conf=transducer_conf["transducer_conf"],
lm_weight=0.0
)

m.export(speech2text, 'onnx_export', quantize=False)

Hi, thx for developing this repo

I am confused about text_length in the code below, why we need text_length in fastspeech2? Isn't text tensor enough?

https://github.com/Masao-Someki/espnet_onnx/blob/8bb9d7798b3b52b3b04c966a978d35c036c70ed7/espnet_onnx/export/tts/models/tts_models/fastspeech2.py#L136-L144

https://github.com/Masao-Someki/espnet_onnx/blob/8bb9d7798b3b52b3b04c966a978d35c036c70ed7/espnet_onnx/tts/tts_model.py#L48-L55

Hi Masao!

I was wondering if you could consider adding support for the standalone version of ESPnet2 Transducer? See the doc.
I'm quite interested in ONNX but I have too much at hands right now to work on this project... I can help on the code understanding or similar things though.

Hi, I am getting the following error

from espnet_onnx.espnet_onnx.export import TTSModelExport

Hi have exported the espnet model trained on my custom dataset using espnet_onnx. Model fails to work properly on some audios. Below is the error which i am getting

Non-zero status code returned while running Add node. Name:'/encoders/encoders.0/self_attn/Add' Status Message: /encoders/encoders.0/self_attn/Add: right operand cannot broadcast on dim 3 LeftShape: {1,8,171,171}, RightShape: {1,1,1,127}

Any idea what could be the issue here. I have infered model on 1500 audio clips and i am getting exactly same error on around 400 audio clips.

Hi @Masao-Someki , have you tried converting xvector-vits to onnx ? when I try to do this, I run into this problem:

/root/anaconda3/envs/espnet/lib/python3.7/site-packages/torch/onnx/symbolic_helper.py:719: UserWarning: allowzero=0 by default. In order to honor zero value in shape use allowzero=1
warnings.warn("allowzero=0 by default. In order to honor zero value in shape use allowzero=1")
WARNING: The shape inference of prim::Constant type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Floating point exception(core dump)

Hi, Thanks to this repository, I made the onnx model what I trained and I curious about the model is on cpu or gpu
Now I'm trying to convert espnet onnx model to triton server model
How can I do this? Let me some hint..
Thank you

1. exported model sucessfully

/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/export/asr/models/language_models/embed.py:360: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/export/asr/models/multihead_att.py:116: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
/home/miniconda3/envs/espnet/lib/python3.8/site-packages/torch/onnx/symbolic_helper.py:719: UserWarning: allowzero=0 by default. In order to honor zero value in shape use allowzero=1
  warnings.warn("allowzero=0 by default. In order to honor zero value in shape use allowzero=1")

2. decode with exported model

Traceback (most recent call last):
  File "test_text_e2e_espnet.py", line 20, in <module>
    speech2text = Speech2Text("wenet")
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/asr/asr_model.py", line 43, in __init__
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/asr/abs_asr_model.py", line 53, in _build_model
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/asr/model/encoder.py", line 11, in get_encoder
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.9-py3.8.egg/espnet_onnx/asr/model/encoders/encoder.py", line 35, in __init__
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 280, in __init__
    self._create_inference_session(providers, provider_options)
  File "/home/miniconda3/envs/espnet/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 307, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from espnet_model_onnx/wenet/full/xformer_encoder.onnx failed:/onnxruntime_src/onnxruntime/core/graph/model.cc:111 onnxruntime::Model::Model(onnx::ModelProto&&, const PathString&, const IOnnxRuntimeOpSchemaRegistryList*, const onnxruntime::logging::Logger&) Unknown model file format version.

@Masao-Someki Hi Masao, I found a bug, when I use the 1.11.0 version of torch to export the vits model, everything works fine, but when I upgrade the torch version to 1.12.1 (required by torchaudio), I get a core dump when exporting the vits.
Can you reproduce this bug ? BTW, everything works fine when export jets model using torch version 1.12.1

Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Warning: Constant folding - Only steps=1 can be constant folded for opset >= 10 onnx::Slice op. Constant folding not applied.
Floating point exception(core dump)

Hi,

Could you please help me with bellow issue, when I want to export TTS model using export_from_zip? Thanks in advance!
Traceback (most recent call last): File "convert_to_onnx.py", line 18, in <module> m.export_from_zip( File "/home/jetson/.virtualenvs/espnet2/lib/python3.8/site-packages/espnet_onnx/export/tts/export_tts.py", line 119, in export_from_zip self.export(model, tag_name, quantize) File "/home/jetson/.virtualenvs/espnet2/lib/python3.8/site-packages/espnet_onnx/export/tts/export_tts.py", line 60, in export model_config = self._create_config(model, export_dir) File "/home/jetson/.virtualenvs/espnet2/lib/python3.8/site-packages/espnet_onnx/export/tts/export_tts.py", line 127, in _create_config ret.update(get_preprocess_config(model.preprocess_fn, path)) File "/home/jetson/.virtualenvs/espnet2/lib/python3.8/site-packages/espnet_onnx/export/tts/get_config.py", line 47, in get_preprocess_config 'cleaner_types': model.text_cleaner.cleaner_types[0] IndexError: list index out of range

Great job, is there any plan to support tts?

hi, I initial text2speech using my own am_model and vocoder and export onnx model, but sound quality drops significantly, I just modify hifigan inference code in https://github.com/Masao-Someki/espnet_onnx/blob/feature/add_PWGVocoder/espnet_onnx/export/tts/models/vocoders/parallel_wavegan.py because hifigan code in repo ParalleWaveGAN does not support parameter x,
and i checked Espnet am and vocoder and onnx am and vocoder, they look the same
could you please offer some advises?

hi ,
i try to inference wav as the document
"import librosa
from espnet_onnx import Speech2Text
speech2text = Speech2Text(tag_name='')
y, sr = librosa.load('sample.wav', sr=16000)
nbest = speech2text(y)"
but when run "nbest = speech2text(y)" , occur a error like
"onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'Gather_4' Status Message: indices element out of data bounds, idx=525 must be within the inclusive range [-512,511]"
my feats_lengths is 526, why this error occurs

@Masao-Someki Hi, thank you very much for completing the onnx export of vits. After the test of my own training model, when I set cleaner=none, there was a small problem, and after solving everything works fine !
File "/work/ysj/espnet_onnx/espnet_onnx/export/tts/get_config.py", line 46, in get_preprocess_config 'cleaner_types': model.text_cleaner.cleaner_types[0] IndexError: list index out of range

Hi @Masao-Someki,

I am testing my streaming Conformer-Transformer model after converting it from espnet2 to onnx format. I noticed that the first word in the best hypo is often missing although it got recognized correctly (debug log is showing it via ("batch_beam_search:271) DEBUG: best hypo: ..." message).

I found some suspicious code in this line https://github.com/espnet/espnet_onnx/blob/master/espnet_onnx/asr/asr_streaming.py#164 where the _get_result() function always removes the first two tokens:

def _get_result(self, hyp: Hypothesis):
        token_int = hyp.yseq[2:-1].astype(np.int64).tolist()

I think _get_result() should just remove the first & last token (assumed to be "<sos/eos>") like so:

def _get_result(self, hyp: Hypothesis):
        token_int = hyp.yseq[1:-1].astype(np.int64).tolist()

After the change my issue with missing word at beginning was resolved. So, this looks like a bug to me.

Could you please double-check and let me know if I should create a PR?

Thanks!

While running the following colab notebook,
https://colab.research.google.com/github/espnet/espnet_onnx/blob/master/demo/tts_onnx_demo.ipynb

I get the following error:

@Masao-Someki Could you have a look? Thanks!

@Masao-Someki I tried changing the max_seq_length for fastspeech2, but it gives error

I used the command while exporting and only changed the tag name

m.set_export_config(
    max_seq_len=3000
)

If i use the command with vits, it works well. Can you please check

espnet2-tts already supports end-to-end model jets, do you have any plans to support jets to onnx? I think with experience with fastspeech2 and hifigan, this will be done very quickly

@Masao-Someki I encountered an error when inferring vits using the latest version
Traceback (most recent call last):
File "export2onnx.py", line 21, in
output_dict = tts("hello how are you")
File "/work/espnet_onnx/espnet_onnx/tts/tts_model.py", line 86, in call
output_dict = self.tts_model(text, **options)
File "/work/espnet_onnx/espnet_onnx/tts/model/tts_models/vits.py", line 58, in call
wav, att_w, dur = self.model.run(output_names, input_dict)
File "/root/anaconda3/envs/espnet/lib/python3.7/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 188, in run
raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 2 inputs. Input Feed contains 1

Thanks for developing this repo.

I tested espnet_onnx by using CSJ corpus.
There was a difference in accuracy between a onnx model and a torch model.

I describe how I tested the onnx model by using CSJ corpus.
I converted the CSJ torch model (https://zenodo.org/record/4037458#.YsVJT-zP30o) to the onnx model by using a script shown below.

from espnet2.bin.asr_inference import Speech2Text
from espnet_onnx.export import ASRModelExport

m = ASRModelExport()
m.set_export_config(max_seq_len=3000)
m.export_from_pretrained('kan-bayashi/csj_asr_train_asr_transformer_raw_char_sp_valid.acc.ave', quantize=False)

I transcribed CSJ eval sets with the onnx model by using a script shown below.

import librosa
import sys
from espnet_onnx import Speech2Text

speech2text = Speech2Text(tag_name='kan-bayashi/csj_asr_train_asr_transformer_raw_char_sp_valid.acc.ave')

target = sys.argv[1]
# /opt/csj_eval_dump/eval[1-3] are copied from dump/raw/eval[1-3] generated by egs2/csj/asr1/run.sh
root_path = "/opt/csj_eval_dump/"
with open(f"{root_path}/{target}/wav.scp") as f:
    for line in f:
        t = line.strip().split(" ")
        utterance_id = t[0]
        filepath = t[1].split("dump/raw/")[1]
        y, sr = librosa.load(f"{root_path}/{filepath}", sr=16000)
        nbest = speech2text(y)[0][0]
        print(f"{utterance_id} {nbest}")

An example of transcribed texts is shown below.

A01M0097_0000211_0001695 えー内容としましては
A01M0097_0002761_0009322 生成過程のモデルモデルに基づくＦ０パターンの分析各音節の分節的特徴と
A01M0097_0009867_0020254 えー音調指令との間の相対的な時間関係の規則化え音調指令の大きさの比の規則化えー音調指令とフレーズ指令の大きさの量子化
A01M0097_0020642_0026743 規則によって生成されたＦ０パターンを持つ合成音声のデモンストレーションの順でお話ししたいと思います
A01M0097_0033771_0045423 えーこの図は日本語音声についてのとてもよく知られた生成過程モデルまー藤崎モデルなんですがえーこのモデルはフレーズ指令と正のアクセント指令によって実測のＦ０パターンをよく近似できることが

Transcribed texts are copied to egs2/csj/asr1/exp/[...]/decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval[1-3]/text and are scored by local/score.sh .
I obtained results shown below.

|dataset|Snt|Wrd|Corr|Sub|Del|Ins|Err|S.Err|
|---|---|---|---|---|---|---|---|---|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval1|1272|43897|95.0|2.9|2.1|1.0|6.0|60.4|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval2|1292|43623|95.8|2.5|1.7|0.7|4.9|61.8|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval3|1385|28225|95.4|2.5|2.1|1.0|5.6|43.5|

According to https://zenodo.org/record/4037458#.YsVJT-zP30o the torch model can obtain results shown below.

|decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.ave/eval1|1272|43897|95.9|2.5|1.6|0.8|4.9|50.5|
|decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.ave/eval2|1292|43623|96.9|2.0|1.0|0.6|3.7|50.3|
|decode_asr_lm_lm_train_lm_char_valid.loss.ave_asr_model_valid.acc.ave/eval3|1385|28225|96.8|2.1|1.1|0.7|3.9|32.6|

My environment is as follows.

Python 3.7.13

torch==1.11.0+cpu
torch-complex==0.4.3
torch-optimizer==0.3.0
torchaudio==0.11.0+cpu
espnet-model-zoo==0.1.7
espnet-onnx==0.1.9
espnet-tts-frontend==0.0.3
-e git+https://github.com/espnet/espnet.git@a2abaf11c81e58653263d6cc8f957c0dfd9677e7#egg=espnet

Could you give me any hint to improve accuracy?
Thanks for your help.

Hello, I follow the tutorial for installation on colab.
But I can import the TTSModelExport successfully.
The running example can be show as follow

Hi there.

I am using this code, stright from the usage section:

from espnet2.bin.asr_inference import Speech2Text
from espnet_onnx.export import ASRModelExport

m = ASRModelExport()

# download with espnet_model_zoo and export from pretrained model
m.export_from_pretrained('espnet/kan-bayashi_vctk_full_band_multi_spk_vits', quantize=True)

And getting this result:

Downloading (…)de1ff/.gitattributes: 100%|████████████████████████████████████████████████| 1.18k/1.18k [00:00<?, ?B/s]
Downloading (…)7a4efde1ff/README.md: 100%|████████████████████████████████████████████████| 1.91k/1.91k [00:00<?, ?B/s]
Downloading (…)rg/tr_no_dev/spk2sid: 100%|████████████████████████████████████████████████████| 872/872 [00:00<?, ?B/s]
Downloading (…)no_space/config.yaml: 100%|████████████████████████████████████████████████| 8.23k/8.23k [00:00<?, ?B/s]
Downloading (…)or_backward_time.png: 100%|█████████████████████████████████████████| 33.1k/33.1k [00:00<00:00, 353kB/s]
Downloading (…)scriminator_loss.png: 100%|█████████████████████████████████████████| 45.5k/45.5k [00:00<00:00, 486kB/s]
Downloading (…)tor_forward_time.png: 100%|█████████████████████████████████████████| 32.4k/32.4k [00:00<00:00, 346kB/s]
Downloading (…)inator_fake_loss.png: 100%|█████████████████████████████████████████| 59.9k/59.9k [00:00<00:00, 640kB/s]
Downloading (…)_optim_step_time.png: 100%|█████████████████████████████████████████| 59.7k/59.7k [00:00<00:00, 637kB/s]
Downloading (…)nator_train_time.png: 100%|█████████████████████████████████████████| 28.7k/28.7k [00:00<00:00, 307kB/s]
Downloading (…)nerator_adv_loss.png: 100%|█████████████████████████████████████████| 59.5k/59.5k [00:00<00:00, 476kB/s]
Downloading (…)inator_real_loss.png: 100%|█████████████████████████████████████████| 60.4k/60.4k [00:00<00:00, 322kB/s]
Downloading (…)or_backward_time.png: 100%|█████████████████████████████████████████| 30.1k/30.1k [00:00<00:00, 321kB/s]
Downloading (…)nerator_dur_loss.png: 100%|█████████████████████████████████████████| 42.3k/42.3k [00:00<00:00, 451kB/s]
Downloading (…)tor_forward_time.png: 100%|█████████████████████████████████████████| 54.0k/54.0k [00:00<00:00, 494kB/s]
Downloading (…)_feat_match_loss.png: 100%|█████████████████████████████████████████| 40.1k/40.1k [00:00<00:00, 427kB/s]
Downloading (…)rator_train_time.png: 100%|█████████████████████████████████████████| 31.0k/31.0k [00:00<00:00, 283kB/s]
Downloading (…)ax_cached_mem_GB.png: 100%|█████████████████████████████████████████| 28.9k/28.9k [00:00<00:00, 231kB/s]
Downloading (…)images/iter_time.png: 100%|█████████████████████████████████████████| 57.1k/57.1k [00:00<00:00, 406kB/s]
Downloading (…)s/generator_loss.png: 100%|█████████████████████████████████████████| 28.7k/28.7k [00:00<00:00, 102kB/s]
Downloading (…)_optim_step_time.png: 100%|█████████████████████████████████████████| 56.8k/56.8k [00:00<00:00, 298kB/s]
Downloading (…)nerator_mel_loss.png: 100%|█████████████████████████████████████████| 30.9k/30.9k [00:00<00:00, 165kB/s]
Downloading (…)enerator_kl_loss.png: 100%|████████████████████████████████████████| 23.4k/23.4k [00:00<00:00, 83.1kB/s]
Downloading (…)mages/optim0_lr0.png: 100%|█████████████████████████████████████████| 25.2k/25.2k [00:00<00:00, 269kB/s]
Downloading (…)7a4efde1ff/meta.yaml: 100%|████████████████████████████████████████████████████| 360/360 [00:00<?, ?B/s]
Downloading (…)mages/train_time.png: 100%|█████████████████████████████████████████| 29.1k/29.1k [00:00<00:00, 310kB/s]
Downloading (…)mages/optim1_lr0.png: 100%|█████████████████████████████████████████| 25.0k/25.0k [00:00<00:00, 266kB/s]
Downloading (…)count.ave_10best.pth: 100%|██████████████████████████████████████████| 387M/387M [01:50<00:00, 3.49MB/s]
Fetching 28 files: 100%|███████████████████████████████████████████████████████████████| 28/28 [01:54<00:00,  4.10s/it]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\Enrique.Navarro\Desktop\Pruebas_TTS\espnet_onnx\env_espnet\lib\site-packages\espnet_onnx\export\asr\export_asr.py", line 176, in export_from_pretrained
    model = Speech2Text.from_pretrained(tag_name, **pretrained_config)
  File "C:\Users\Enrique.Navarro\Desktop\Pruebas_TTS\espnet_onnx\env_espnet\lib\site-packages\espnet2\bin\asr_inference.py", line 516, in from_pretrained
    return Speech2Text(**kwargs)
TypeError: Speech2Text.__init__() got an unexpected keyword argument 'train_config'
>>> speech2text = Speech2Text(args)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'args' is not defined

I simply test the espnet_onnx function by train the Espnet2 model using recipe "librispeech_100".
The ONNX conversion working in model train using default features (FBank), which is good.

But when I change the feature_type from default to "fbank_pitch", which will pre-generate fbank+pitch features using Kaldi extractor and use it in subsequent training and decode, error pop up when I using espnet_onnx to convert that trained model.

Can suggest if there is any problem and how to fix it ?

Command I am using is as below
python3 -m espnet_onnx.export --model_type asr --tag conformer_ext_feature --input asr_conformer_lr2e-3_warmup15k_amp_nondeterministic_valid.acc.ave.zip

And here is the error shown
File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/__main__.py", line 91, in <module> m.export_from_zip( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 191, in export_from_zip self.export(model, tag_name, quantize, optimize) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 91, in export self._export_encoder(enc_model, export_dir, verbose) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 246, in _export_encoder self._export_model(model, verbose, path) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/export_asr.py", line 226, in _export_model torch.onnx.export( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/__init__.py", line 350, in export return utils.export( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 163, in export _export( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 1074, in _export graph, params_dict, torch_out = _model_to_graph( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 727, in _model_to_graph graph, params, torch_out, module = _create_jit_graph(model, args) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 602, in _create_jit_graph graph, torch_out = _trace_and_get_graph_from_model(model, args) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/onnx/utils.py", line 517, in _trace_and_get_graph_from_model trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 1175, in _get_trace_graph outs = ONNXTracedModule(f, strict, _force_outplace, return_inputs, _return_inputs_states)(*args, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 127, in forward graph, out = torch._C._create_graph_by_tracing( File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/jit/_trace.py", line 118, in wrapper outs.append(self.inner(*trace_inputs)) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/encoders/conformer.py", line 104, in forward xs_pad, mask = self.embed(feats, mask) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/language_models/embed.py", line 74, in forward return self.model(x, mask) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/espnet_onnx/export/asr/models/language_models/subsampling.py", line 48, in forward x = self.out(x.transpose(1, 2).contiguous().view(b, t, c * f)) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward input = module(input) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1118, in _slow_forward result = self.forward(*input, **kwargs) File "/home/cli/miniconda3/envs/espnet-curr-test/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward return F.linear(input, self.weight, self.bias) RuntimeError: mat1 and mat2 shapes cannot be multiplied (24x4864 and 5120x256)

export codes:

from espnet2.bin.tts_inference import Text2Speech
from espnet_onnx.export import TTSModelExport

m = TTSModelExport()

# download with espnet_model_zoo and export from pretrained model
m.export_from_pretrained('kan-bayashi/csmsc_tts_train_conformer_fastspeech2_raw_phn_pypinyin_g2p_phone_train.loss.ave', quantize=True)

# export from trained model
text2speech = Text2Speech(args)
m.export(text2speech, 'kan-bayashi/csmsc_tts_train_conformer_fastspeech2_raw_phn_pypinyin_g2p_phone_train.loss.ave', quantize=True)

error:

WARNING:root:Fallback to conformer_pos_enc_layer_type = 'legacy_rel_pos' due to the compatibility. If you want to use the new one, please use conformer_pos_enc_layer_type = 'latest'.
WARNING:root:Fallback to conformer_self_attn_layer_type = 'legacy_rel_selfattn' due to the compatibility. If you want  to use the new one, please use conformer_pos_enc_layer_type = 'latest'.
Traceback (most recent call last):
  File "export_espnet_tts.py", line 7, in <module>
    m.export_from_pretrained('kan-bayashi/csmsc_tts_train_conformer_fastspeech2_raw_phn_pypinyin_g2p_phone_train.loss.ave', quantize=True)
  File "/home/ybZhang/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.10-py3.8.egg/espnet_onnx/export/tts/export_tts.py", line 114, in export_
  File "/home/ybZhang/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.10-py3.8.egg/espnet_onnx/export/tts/export_tts.py", line 127, in _create_config
  File "/home/ybZhang/miniconda3/envs/espnet/lib/python3.8/site-packages/espnet_onnx-0.1.10-py3.8.egg/espnet_onnx/export/tts/get_co

Error encountered when trying to export a model to a custom output directory:

Traceback (most recent call last):
  File "xxx.py", line 59, in <module>
    main()
  File "xxx.py", line 33, in main
    m.export(
  File "E:\repos\on-device-speech-translation\onnx_utils\export_st.py", line 141, in export
    update_model_path(tag_name, base_dir)
  File "D:\Anaconda\Lib\site-packages\espnet_onnx\utils\config.py", line 55, in update_model_path
    save_config(config, tag_config_path)
  File "D:\Anaconda\Lib\site-packages\espnet_onnx\utils\config.py", line 32, in save_config
    with open(path, "w", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\xxx\\.cache\\espnet_onnx\\tag_config.yaml'

Seems like the path is hardcoded to home cache dir here

Hi,

Supported vocoders include HiFiGAN in https://github.com/espnet/espnet_onnx/blob/master/docs/TTSSupported.md, but in the code HiFiGAN seems to be not supported in the vocoder loader function below.

Any idea how to use the HiFiGAN vocoder? I am also attaching the config.yaml file that TTSModelExport created -> config.yaml

Hi @Masao-Someki,

In the readme the example for streaming asr shows the use of start() and end() methods:

from espnet_onnx import StreamingSpeech2Text

stream_asr = StreamingSpeech2Text(tag_name)

# start streaming asr
stream_asr.start()
while streaming:
  wav = <some code to get wav>
  assert len(wav) == stream_asr.hop_size
  stream_text = stream_asr(wav)[0][0]

# You can get non-streaming asr result with end function
nbest = stream_asr.end()

In a real streaming scenario should the start() and end() methods be called whenever the microphone is opened and closed?

I am asking because I noticed that the end() function in https://github.com/espnet/espnet_onnx/blob/master/espnet_onnx/asr/asr_streaming.py#151 calls the self.batch_beam_search() function which will restart decoding from postion 0 again causing a rather large delay for longer speech inputs. If I change https://github.com/espnet/espnet_onnx/blob/master/espnet_onnx/asr/asr_streaming.py#151 to use self.beam_search() method instead it avoids decoding the entire utterance at the end again and thus the delay.

Could you please clarify why self.batch_beam_search() is used in stream_asr.end() function?

Thanks!

Trying to download the pretrained model using following code -

`from espnet2.bin.asr_inference import Speech2Text
from espnet_onnx.export import ASRModelExport

m = ASRModelExport()

tag_name = 'asr_train_asr_transformer2_raw_zh_char_batch_bins20000000_ctc_confignore_nan_gradtrue_sp_valid.acc.ave'

download with espnet_model_zoo and export from pretrained model

m.export_from_pretrained(tag_name, quantize=True)

export from trained model

speech2text = Speech2Text(args)
m.export(speech2text, '', quantize=True)`

and ends up with the error

Traceback (most recent call last): File "espnet_onnx_export.py", line 7, in <module> from espnet_onnx.export import ASRModelExport File "/home/neso/espnet_onnx/espnet_onnx/export/__init__.py", line 1, in <module> from .asr.export_asr import ASRModelExport File "/home/neso/espnet_onnx/espnet_onnx/export/asr/export_asr.py", line 14, in <module> from espnet_model_zoo.downloader import ModelDownloader File "/home/neso/espnet_onnx/espnet_model_zoo.py", line 8, in <module> speech2text = Speech2Text.from_pretrained( File "/home/neso/anaconda3/envs/espnet_onnx/lib/python3.8/site-packages/espnet2/bin/asr_inference.py", line 556, in from_pretrained from espnet_model_zoo.downloader import ModelDownloader ModuleNotFoundError: No module named 'espnet_model_zoo.downloader'; 'espnet_model_zoo' is not a package

when I try to use inference demo, I got an error
import librosa
from espnet_onnx import Speech2Text

speech2text = Speech2Text(model_dir='/root/autodl-tmp/.cache/espnet_onnx/librispeech_100-asr-conformer-aed')

wav_file = '../wav_test/121-121726-0000.wav'
y, sr = librosa.load(wav_file, sr=16000)
nbest = speech2text(y)

error info
/root/miniconda3/envs/espnet-onnx/lib/python3.8/site-packages/espnet_onnx/utils/abs_model.py:63: UserWarning: Inference will be executed on the CPU. Please provide gpu providers. Read How to use GPU on espnet_onnx in readme in detail.
warnings.warn(
2023-04-15 19:34:26.289677552 [E:onnxruntime:, sequential_executor.cc:368 Execute] Non-zero status code returned while running Add node. Name:'Add_398' Status Message: /hdd/doc/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:523 void onnxruntime::BroadcastIterator::Append(ptrdiff_t, ptrdiff_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 211 by 421

Traceback (most recent call last):
File "inference.py", line 9, in
nbest = speech2text(y)
File "/root/miniconda3/envs/espnet-onnx/lib/python3.8/site-packages/espnet_onnx/asr/asr_model.py", line 79, in call
enc, _ = self.encoder(speech=speech, speech_length=lengths)
File "/root/miniconda3/envs/espnet-onnx/lib/python3.8/site-packages/espnet_onnx/asr/model/encoders/encoder.py", line 70, in call
self.forward_encoder(feats, feat_length)
File "/root/miniconda3/envs/espnet-onnx/lib/python3.8/site-packages/espnet_onnx/asr/model/encoders/encoder.py", line 87, in forward_encoder
self.encoder.run(["encoder_out", "encoder_out_lens"], {
File "/root/miniconda3/envs/espnet-onnx/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running Add node. Name:'Add_398' Status Message: /hdd/doc/onnxruntime/onnxruntime/core/providers/cpu/math/element_wise_ops.h:523 void onnxruntime::BroadcastIterator::Append(ptrdiff_t, ptrdiff_t) axis == 1 || axis == largest was false. Attempting to broadcast an axis by a dimension other than 1. 211 by 421

please help me. thank you very much!

When I tested the demo of ljspeech, I encountered the following problems

Traceback (most recent call last): File "export2onnx.py", line 6, in <module> m.export_from_pretrained(tag_name) File "/work/ysj/espnet_onnx/espnet_onnx/export/tts/export_tts.py", line 109, in export_from_pretrained self.export(model, tag_name, quantize) File "/work/ysj/espnet_onnx/espnet_onnx/export/tts/export_tts.py", line 64, in export self._export_tts(tts_model, export_dir, verbose) File "/work/ysj/espnet_onnx/espnet_onnx/export/tts/export_tts.py", line 174, in _export_tts self._export_model(model, verbose, path) File "/work/ysj/espnet_onnx/espnet_onnx/export/tts/export_tts.py", line 163, in _export_model dynamic_axes=model.get_dynamic_axes(), File "/root/anaconda3/envs/espnet/lib/python3.7/site-packages/torch/onnx/__init__.py", line 320, in export custom_opsets, enable_onnx_checker, use_external_data_format) File "/root/anaconda3/envs/espnet/lib/python3.7/site-packages/torch/onnx/utils.py", line 111, in export custom_opsets=custom_opsets, use_external_data_format=use_external_data_format) File "/root/anaconda3/envs/espnet/lib/python3.7/site-packages/torch/onnx/utils.py", line 707, in _export _set_opset_version(opset_version) File "/root/anaconda3/envs/espnet/lib/python3.7/site-packages/torch/onnx/symbolic_helper.py", line 849, in _set_opset_version raise ValueError("Unsupported ONNX opset version: " + str(opset_version)) ValueError: Unsupported ONNX opset version: 15

My pytorch version is 1.10.1, is it related to this?

I test espnet_onnx with a conformer model, I eval 100 wav 10 times and calculate the RTF only forward time, the result is

quantize model is very slower than fp32 model on GPU and just a litter bit faster on cpu

System information:
torch /cuda / GPU: 11.0 / 11.6 / A100
cpu: AMD EPYC 7402 24-Core Processor
onnx: 1.10.1
onnxruntime-gpu : 1.13.1
espnet_onnx: 0.1.9

Have you tested the speed of the quantize model on GPU

Sorry to bother you again.

I tested the conformer model of espnet_onnx by using CSJ corpus.
There was a difference in accuracy between a onnx conformer model and a torch conformer model.

I describe how I tested the conformer model of espnet_onnx by using CSJ corpus.
I downloaded the CSJ torch conformer model ( https://zenodo.org/record/4065140#.YsurAezP30o ), and converted it to the onnx model by using a script shown below.

from espnet2.bin.asr_inference import Speech2Text
from espnet_onnx.export import ASRModelExport

m = ASRModelExport()

m.set_export_config(max_seq_len=3000)
speech2text = Speech2Text.from_pretrained('asr_train_asr_conformer_raw_char_sp_valid.acc.ave.zip')
m.export(speech2text, 'kan-bayashi/csj_asr_train_asr_conformer_raw_char_sp_valid.acc.ave', quantize=False)

I modified ~/.cache/espnet_onnx/kan-bayashi/csj_asr_train_asr_conformer_raw_char_sp_valid.acc.ave/config.yaml as shown below.

weights:
  ctc: 0.3
  decoder: 0.7
  length_bonus: 0.0
  lm: 0.3
  ngram: 0.0

I transcribed CSJ eval sets with the onnx conformer model in the same manner as https://github.com/Masao-Someki/espnet_onnx/issues/36#issue-1295560021

I obtained results shown below.

### CER

|dataset|Snt|Wrd|Corr|Sub|Del|Ins|Err|S.Err|
|---|---|---|---|---|---|---|---|---|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval1|1272|43897|95.0|3.3|1.7|1.2|6.2|56.3|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval2|1292|43623|96.5|2.3|1.2|0.8|4.3|53.3|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval3|1385|28225|96.0|2.6|1.4|1.0|5.0|38.1|

I transcribed CSJ eval sets with the torch conformer model( https://zenodo.org/record/4065140#.YsurAezP30o ).
I obtained a result shown below.

### CER

|dataset|Snt|Wrd|Corr|Sub|Del|Ins|Err|S.Err|
|---|---|---|---|---|---|---|---|---|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval1|1272|43897|96.3|2.3|1.5|0.7|4.5|49.4|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval2|1292|43623|97.2|1.7|1.0|0.5|3.3|47.6|
|decode_asr_lm_lm_train_lm_jp_char_valid.loss.ave_asr_model_valid.acc.ave/eval3|1385|28225|97.2|1.8|1.0|0.7|3.6|32.5|

Could you give me any hint to improve accuracy?
Thanks for your help.

I was testing the encoder module.
https://github.com/espnet/espnet_onnx/blob/18eb341d44ccf83c3ab35bc040b4102a73602922/tests/unit_tests/test_inference_asr.py.
raise ValueError("Model requires {} inputs. Input Feed contains {}".format(num_required_inputs, num_inputs))
ValueError: Model requires 9 inputs. Input Feed contains 1.
The dummy input is for the feats. do we need to create dummy inputs for rest of the inputs.
{ 'xs_pad',
'mask
'buffer_before_downsampling'
'buffer_after_downsampling'
'prev_addin'
'pos_enc_xs':
'pos_enc_addin'
'past_encoder_ctx'
'indicies'}

BTW the decoder module works fine.

Hi, thanks for the great system!

I've tried to install custom-onnxrutime following the documentation.
https://github.com/espnet/espnet_onnx/blob/master/docs/Optimization.md#install-custom-onnxruntime

However, the link https://github.com/Masao-Someki/espnet_onnx/releases/download/custom_ort_v1.11.1-espnet_onnx/onnxruntime-1.11.1_espnet_onnx-cp38-cp38-linux_x86_64.whl seems not to exist.

Thank you for your always kind reply. Also, your research has really helped me a lot.

In general, the purpose of onnx conversion is to make the model lighter and get faster speed. Have you ever checked the difference in decoding speed before and after conversion onnx model conversion in the asr task? I would also like to know if there is also a difference in accuracy. It doesn't matter which model you've been running.

Hi..
I am trying to convert pretrained LJSpeech TTS model based on kan-bayashi/ljspeech_fastspeech2 and parallel_wavegan/ljspeech_parallel_wavegan.v1 using the below code:

########################### ONNX Conversion ############################

from espnet2.bin.tts_inference import Text2Speech
from espnet_onnx.export import TTSModelExport

m = TTSModelExport()

tag_exp = "exp/tts_train_fastspeech2_raw_phn_tacotron_g2p_en_no_space/train.loss.ave_5best.pth"
train_config="exp/tts_train_fastspeech2_raw_phn_tacotron_g2p_en_no_space/config.yaml"

vocoder_tag = 'parallel_wavegan.v1/checkpoint-400000steps.pkl'
vocoder_config= 'parallel_wavegan.v1/config.yml'

text2speech = Text2Speech.from_pretrained(
train_config=train_config,
model_file=tag_exp,
vocoder_file=vocoder_tag,
vocoder_config=vocoder_config,
speed_control_alpha=1.0,
always_fix_seed=False
)

tag_name = 'ljspeech_pretrained'
m.export(text2speech, tag_name, quantize=True)

########################### Inference ############################

from espnet_onnx import Text2Speech
import soundfile
import numpy as np
import time

text2speech = Text2Speech(tag_name)

text = 'hello world!'
wav = wav['wav']

soundfile.write("ljspeech_pretrained_test.wav", wav, 22050, "PCM_16")

######################################################################

On synthesizing, the audio quality is very low.
I realized that the converted ONNX folder did not have stats.h5 file from the pwg vocoder folder.
~/.cache/espnet_onnx/ljspeesch_pretrained/: config.yaml feats_stats.npz full quantize

Can anyone please help how to include the stats.h5 during inference using espnet_onnx

I've test a sentence with these three models, the quantize model seems abnormal.

pytorch vits
infer time : 10.691850900650024
RTF = 0.349893

onnx vits
infer time : 4.495715618133545
RTF = 0.146124

onnx quantize vits
infer time : 83.49901461601257
RTF = 2.672614

Error:

[ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running ScatterND node. Name:'ScatterND_70' Status Message: invalid indice found, indice = -1
File "D:\recognition\espnet\esp2onnx\espnet_onnx\asr\model\decoders\xformer.py", line 84, in batch_score
input_dict
File "D:\recognition\espnet\esp2onnx\espnet_onnx\asr\beam_search\batch_beam_search.py", line 136, in score_full
scores[k], states[k] = d.batch_score(hyp.yseq, hyp.states[k], x)
File "D:\recognition\espnet\esp2onnx\espnet_onnx\asr\beam_search\batch_beam_search.py", line 195, in search
[x for _ in range(n_batch)]).reshape(n_batch, *x.shape))
File "D:\recognition\espnet\esp2onnx\espnet_onnx\asr\beam_search\beam_search.py", line 334, in call
best = self.search(running_hyps, x)
File "D:\recognition\espnet\esp2onnx\espnet_onnx\asr\asr_model.py", line 84, in call
nbest_hyps = self.beam_search(enc[0])[:1]
File "D:\recognition\espnet\esp2onnx\demo.py", line 12, in
res = speech2text(speech=data)

Code:

from espnet_onnx import Speech2Text
import librosa
import numpy as np

if __name__ == "__main__":
    tag = 'espnet/Shinji_Watanabe_laborotv_asr_train_asr_conformer2_latest33_raw_char_sp_valid.acc.ave'

    speech2text = Speech2Text(tag)

    data, _ = librosa.load('1.wav',sr=16000)

    res = speech2text(speech=data)
    print(res)

Enviroment:
win11,python3.7.7,onnx==1.12.0,onnxruntime==1.10.0,espnet==202205

hi, i met a problem when onnx inference on gpu

1. onnx inference on gpu slower than onnx cpu inference much time and sometimes faster than gpu pt inference(2 times acceleration)
2. when i inference same text twice or more, inference achieves 2 time acceleration compare to gpu pt inference
   any advicec?
   thanks

Hi, I was wondering if you test the accuracy using espnet onnx. I have encountered accuracy drop issue from ~5% wer to 12% wer after using onnx. Just wondering if anyone encounter the same issue.

Hi, thanks for your work on great project!

I tried exporting as specified in example:

from espnet_onnx.export import TTSModelExport

m = TTSModelExport()

tag_name = 'kan-bayashi/ljspeech_vits'
# download with espnet_model_zoo and export from pretrained model
m.export_from_pretrained(tag_name, quantize=True)

Then I tried to use model:

from espnet_onnx import Text2Speech
import IPython

tag_name = 'kan-bayashi/ljspeech_vits'
text2speech = Text2Speech(tag_name, use_quantized=True)

text = 'This model is so small it can be run on a smartwatch! This model is so small it can be run on a smartwatch!'
output_dict = text2speech(text) # inference with onnx model.
wav = output_dict['wav']
IPython.display.Audio(data=wav, rate=22050)

But I get the following message:

2022-10-01 12:31:15.826208114 [E:onnxruntime:, sequential_executor.cc:364 Execute] Non-zero status code returned while running Gather node. Name:'Gather_3861' Status Message: indices element out of data bounds, idx=712 must be within the inclusive range [-512,511]
Traceback (most recent call last):
  File "<path>/onnx-tts/use_onnx.py", line 8, in <module>
    output_dict = text2speech(text) # inference with onnx model.
  File "<path>/onnx-tts/.venv/lib/python3.9/site-packages/espnet_onnx/tts/tts_model.py", line 86, in __call__
    output_dict = self.tts_model(text, **options)
  File "<path>/onnx-tts/.venv/lib/python3.9/site-packages/espnet_onnx/tts/model/tts_models/vits.py", line 58, in __call__
    wav, att_w, dur = self.model.run(output_names, input_dict)
  File "<path>/onnx-tts/.venv/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'Gather_3861' Status Message: indices element out of data bounds, idx=712 must be within the inclusive range [-512,511]

I believe it's because inputs are fixed.
I can try to fix this, but maybe do you know how to handle that?

First of all, thank you for you great work, I've converted the model to onnx successfully, but I found an issue ,When I synthesis a long sentence, will cause onnx runtime error.

Traceback (most recent call last):
File "infer_onnx.py", line 39, in
wav = text2speech(text)["wav"]
File "/opt/conda/lib/python3.8/site-packages/espnet_onnx/tts/tts_model.py", line 79, in call
output_dict = self.tts_model(text, options)
File "/opt/conda/lib/python3.8/site-packages/espnet_onnx/tts/model/tts_models/vits.py", line 59, in call
wav, att_w, dur = self.model.run(output_names, input_dict)
File "/opt/conda/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 192, in run
return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Non-zero status code returned while running Gather node. Name:'Gather_3704' Status Message: indices element out of data bounds, idx=2595 must be within the inclusive range [-512,511]