josephrocca / lyra-v2-soundstream-web Goto Github PK

IIUC, the above-linked repos are direct Emscripten ports of the C++ Lyra code - so they don't use TFjs-TFLite or ONNX Runtime Web. They seem to work great, so I'll probably not update this repo unless it turns out it'd be handy to have a WebGL version, or if there's reason to believe that using TFjs-TFLite or ONNX Runtime Web would be significantly faster in general.

Lyra V2 (SoundStream) running in the browser.

• https://josephrocca.github.io/lyra-v2-soundstream-web/onnx-simple.html
• https://josephrocca.github.io/lyra-v2-soundstream-web/tflite-simple.html
• https://josephrocca.github.io/lyra-v2-soundstream-web/onnx.html
• https://josephrocca.github.io/lyra-v2-soundstream-web/tflite.html

• ONNX:
  • TFLite-to-ONNX conversion process is producing a LyraGAN ORT model which doesn't have matching outputs between ONNX Web and ONNX Python runtimes at inference. microsoft/onnxruntime#13670
  • TFLite "VAR" ops conversion issues? onnx/tensorflow-onnx#2059 (comment) and google/lyra#99 (comment)
• tfjs-tflite:
  • Errors (and infinite loop if cross-origin isolated) on soundstream_encoder: tensorflow/tfjs#7055
  • tensorflow/tfjs#6094 (comment) (can likely work around this by running each in its own worker, or worst case, in its own iframe)

• https://colab.research.google.com/gist/josephrocca/401efe82bf18ffe93d5f3deca7ed7515

• Python ONNX: https://colab.research.google.com/gist/josephrocca/605006bbeb49df6c9ab7efd18a9cdd0d
• Web ONNX: https://josephrocca.github.io/lyra-v2-soundstream-web/onnx-simple.html
• Python TFLite: https://colab.research.google.com/gist/josephrocca/ba3021195dfb4f91ffc7898cea6f8596

• ONNX: https://huggingface.co/rocca/lyra-v2-soundstream/tree/main/onnx
• ORT: https://huggingface.co/rocca/lyra-v2-soundstream/tree/main/ort
• TFLite: https://huggingface.co/rocca/lyra-v2-soundstream/tree/main/tflite

• Update this thread RE correctness when you have both tflite and onnx working.