Exploring "linear attention" for text-to-speech.

It predicts audio codec "à la" MusicGen : delayed residual vector quantizers so that we do not need multiple models.

Featuring RWKV, Mamba, Gated Linear Attention.

Compared to other LM TTS model :

• Can be easily pretrained and finetuned on midrange GPUs.
• Tiny memory footprint.
• Trained on long context (up to 2000 tokens : ~27s).

Following the linear complexity LM you choose, follow respective instructions first:

• For Mamba check the official repo.
• For GLA/RWKV inference check flash-linear-attention.
• For RWKV training check RWKV-LM

Download configuration and weights above, then check Inference.ipynb.

• Fix RWKV6 inference and/or switch to FLA implem.
• Provide a Datamodule for training (lhotse might also work well).
• Implement CFG.
• Scale up.

• The RWKV authors and the community around for carrying high-level truly opensource research.
• @SmerkyG for making my life easy at testing cutting edge language model.
• @lucidrains for its huge codebase.
• @sustcsonglin who made GLA and FLA.
• @harrisonvanderbyl fixing RWKV inference.

@software{lemerle2024linaspeech,
  title  = {LinaSpeech: Exploring "linear attention" for text-to-speech.},
  author = {Lemerle, Théodor},
  url    = {https://github.com/theodorblackbird/lina-speech},
  month  = april,
  year   = {2024}
}

This work is performed in the Analysis/Synthesis team of the STMS Laboratory at IRCAM, and is part of the following project: ANR Exovoices