Giter VIP home page Giter VIP logo

audioldm's Introduction

Text-to-Audio Generation

arXiv githubio Hugging Face Spaces Open In Colab Replicate

Generate speech, sound effects, music and beyond.


Important tricks to make your generated audio sound better

  1. Try to use more adjectives to describe your sound. For example: "A man is speaking clearly and slowly in a professional studio" is better than "A man is speaking". This can make sure AudioLDM understand what you want.
  2. Try to use different random seeds, which can affect the generation quality significantly sometimes.
  3. It's best to use general terms like 'man' or 'woman' instead of specific names for individuals or abstract objects that humans may not be familiar with, such as 'mummy'.

Change Log

2023-02-15: Add audio style transfer. Add more options on generation.

Web APP

  1. Prepare running environment
conda create -n audioldm python=3.8; conda activate audioldm
pip3 install audioldm==0.0.8
git clone https://github.com/haoheliu/AudioLDM; cd AudioLDM
  1. Start the web application (powered by Gradio)
python3 app.py
  1. A link will be printed out. Click the link to open the browser and play.

Commandline Usage

  1. Prepare running environment
# Optional
conda create -n audioldm python=3.8; conda activate audioldm
# Install AudioLDM
pip3 install audioldm==0.0.8
  1. text-to-audio generation
# Test run
audioldm -t "A hammer is hitting a wooden surface" # The default --mode is "generation"
  1. audio-to-audio style transfer
# Test run
# --file_path is the original audio file for transfer
# -t is the text AudioLDM uses for transfer. 
# Please make sure that --file_path exist
audioldm --mode "transfer" --file_path trumpet.wav -t "Children Singing" 

# Tune the value of --transfer_strength is important!
# --transfer_strength: A value between 0 and 1. 0 means original audio without transfer, 1 means completely transfer to the audio indicated by text
audioldm --mode "transfer" --file_path trumpet.wav -t "Children Singing" --transfer_strength 0.25

For more options on guidance scale, batchsize, seed, ddim steps, etc., please run

audioldm -h

For the evaluation of audio generative model, please refer to audioldm_eval.

Web Demo

Integrated into Hugging Face Spaces ๐Ÿค— using Gradio. Try out the Web Demo Hugging Face Spaces

TODO

  • Update the checkpoint with more training steps.
  • Add AudioCaps finetuned AudioLDM-S model
  • Build pip installable package for commandline use
  • Build Gradio web application
  • Add text-guided style transfer
  • Add audio super-resolution
  • Add audio inpainting

Cite this work

If you found this tool useful, please consider citing

@article{liu2023audioldm,
  title={AudioLDM: Text-to-Audio Generation with Latent Diffusion Models},
  author={Liu, Haohe and Chen, Zehua and Yuan, Yi and Mei, Xinhao and Liu, Xubo and Mandic, Danilo and Wang, Wenwu and Plumbley, Mark D},
  journal={arXiv preprint arXiv:2301.12503},
  year={2023}
}

Hardware requirement

  • GPU with 8GB of dedicated VRAM
  • A system with a 64-bit operating system (Windows 7, 8.1 or 10, Ubuntu 16.04 or later, or macOS 10.13 or later) 16GB or more of system RAM

Reference

Part of the code is borrowed from the following repos. We would like to thank the authors of these repos for their contribution.

https://github.com/LAION-AI/CLAP

https://github.com/CompVis/stable-diffusion

https://github.com/v-iashin/SpecVQGAN

https://github.com/toshas/torch-fidelity

We build the model with data from AudioSet, Freesound and BBC Sound Effect library. We share this demo based on the UK copyright exception of data for academic research.

audioldm's People

Contributors

eltociear avatar haoheliu avatar hysts avatar jagilley avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.