kevinmin95 / stylespeech Goto Github PK
View Code? Open in Web Editor NEWOfficial implementation of Meta-StyleSpeech and StyleSpeech
License: MIT License
Official implementation of Meta-StyleSpeech and StyleSpeech
License: MIT License
Dear author @KevinMIN95
Thank you for sharing the interesting project.
I use pretrained model (Stylespeech and Meta-Stylespeech) and Melgan (pretrained model).
I also use the same people in the page (https://stylespeech.github.io/) to evaluate Trained Speaker and Unseen Speaker. However, the synthesized audio is not good as you report.
Could you give me some advice to reproduce the similar result that you report.
Best Regard
How do you train the vocoder? Have you used the GTA training?
@KevinMIN95 I use this pretrained model: https://huggingface.co/Guan-Ting/StyleSpeech-MelGAN-vocoder-16kHz. And I also use this repos https://github.com/descriptinc/melgan-neurips for inference, but the output audio sound very bad, output pitch may changed.
1463_infer.zip
@KevinMIN95 Could you please tag the author of this model if it necessary.
Hello, please check and help me fix it.
python preprocess.py -> AttributeError: module 'preprocessors.libritts' has no attribute 'write_metadata'
Hi,
I'm probably missing something, but how can I generate the synthesized audio file from the mel-spectrogram in the synthesize.py demo?
Thanks
Hi. I notice that in the demo page (Section 4.3), you only did parallel voice cloning with unseen speakers, have you tried testing with different text with these unseen speakers?
Besides, is it possible to get some pretrained models?
Thank you very much.
Do you have chinese-data model?
As I understand, you train your own version of MelGAN for multi-speaker synthesis, as the official code supports the sampling rate of 22.05 kHz, while StyleSpeech operates at 16 kHz.
Could you share the details for reproducibility purposes: which dataset did you use, which parameters did you change? Or you can maybe upload the trained vocoder itself? It would be great!
I use two speakers, but the result is bad!
StyleSpeech/preprocessors/libritts.py
Line 130 in ddff11e
Issue: Using the extracted pre-trained style encoder results in output embeddings with NaN values, but only during training.
Resolution: Training was initially running on mixed precision at 16 bits. However, changing it to 32 bits resolved the issue, and only correct embeddings were generated.
When running train.py,
models/VarianceAdaptor.py line 52: x = self.ln(x) + pitch_embedding + energy_embedding
returns an error.
The shape of x seems to be [Batch_size, max_text_input_length, 256].
The shape of the other two seems to be [Batch_size, ??????, 256] (I don't know what pitch_embedding.shape(1) should be.)
Is there a solution to this?
For "unseen Speaker Adaptation" , did you refine the model using the data of target speaker, or just using MelStyleEncoder to get ws which adjust the output (like zero-shot/one-shot)?
I have tried to train a lot of model with sampling rate 22050 but, it can not reproduce quality of 16000 hz model. Can you explain why you use 16000 in your research ?
@KevinMIN95 Why you use model_without_ddp and discriminator_without_ddp to calculate some tensors participating the losses calculation? I think the gradients of model_without_ddp will not be synchronized and reduced accross the device, and could this lead to mistakes in distributed training?
When I ran the preprocess.py, I would get this error: IndexError: cannot do a non-empty take from an empty axes.
Traceback (most recent call last):
File "preprocess.py", line 54, in
main(preprocessor, args.data_path, args.output_path)
File "preprocess.py", line 39, in main
datas = preprocessor.build_from_path(data_dir, out_dir)
File "/media/fish-bsp/fish_4TB/Audio/StyleSpeech-main/preprocessors/libritts.py", line 132, in build_from_path
f0 = remove_outlier(f0)
I think the problem is from here:
How can I revise the code to let it run successfully? Thanks.
Same as the title, I use the HiFi-GAN vocoder to generate the audio. But there is full of noice in the audio. How could you make the qualified audio as the demo page. Could you pls share some experinece.
Thanks a lot.
After running the inference command, the result is a png file showing spectrograms. So I tried to do some changes in the synthesize.py file to covert generated mel -spectrogram to audio file. I used librosa for this purpose. Below is the code snippet to do so.
But the audio obtained is blank.
Can you please help me out to convert the mel to proper audio? I am not yet an expert in this field. @KevinMIN95 your help is appreciated here.
Thank you
I have trained a stylespeech model use LibriTTS, but the quality was far worse than pretrain stylespeech model of author. I use default config and parameter and train the model within 100k step. The loss like bellow:
I also upload my audio sample of the text same as demo page in folder Train_LibriTTS_StyleSpeech in attached fille. There are always strange sounds at the end of each audio file, i can't explain that.
meta_stylespeech_results.zip
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.