Using own text to generate speech using Mellotron about mellotron HOT 8 CLOSED

Specifically, I'm trying to generate my own embedded_text and use that instead of the one generated from the input audio. This might be related to #57.

from mellotron.

More specifically, it seems I'm not able to match some matrix dimensions. I'm not sure what I need to pad.

/data/mellotron/model.py in forward(self, attention_hidden_state, memory, processed_memory, attention_weights_cat, mask, attention_weights)
94
95 attention_weights = F.softmax(alignment, dim=1)
---> 96 attention_context = torch.bmm(attention_weights.unsqueeze(1), memory)
97 attention_context = attention_context.squeeze(1)
98

RuntimeError: invalid argument 6: wrong matrix size at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:534

from mellotron.

@astricks
What you're asking for is slight madness 😄
Try inputting text of the same length as the original text and see what happens.

from mellotron.

I got something to work. Seems like the word "en-han-ces" doesn't play well with the dictionary, removed that change and it worked.
@CookiePPP could you expand a bit on the madness part? 😄 What I want to do is generate sentences spoken in the style of reference sentence. My understanding is that Tacotron does prosody transfer, and mellotron will give me rhythmic transfer as well.

from mellotron.

@astricks
The source rhythm is an attention map that is ran over the text (encoder_outputs).
I don't expect you to be able to change the input and have it still sound natural, though it would work for same length inputs.
Also f0 would be teacher forced so I'd expect that to also effect the naturalness when changing the text.

If you try it (with same length inputs), can you upload the audio file you generate as well as the original and updated texts?
I'm curious what it sounds like

from mellotron.

@CookiePPP
I used the source example sentence and changed the text a little to generate audio below.

Source text: "exploring the expanses of space to keep our planet safe"
Modified text: "exploding the expanses of grace to keep our planet sane"

https://drive.google.com/open?id=1w-i_T9hzwzgXOVVouXVP039YuD02ROP3

Not a bad transfer, the sentences are very similar though.

from mellotron.

@astricks
Sounds pretty good, now I wonder how far it can be changed? 😄

from mellotron.

Not too much, even if i try to keep the same number of syllables. I think I can close this bug as resolved. Thanks the help @CookiePPP!

from mellotron.