Training stuck about vits2_pytorch HOT 18 OPEN

@p0p4k bug is on line
scaler.scale(loss_gen_all).backward()

Seems like GradScalar has issues with multi-gpu. I removed it and replaced it with standard backprop. The issue persists. Looks like a multi GPU issue.

from vits2_pytorch.

Haven't tested yet. Trying a run with fp16 enabled.

from vits2_pytorch.

Hi, temporarily turn off duration discriminator and tell me if it works.

from vits2_pytorch.

Yes it seems to be working for now. Any reason with the duration discriminator is causing the issue ?

from vits2_pytorch.

I feel my implementation was too naive. Might need to correct it with some testing. Busy on other models now, will do it when I have some time on me. Let me know about the audio quality after you train. Thanks.

from vits2_pytorch.

Hello,thank you for your great effort !
I meet the same problem and I want to know will you correct it recently or still busy on other models?

from vits2_pytorch.

I have moved to improving pflowtts.

from vits2_pytorch.

I have moved to improving pflowtts.

Hi, p0p4k,
How is the pflowtts growing now? is it a better choice then vits2？ can it support both normal tts and zero-shot tts?

from vits2_pytorch.

I think better than vits/vits2. Only downside it not being e2e.

from vits2_pytorch.

I think better than vits/vits2. Only downside it not being e2e.

ok, thank you.

from vits2_pytorch.

@p0p4k do you know the bug here?

from vits2_pytorch.

@codeghees which part ? training stuck part?

from vits2_pytorch.

yep

from vits2_pytorch.

In the same boat.

from vits2_pytorch.

@codeghees did not look into this personally cause of no gpu yet. Maybe you can try to debug and send a PR. I can assist you. Thanks a lot!

from vits2_pytorch.

Yep, will do! Trying to debug this.

from vits2_pytorch.

Works on single gpu?

from vits2_pytorch.

@p0p4k I have no issues for single GPU training. But it will stuck if I do multiple GPU training. Any success for resolving the issue?

from vits2_pytorch.