negative loss and failed to load model for retraining about waveglow HOT 13 CLOSED

@yoyololicon

1. When training with default model given by this repo, I encounter NAN.
   [Solve]: I initialize the upsample layer weight to be 1.0, bias to be 0.0.
2. When training with multi-gpus, I encounter NAN again.
   [Solve]: Use one gpu, batch_size = 4

I'm not sure how to avoid nan.
It works to train with inited upsample layer and on one gpu.

from waveglow.

I also got negative loss and 'nan' during training.

I moved the waveglow into my training framework and trained it with DataParallel from scratch. It seems that the nll recovered from nan and then became nan again.

Besides, the infered samples even reach 1e17.

I used hparams:

        sigma                       = 1.0,
        n_flows                     = 12,
        n_group                     = 8,
        n_early_every               = 4,
        n_early_size                = 2,
        wn                          = Config(n_layers=8, n_channels=256, kernel_size=3)

from waveglow.

@chaiyujin so, what's your training framework? retrain from the provided model, did it work? I am lost your words

from waveglow.

@benlaitang Sorry about my english. I have my own training framework. I trained glow from scratch. Never test fine-tuning from provided model.

from waveglow.

I got exactly the same problem.

from waveglow.

@wenyong-h did you solve the problem?

from waveglow.

No, I'm training from scratch now.

from waveglow.

1. Yes, the loss should be negative. We trained the model for 580 iterations with batch size 24. 22k iters with batch size 3 is probably not enough to produce intelligible speech.
2. The second model is shared for inference only, not resuming training.

from waveglow.

@rafaelvalle thanks a lot. I will try from scratch again.

from waveglow.

Closing. Please re-open if necessary.

from waveglow.

@chaiyujin Did you solve the issue of nan loss? Because I encounter similar issue. My training curve is something like these:

from waveglow.

@rafaelvalle is there anyway to fine-tune or re-traing or resume training of the model ?

from waveglow.

@chaiyujin thanks for your solution. I also met the NAN problem. When I set the one GPU with batch size 1, the training loss is fine. But when I set 8 GPU by using torch.nn.parallel.data_parallel with batch size 8, I got NAN loss after a few thousand steps. I adjust the learning rate from 1e-4 to 1e-5, then solved the NAN problem.

from waveglow.