Gradient overflow and negative loss about glow-tts HOT 5 CLOSED

I think the negative loss is normal, since the loss is log- scalar, which can be negative.

However, I encountered another issue that the grad_norm is larger than you (starting from ~50 and decrease to ~5, then increase to ~10), while it seems that your grad norm is always ~1.

from glow-tts.

add_noise is about whether to add noise in the input data or not. It is not related with noise_scale. At inference time, you can change noise_scale between 0 to 1 for finding best sample quality.

from glow-tts.

Besides, I successfully trained the model by using the LJ dataset and the synthezised voice is quite well.
I also noticed that during the training process, the l_mle_normal is negative:

My question is, why the loss will become negative?

Thank you!

from glow-tts.

@Charlottecuc Yes, the negative loss is normal, as it stans for negative log-likelihood of data. If you encounter NAN loss, I have no perfect solution, but I can give you a suggestion.

You can change configs add_noise to false, and fp16_run to false. Switching on/off theses configs may help to increase numerical stability.

from glow-tts.

@jaywalnut310 Hi. I changed add_noiseto false and it seems that the model is more stable than before. Besides, after changing the "add_noise" to false, do I also need to change the value of "noise_scale" in the inference time? Thanks!

from glow-tts.