Why you do not use pytorch native layer norm? about glow-tts HOT 7 CLOSED

You can use it like this without transposing:
nn.GroupNorm(1, channels)

from glow-tts.

There was no critical reason not to use native implementation of layer norm. You can replace my implementation with the native one. You may need to transpose the last two dimensions for replacing it.

When I decided to implement by my self, I just found the native layer norm operates on last dimensions, whereas I wanted it to operate on the channel dimension only (among the shape [batch_size, channel, time_step]).

Closing the issue.

from glow-tts.

@jaywalnut310
Why dont you use pytorch vanilla gradient clipping?

from glow-tts.

@hadaev8
I didn't know GroupNorm can replace my implementation of LayerNorm, thank you:)
Actually, the gradient clipping is same with the pytorch vanilla clip_grad_value_.
When I implemented it, I wanted to clip gradients by value, and at the same time, I also wanted to get the gradient norm. As executing two clipping methods (clip_grad_value_ for gradient clipping, and clip_grad_norm_ for getting gradient norm) is time-consuming, I implemented the clipping method by my self.

from glow-tts.

@jaywalnut310
I tried to start train (private russian dataset, text tokenization changed), but in fp16 model crush at some point: apex gradient scaling goes to zero.
Any ideas why it should happen?

Also, does audio noise really make train better? Given, it divided by max wav value, noise value is on fp16 border.

from glow-tts.

@hadaev8
It seems that you encountered in the problem of numerically unstable loss. What I could tell is, you can switch on/off add_noise and fp16_run configs. fp32 training could increase numerical stability, and audio noise is also unnecessary. I add audio noise just for dequantizing 16bit-quantized wav.

If the unstable training process is similar to #15,
would you discuss in the issue?

from glow-tts.

@jaywalnut310
What's the benefit of dequantization?
This is sample spectrogram
https://i.imgur.com/WL40jpX.png

And this is the difference between 2 spectrograms
https://i.imgur.com/WITFuT1.png

So basically it affects no sound silence parts.

Also, shouldn't it be torch.rand_like(audio) - 0.5?

from glow-tts.