Training fails with torch.jit.script about waveglow HOT 6 CLOSED

Thanks for letting us know,. We reverted the PR while working on a solution.

from waveglow.

update to pytorch-1.0.0 but another question appears ,"ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead."

from waveglow.

@dchaws Please pull from master and update your pytorch to 1.0

from waveglow.

Update pytorch but still facing same issue

from waveglow.

slicing multiple dimensions at the same time isn't supported yet
@torch.jit.script
def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
n_channels_int = n_channels[0]
in_act = input_a+input_b
t_act = torch.nn.functional.tanh(in_act[:, :n_channels_int, :])
~~~~~~ <--- HERE
s_act = torch.nn.functional.sigmoid(in_act[:, n_channels_int:, :])
acts = t_act * s_act
return acts

from waveglow.

operation failed in interpreter:
@torch.jit.script
def fused_add_tanh_sigmoid_multiply(input_a, input_b, n_channels):
n_channels_int = n_channels[0]
in_act = input_a+input_b
~~~~~~~~~~~~~~~ <--- HERE
t_act = torch.tanh(in_act[:, :n_channels_int, :])
s_act = torch.sigmoid(in_act[:, n_channels_int:, :])
acts = t_act * s_act
return acts

pytorch 1.1

Epoch: 0
Traceback (most recent call last):
File "train.py", line 182, in
train(num_gpus, args.rank, args.group_name, **train_config)
File "train.py", line 121, in train
outputs = model((mel, audio))
File "/home/cory/anaconda3/envs/nvc/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/cory/ssd_dsk/waveglow/glow.py", line 240, in forward
output = self.WN[k]((audio_0, spect))
File "/home/cory/anaconda3/envs/nvc/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
File "/media/cory/ssd_dsk/waveglow/glow.py", line 162, in forward
torch.IntTensor([self.n_channels]))
File "/home/cory/anaconda3/envs/nvc/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, **kwargs)
RuntimeError: /pytorch/torch/csrc/jit/fuser/cuda/fused_kernel.cpp:137: a PTX JIT compilation failed

pytorch 1.0.1

Update

It seems the root cause for me was CUDA OOM and was showing the jit error message for some reason. Using PT 1.1 with jit now

from waveglow.