Cannot train with multi GPUs about mellotron HOT 7 CLOSED

Pull from master and try again with FP16 enabled and disabled.

from mellotron.

Hi, rafaelvalle. I tried and it seems to be stuck here for a long time.
I change nothing in the hparams but turn the "fp16_run" and "distributed_run" to be true.

FP16 Run: True
Dynamic Loss Scaling: False
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing Distributed
Done initializing distributed
Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

Defaults for this optimization level are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic
Processing user overrides (additional kwargs that are not None)...
After processing overrides, optimization options are:
enabled                : True
opt_level              : O2
cast_model_type        : torch.float16
patch_torch_functions  : False
keep_batchnorm_fp32    : True
master_weights         : True
loss_scale             : dynamic

from mellotron.

Try with fp16_run=False

from mellotron.

Hi, rafaelvalle. I also got a same error.
I copied from newest code. And I modified distributed_run=True in hparams.py.
Then I execute blow command.
python train.py -o out_dir -l logdir -g

After a few minutes, below log appeared and it seemed to stop.

FP16 Run: False
Dynamic Loss Scaling: True
Distributed Run: True
cuDNN Enabled: True
cuDNN Benchmark: False
Initializing Distributed
Done initializing distributed

I checked my netword status with netstat -atno .
Then I find "localhost:54321 LISTEN" and "localhost => localhost:54321".
But process seems to stop...

from mellotron.

@n5-suzuki : for multi-gpu you should be running multiproc.
python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True

from mellotron.

I got the same error. It's the same problem for tacotron-pytorch.So sad!

from mellotron.

I think we can learn from this project and see how it is done to synthesis music rather than running this project. So I manually close this for lack of activity. If anybody has a solution, welcome to reopen and share it below.

from mellotron.