Problems creating Tensorflow-based Dataloader about tensorflowtts HOT 38 CLOSED

Using csmsc data, amended the sampling rate, it seems invalid?

from tensorflowtts.

@dathudeptrai

from tensorflowtts.

@rgzn-aiyun the error said that the file 000001.wav seem to have a different sampling rate compared with sampling_rate on ljspeech_preprocess.yaml. Pls check the sampling rate.

from tensorflowtts.

@rgzn-aiyun pls change sampling rate on ljspeech_processes.yaml to 24000

from tensorflowtts.

@rgzn-aiyun your bug here https://github.com/dathudeptrai/TensorflowTTS/blob/master/tensorflow_tts/bin/preprocess.py#L164-L165. That mean a file 000001.wav has a sampling rate difference with config["sampling_rate"].

from tensorflowtts.

The BZNSYP.rar data used should be reasonable.

from tensorflowtts.

@rgzn-aiyun your bug here https://github.com/dathudeptrai/TensorflowTTS/blob/master/tensorflow_tts/bin/preprocess.py#L164-L165. That mean a file 000001.wav has a sampling rate difference with config["sampling_rate"].

Looks like uncompressed PCM WAV format, sampling rate is 48kHz, 16bit？Before ParallelWaveGAN adopted 24000, it seemed no problem.

from tensorflowtts.

@rgzn-aiyun you can ignore those sample, just modify tensorflow_tts/bin/preprocess.py. see https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/parallel_wavegan/bin/preprocess.py#L133-L134, my code is same.

from tensorflowtts.

@rgzn-aiyun you can ignore those sample, just modify tensorflow_tts/bin/preprocess.py. see https://github.com/kan-bayashi/ParallelWaveGAN/blob/master/parallel_wavegan/bin/preprocess.py#L133-L134, my code is same.

Modified the code, it seems to run, but the data display seems to be a problem?
[Preprocessing]: 6% 625/10000 [01:30<22:35, 6.92it/s]
[Preprocessing]: 6% 625/10000 [01:39<24:59, 6.25it/s]
[Preprocessing]: 6% 625/10000 [01:43<25:47, 6.06it/s]
[Preprocessing]: 6% 625/10000 [01:48<27:01, 5.78it/s]
[Preprocessing]: 6% 625/10000 [03:13<48:27, 3.22it/s]
[Preprocessing]: 6% 625/10000 [03:20<50:00, 3.12it/s]
[Preprocessing]: 6% 625/10000 [03:23<50:58, 3.07it/s]
[Preprocessing]: 6% 625/10000 [03:27<51:52, 3.01it/s]
[Preprocessing]: 6% 625/10000 [04:59<1:14:56, 2.09it/s]
[Preprocessing]: 6% 625/10000 [05:05<1:16:28, 2.04it/s]
[Preprocessing]: 6% 625/10000 [05:08<1:17:05, 2.03it/s]
[Preprocessing]: 6% 625/10000 [05:10<1:17:35, 2.01it/s]
[Preprocessing]: 6% 625/10000 [06:45<1:41:22, 1.54it/s]
[Preprocessing]: 6% 625/10000 [06:47<1:41:47, 1.53it/s]
[Preprocessing]: 6% 625/10000 [06:53<1:43:15, 1.51it/s]
[Preprocessing]: 0% 0/10000 [06:55<?, ?it/s]
[Preprocessing]: 6% 625/10000 [06:55<1:43:58, 1.50it/s]

from tensorflowtts.

@rgzn-aiyun it's multi-processing so it's ok. BTW, what is a code you modify?

from tensorflowtts.

@rgzn-aiyun it's multi-processing so it's ok. BTW, what is a code you modify?

The direct sampling rate is changed to 24000, the error should be in the last step of saving data in p.map(save_to_file, range(len(processor.items))).

from tensorflowtts.

There is an error:

--use-norm 1 is missing a symbol /

from tensorflowtts.

Traceback (most recent call last):
File "examples/fastspeech/train_fastspeech.py", line 31, in
from examples.fastspeech.fastspeech_dataset import CharactorDurationMelDataset
ModuleNotFoundError: No module named 'examples.fastspeech'

Can't find this file from the beginning of training?

from tensorflowtts.

you should run the command line outside of the folder examples so it can understand module named examples.fastspeech, i guess you are running the script inside a folder example/fastspeech

from tensorflowtts.

you should run the command line outside of the folder examples so it can understand module named examples.fastspeech, i guess you are running the script inside a folder example/fastspeech

Run in the TensorflowTTS directory

from tensorflowtts.

@dathudeptrai
tensorflow-tts-normalize --rootdir ./dump --outdir ./dump --stats ./dump/stats.npy --config preprocess/ljspeech_preprocess.yaml
2020-06-06 07:29:40.990457: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-06 07:29:40.992572: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-06-06 07:29:40.992626: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: 9b3e55fb2443
2020-06-06 07:29:40.992644: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: 9b3e55fb2443
2020-06-06 07:29:40.992731: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 418.67.0
2020-06-06 07:29:40.992769: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 418.67.0
2020-06-06 07:29:40.992782: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 418.67.0
2020-06-06 07:29:40.993082: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2020-06-06 07:29:40.997299: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2000165000 Hz
2020-06-06 07:29:40.997480: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x29df2c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-06 07:29:40.997507: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
0it [00:00, ?it/s]
Traceback (most recent call last):
File "/usr/local/bin/tensorflow-tts-normalize", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/tensorflow_tts/bin/normalize.py", line 115, in main
np.save(os.path.join(args.outdir, subdir, "norm-feats", f"{utt_id}-norm-feats.npy"),
UnboundLocalError: local variable 'subdir' referenced before assignment

It seems a lot of errors.

from tensorflowtts.

I cannot reproduce those error. Pls see #19

from tensorflowtts.

I cannot reproduce those error. Pls see #19

I've replaced utt_id = utt_id[0].numpy().decode("utf-8") with utt_id = utt_id[0].numpy().decode("utf-8").replace("-raw","") and everything worked.

There seems to be an error when starting training?
2020-06-06 08:10:26,781 (train_fastspeech:341) INFO: mixed_precision = False
2020-06-06 08:10:26,781 (train_fastspeech:341) INFO: version = 0.6.1
Traceback (most recent call last):
File "train_fastspeech.py", line 440, in
main()
File "train_fastspeech.py", line 369, in main
return_utt_id=False
File "/ai/TensorflowTTS/fastspeech_dataset.py", line 72, in init
duration_files = [duration_files[idx] for idx in idxs]
File "/ai/TensorflowTTS/fastspeech_dataset.py", line 72, in
duration_files = [duration_files[idx] for idx in idxs]
IndexError: list index out of range

from tensorflowtts.

Pls see the last sentence of step 1. You need extract duration from teacher model like tacotron to be able train fastspeech

from tensorflowtts.

Pls see the last sentence of step 1. You need extract duration from teacher model like tacotron to be able train fastspeech

*-durations.npy？It seems that there are only four directories: id, raw-feat, norm-feats, and wave.

from tensorflowtts.

Pls read step 1 again. You need extract duration files base on tacotron 2. You should train tacotron before you train fastspeech.

from tensorflowtts.

Pls read step 1 again. You need extract duration files base on tacotron 2. You should train tacotron before you train fastspeech.

What is the connection? Training tacotron alone will undoubtedly add more time!

from tensorflowtts.

The duration files are extracted from tacotron alignments

from tensorflowtts.

Traceback (most recent call last):
File "examples/fastspeech/train_fastspeech.py", line 31, in
from examples.fastspeech.fastspeech_dataset import CharactorDurationMelDataset
ModuleNotFoundError: No module named 'examples.fastspeech'

Can't find this file from the beginning of training?

TensorflowTTS/examples is missing a init.py file.

from tensorflowtts.

Ok i will fix those bugs tonight.

from tensorflowtts.

@dathudeptrai
2020-06-06 10:06:22.365322: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 9460 of 9500
2020-06-06 10:06:31.056826: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1986, in execution_mode
yield
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 655, in _next_internal
output_shapes=self._flat_output_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2363, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.DataLossError: Attempted to pad to a smaller size than the input element. [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train_tacotron2.py", line 507, in
main()
File "train_tacotron2.py", line 500, in main
resume=args.resume)
File "train_tacotron2.py", line 343, in fit
self.run()
File "/ai/TensorflowTTS/tensorflow_tts/trainers/base_trainer.py", line 72, in run
self._train_epoch()
File "/ai/TensorflowTTS/tensorflow_tts/trainers/base_trainer.py", line 92, in _train_epoch
for train_steps_per_epoch, batch in enumerate(self.train_data_loader, 1):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 631, in next
return self.next()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 670, in next
return self._next_internal()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 661, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1989, in execution_mode
executor_new.wait()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.DataLossError: Attempted to pad to a smaller size than the input element.
[train]: 0% 0/200000 [22:42<?, ?it/s]

Just started to report errors after a while?

from tensorflowtts.

What is ur version of tf. I never have this problem. @ZDisket do u have this problem ? I see u can be train tacotron 2

from tensorflowtts.

@dathudeptrai Never heard of it, and I've trained two Tacotron2 models (well actually three since I redid my first model on Tacotron2-120k), along with two MelGAN-STFTS
Although I did have the can't find 'examples.xxx' error in Google Colaboratory, but I solved it by renaming the folder to ttsexamples and doing a simple search and replace on all the .py files to adjust this accordingly.

from tensorflowtts.

i fixed utt_ids wrong return, close issue

from tensorflowtts.

What is ur version of tf. I never have this problem. @ZDisket do u have this problem ? I see u can be train tacotron 2

Use tacotron 2,I ran it again today and still had the same problem.

from tensorflowtts.

@rgzn-aiyun did u use tf 2.2 ?

from tensorflowtts.

@rgzn-aiyun did u use tf 2.2 ?

Use the latest tacotron2 version.

from tensorflowtts.

@rgzn-aiyun i mean tensorflow version

from tensorflowtts.

@rgzn-aiyun i mean tensorflow version

yes,tensorflow-2.2.0.dist-info.

from tensorflowtts.

@rgzn-aiyun https://github.com/dathudeptrai/TensorflowTTS/blob/master/examples/tacotron2/conf/tacotron2.v1.yaml#L53-L54. Please modify max_char_length and max_mel_length based on ur dataset.

from tensorflowtts.

@rgzn-aiyun https://github.com/dathudeptrai/TensorflowTTS/blob/master/examples/tacotron2/conf/tacotron2.v1.yaml#L53-L54. Please modify max_char_length and max_mel_length based on ur dataset.

Thanks, how to get this value through calculation?

from tensorflowtts.

@rgzn-aiyun you can read all ur -ids.npy files and -norm-feats.npy files the u return maximum len ids and maximum len norm-feats.

from tensorflowtts.

@rgzn-aiyun you can read all ur -ids.npy files and -norm-feats.npy files the u return maximum len ids and maximum len norm-feats.

Hi, it seems to be a problem with batch_size: 30, set to batch_size: 16 and run successfully.

from tensorflowtts.