Hi, thank you for the nice work. When Im trying to recreate the experiment on my computer I encountered some failures:
$ bash pytorch_run.sh
utils/fix_data_dir.sh: file data/voxceleb2_train/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb2_train/spk2utt is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb2_train/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 1092009 utterances.
fix_data_dir.sh: old files are kept in data/voxceleb2_train/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/voxceleb2_train
utils/fix_data_dir.sh: file data/voxceleb2_test/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb2_test/spk2utt is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb2_test/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 36237 utterances.
fix_data_dir.sh: old files are kept in data/voxceleb2_test/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/voxceleb2_test
utils/fix_data_dir.sh: file data/voxceleb1_train/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb1_train/spk2utt is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb1_train/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 148642 utterances.
fix_data_dir.sh: old files are kept in data/voxceleb1_train/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/voxceleb1_train
utils/fix_data_dir.sh: file data/voxceleb1_test/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb1_test/spk2utt is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/voxceleb1_test/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 4874 utterances.
fix_data_dir.sh: old files are kept in data/voxceleb1_test/.backup
utils/validate_data_dir.sh: Successfully validated data-directory data/voxceleb1_test
utils/combine_data.sh /home/tjw/pytorch_xvectors/data/train /home/tjw/pytorch_xvectors/data/voxceleb2_train /home/tjw/pytorch_xvectors/data/voxceleb2_test /home/tjw/pytorch_xvectors/data/voxceleb1_train
utils/combine_data.sh [info]: not combining utt2uniq as it does not exist
utils/combine_data.sh [info]: not combining segments as it does not exist
utils/combine_data.sh: combined utt2spk
utils/combine_data.sh [info]: not combining utt2lang as it does not exist
utils/combine_data.sh [info]: not combining utt2dur as it does not exist
utils/combine_data.sh [info]: not combining utt2num_frames as it does not exist
utils/combine_data.sh [info]: not combining reco2dur as it does not exist
utils/combine_data.sh [info]: not combining feats.scp as it does not exist
utils/combine_data.sh [info]: not combining text as it does not exist
utils/combine_data.sh [info]: not combining cmvn.scp as it does not exist
utils/combine_data.sh [info]: not combining vad.scp as it does not exist
utils/combine_data.sh [info]: not combining reco2file_and_channel as it does not exist
utils/combine_data.sh: combined wav.scp
utils/combine_data.sh [info]: not combining spk2gender as it does not exist
fix_data_dir.sh: kept all 1276888 utterances.
fix_data_dir.sh: old files are kept in /home/tjw/pytorch_xvectors/data/train/.backup
steps/make_mfcc.sh --write-utt2num-frames true --mfcc-config conf/mfcc.conf --nj 40 --cmd run.pl /home/tjw/pytorch_xvectors/data/train /home/tjw/pytorch_xvectors/exp/make_mfcc mfcc
utils/validate_data_dir.sh: Successfully validated data-directory /home/tjw/pytorch_xvectors/data/train
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
run.pl: 40 / 40 failed, log is in /home/tjw/pytorch_xvectors/exp/make_mfcc/make_mfcc_train.*.log
fix_data_dir.sh: kept all 1276888 utterances.
fix_data_dir.sh: old files are kept in /home/tjw/pytorch_xvectors/data/train/.backup
sid/compute_vad_decision.sh --nj 40 --cmd run.pl /home/tjw/pytorch_xvectors/data/train exp/make_vad mfcc
compute_vad_decision.sh: no such file /home/tjw/pytorch_xvectors/data/train/feats.scp
fix_data_dir.sh: kept all 1276888 utterances.
fix_data_dir.sh: old files are kept in /home/tjw/pytorch_xvectors/data/train/.backup
steps/make_mfcc.sh --write-utt2num-frames true --mfcc-config conf/mfcc.conf --nj 40 --cmd run.pl /home/tjw/pytorch_xvectors/data/voxceleb1_test /home/tjw/pytorch_xvectors/exp/make_mfcc mfcc
utils/validate_data_dir.sh: Successfully validated data-directory /home/tjw/pytorch_xvectors/data/voxceleb1_test
steps/make_mfcc.sh: [info]: no segments file exists: assuming wav.scp indexed by utterance.
run.pl: 40 / 40 failed, log is in /home/tjw/pytorch_xvectors/exp/make_mfcc/make_mfcc_voxceleb1_test.*.log
fix_data_dir.sh: kept all 4874 utterances.
fix_data_dir.sh: old files are kept in /home/tjw/pytorch_xvectors/data/voxceleb1_test/.backup
sid/compute_vad_decision.sh --nj 40 --cmd run.pl /home/tjw/pytorch_xvectors/data/voxceleb1_test exp/make_vad mfcc
compute_vad_decision.sh: no such file /home/tjw/pytorch_xvectors/data/voxceleb1_test/feats.scp
fix_data_dir.sh: kept all 4874 utterances.
fix_data_dir.sh: old files are kept in /home/tjw/pytorch_xvectors/data/voxceleb1_test/.backup
awk: cannot open data/train/utt2num_frames (No such file or directory)
steps/data/reverberate_data_dir.py --rir-set-parameters 0.5, /home/tjw/pytorch_xvectors/RIRS_NOISES/simulated_rirs/smallroom/rir_list --rir-set-parameters 0.5, /home/tjw/pytorch_xvectors/RIRS_NOISES/simulated_rirs/mediumroom/rir_list --speech-rvb-probability 1 --pointsource-noise-addition-probability 0 --isotropic-noise-addition-probability 0 --num-replications 1 --source-sampling-rate 16000 data/train data/train_reverb
Number of RIRs is 40000
Traceback (most recent call last):
File "steps/data/reverberate_data_dir.py", line 682, in <module>
main()
File "steps/data/reverberate_data_dir.py", line 675, in main
max_noises_per_minute = args.max_noises_per_minute)
File "steps/data/reverberate_data_dir.py", line 433, in create_reverberated_copy
pointsource_noise_addition_probability, max_noises_per_minute)
File "steps/data/reverberate_data_dir.py", line 352, in generate_reverberated_wav_scp
speech_dur = durations[recording_id]
KeyError: 'id00012-21Uxsk56VDQ-00001'
cp: cannot stat 'data/train/vad.scp': No such file or directory
copy_data_dir.sh: no such file data/train_reverb/utt2spk
mv: cannot stat 'data/train_reverb.new': No such file or directory
steps/data/make_musan.sh --sampling-rate 16000 /home/tjw/pytorch_xvectors/musan data
steps/data/make_musan.py --use-vocals true --sampling-rate 16000 /home/tjw/pytorch_xvectors/musan data/musan
Preparing data/musan/musan...
In music directory, processed 645 files; 0 had missing wav data
In speech directory, processed 426 files; 0 had missing wav data
In noise directory, processed 930 files; 0 had missing wav data
utils/fix_data_dir.sh: file data/musan/utt2spk is not in sorted order or not unique, sorting it
utils/fix_data_dir.sh: file data/musan/wav.scp is not in sorted order or not unique, sorting it
fix_data_dir.sh: kept all 2001 utterances.
fix_data_dir.sh: old files are kept in data/musan/.backup
utils/subset_data_dir.sh: reducing #utt from 2001 to 645
utils/subset_data_dir.sh: reducing #utt from 2001 to 426
utils/subset_data_dir.sh: reducing #utt from 2001 to 930
fix_data_dir.sh: kept all 645 utterances.
fix_data_dir.sh: old files are kept in data/musan_music/.backup
fix_data_dir.sh: kept all 426 utterances.
fix_data_dir.sh: old files are kept in data/musan_speech/.backup
fix_data_dir.sh: kept all 930 utterances.
fix_data_dir.sh: old files are kept in data/musan_noise/.backup
utils/data/get_reco2dur.sh: obtaining durations from recordings
utils/data/get_reco2dur.sh: could not get recording lengths from sphere-file headers, using wav-to-duration
utils/data/get_reco2dur.sh: wav-to-duration is not on your path
utils/data/get_utt2dur.sh: segments file does not exist so getting durations from wave files
utils/data/get_utt2dur.sh: could not get utterance lengths from sphere-file headers, using wav-to-duration
utils/data/get_utt2dur.sh: wav-to-duration is not on your path
utils/data/get_utt2dur.sh: segments file does not exist so getting durations from wave files
utils/data/get_utt2dur.sh: could not get utterance lengths from sphere-file headers, using wav-to-duration
utils/data/get_utt2dur.sh: wav-to-duration is not on your path
utils/data/get_utt2dur.sh: segments file does not exist so getting durations from wave files
utils/data/get_utt2dur.sh: could not get utterance lengths from sphere-file headers, using wav-to-duration
utils/data/get_utt2dur.sh: wav-to-duration is not on your path
steps/data/augment_data_dir.py --utt-suffix noise --fg-interval 1 --fg-snrs 15:10:5:0 --fg-noise-dir data/musan_noise data/train data/train_noise
Traceback (most recent call last):
File "steps/data/augment_data_dir.py", line 298, in <module>
main()
File "steps/data/augment_data_dir.py", line 242, in main
dur = reco2dur[utt]
KeyError: 'id00012-21Uxsk56VDQ-00001'
steps/data/augment_data_dir.py --utt-suffix music --bg-snrs 15:10:8:5 --num-bg-noises 1 --bg-noise-dir data/musan_music data/train data/train_music
Traceback (most recent call last):
File "steps/data/augment_data_dir.py", line 298, in <module>
main()
File "steps/data/augment_data_dir.py", line 242, in main
dur = reco2dur[utt]
KeyError: 'id00012-21Uxsk56VDQ-00001'
steps/data/augment_data_dir.py --utt-suffix babble --bg-snrs 20:17:15:13 --num-bg-noises 3:4:5:6:7 --bg-noise-dir data/musan_speech data/train data/train_babble
Traceback (most recent call last):
File "steps/data/augment_data_dir.py", line 298, in <module>
main()
File "steps/data/augment_data_dir.py", line 242, in main
dur = reco2dur[utt]
KeyError: 'id00012-21Uxsk56VDQ-00001'
utils/combine_data.sh data/train_aug data/train_reverb data/train_noise data/train_music data/train_babble
utils/combine_data.sh: no such file data/train_reverb/utt2spk
utils/subset_data_dir.sh: no such file data/train_aug/utt2spk
utils/fix_data_dir.sh: no such file data/train_aug_1m/utt2spk
steps/make_mfcc.sh --mfcc-config conf/mfcc.conf --nj 40 --cmd run.pl data/train_aug_1m exp/make_mfcc mfcc
steps/make_mfcc.sh: no such file data/train_aug_1m/wav.scp
utils/combine_data.sh data/train_combined data/train_aug_1m data/train
utils/combine_data.sh: no such file data/train_aug_1m/utt2spk
local/nnet3/xvector/prepare_feats_for_egs.sh --nj 40 --cmd run.pl data/train_combined data/train_combined_no_sil exp/train_combined_no_sil
local/nnet3/xvector/prepare_feats_for_egs.sh: No such file data/train_combined/feats.scp
utils/fix_data_dir.sh: no such file data/train_combined_no_sil/utt2spk
local/nnet3/xvector/prepare_feats_for_egs.sh --nj 10 --cmd run.pl data/voxceleb1_test data/voxceleb1_test_no_sil exp/voxceleb1_test_no_sil
local/nnet3/xvector/prepare_feats_for_egs.sh: No such file data/voxceleb1_test/feats.scp
utils/fix_data_dir.sh: no such file data/voxceleb1_test_no_sil/utt2spk
mv: cannot stat 'data/train_combined_no_sil/utt2num_frames': No such file or directory
awk: cannot open data/train_combined_no_sil/utt2num_frames.bak (No such file or directory)
Can't open data/train_combined_no_sil/utt2spk: No such file or directory at utils/filter_scp.pl line 65.
fix_data_dir.sh: no utterances remained: not proceeding further.
fix_data_dir.sh: no utterances remained: not proceeding further.
sid/nnet3/xvector/get_egs.sh --cmd run.pl --nj 8 --stage 0 --frames-per-iter 1000000000 --frames-per-iter-diagnostic 100000 --min-frames-per-chunk 200 --max-frames-per-chunk 400 --num-diagnostic-archives 3 --num-repeats 50 data/train_combined_no_sil exp/xvector_nnet_1a/egs/
sid/nnet3/xvector/get_egs.sh: expected file data/train_combined_no_sil/feats.scp
Namespace(baseLR=0.001, batchSize=32, featDim=30, featDir='exp/xvector_nnet_1a/egs/', local_rank=0, logStepSize=200, maxLR=0.002, modelType='xvecTDNN', noiseEps=1e-05, numArchives=84, numEgsPerArk=366150, numEpochs=2, numSpkrs=7323, optimMomentum=0.5, pDropMax=0.2, preFetchRatio=30, preTrainedModelDir=None, protoEpisodesPerArk=25, protoMaxClasses=35, protoMinClasses=5, resumeModelDir=None, stepFrac=0.5, supportFrac=0.7, totalEpisodes=100, trainingMode='init')
Initializing Model..
Traceback (most recent call last):
File "train_xent.py", line 32, in <module>
net, optimizer, step, saveDir = prepareModel(args)
File "/home/tjw/pytorch_xvectors/train_utils.py", line 211, in prepareModel
output_device=0)
File "/home/tjw/anaconda3/envs/xvec/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 250, in __init__
).format(device_ids, output_device, {p.device for p in module.parameters()})
AssertionError: DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules, but got device_ids [0], output_device 0, and module parameters {device(type='cpu')}.
Traceback (most recent call last):
File "/home/tjw/anaconda3/envs/xvec/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/tjw/anaconda3/envs/xvec/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/tjw/anaconda3/envs/xvec/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
main()
File "/home/tjw/anaconda3/envs/xvec/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/tjw/anaconda3/envs/xvec/bin/python', '-u', 'train_xent.py', '--local_rank=0', 'exp/xvector_nnet_1a/egs/']' returned non-zero exit status 1.
Traceback (most recent call last):
File "extract.py", line 96, in <module>
main()
File "extract.py", line 39, in main
key=getSplitNum)[-1].split('/')[-1].lstrip('split'))
IndexError: list index out of range
Traceback (most recent call last):
File "extract.py", line 96, in <module>
main()
File "extract.py", line 39, in main
key=getSplitNum)[-1].split('/')[-1].lstrip('split'))
IndexError: list index out of range
run.pl: job failed, log is in xvectors/xvec_preTrained/train/log/compute_mean.log
run.pl: job failed, log is in xvectors/xvec_preTrained/train/log/lda.log
run.pl: job failed, log is in xvectors/xvec_preTrained/train/log/plda.log
run.pl: job failed, log is in xvectors/xvec_preTrained/test/log/voxceleb1_test_scoring.log
Traceback (most recent call last):
File "local/prepare_for_eer.py", line 12, in <module>
scores = open(sys.argv[2], 'r').readlines()
FileNotFoundError: [Errno 2] No such file or directory: 'xvectors/xvec_preTrained/test/scores_voxceleb1_test'
EER: %
minDCF(p-target=0.01):
minDCF(p-target=0.001):
Before running this I have modified 'voxceleb1_root' and 'voxceleb2_root' to my local dataset path, and set stage = 0.
The first failure seems to occur in run.pl when it outputs
run.pl: 40 / 40 failed, log is in /home/tjw/pytorch_xvectors/exp/make_mfcc/make_mfcc_train.*.log
# compute-mfcc-feats --write-utt2dur=ark,t:/home/tjw/pytorch_xvectors/exp/make_mfcc/utt2dur.17 --verbose=2 --config=conf/mfcc.conf scp,p:/home/tjw/pytorch_xvectors/exp/make_mfcc/wav_train.17.scp ark:- | copy-feats --write-num-frames=ark,t:/home/tjw/pytorch_xvectors/exp/make_mfcc/utt2num_frames.17 --compress=true ark:- ark,scp:/home/tjw/pytorch_xvectors/mfcc/raw_mfcc_train.17.ark,/home/tjw/pytorch_xvectors/mfcc/raw_mfcc_train.17.scp
# Started at Wed Dec 9 14:43:22 CST 2020
#
bash: line 1: compute-mfcc-feats: command not found
bash: line 1: copy-feats: command not found
# Accounting: time=0 threads=1
# Ended (code 127) at Wed Dec 9 14:43:22 CST 2020, elapsed time 0 seconds
Do you know what may cause this problem? Thanks in advance.