speaker selection on inference on finetuned libritts about styletts2 HOT 3 CLOSED

How many speakers do you have? If it is a single speaker dataset you are finetuning, you do not need any reference (and can even set multispeaker flag to false and do not load the pretrained diffusion model). Otherwise you do need a reference in the same way as the base model because the model needs to know the target speaker you want to synthesize. Or you can just hard code the speaker embeddings as a part of the model weights if you do not want any reference.

from styletts2.

My dataset is 1 speaker but libritts is many speakers. I used the libritts model you shared to finetune. I will try setting multispeaker flag to false

from styletts2.

It should work even if you set multispeaker flag to true. You just need a arbitrary reference audio from the training set. You can save this as a part of the parameters. For example,

text = '''Maltby and Company would issue warrants on them deliverable to the importer, and the goods were then passed to be stored in neighboring warehouses.
'''

reference_dicts = {}
reference_dicts['LJSpeech'] = "data/LJSpeech-1.1/wavs/LJ001-0001.wav"

start = time.time()
noise = torch.randn(1,1,256).to(device)
for k, path in reference_dicts.items():
    ref_s = compute_style(path)
    
    wav = inference(text, ref_s, alpha=0.9, beta=0.9, diffusion_steps=10, embedding_scale=1)
    rtf = (time.time() - start) / (len(wav) / 24000)
    print(f"RTF = {rtf:5f}")
    import IPython.display as ipd
    print(k + ' Synthesized:')
    display(ipd.Audio(wav, rate=24000, normalize=False))
    print('Reference:')
    display(ipd.Audio(path, rate=24000, normalize=False))

from styletts2.