Unfortunately your reference concerning phonemes does not provide a reference other than the link to CMU Sphinx.

I did a bit of research and ended up with the following code:

def create_phoneme(audio_wave_file):
    with wave.open(audio_wave_file, "rb") as audio:
        decoder = Decoder(samprate=audio.getframerate(), allphone=ps.get_model_path("en-us/en-us-phone.lm.bin"))
        decoder.start_utt()
        decoder.process_raw(audio.getfp().read(), full_utt=True)
        decoder.end_utt()

    input_phoneme_list = []
    if decoder.hyp():
        segments = decoder.seg()
        for seg in segments:
            input_phoneme_list.append({'phone': seg.word, 'phone_end_frame': seg.end_frame})
    else:
        raise Exception('Phoneme recognition failed')

    total_number_of_frames_in_audio = int(input_phoneme_list[-1]['phone_end_frame'] / 100 * ASSUMED_FRAME_RATE)
    print(total_number_of_frames_in_audio)

    frame_index = 0
    phone_list = []
    phone_index = 0

    while frame_index < total_number_of_frames_in_audio:
        if (frame_index * 100 / ASSUMED_FRAME_RATE) < input_phoneme_list[phone_index]['phone_end_frame']:
            phone_list.append(input_phoneme_list[phone_index]['phone'])
            frame_index += 1
        else:
            phone_index += 1

    with open(str("phindex.json")) as f:
        ph2index = json.load(f)
    phonemes = []
    for p in phone_list:
        if p in ph2index:
            phonemes.append(ph2index[p])
        else:
            print(f"Weird Phoneme found: {p}. Ignoring...")
            phonemes.append(31) # Silence

    phone_list = phonemes

    print("Phoneme generation done")

    return phone_list

I'm using the phindex.json file from https://github.com/FuxiVirtualHuman/AAAI22-one-shot-talking-face/blob/main/phindex.json and a ASSUMED_FRAME_RATE of 30 (this seems to match the number of phonemes you have in the samples rather than 25 as referenced in the papers).

However my phonemes look a lot different as compared to your samples for the sample wave files. What am I doing wrong?

Where can I download the supplementary materials? Thanks.

Thanks you for your great research.
What is the license of code?

@FuxiVirtualHuman hey i have made a colab noteook if you find it useful you can add on your page https://colab.research.google.com/drive/1tqpbUBYNaw0CYpaZEo2ONNm8lpj375F4?usp=sharing

Hi, I have added my own wav file (16k, pcm, 16000hz) and get this error.

Traceback (most recent call last):
File "/content/styletalk/inference_for_demo.py", line 168, in
generate_expression_params(
File "/usr/local/lib/python3.10/dist-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/content/styletalk/inference_for_demo.py", line 107, in generate_expression_params
audio_win = torch.tensor(audio_win).cuda()
TypeError: can't convert np.ndarray of type numpy.object. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
video created!

OK so it wasnt the wav file it was the json not being in the correct format.

Amazing work!
When do you plan to publish the code? Or the inference code and weights should be public for comparison. Thanks!

hi sir,

as i have seen above you tell that code release on feb 2023,but now march also gone so please upload code as soon as possible

Thanks ,public is waiting for your code please release as soon as possible

Is there any plan to release the training code?

@YifengMa9
In the paper, the param was divided into 13 and 51.

We select 13 out of 64 expression parameters that are highly related to mouth movements as the lower
face group, and the other parameters as the upper face group.

But in the code: in core.networks.disentangle_decoder import DisentangleDecoder

upper_face3d_indices=tuple(list(range(19)) + list(range(46, 51))),
lower_face3d_indices=tuple(range(19, 46)),

upper and lower gorup has 24 and 27 elements, 51 in total, 13 were missing from 64.

Can you help me with this question?
Many thanks

请问，怎么在中文的音频上测试，目前开源的模型在别的语种效果好么

Thanks for publishing the code

https://huggingface.co/spaces/camenduru-com/one-shot-talking-face

Hi there, can you share any more details about the discriminators you used? E.g how many layers do they have? I can't seem to find in the paper or supplementary details. Thanks for the great work.

I'm getting very strange results with the face being distorted and I'm unsure if it's a matter of not being aligned better, or there's some additional step per-identity that needs to happen first
All I did here, compared to the sample Reagan and Andrew Ng sample, is to replace the input image