How to do facial reenactment? about 4d-facial-avatars HOT 6 OPEN

Hi, I tried to implement the facial reenactment and got the results below. But they don't look as neat as Fig.7 in the paper.

I use person_1 as target and person_2 as driving. I added custom_seq_driving before line 367 in eval_transformed_rays.py to compute the transferred expressions and poses. Then I use the obtained expressions and poses inside the for loop.

I tried either transferring the raw expressions or the expression deltas (where I manually choose frame 973 in person_1 and frame 990 in person_2 as neutral expression).

• Results by directly transferring expression (by setting transfer_delta_expressions_from_netural to False.)

• Results by transferring expression delta (by setting transfer_delta_expressions_from_netural to True.)

Any help and suggestion will be appreciated. Thanks for your time!

from 4d-facial-avatars.

Hey, if you use the expressions and poses from one actor with a trained model of another, you will basically get reenactment.
There are a few things to be careful with:

• provide the correct background image you want to use
• the expression and face identity coefficients are quite entangled (this is inherent in the 3DMM face model), so instead of using the expressions themselves, it looks better if you just transfer the 'expression delta' from one person's "neutral expression" to the other person's "netural expression". This logic for this is in real_to_nerf.py.
• If you go for head angles that are beyond the angles in the training of the model, it obviously won't look good.

from 4d-facial-avatars.

Thanks for the reply, is there any plan to release related code in the near future?

from 4d-facial-avatars.

Hi, I tried to implement the facial reenactment and got the results below. But they don't look as neat as Fig.7 in the paper.

I use person_1 as target and person_2 as driving. I added custom_seq_driving before line 367 in eval_transformed_rays.py to compute the transferred expressions and poses. Then I use the obtained expressions and poses inside the for loop.

I tried either transferring the raw expressions or the expression deltas (where I manually choose frame 973 in person_1 and frame 990 in person_2 as neutral expression).

• Results by directly transferring expression (by setting transfer_delta_expressions_from_netural to False.)

• Results by transferring expression delta (by setting transfer_delta_expressions_from_netural to True.)

Any help and suggestion will be appreciated. Thanks for your time!

Hi Thanks for sharing the solution. But I meet the index out of bound error when I add code before line 367. Could you share more information how should I change the code? Thanks.
_, posesD, _, _, _, expressionsD, _, _ = load_flame_data(
"nerface_dataset/person_2",
half_res=cfg.dataset.half_res,
testskip=cfg.dataset.testskip, test=True) #i_train, i_val, i_test = i_split
i_test = i_split
rigid_poses_driving = posesD[i_test].float().to(device)
expressions_driving = expressionsD[i_test].float().to(device)
render_expressions,render_poses = custom_seq_driving(rigid_poses_driving,render_poses,expressions_driving,render_expressions)

from 4d-facial-avatars.

Hi, I tried to implement the facial reenactment and got the results below. But they don't look as neat as Fig.7 in the paper.
I use person_1 as target and person_2 as driving. I added custom_seq_driving before line 367 in eval_transformed_rays.py to compute the transferred expressions and poses. Then I use the obtained expressions and poses inside the for loop.
I tried either transferring the raw expressions or the expression deltas (where I manually choose frame 973 in person_1 and frame 990 in person_2 as neutral expression).

• Results by directly transferring expression (by setting transfer_delta_expressions_from_netural to False.)
• Results by transferring expression delta (by setting transfer_delta_expressions_from_netural to True.)

Any help and suggestion will be appreciated. Thanks for your time!

Hi Thanks for sharing the solution. But I meet the index out of bound error when I add code before line 367. Could you share more information how should I change the code? Thanks. _, posesD, _, _, _, expressionsD, _, _ = load_flame_data( "nerface_dataset/person_2", half_res=cfg.dataset.half_res, testskip=cfg.dataset.testskip, test=True) #i_train, i_val, i_test = i_split i_test = i_split rigid_poses_driving = posesD[i_test].float().to(device) expressions_driving = expressionsD[i_test].float().to(device) render_expressions,render_poses = custom_seq_driving(rigid_poses_driving,render_poses,expressions_driving,render_expressions)

See details #57 and #37

from 4d-facial-avatars.

Hi, I tried to implement the facial reenactment and got the results below. But they don't look as neat as Fig.7 in the paper.
I use person_1 as target and person_2 as driving. I added custom_seq_driving before line 367 in eval_transformed_rays.py to compute the transferred expressions and poses. Then I use the obtained expressions and poses inside the for loop.
I tried either transferring the raw expressions or the expression deltas (where I manually choose frame 973 in person_1 and frame 990 in person_2 as neutral expression).

• Results by directly transferring expression (by setting transfer_delta_expressions_from_netural to False.)
• Results by transferring expression delta (by setting transfer_delta_expressions_from_netural to True.)

Any help and suggestion will be appreciated. Thanks for your time!

Hi Thanks for sharing the solution. But I meet the index out of bound error when I add code before line 367. Could you share more information how should I change the code? Thanks. _, posesD, _, _, _, expressionsD, _, _ = load_flame_data( "nerface_dataset/person_2", half_res=cfg.dataset.half_res, testskip=cfg.dataset.testskip, test=True) #i_train, i_val, i_test = i_split i_test = i_split rigid_poses_driving = posesD[i_test].float().to(device) expressions_driving = expressionsD[i_test].float().to(device) render_expressions,render_poses = custom_seq_driving(rigid_poses_driving,render_poses,expressions_driving,render_expressions)

Have you solved this problem yet？

from 4d-facial-avatars.