Expression Retargeting about spectre HOT 10 CLOSED

Hey, thanks for your interest. Yes, it is possible if you only transfer the predicted expression and jaw pose parameters (i.e. pose[:,3:6]). You can find an example of that in the original DECA (https://github.com/YadiraF/DECA/blob/master/demos/demo_transfer.py).

from spectre.

@filby89 , What is value range of expression data predicted by Spectre model?
Since, It's based on FLAME face model, Their value range is (-3,3) i think. How can i map these value to lower range having same effect.

from spectre.

Hey @ujjawalcse. Yes it follows the FLAME model. I have tested the ranges of DECA, EMOCA, and SPECTRE and found out that generally the values of expression in EMOCA and SPECTRE are much large than DECA. On a small subset of videos I have found that the maximum value of DECA for the first component is around 3.5, EMOCA 6.5 and SPECTRE 9. However, in some other components SPECTRE can be as large as 35 in some cases (although I have not seen how this component affects the output specifically).
Can you list more details regarding what you wanna do ?

from spectre.

Sure @filby89 ,
Actually, I wanna assign these expressions values to a FLAME model in 3D software (like Blender) so that the mesh can be deformed and exported into AR/VR friendly format. But when I checked with the Spectre expression values, sometimes it's deforming the mesh too much such that the faces are looking awkward.

from spectre.

@ujjawalcse Can you post an example of parameters ?

from spectre.

Sure,
Please find the below .zip file containing one .npy file.
https://drive.google.com/file/d/1zrlIj0pNktoYkwFZ2MH3f9HOdyndcmPR/view?usp=sharing

It contains all the parameters saved after adding Spectre predicted parameter with the DECA initials parameters.

Here you can see,how i saved them.
Line no. 192, Just keep appending the final parameters to one empty list codedicts defined outside the loop.
Line 197, Saved the updated list to a .npy file.
While checking this .npy file, First 2 frames and last 2 frames had unusual values as they are corresponding to the padded frames i think, so I removed them.

from spectre.

Hey,
you are correct, first 2 and last frames correspond to padded frames and contain garbage. I reconstructed the .npy file and got the following result:

I guess that the problems you are talking about are frames like the next one ?

We have found out that this happens in a minority of videos and it is due to DECA, not estimating correctly the camera parameters. Because SPECTRE only predicts the expression parameters and jaw, it relies on DECA for the camera matrix and if DECA fails, artifacts like this can appear.

I am not sure if that will fix the problem, but you could tweak a little the scale parameter in the dataset class and zoom out a bit. We have it fixed at 1.6 for testing, but during training we use a range from [1.4,1.8].

from spectre.

Thanks @filby89 for your clear expression.
The scale parameter in DECA is 1.25.
Should I test with 1.6 or any value between 1.4 and 1.8 ?

from spectre.

@ujjawalcse I would suggest running the demo in this repository with multiple values (either 1.25 or between 1.4 and 1.8) and see which one fits your case :).

from spectre.

Right! Thanks, @filby89.

from spectre.