georgeretsi / smirk Goto Github PK

Hello!
Thank you for Amazing work!
I know it's a stupid question,
An error occurred when I generated demo.py : Missing file pretrained_models/SMIRK_em1.pt

Hi!

• Would you let me know each mediapipe 105 landmark represent which part of face?

  • mediapipe landmarks return 105 landmark (

    smirk/src/FLAME/FLAME.py

    Line 314 in b7c08bd

    'landmarks_mp': landmarksmp

    )

  • Like this figure.

    

can u fix some errors about emotion_loss?

Hi,
Thanks for sharing this amazing repo 😄
I was trying to retrain the model with a couple of modifications.
In the processes I faced a couple of hitches while trying to run the pre-processing code on the mentioned datasets
Would help me a lot if you can guide me through them

Errors:

1. In apply_mediapipe_to_dataset.py, inside the def preprocess_sample() function,
   you used model_asset_path='assets/face_landmarker.task' but the file is not provided within the assets folder.
   Could you share a link from where I can download this / update the repo with this file ?
2. In apply_fan_to_dataset.py I noticed you os.walk() the root, store the paths and then in L36 we loop on those pairs
   In the process we were also storing .mp4 and .avi but trying to read them with cv2.imread() in L44.
   This would in the end throw None type has no shape attribuite error. How do I handle this case / video datasets ?

Generic Doubts:

1. Since LRS3 is not available anymore, I was trying to make it work for LRS2. There were 2 subsets in it - pretrain and main. Should I combine them together and then preprocess it ?
2. I have not used MEAD dataset before but from the config files, I noticed that the code uses MEAD_front and MEAD_sides
   But the configs do not have provision landmark paths for MEAD_sides. Are they already provided with the dataset ?
3. I noticed that the output flame params from the model just has 50 expression params. I wanted retrain the model so that it outputs 100 expr params instead. For this, is it sufficient to just modify in the config and retrain or should I replace the flame implementation as a whole ?

Hello George!
     Thank you for your work and open-source contributions; it's indeed very enlightening.  I have a few simple questions about your work:
      1) Besides the common encoding of 3DMM params such as shape, expression, global pose+jaw pose, and caw, an additional 'eyelip' parameter is used in smirk. I'm not sure if this corresponds to the eye_pose parameter in the generic FLAME model. Has the FLAME model used in your open-source code been modified as shown in Figures 1 and 2, or does this parameter serve an additional purpose? Can it be used for decoding with the generic FLAME model?
      2) For the same input image (as shown in Figure 3), SMIRK's reconstruction shows a better eye closure effect compared to DECA and Emoca, as shown in Figures 4 and 5. Since SMIRK does not provide code for generating meshes, I used the decoded FLAME vertices from SMIRK combined with the generic FLAME faces to generate an OBJ file. Although SMIRK's reconstruction is closer to the input, a noticeable issue is the overlapping of eyelids and eyeballs, which is not an isolated case (as seen in the zoom in of image). What could be causing this problem, and is there a solution?

                                                                         fig1: the 3DMM params in smirk

                                                                         fig2: the 3DMM params in deca and emoca

fig3: the input image

                            fig4: the smirk result

                        fig5: the emoca result

Since the pre-training is based on MICA, Is there a way to use Metrical_Tracker that comes with MICA, with SMIRK?

If there is no direct way and need a bit of development effort, I am happy to do it and send a PR. Just need directions on how to go about it.

Hi, I'm looking forward to giving this a test drive. This is a general question about 3DMM's and face tracking. Having read this board, It sounds like smirk won't yet produce smooth results on video input. Outside of EMOCAv2 and MICA, are you aware of any repo's that have pushed that work further?

I've seen FlawlessAI's new paper improving results on 2D and 3D landmarks, but that's not publicly available code.

Hi!I'm very interested in smirk_generator.

it could re-generate the entire face very well.I noticed that the covered part is not completely black. It randomly samples the relevant information of the original image. I want to know what the training process of this model is.

The current results exist as monochromatic rendered mesh images. How can I directly obtain the mesh source files, for instance, stored in a common 3D file format like OBJ or others? Additionally, the paper mentions using FLAME as the facial prior, but the results do not appear to have FLAME topology. Is additional post-processing required?

Hi!

Amazing work!

Is there any smoothing operation?

Hello!

Amazing work!

Why take negative for transformed_vertices：transformed_vertices[:, :, 1:] = -transformed_vertices[:, :, 1:]

Hi! Thanks for sharing great work!

What FLAME version used in SMIRK??

• FLAME 2020 / general .pkl ?

• I am asking since advanced FLAME versions are updated in https://flame.is.tue.mpg.de/download.php

  • 

• If I want to use FLAME 2023, need to re-train and re-adjust FLAME parameter corresponding 2023 version??

Thanks!

Hello, would you like to let me know how to create mask to reconstruct only mouth area not full face?
In addition to this we can reconstruct lips part with different jaw pose?
Looking forward to hearing from you!
Thanks

Hi, will you release the pre-trained weights of smirk?

Hello, everybody.
I have got jaw_pose parameter using smirk.
([[[ 0.0809, -0.0012, -0.0506]]], device='cuda:0')

But jaw_pose in metrical_tacker has different format.
([[1., 0., 0., 0., 1., 0.]], device='cuda:0', requires_grad=True)

Would you like to let me know which format is correct and how to convert with each other?
Thanks

Here are the obtained pose and cam. Does the cam here refer to the camera parameters? Is it possible to select a camera parameter with a front-facing orientation?

Can the following be made to face forward?

hope you can answer my question, thank you.

Hello!

Great work!

Is there a way to input videos/track SMIRK over frames?