Detecting face landmarks using previous frame's landmarks about superviseddescent HOT 8 CLOSED

Hi,

What you're trying to do with initialising using the previous points was also my first intuition, and similar to you, it didn't work for the reason you mentioned. I think what's happening is the following:

• Initialising with the mean landmarks in each frame acts as a kind of regularisation, we go back near the space of valid faces. If you're curious, what you could try is train a model and train it with a high regularisation value, so the training error would be quite large - and then see if the landmarks also get scattered in your testing. I guess this would help analyse this behaviour.
• If you're initialising with the previous frame, the landmarks will be fairly close to the true location already (depending on how much the subject moves obviously). This is something that the regressor (or at least the first) actually has not learned from the training data. In the training, we initialise from the face-boxes, and thus quite far away from the true location. So if we're already close, the regressor in a sense can't really know the right thing to do. What I would do here is re-train a model with a different initialisation strategy, i.e. perturb around the ground-truth location of the training landmarks, and not around the face box. I've started this because I think it's a neat idea but never finished it so far.

I hope this helps, let me know how it goes and I'm glad to help more, I'm quite curious as well. (It's unfortunately the stuff that never gets mentioned in papers, for example Xiong & De La Torre don't disclose this information in their paper. But I'm actually pretty sure my idea should work (and might also be exactly how they do it)).

from superviseddescent.

Thank you for your detailed response. I'm working to implement your ideas. I'll post here if I manage to make any headway. Thanks again!

from superviseddescent.

Cool to hear that! Just let me know if you need any help.

from superviseddescent.

I have ever implement the idea "perturb around the ground-truth location of the training landmarks", however the tracking for rotation was week. So, did you have some better idea for simulating the perturbation of rotation.

from superviseddescent.

You can correct for rotation by in-plane rotating the input image. That way you don't need to train it. I think there's a discussion about it in some of the (possibly closed) issues here.
Alternatively you can try perturbing in the training with a 2D rotation.

I'm closing this since the original question is resolved, and the new question is kind of off-topic.

from superviseddescent.

from superviseddescent.

I see.
The tracking works well enough for my purposes. Landmark detection isn't my main topic. I am focussing my effort on eos, the 3D fitting library.

from superviseddescent.

from superviseddescent.