geronymos / koh Goto Github PK

Problem

Mediapipe is primarely for tracking faces in 3d-space and it is good at that, but on sertain facial expressions (mouth wide open) the face-mesh does not open the mouth as wide as it really is.

Solution

Two-dimensional tracking has proofen itself worthy with applications like FaceRig. A famous python library for tracking face in two dimensions is dlib.

Problem

To come true to the claim Virtual camera that displays closest face match from a data set we need to implement it.

Solution

Maybe use pyvirtualcam

Problem

The programm takes a lot if start-up time and also lags in general with larger datasets and if you chose to match the whole face.

Solution

See what functions take the most amount if time, so they could be improved.

There are a couple of libraries for python that can measure the time a funtion takes and created a report either in the console or as a file you could see in a web-interface.

Problem

With small datasets it is less likely to find a image in the dataset that matched the webcam well.

Solution

Normalize the match, so that it ignores the size and orientation of the face.

In the output either only show the normalized webcam and the normalized match. Or transform the image if the match and apply the size and rotation of the webcams face to the match.

How it works now

Now the faces are matched by calculating the average distance fom a point of the webcam-face to a point of a face of the dataset. This goes on for every point in the face and every face in the dataset. This list of distances then get sorted. The first result (with least distance) gets choosen as a match.

Problem

The sorting may need a lot of time. After we have the match with the least distance, we do not need the other faces in the list.

Solution

Replace the sort-function with replace or even just use a variable and set it if the new comparison has a smaller distance than the one that is in the function.