This program provides demo of live stream hand gesture classification. I contains 3 parts: 1) dataset selection, 2) model training, 3) live recogniation demo.

The dataset is selected from HaGRID - HAnd Gesture Recognition Image Dataset, which contains the hand gestures below:

In dataset selection, we select part of hand gestures, and match each to an emoji.

We prepare the training dataset into HandLandmarks.csv. The process is done by DataPrepare.py. We first detection the hand landmarks of images in HaGRID dataset by mediapipe landmarker model. Each hand landmark will be preprocessed and converted into one row (65 columns) in csv file. The structure of one row is: 1) index 0 is label, same as the table above; 2) index 1 is handerness, right is 0 and left is 1; 3) index 2 to index 64 are positions (x,y,z) of 21 landmarks of a hand, shown in image below. The hand landmarks will be preprocessed, steps are: 1) shift all landmark points so point0 is at (0,0,0) 2) rotate all (x,y) to set the line between point0 and point5 vertical.

The model is trained by gesture_classification.ipynb, which output a model gesture_classifier.tflite. The demo gesture_classifier.py use the model to preform live stream hand gesture classification.