A high performance C++ library with node bindings to easily incorporate gesture recognition into any project. It uses a trained object detection method (with over 100,000 samples per gesture!) and is therefore robust to different skin colors and lighting changes.

For the initial release the SDK is only linux compatible, but it can be extended to other systems via Docker and a WebSocket.

There are currently three detectable gestures: open PALM, THUMB UP, and FIST. Tracking options are available so it can track changes between hands. Smoothing filters are available so fine control can be used.

gcc 4.9.3+, OpenCV 3.1.0, Node 0.12+

MSVC C++ distribution MSVC 14 (May work on others, only tested with VS 14)

OpenCV 3.1.0 Clang 3.4+

As it is currently only linux compatible we have created an image on DockerHub under admobilize/gesture which includes all required dependencies. If you would like to build the Dockerfile yourself it is included in the repository.

• First give root access to Docker as an X server. This will allow the camera feed to be accessed.

xhost local:root

• Download the image and create and run a daemon container, giving it access to your default camera and a port through which to host a webserver.

sudo docker run -itd -p 8080:8080 -p 3000:3000 --privileged \
--name gesture \
-v /dev/video0:/dev/video0 \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-e DISPLAY=$DISPLAY admobilize/gesture

• Execute a command inside the docker, to run the node sample handTracker.js

sudo docker exec -it gesture /bin/sh -c 'cd ~/matrix-gesture-node-sdk/; examples/handTracker.js'

This pops up a window with the feed from your default camera and runs the tracking algorithm looking for and boxing whenever it detects a flat palm. It will also look for a thumbs up or a closed fist if the palm briefly disappears. Try out some of the examples or write your own using the documentation!

It will return a JSON object with the location (both center (xc,yc) and top left corner (x,y)), size, hand type, and unique id.

{
  "results":
    [{
        "hand_type":0,
        "height":184,
        "id":5,
        "width":184,
        "x":181,
        "xc":273,
        "y":196,
        "yc":288
      }]
}

For those running mac/windows, you would have to create a VirtualBox running linux to be able to access the camera from docker. Docker-engine runs a basic linux that doesn't recognize the camera. Docker is currently beta testing their new mac/windows docker which will be compatible with this library and you will be able to access your camera. Alternatively you can run detection through a node server in the node SDK where you can access it through localhost:3000 in a browser.

sudo docker exec -it gesture /bin/sh -c 'cd ~/matrix-gesture-node-sdk/; npm run start-test-server'

(We have had some permission problems with google chrome and safari blocking the server on mac, but firefox works fine)

1. Err fix

   Error response from daemon: Conflict.

   1-1. Stop a running gesture container:

   docker stop gesture

   1-2. Force remove a running gestrue container:

   docker rm -f gestrue

2. webcam in VirtualBox Guest OS on Windows Host. https://reurl.cc/rQGAN

   Windows Host

   2-1. Install 'Oracle VM VirtualBox Extrnsion Pack'.

   2-2. open cmd.

   cd c:\Program Files\Oracle\VirtualBox

   2-3. open VirtualBox.

   VboxManage controlvm "Ubuntu16" webcam attach .1

   *Replace "Ubuntu16" with your guest OS name.

As long as you have the required dependencies enter the node sdk directory and run:

npm install && npm run setup

or if you are root

npm install && npm run-root-setup

1. A secure channel is required for the video stream, i.e. https. You'll need to get/configure the Nodejs ssl key pair.
2. The keys path are specified in file examples/detection-server/server.js

const privateKey = fs.readFileSync('YOUR_KEY_PATH/privkey.pem', 'utf8');
const certificate = fs.readFileSync('YOUR_KEY_PATH/cert.pem', 'utf8');
const ca = fs.readFileSync('YOUR_KEY_PATH/fullchain.pem', 'utf8');

3. check if the port for https (3000 in this example) is allowed to access from the firewall.
4. start the server by run the following command:

npm run start-test-server

5. You'll see message listening on *:3000
6. Launch your browser and access the web page at https://YOUR_DOMAIN:3000
7. You should be able to see the page as below:

Run some other examples!

node examples/handTracker.js or npm run start-test-server

The documentation can be found here

Gesture.ai anonymously aggregates data from people using the SDK to improve the detection of gestures and to provide the best SDK possible. This data aggregation can be turned off from the client side but it is opt-out. To opt out, run set_data_submit(false) on creation of a Gesture class.

Visit and join our community here