yanatan16 / robonui Goto Github PK

Robotic Natural User Interface using Kinect and COTS Robot Kits in C#

License: Apache License 2.0

C# 100.00%

robonui's Introduction

****************************************************
****                                            ****
****       Robotic Natural User Interface       ****
****                                            ****
****                aka RoboNUI                 ****
****                                            ****
****************************************************

Authors: 
  Jon Eisen 		(yanatan16 @ GitHub)

Description:
 A natural user interface for controlling robotic 
manipulator arms and human-like robots. The system
uses the Microsoft Kinect to interpret the 
controller's skeletal motion and map that to moving
the robot's servos.

 The system is implemented in C# and requires the
platform running the system to have Windows 7 as
well as the Kinect for Windows drivers installed.
It must be compiled on Visual Studio 2010 with the 
Kinect for Windows SDK installed.

RoboNUI is licensed with the Apache 2.0 Open Source
License, a copy of which can be viewed in this directory.

robonui's People

Contributors

Stargazers

Watchers

Forkers

akirkby

robonui's Issues

Update comments

Make sure all comments are up to date and are actually there.

Get Software installed on NGC laptop

Sdk beta 2
OpenNI
Voice commands needed software (see Kinect SDK documentation)

Update QuadChart with 'tested technologies' section

Kinect, windows SDK, OpenNI? OpenCV?

Evangelize!

Show to list:
Jack Riddle
Tom Jones
Coworkers
Undersea robotics CoP

Marionette Integration

Integrate with the marionette and make the angles move appropriately.

Update configuration parameters

Update configuration parameters to include all commands

Hand Resolution

Integrate Hand tracking and fingertip tracking into RoboNUI to control the operation of the manipulator.

This can be done in one of two ways:

Integrate OpenNI (another pending issue) and use built in hand and finger tracker

Use an OpenCV derivative to find it myself

Hand Tracking (see ref section for places to get more info on this) :
#1: Get hand position (also convert raw depth to z coordinate depth)
#2: cut out hand blob using threshold on depth data. (70cm in depth?)
#3: use opencv to get defect points and lines
#5: determine that distance minus filter radius to open/close palm
#6: find thumb ???
#7: use thumb/palm position in wrist rotation angle

We can use OpenCV for comp vision (.net version?): engucv or opencvsharp or sharpcv

Reference:
Tracking of Fingertips and Centers of Palm Using KINECT (I have a copy)
Frati, V., Prattichizzo, D., Using Kinect for hand tracking and rendering in wearable haptics, IEEE World Haptics Conference (WHC 2011), 21-24 June 2011, pp. 317-321.