Game-Link

• This Project takes hand gesture (through webcam of your laptop) as an input to move a 3D character model in a 3D environment build in Unity 3D.

• Currently there are 6 hand gestures used in the project:
  • Move Front
  • Stop
  • Move Left
  • Move Right
  • Jump
  • Crouch
• Background music is also integrated with the environment.

1. Clone the repository
2. In the Clonned folder terminal, run command 'npm install' to install all the dependencies required for the project(make sure npm package manager is installed)
3. Open /src/app.js in any js editor(example Visual Studio Code).
4. Run command 'npm run'.
5. Enjoy 😎.

• Using the Tensorflow Hand Recognition Model for hand and gesture detection.

  • Mediapipe HandPose Model: MediaPipe Handpose is a lightweight ML pipeline consisting of two models: A Balze-palm detector and a hand-skeleton finger tracking model(Hand Landmark model). It predicts 21 3D hand keypoints per detected hand. For more details, please read Google AI blogpost.
  • Fingerpose library for building user generated gesture (6 hand gesture, specified above)

• Building the 3D Environment in Unity 3D

  • Creating character in Unity 3D
  • Giving Animations to the Character using Adobe's Mixamo.
  • Developing C# Script for movement of the 3D character.
  • Creating 3D Environment by adding Objects such as trees, elevated lands, bridge, pond
  • Integrating this environment with the character created.
  • Adding sound to the environment.

• Integrating the 3D Environment and the Gesture Recognition Model

  • Using WebGL library to convert Unity Model to HTML.
  • Calling script functions in C# from React.js using Unity Model Instance.

• Hosting the project website on Github through Github Pages.

You can play this game by clicking on Game Link.
Note: This game can be played on Laptop/PC only.
To watch how the Project works: Project Demo

Unity-Environment

Unity-Character

LinkedIn - Github
Chaitanya Chafale - chaitanya-chafale
Atharva Sawarkar - Atharva787
Ashutosh Gupta - Ashutosh0212
Apoorva Mandavgane - alpha-maverick
Karan Jadhav - karanjadhav12

Also I would like to thanks Nicholas Renotte for helping us to complete this project through his youtube videos and its explanations.
Video: Real Time AI GESTURE RECOGNITION with Tensorflow.JS + React.JS + Fingerpose
Github: Gesture recognition