legendkartik45 / virtualmouse Goto Github PK

Abstract

Technology today has changed our outlook the way we experience this real world. In couple of years’ technologies has changed how we do small and large number of things. Technology came out with new advance level where user can use gestures and virtually control interaction with the Computer.

For example, during this pandemic we all have learned to get Digitalized and avoid contact with One another.

Considering the current Scenario, we thought of creating an application that can benefit users for contactless interactions with the systems such as ATM, Flight Boarding etc.

The concept is to use a simple camera instead of a classic or standard mouse to control mouse cursor functions. The Virtual Mouse provides an infrastructure between the user and the system using only a camera. It allows users to interface with machines without the use of mechanical or physical devices, and even control mouse functionalities. This study presents a method for controlling the cursor’s position without the need for any electronic equipment. While actions such as clicking and dragging things will be carried out using various hand gestures. As an input device, the suggested system will just require a webcam. The suggested system will require the use of OpenCV and Python as well as other tools. The camera’s output will be presented on the system’s screen so that the user can further calibrate it.

1. Problem Statement

To design a virtual mouse that detects hand gestures and performs mouse operations only using the fingers. Different combinations of fingers can be used to perform different operations of the mouse based on the particular combination of fingers. In the Proposed system, users don’t have to color their fingers with a specific color and are not required to use any device or sensors

2. Introduction

Hand gesture technology is applied in many different fields in today’s world of automation, including medical applications, industry applications, IT hubs, banking sectors, and so on. This idea is based on the common notion of using hand gestures to manage a laptop or computer. The Human Machine Interface (HMI) is a hardware and software system that aids in the communication and exchange of information between the humans and the machine. As part of HMI devices, we commonly employ numerous indicators such as LEDs, Switches, Touch Screens, and LCDs. Hand Gestures are another unique means of communicating with devices such as robots or computers. Have you ever wished to operate your PC or Laptop by simply waving your finger in the air? We address the challenges of using gestures to control computer applications that include both static symbols and dynamic motions. Each gesture is modeled using either static model information or a dynamic system that is linear in parameters. Instead of using numerous devices such as keyboards, mice, joysticks, and so on, we may now simply utilize hand moments/hand gestures to control the laptop’s functionalities. We used hand gesture control in this project, which allows users to control music, video, browser, documents, games, navigation, and many other functions. Recognition takes place in real-time, with only a tiny amount of processing time and memory required. We investigate which motions are appropriate, how to recognize them, and in which order they should control. Myriads of technological advancements are occurring in today’s society, such as natural language processing, biometric authentication, and face detection, which can be found on our tablets, iPads, computers, and smartphones. Similarly, Hand Movement Identification was a modern form of Human–Machine Interconnection, in which the system’s mouse cursor could be controlled simply by placing our figure in front of the computer’s web camera.

3. Motivation

The motivation was to create a human interaction application to interact with the computer and develop a virtual human-computer interaction device. The existing system was costly and used some extra sensors or material on hand to make the virtual device.

4. Significance
5. It is cost-effective as no external mouse is required.
6. It can be used in airports for virtual touch while generating boarding pass.
7. It can be used in ATMs and avoid the bacterial transmission of touch as we have seen in the Covid pandemic.
8. Literature Survey

6. Objectives

Following are the objectives of the proposed dissertation work:

1. To study different research papers on hand gesture recognition and get an idea about the previous work related to the project.
2. To study different algorithms that can be used for hand gesture recognition
3. To design or develop a system/model that can be used to recognize the different classes of combinations of hand gestures and control the mouse pointer using different hand gestures.
4. To combine the mouse functionality with the different classes of the model developed.
5. To analyze and evaluate the performance and accuracy of the system.
6. Methodology

In order to create a virtual mouse using hand gestures we need to have an algorithm/model using which we can recognize the hand gesture and based on different kind of hand gestures we can perform different kind of operations like increase volume, decrease volume, zoom in, zoom out, move mouse pointer, click.

Following is the methodology proposed for the project:

• The main problem that we are facing is detecting and separating the hand from the image. There are many problems like the background color, different sizes of hands, and other body parts like the face, arm etc. So instead of creating our own model to detect the hand we can use the already-built Google media pipe that is trained to detect and separate the hand from the background.
• We will be using Google’s Mediapipe to detect and track the hand from the image frame and the Google’s media pipe returns the landmark of the hand. There are 21 landmarks of one hand that are returned. The landmark returned by the media pipe are shown below.

• After the landmarks are obtained we are creating a model that takes landmarks of the hands at different positions and each of these landmarks at different positions is classified to represent a particular function of the mouse.
• After the landmarks are obtained we need to normalize the points obtained. So that the distance of the hand from the screen is not a problem and it becomes easy for us to train the new model.
• Then these landmarks are passed as input to the model and the model is trained to classify the different landmarks to different classes. 75% of the dataset is used to train the model and the remaining 25% is used to test the model.
• After the model is able to classify the hand to different classes accurately. We can use the model to detect the hand class and after the class is detected we can write a python script to perform different functions of the mouse using the python library pyautogui.

Functional Block Diagram:

8. Implementation and results

Python library used:

Mediapipe Model Loading

Landmarks returned by the mediapipe are show above.

These landmarks are processed and then send to keypoint_classifier.csv. This csv file is then used to train and test our model.

Data in keypoint_classifier.csv

Training and Testing of the model:

ACCURACY :- 90%

Running the model with real-time images from the camera:

The obtained classification is shown in the image at real-time and is shown below:

We represent these classes with the numbers and are stored in Keypoint_classifier_csv.

Based on the id of the class obtained we perform the mouse functionality using pyautogui module of python.

9. Conclusion

This project illustrates how to use computer operations through hand gestures and movements, which is one of the most efficient methods to utilize a computer. It’s one of the most fundamental ways of human-computer communication. It’s a budget-friendly strategy. Using this strategy, we may quickly and simply accomplish practically any fundamental operation, as well as keyboard shortcuts. We may use this method to develop our own code to do whatever action we want. However, this system cannot recognize the movement of the hand as a gesture. Different combinations of hand gestures are used to perform different mouse operations.

References

1. Amornched Jinda-apiraksa, Warong Pongstiensak, and Toshiaki Kondo, "A Simple Shape-Based Approach to Hand Gesture Recognition", in Proceedings of IEEE International Conference on Electrical Engineering/ Electronics Computer Telecommunications and Information Technology (ECTI-CON), Pathumthani, Thailand, pages 851-855, May 2010
2. N. Yasukochi, A. Mitome and R. Ishii, "A Recognition Method of Restricted Hand Shapes in Still Image and Moving Image as a Man-Machine Interface", IEEE Conference on Human System Interactions, pp. 306-310, 2008.
3. H. Xu, X. Hou, R. Su and Q. Ni, "Real-time Hand Gesture Recognition System based on Associative Processors", The 2nd IEEE International Conference on Computer Science and Information Technology, pp. 14-18, August 2009.
4. Q. Zhang, F. Chen and X. Liu, "Hand Gesture Detection and Segmentation Based on Difference Background Image with Complex Background", 2008 International Conference on Embedded Software and Systems (ICESS2008), pp. 338-343, July 2008.
5. V.I.Pavlovic, R.sharma, and T.S.Huang, "Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review," IEEE Trans. On PAMI, vol 19,pp.677-695,1997