A Medical Image Navigation prototype application powered by hand gesture commands, created to demonstrate the feasibility of a Deep Learning and Computer Vision-based Hand Gesture Recognition System for Operating Room usage. A deliverable requirement for CS 300 Thesis course, successfully defended Midyear 2020 at the University of the Philippines Diliman under Sir Prospero Naval.

For more details, please check the published paper and YouTube video on how the system would be used.

• Python 3.5
• OpenCV
• imutils
• Lasagne
• Plotly

Trained network with Jester Dataset named as network\jester.npz. For more details on training the network, check training folder.

To run the prototype application using your webcam to capture the gestures:

python DemoMain.py

Alternatively, if we have video recording of the gesture sequence, run the following to visualize how the application responds:

python DemoVideo.py -v [video_file]

# example
python DemoVideo.py -v testOR.mp4

1. Annotate the video recording with the gestures by frame level. Follow the format of the sample annotation file.
2. Optionally, specify your preference of the colors of the plot per gesture. Follow the format of the sample plot colors configuration file
3. Plot the confidence of the network and measure performance against the ground truth.

python VisualizePerf.py -v [video_file] -a [annotation file]

# example
python VisualizePerf.py -v testOR.mp4 -a testOrAnnotation.csv

Please cite this paper.

@article{salvador2022,
  title={Towards a Feasible Hand Gesture Recognition System as Sterile Interface in the Operating Room with 3D Convolutional Neural Network},
  author={Salvador, Roy Amante and Naval Jr, Prospero},
  journal={Informatica},
  year=2022,
  month=3,
  volume=46,
  number=1,
  pages={1-12},      
  doi={10.31449/inf.v46i1.3442},
  url={https://www.informatica.si/index.php/informatica/article/view/3442}
}