Real time Fight Detection Based on 2D Pose Estimation and RNN Action Recognition
License: MIT License
C++ 94.67%Python 4.00%Makefile 1.33%
fight_detection's Introduction
Overview
Real time Fight Detection Based on 2D Pose Estimation and RNN Action Recognition.
This project is based on darknet_server. if you want to run this experiment take a look how to build here.
Fight Detection System Pipeline
Pose Estimation and Object Tracking
Made pipeline to get 2D pose time series data in video sequence.
In worker process, Pose Estimation is performed using OpenPose. Input image pass through the Pose_detector, and get the people object which packed up people joint coordinates. People object serialized and send to sink process.
In Sink process, people object convert to person objects. and every person object are send to Tracker. Tracker receive person object and produces object identities Using SORT(simple online and realtime tracking algorithm).
Finally, can get the joint time series data per each person. each person's Time series data is managed by queue container. so person object always maintain recent 32 frame.
Tracking Pipeline
Time series data
Examples of Result (Pose Estimation and Object Tracking)
Collect action video
Collected Four Action type (Standing, Walking, Punching, Kicking) video.
Punching Action type video is a subset of the Berkeley Multimodal Human Action Database (MHAD) dataset.
This video data is comprised of 12 subjects doing the punching actions for 5 repetitions, filmed from 4 angles. (http://tele-immersion.citris-uc.org/berkeley_mhad)
Others (Standing, Walking, Kicking) are subsets of the CMU Panoptic Dataset. In Range of Motion videos(171204_pose3, 171204_pose5, 171204_pose6), cut out 3 action type. I recorded timestamp per action type and cut out the video using python script(util/concat.py). This video data is comprised of 13 subjects doing the three actions, filmed from 31 angles. (http://domedb.perception.cs.cmu.edu/index.html)
0,1,10// 0 : Standing, 1 : begin timestamp, 10: end timestamp1,11,15// 1 : Walking, 11 : begin timestamp, 15: end timestamp3,39,46// 3 : Kicking, 39 : begin timestamp, 46: end timestamp
Examples of Dataset (Stand & Walk)
Stand (CMU Panoptic Dataset)
Walk (CMU Panoptic Dataset)
Examples of Dataset (Punch & Kick)
Punch (Berkeley MHAD Dataset)
Kick (CMU Panoptic Dataset)
Make training dataset
Put action video data to tracking pipeline and get joint time series data per each person. this results (joint position) are processed to feature vector.
Angle : current frame joint angle
ΔPoint : A distance of prior frame joint point and current frame
ΔAngle : A change of prior frame joint angle and current frame.
Finally get each frame feature vector and then make action training data which consist of 32 frames feature vector. training datas are overlapped by 26 frames. so we got the four type action data set.
A summary of the dataset is:
training was run for 300 epochs with a batch size of 1024. (weights/action.h5)
After training, To get action recognition result in real time, made action recognition pipeline. Sink process send each person's time series feature vector to action process as string. Action process put received data into RNN network and send back results of prediction. (0 : Standing, 1 : Walking, 2 : Punching, 3 : Kicking)
This stage check that person who kick or punch is hitting someone. if some person has hit other, those people set enemy each other.
System count it as fight and then track them until they exist in frame.