Exploiting Leap Motion and Microsoft Kinect Sensors for Static and Dynamic Sign Gesture Recognition

Abstract: One of the primary ways of communication between humans is verbal communication. Among hearing-impaired persons, the traditional way of communication is through sign language. Sign gestures are the atomic actions used in sign language for non-verbal communication. With the advancement in both sensor-based devices and machine learning techniques, there is a lot of scope in identifying and developing an efficient, robust, and low-cost sign gesture recognition system that can help hearing-impaired individuals to communicate among themselves and people with usual hearing ability. My focus will be on classical machine learning algorithms for static the recognition of sign gestures recorded using leap motion sensors, and deep learning methods for the recognition of dynamic sign gestures recorded using Microsoft Kinect sensors. I addressed the Leap motion-based static palm sign gesture recognition by applying three machine learning algorithms: Support Vector Machine (SVM), Random Forest (RF), and Naive Bayes (NB), and used Genetic Algorithm (GA) for feature selection. Genetically selected features are fed to different classifiers for gesture recognition. Whereas, we addressed the Microsoft Kinect-based dynamic sign gesture recognition using an end-to-end deep learning approach. The dynamic sign gestures are sequential that deep learning methods are proven to model effectively without handcrafted features. Deep learning models can directly work on raw data and learn higher-level representations (features) by themselves. To test our hypothesis, we have used two latest and promising deep learning models, Gated Recurrent Unit (GRU) and Bidirectional Long Short-Term Memory (BiLSTM), and trained them using only raw data for dynamic gestures; the dynamic gestures were collected from three views namely front, mid, side views. With regard to static palm sign gesture recognition, an accuracy of 74.00% is recorded with RF classifier on the Leap motion sign gesture dataset. With regard to dynamic sign gesture recognition, we have performed comparative analysis among both models and also with the base paper results. Conducted experiments reflected that the proposed method outperforms the existing work, where GRU successfully concluded with 70.78% average accuracy with front view training.