A two stream convolutional neural network with bi-directional GRU model to classify dynamic hand gesture

Dynamic hand gesture recognition is still an interesting topic for the computer vision community. A set of feature vectors can represent any hand gesture. A Recurrent Neural Network (RNN) can recognize these feature vectors as a hand gesture that analyzes the temporal and contextual information of the gesture sequence. Thus, we proposed a hybrid deep learning framework to recognize dynamic hand gestures. In the Hybrid model GoogleNet is pipelined with a Bidirectional GRU unit to recognize the dynamic hand gesture. Dynamic hand gestures consist of many frames, and features of each frame need to be extracted to get the temporal and dynamic information of the performed gesture. As RNN takes input as a sequence of feature vectors, we extract features from videos using pretrained GoogleNet. As Gated Recurrent Unit is one of the variants of RNN to classify the sequential data, we created a feature vector that corresponds to each video and passed it to the bidirectional GRU (BGRU) network to classify the gestures. We evaluate our model on four publicly available hand gesture datasets. The proposed method performs well and is comparable with the existing methods. For instance, we achieved 98.6% accuracy on Northwestern University Hand Gesture(NWUHG), 99.6% on SKIG, 99.4% on Cambridge Hand Gesture (CHG) datasets respectively. We performed our experiments on DHG14/28 dataset and achieved an accuracy of 97.8% with 14-gesture classes and 92.1% on 28-gesture classes. DHG14/28 dataset contains skeleton and depth data, and our proposed model used depth data and achieved comparable accuracy.     

基于多特征融合与支持向量机的手势识别

针对手势识别中人的手部特征描述易受到环境因素影响,手势识别率低等问题,并考虑到单个特征的局限性,提出了一种基于Hu矩和HOG特征融合的支持向量机手势识别新方法。该方法首先对处理后的手势图像提取局部的HOG特征,然后针对手势的轮廓提取全局Hu矩特征,再将两种特征融合成混合特征,并通过主成分分析法对混合特征进行降维形成最终分类特征,并将新特征输入到支持向量机中进行识别。实验表明,该方法具有较好的鲁棒性和较高的识别率。     

Real-time Hand Gesture Recognition from Depth Images Using Convex Shape Decomposition Method

Hand gesture recognition is one of the most natural and intuitive ways to communicate between people and machines, since it closely mimics how human interact with each other. This paper presents a novel method for real-time markerless hand gesture recognition from depth images. The proposed method encompasses a collection of techniques that enable the detection, segmentation and recognition of hand gestures. A Hand detection and location method is employed using the depth information acquired from a depth sensor. Then, the hand is robustly segmented in cluttered background without any marker around. A convex shape decomposition method based on Radius Morse function is proposed for hand shape decomposition in real-time. Hand palm, fingertips and hand skeleton are recognized based on the hand shape decomposition and hand features. Moreover, we present a method for recognition of two-hand gestures. Representative experimental results demonstrate qualitatively and quantitatively that accurate hand gesture recognition can be achieved for real-time applications.     

融合图像轮廓矩和傅立叶描叙子的手势识别

针对手势识别中人手是复杂变形体,手部特征描叙容易受到环境因素影响的特点,提出了一种基于混合轮廓特征的手势识别的新方法.首先根据手的颜色信息将手从复杂的背景中分离出来,然后提取出手轮廓的矩不变量和Fourier描叙子特征,然后将2种特征融合构成混合特征,最后将新特征输入BP神经网络进行识别.实验表明:此方法具有较好的鲁棒性和较高的识别率.     

Hand gesture understanding by weakly-supervised fusing shallow/deep image attributes

Accurately recognizing human hand gestures is a useful component in many modern intelligent systems, such as identification authentication, human–computer interaction, and sign language recognition. Conventional approaches are typically based on shallow visual features and relatively simple backgrounds, which cannot readily recognize partially occluded hand gestures with sophisticated backgrounds. In this work, we propose a unified hand gesture recognition framework by optimally fusing a set of shallow/deep finger-level image attributes, based on which a weakly-supervised ranking algorithm is designed to select semantically salient regions for gesture understanding. More specifically, given a rich number of hand gesture images, we employ the well-known BING object proposal generator to extract hundreds of object patches that potentially draw human visual attention. Since the hundreds of object patches are still too many for building an effective recognition system, a weakly-supervised metric is proposed to rank them by extracting multiple shallow/deep features. And visual semantics are encoded at region-level by transferring the image-level semantic tags into various human gesture image regions by a weakly-supervised learning paradigm Apparently, the top-ranking highly salient object patches are highly indicative to human visual perception of human hand gesture, Thus we extract their ImageNet-CNN features and further concatenate them. Finally, the concatenated deep feature is fed into a multi-class SVM for classifying each hand gesture image into a particular type. Comprehensive experimental validations have demonstrated the effectiveness and robustness of our proposed hybrid-feature-based hand gesture categorization.     

Research on unified recognition model and algorithm for multi-modal gestures

In gesture recognition,static gestures,dynamic gestures and trajectory gestures are collectively known as multi-modal gestures.To solve the existing problem in different recognition methods for different modal gestures,a unified recognition algorithm is proposed.The angle change data of the finger joints and the movement of the centroid of the hand were acquired respectively by data glove and Kinect.Through the preprocessing of the multi-source heterogeneous data,all hand gestures were considered as curves while solving hand shaking,and a uniform hand gesture recognition algorithm was established to calculate the Pearson correlation coefficient between hand gestures for gesture recognition.In this way,complex gesture recognition was transformed into the problem of a simple comparison of curves similarities.The main innovations:1) Aiming at solving the problem of multi-modal gesture recognition,an unified recognition model and a new algorithm is proposed;2) The Pearson correlation coefficient for the first time to construct the gesture similarity operator is improved.By testing 50 kinds of gestures,the experimental results showed that the method presented could cope with intricate gesture interaction with the 97.7% recognition rate.     

结合手势二进制编码和类-Hausdorff距离的手势识别

针对目前动态手势识别方法受手势旋转、平移、缩放的影响,并解决手势识别的实时性问题,提出一种基于手势二进制编码和类-Hausdorff距离模板匹配的手势识别方法.首先,把分割好的手势图像进行标准化处理,求出标准化图像中的手势主方向,建立二维手势直角坐标系,提取空间手势特征;其次,根据前五帧手势图像中手势像素点个数的变化量识别出动态手势类型;然后,用手势二进制描述子从动态手势类型中再筛选出可能的候选手势集合;最后,用类-Hausdorff距离模板匹配方法从候选手势集合中识别出最终手势.主要创新点在于：提出的动态手势类型识别和手势二进制描述子匹配的方法,大大缩短了动态手势识别的时间;提出的结合手势主方向的类-Hausdorff距离方法,不仅对旋转、平移和缩放手势具有不变性,而且对区分度较小的手势也具有较高的识别准确率.实验结果表明,在光照相对稳定的条件下,该方法能够实时准确的实现动态手势识别,总体识别率达到95%以上,对发生缩放的手势识别率能达到92%以上,对发生旋转的手势识别率能达到87%以上.本文算法已经在一个基于手势的人机交互界面中得到应用.     

空间上下文与时序特征融合的交警指挥手势识别技术

针对无人驾驶汽车快速准确识别交警指挥手势的需求,本文在分析交警指挥手势的关节铰接特征基础上,建立基于关节点和骨架的交警指挥手势模型;其次,引入卷积姿势机（Convolutional Pose Machine,CPM）提取交警指挥手势的关键节点,进而提取交警指挥手势中骨架的相对长度及其与重力加速度的夹角作为空间上下文特征,并引入长短时记忆网络（Long Short Term Memory,LSTM）提取交警指挥手势的时序特征;最后,设计了融合空间上下文和时序特征的交警指挥手势识别机（Chinese Traffic Police Gesture Recognizer,CTPGR）,创建了包含8种交警指挥手势、时长约2小时的交警指挥手势视频库对CTPGR进行训练验证,并通过实验将CTPGR与已有交警手势识别算法进行了对比分析.实验证明CTPGR可以快速准确地识别交警指挥手势,系统对复杂背景和动态交警指挥手势具有较强的适应能力.     

A fuzzy rule-based approach to spatio-temporal hand gesturerecognition

Gesture based applications widely range from replacing the traditional mouse as a position device to virtual reality and communication with the deaf. The article presents a fuzzy rule based approach to spatio-temporal hand gesture recognition. This approach employs a powerful method based on hyperrectangutar composite neural networks (HRCNNs) for selecting templates. Templates for each hand shape are represented in the form of crisp IF-THEN rules that are extracted from the values of synaptic weights of the corresponding trained HRCNNs. Each crisp IF-THEN rule is then fuzzified by employing a special membership function in order to represent the degree to which a pattern is similar to the corresponding antecedent part. When an unknown gesture is to be classified, each sample of the unknown gesture is tested by each fuzzy rule. The accumulated similarity associated with all samples of the input is computed for each hand gesture in the vocabulary, and the unknown gesture is classified as the gesture yielding the highest accumulative similarity. Based on the method we can implement a small-sized dynamic hand gesture recognition system. Two databases which consisted of 90 spatio-temporal hand gestures are utilized for verifying its performance. An encouraging experimental result confirms the effectiveness of the proposed method     

A two-electrode frequency-scan system for gesture recognition

In this paper, we proposed a novel two-electrode, frequency-scan gesture recognition system based on bio-impedance measurement. This method not only achieves a high accuracy in recognizing common gestures and pinch gestures, but also reduces the measurement complexity and the number of electrodes. We developed a bespoke circuit with two medical electrodes to collect data from the back of the hand and presented a frequency-scan method to increase the diversity of impedance data. Feature extraction method was adapted to explore the representative features for gesture recognition, and machine learning classification models with five-fold cross-validation were used to train and realize accurate gesture recognition. To verify the effectiveness of this system, we designed two groups of nine gestures in a hand-gesture recognition experiment. The results showed that the system achieved a recognition accuracy of 98.3% with a group of four common gestures and an accuracy of 98.5% with a group of six pinch gestures. The proposed method realized a higher accuracy in pinch gesture set while using fewer electrodes. Additionally, we designed two real-time proof-of-concept interactive scenarios to demonstrate the general applications of this system.     

基于时空压缩特征表示学习的毫米波雷达手势识别算法

针对现有无线射频信号的手势识别研究中的数据预处理和特征利用问题,该文提出一种用于调频连续波(FMCW)雷达的时空压缩特征表示学习的手势识别算法。首先对手部反射的毫米波雷达回波信号的距离-多普勒(RD)图进行静态干扰去除和动目标点筛选,减少杂波对手势信号的干扰,同时减少计算数据量;然后提出一种压缩手势时空特征的表示方法,利用动目标点的主导速度来表示手势的运动特征,实现多维特征的压缩映射,并保留手势运动的关键特征信息;最后设计了一个单通道的卷积神经网络(CNN)来学习和分类多维手势特征信息并应用于多用户和多位置的手势识别。实验结果表明,与现有其他手势识别算法相比,该文提出的手势识别方法在识别精度、实时性以及泛化能力上都具有明显的优势。     

Chronological pattern indexing: An efficient feature extraction method for hand gesture recognition with Leap Motion

Recently, Hand-Gesture-Recognition (HGR) systems has appreciably change the way of interaction between humans and computers thanks to advanced sensor technologies like the Leap-Motion-Controller (LMC). Despite the success achieved by many state-of-the-art methods, they have not worked on the rich temporal information existing in the sequential hand gesture data and characterizing the discriminative representation of different hand gesture classes. In this paper, we suggest a novel Chronological-Pattern-Indexing (CPI) approach which encodes the temporal orders of patterns for hand gesture time series data acquired by the LMC sensor. We extract a set of temporal patterns from different optimized projections. Then, we compare their temporal order and we encode the whole sequence with the index of the first coming pattern. We repeat these steps until we generate an efficient feature vector modeling the chronological dynamics of the hand gesture. The experiments demonstrate the potential of the proposed CPI approach for HGR systems.     

Face anti-spoofing based on multi-modal and multi-scale features fusion

Face anti-spoofing is used to assist face recognition system to judge whether the detected face is real face or fake face. In the traditional face anti-spoofing methods, features extracted by hand are used to describe the difference between living face and fraudulent face. But these handmade features do not apply to different variations in an unconstrained environment. The convolutional neural network (CNN) for face deceptions achieves considerable results. However, most existing neural network-based methods simply use neural networks to extract single-scale features from single-modal data, while ignoring multi-scale and multi-modal information. To address this problem, a novel face anti-spoofing method based on multi-modal and multi-scale features fusion ( MMFF) is proposed. Specifically, first residual network ( Resnet )-34 is adopted to extract features of different scales from each modality, then these features of different scales are fused by feature pyramid network (FPN), finally squeeze-and-excitation fusion ( SEF) module and self-attention network ( SAN) are combined to fuse features from different modalities for classification. Experiments on the CASIA-SURF dataset show that the new method based on MMFF achieves better performance compared with most existing methods.     

Gesture image recognition method based on DC-Res2Net and a feature fusion attention module

To extract decisive features from gesture images and solve the problem of information redundancy in the existing gesture recognition methods, we propose a new multi-scale feature extraction module named densely connected Res2Net (DC-Res2Net) and design a feature fusion attention module (FFA). Firstly, based on the new dimension residual network (Res2Net), the DC-Res2Net uses channel grouping to extract fine-grained multi-scale features, and dense connection has been adopted to extract stronger features of different scales. Then, we apply a selective kernel network (SK-Net) to enhance the representation of effective features. Afterwards, the FFA has been designed to remove redundant information in features by fusing low-level location features with high-level semantic features. Finally, experiments have been conducted to validate our method on the OUHANDS, ASL, and NUS-II datasets. The results demonstrate the superiority of DC-Res2Net and FFA, which can extract more decisive features and remove redundant information while ensuring high recognition accuracy and low computational complexity.     

On an algorithm for Vision-based hand gesture recognition

A vision-based static hand gesture recognition method which consists of preprocessing, feature extraction, feature selection and classification stages is presented in this work. The preprocessing stage involves image enhancement, segmentation, rotation and filtering. This work proposes an image rotation technique that makes segmented image rotation invariant and explores a combined feature set, using localized contour sequences and block-based features for better representation of static hand gesture. Genetic algorithm is used here to select optimized feature subset from the combined feature set. This work also proposes an improved version of radial basis function (RBF) neural network to classify hand gesture images using selected combined features. In the proposed RBF neural network, the centers are automatically selected using k-means algorithm and estimated weight matrix is recursively updated, utilizing least-mean-square algorithm for better recognition of hand gesture images. The comparative performances are tested on two indigenously developed databases of 24 American sign language hand alphabet.     

融合Inception-LSTM级联网络下的动态手势识别

目前基于视觉的动态手势识别问题仍是研究的难点,在大多数应用背景情况下很难提高手势识别率.传统的动态手势识别手段主要是利用智能传感设备以及单个或多个摄像头进行数据采集的视觉方法来实现,效率低,准确度差.近年来,随着深度神经网络技术的快速发展,利用网络自主学习的方法来提取手势姿态有关特征得到了广泛关注.本文针对传统动态手势...     

Viewpoint-independent hand gesture recognition with Kinect

The advent and popularity of Kinect provide new choice and opportunity for hand gesture recognition research. Aiming at the effective, accurate and freely used hand gesture recognition with Kinect, this paper presents a viewpoint-independent hand gesture recognition method. Firstly, based on the rules about gesturers posture under optimal viewpoint, the gesturers point clouds are built and transformed to the optimal viewpoint with the exploration of the joint information. Then Laplacian-based contraction is applied to extract representative skeletons from the transformed point clouds. A novel partition-based algorithm is further proposed to recognize the gestures. The promising experiment results show that the proposed method performs satisfyingly on scale and rotation variant in HGR with robustness and high accuracy.     

A Sign-Component-Based Framework for Chinese Sign Language Recognition Using Accelerometer and sEMG Data

Identification of constituent components of each sign gesture can be beneficial to the improved performance of sign language recognition (SLR), especially for large-vocabulary SLR systems. Aiming at developing such a system using portable accelerometer (ACC) and surface electromyographic (sEMG) sensors, we propose a framework for automatic Chinese SLR at the component level. In the proposed framework, data segmentation, as an important preprocessing operation, is performed to divide a continuous sign language sentence into subword segments. Based on the features extracted from ACC and sEMG data, three basic components of sign subwords, namely the hand shape, orientation, and movement, are further modeled and the corresponding component classifiers are learned. At the decision level, a sequence of subwords can be recognized by fusing the likelihoods at the component level. The overall classification accuracy of 96.5% for a vocabulary of 120 signs and 86.7% for 200 sentences demonstrate the feasibility of interpreting sign components from ACC and sEMG data and clearly show the superior recognition performance of the proposed method when compared with the previous SLR method at the subword level. The proposed method seems promising for implementing large-vocabulary portable SLR systems.     

基于Kinect的人体动作识别算法研究

针对现有的复杂背景下人体动作识别中存在识别准确率不高和实时性不强等问题,提出基于Kinect骨骼数据的改进动作识别算法。通过Kinect获取骨骼数据,提取出人体关节的特征向量,然后用模板匹配的方法对人体动作进行识别。通过搭建机器人体感控制系统验证了算法的可行性。在相同实验条件下测得算法的平均识别率为95.2%,平均识别时间为32.5ms。与其它动作识别算法比较,证明了算法的识别率较高、实时性较好。