融合宽残差和长短时记忆网络的动态手势识别研究

针对现有的动态手势识别方法对长时间序列的时空特征难以精确匹配的问题,提出了一种基于宽残差和双向长短时记忆网络的时空特征一致手势识别方法。首先使用已经训练好的3D卷积神经网络从视频的空间和时间维度同步提取出短时特征,再经双向空间长短时记忆网络同步解析后形成长时空特征连接单元,并作为残差网络的输入。为了验证算法的有效性,使用Kinect传感器构建了一个全新的多模式手势数据集,在三个手势识别公开数据集SLVM、Montalbano和SKIG上的实验表明,提出的方法有很好的性能表现,识别精度超越了目前已公开的最佳识别率。     

基于异步多时域特征的动态手势识别方法

准确及时地手势识别在增强现实技术中具有重要的意义。针对表征复杂手势序列的时空特征,提出了一种基于异步多时域时空特征的手势识别方法。该方法通过轻量级三维卷积网络提取视频序列的不同时间步态的短期时空特征,通过改进的卷积长短期记忆网络学习长期时空特征,将不同步态的时空特征融合为异步多时域特征,以此来对手势进行分类识别。通过与其他主流方法进行比较,实验结果证明了提出的方法具有较高的动态手势识别率。     

基于Leap Motion的动态手势识别

动态手势识别作为人机交互的一个重要方向，在各个领域具有广泛的需求。相较于静态手势，动态手势的变化更为复杂，对其特征的充分提取与描述是准确识别动态手势的关键。为了解决对动态手势特征描述不充分的问题，利用高精度的Leap Motion传感器对手部三维坐标信息进行采集，提出了一种包含手指姿势和手掌位移的特征在内的、能够充分描述复杂动态手势的特征序列，并结合长短期记忆网络模型进行动态手势识别。实验结果表明，提出的方法在包含16种动态手势的数据集上的识别准确率为98.50%；与其他特征序列的对比实验表明，提出的特征序列，能更充分准确地描述动态手势特征。     

3D-based Deep Convolutional Neural Network for action recognition with depth sequences

Traditional algorithms to design hand-crafted features for action recognition have been a hot research area in the last decade. Compared to RGB video, depth sequence is more insensitive to lighting changes and more discriminative due to its capability to catch geometric information of object. Unlike many existing methods for action recognition which depend on well-designed features, this paper studies deep learning-based action recognition using depth sequences and the corresponding skeleton joint information. Firstly, we construct a 3D-based Deep Convolutional Neural Network (3D²CNN) to directly learn spatio-temporal features from raw depth sequences, then compute a joint based feature vector named JointVector for each sequence by taking into account the simple position and angle information between skeleton joints. Finally, support vector machine (SVM) classification results from 3D²CNN learned features and JointVector are fused to take action recognition. Experimental results demonstrate that our method can learn feature representation which is time-invariant and viewpoint-invariant from depth sequences. The proposed method achieves comparable results to the state-of-the-art methods on the UTKinect-Action3D dataset and achieves superior performance in comparison to baseline methods on the MSR-Action3D dataset. We further investigate the generalization of the trained model by transferring the learned features from one dataset (MSR-Action3D) to another dataset (UTKinect-Action3D) without retraining and obtain very promising classification accuracy.     

融合关节旋转特征和指尖距离特征的手势识别

作为人机交互的重要方式,手势交互和识别由于其具有的高自由度而成为计算机图形学、虚拟现实与人机交互等领域的研究热点.传统直接提取手势轮廓或手部关节点位置信息的手势识别方法,其提取的特征通常难以准确表示手势之间的区别.针对手势识别中不同手势具有的高自由度以及由于手势图像分辨率低、背景杂乱、手被遮挡、手指形状尺寸不同、个体差异性导致手势特征表示不准确等问题,本文提出了一种新的融合关节旋转特征和指尖距离特征的手势特征表示与手势识别方法.首先从手势深度图中利用手部模板并将手部看成链段结构提取手部20个关节点的3D位置信息;然后利用手部关节点位置信息提取四元数关节旋转特征和指尖距离特征,该表示构成了手势特征的内在表示;最后利用一对一支持向量机对手势进行有效识别分类.本文不仅提出了一种新的手势特征表示与提取方法,该表示融合了关节旋转信息和指尖距离特征;而且从理论上证明了该特征表示能唯一地表征手势关节点的位置信息;同时提出了基于一对一SVM多分类策略进行手势分类与识别.对ASTAR静态手势深度图数据集中8类中国数字手势和21类美国字母手势数据集分别进行了实验验证,其分类识别准确率分别为99.71%和85.24%.实验结果表明,本文提出的基于关节旋转特征和指尖距离特征的融合特征能很好地表示不同手势的几何特征,能准确地表征静态手势并进行手势识别.     

基于深度运动图和密集轨迹的行为识别算法

为了融合深度图中不易受光照等环境因素影响的深度信息和RGB视频序列中丰富的纹理信息,提出一种基于深度运动图（Depth Motion Maps,DMMs）和密集轨迹的人体行为识别算法。利用卷积神经网络训练DMMs数据并提取高层特征作为行为视频的静态特征表示,使用密集轨迹来描述RGB视频序列的动态运动信息,将行为视频的静态特征和动态特征串联,作为整个视频的行为特征表示并输入到线性支持向量机（Support Vector Machine,SVM）进行识别。实验结果表明,在公开的动作识别库UTD-MHAD和MSR Daily Activity 3D上,该算法能够有效提取深度信息和纹理信息,并取得了较好的识别效果。     

基于长短时记忆和深度神经网络的视觉手势识别技术

针对基于视觉的动态手势识别易受光照、背景和手势形状变化影响等问题,在分 析人体手势空间上下文特征的基础上,首先建立一种基于人体骨架和部件轮廓特征的动态手势 模型,并采用卷积姿势机和单发多框检测器技术构造深度神经网络进行人体手势骨架和部件轮 廓特征提取。其次,引入长短时记忆网络提取动态人体手势中骨架、左右手和头部轮廓的时序 特征,进而分类识别手势。在此基础上,设计了一种空间上下文与时序特征融合的动态手势识 别机(GRSCTFF),并通过交警指挥手势视频样本库对其进行网络训练和实验分析。实验证明, 该系统 可以快速准确识别动态交警指挥手势,准确率达到94.12%,并对光线、背景和手势形 状变化具有较强的抗干扰能力。     

连续动态手势的时空表观建模及识别

论述了复杂背景下连接动态手势的分割、建模及识别;融合手势运动信息和皮肤颜色信息,进行复杂背景下的手势分割;通过结合手势的时序信息、运动表观以及形状表观,提出动态手势的时空表观模型,并提出基于颜色、运动以及形状等多模式信息的分层次融合策略抽取时空表观模型的参数,最后,提出动态时空规整算法用于手势识别,实验表明,利用上述提出的手势分割、建模、特征参数抽取及识别方法识别１２种手势,平均识别率高达９７％。     

基于Kinect骨骼信息与深度图像的指尖点检测

针对普通摄像头手势识别系统易受复杂环境和光照条件等因素影响,存在对指尖点的漏判、误判问题,提出一种基于Kinect 骨骼信息与深度图像的掌心点提取和指尖点检测的手势识别方法。在DRVI平台上创建Kinect的接口控件,对Kinect传感器获取人体骨骼信息和深度图像进行分析,采用了坐标映射、图像分割、距离变换的关键技术和方法从深度图中分割出手势部分区域,对手势区域形态学处理,结合凸包和K-曲率算法检测不同手势中指尖点的个数和位置,计算不同手势凸包轮廓上的点集生成的HOG(Histogram of Oriented Gradient)特征描述子,最后利用特征描述子对预定的6种数字手势进行识别。经实验测试可以在复杂环境和不同光照情况下正确识别指尖点。     

A Real-Time Approach to the Spotting, Representation, and Recognition of Hand Gestures for Human–Computer Interaction

Aiming at the use of hand gestures for human–computer interaction, this paper presents a real-time approach to the spotting, representation, and recognition of hand gestures from a video stream. The approach exploits multiple cues including skin color, hand motion, and shape. Skin color analysis and coarse image motion detection are joined to perform reliable hand gesture spotting. At a higher level, a compact spatiotemporal representation is proposed for modeling appearance changes in image sequences containing hand gestures. The representation is extracted by combining robust parameterized image motion regression and shape features of a segmented hand. For efficient recognition of gestures made at varying rates, a linear resampling technique for eliminating the temporal variation (time normalization) while maintaining the essential information of the original gesture representations is developed. The gesture is then classified according to a training set of gestures. In experiments with a library of 12 gestures, the recognition rate was over 90%. Through the development of a prototype gesture-controlled panoramic map browser, we demonstrate that a vocabulary of predefined hand gestures can be used to interact successfully with applications running on an off-the-shelf personal computer equipped with a home video camera.     

多视角深度运动图的人体行为识别

目的 使用运动历史点云（MHPC）进行人体行为识别的方法,由于点云数据量大,在提取特征时运算复杂度很高。而使用深度运动图（DMM）进行人体行为识别的方法,提取特征简单,但是包含的动作信息不全面,限制了人体行为识别精度的上限。针对上述问题,提出了一种多视角深度运动图的人体行为识别算法。方法 首先采用深度图序列生成MHPC对动作进行表示,接着将MHPC旋转特定角度补充更多视角下的动作信息;然后将原始和旋转后MHPC投影到笛卡儿坐标平面,生成多视角深度运动图,并对其提取方向梯度直方图,采用串联融合生成特征向量;最后使用支持向量机对特征向量进行分类识别,在MSR Action3D和自建数据库上对算法进行验证。结果 MSR Action3D数据库有2种实验设置,采用实验设置1时,算法识别率为96.8%,比APS_PHOG（axonometric projections and PHOG feature）算法高2.5%,比DMM算法高1.9%,比DMM_CRC（depth motion maps and collaborative representation classifier）算法高1.1%。采用实验设置2时,算法识别率为93.82%,比DMM算法高5.09%,比HON4D（histogram of oriented 4D surface normal）算法高4.93%。在自建数据库上该算法识别率达到97.98%,比MHPC算法高3.98%。结论 实验结果表明,多视角深度运动图不但解决了MHPC提取特征复杂的问题,而且使DMM包含了更多视角下的动作信息,有效提高了人体行为识别的精度。     

Real-time fingertip localization conditioned on hand gesture classification

A method to obtain accurate hand gesture classification and fingertip localization from depth images is proposed. The Oriented Radial Distribution feature is utilized, exploiting its ability to globally describe hand poses, but also to locally detect fingertip positions. Hence, hand gesture and fingertip locations are characterized with a single feature calculation. We propose to divide the difficult problem of locating fingertips into two more tractable problems, by taking advantage of hand gesture as an auxiliary variable. Along with the method we present the ColorTip dataset, a dataset for hand gesture recognition and fingertip classification using depth data. ColorTip contains sequences where actors wear a glove with colored fingertips, allowing automatic annotation. The proposed method is evaluated against recent works in several datasets, achieving promising results in both gesture classification and fingertip localization.     

基于深度信息的手势识别算法研究

针对在复杂背景中传统手势识别算法的识别率低问题,利用Kinect的深度摄像头获取深度图像,分割出手势区域后进行预处理;提取手势的几何特征,并提出深度信息的同心圆分布直方图特征,融合手势的几何特征和深度信息的同心圆分布直方图特征;学习训练随机森林分类器进行手势识别.文中通过在复杂背景条件下对常见的“石头”、“剪刀”、“布”3种手势进行测试,实验结果表明:文中所提方法具有很好的平移,旋转和缩放不变性,能适应复杂环境的变化.     

基于时空上下文的手势跟踪与识别

针对复杂背景下手势运动过程中出现的手势形态变化、遮挡、光照变化等问题,提出了一种基于时空上下文的手势跟踪与识别方法。使用机器学习方法离线训练手势样本分类器,实现对手势的检测和定位;利用时空上下文跟踪算法对动态手势进行跟踪,同时为了避免跟踪过程中出现的漂移、目标丢失等情况,使用手势检测算法对手势位置信息进行实时校准;根据手势运动轨迹对手势运动进行跟踪与识别。实验表明,提出的方法可以实现对手势运动快速、准确、连续识别,满足人机交互的要求。     

基于骨架的快速手势识别模型

对于手势识别来说,骨架数据是一种紧凑且对环境条件稳健的数据模态.最近基于骨架的手势识别研究多使用深度神经网络去提取空间和时间的信息,然而这些方法可能存在复杂的计算和大量的模型参数的问题.为了解决这个问题,我们提出一种轻量高效的手势识别模型.该模型使用从骨架序列上计算出的两种空间几何特征,以及自动学习的运动轨迹特征,然后只使用卷积网络作为骨干网络实现手势分类.最终我们的模型参数量最少情况下仅为0.16 M,计算复杂度最大情况为0.03 GFLOPs.我们在公开的两个数据集上评估了我们的方法,与其他输入为骨架模态的方法相比,我们的方法取得了相应数据集上最好的结果.     

A natural and synthetic corpus for benchmarking of hand gesture recognition systems

The use of hand gestures offers an alternative to the commonly used human–computer interfaces (i.e. keyboard, mouse, gamepad, voice, etc.), providing a more intuitive way of navigating among menus and in multimedia applications. This paper presents a dataset for the evaluation of hand gesture recognition approaches in human–computer interaction scenarios. It includes natural data and synthetic data from several State of the Art dictionaries. The dataset considers single-pose and multiple-pose gestures, as well as gestures defined by pose and motion or just by motion. Data types include static pose videos and gesture execution videos—performed by a set of eleven users and recorded with a time-of-flight camera—and synthetically generated gesture images. A novel collection of critical factors involved in the creation of a hand gestures dataset is proposed: capture technology, temporal coherence, nature of gestures, representativeness, pose issues and scalability. Special attention is given to the scalability factor, proposing a simple method for the synthetic generation of depth images of gestures, making possible the extension of a dataset with new dictionaries and gestures without the need of recruiting new users, as well as providing more flexibility in the point-of-view selection. The method is validated for the presented dataset. Finally, a separability study of the pose-based gestures of a dictionary is performed. The resulting corpus, which exceeds in terms of representativity and scalability the datasets existing in the State Of Art, provides a significant evaluation scenario for different kinds of hand gesture recognition solutions.     

复杂背景下基于傅立叶描述子的手势识别

人的手势是人们日常生活中最广泛使用的一种交流方式。由于在人机交互界面和虚拟现实环境中的应用,手势识别的研究受到了越来越广泛的关注。但是目前基于单目视觉的手势识别技术中,手势分割要求背景简单或者要求识别者戴着笨重的数据手套。而该文结合了运动信息和基于KL变换的肤色模型,在复杂背景下进行手势分割,与传统的基于RGB肤色模型的手势分割相比,在复杂背景环境下得到了很好的分割效果。在对分割的手势区域进行预处理后,该文使用了一种归一化的傅立叶描述子进行手势的特征提取,相比传统的傅立叶描述子更加准确,最后采用了传统的三层BP网络作为模式识别器,手势训练集和测试集的识别率分别达到了95．9％和95％。     

Recognition of dynamic hand gestures

This paper is concerned with the problem of recognition of dynamic hand gestures. We have considered gestures which are sequences of distinct hand poses. In these gestures hand poses can undergo motion and discrete changes. However, continuous deformations of the hand shapes are not permitted. We have developed a recognition engine which can reliably recognize these gestures despite individual variations. The engine also has the ability to detect start and end of gesture sequences in an automated fashion. The recognition strategy uses a combination of static shape recognition (performed using contour discriminant analysis), Kalman filter based hand tracking and a HMM based temporal characterization scheme. The system is fairly robust to background clutter and uses skin color for static shape recognition and tracking. A real time implementation on standard hardware is developed. Experimental results establish the effectiveness of the approach.