A framework for heading-guided recognition of human activity

A combined 2D, 3D approach is presented that allows for robust tracking of moving people and recognition of actions. It is assumed that the system observes multiple moving objects via a single, uncalibrated video camera. Low-level features are often insufficient for detection, segmentation, and tracking of non-rigid moving objects. Therefore, an improved mechanism is proposed that integrates low-level (image processing), mid-level (recursive 3D trajectory estimation), and high-level (action recognition) processes. A novel extended Kalman filter formulation is used in estimating the relative 3D motion trajectories up to a scale factor. The recursive estimation process provides a prediction and error measure that is exploited in higher-level stages of action recognition. Conversely, higher-level mechanisms provide feedback that allows the system to reliably segment and maintain the tracking of moving objects before, during, and after occlusion. Heading-guided recognition (HGR) is proposed as an efficient method for adaptive classification of activity. The HGR approach is demonstrated using “motion history images” that are then recognized via a mixture-of-Gaussians classifier. The system is tested in recognizing various dynamic human outdoor activities: running, walking, roller blading, and cycling. In addition, experiments with real and synthetic data sets are used to evaluate stability of the trajectory estimator with respect to noise.     

A real-time system for motion retrieval and interpretation

This paper proposes a new examplar-based method for real-time human motion recognition using Motion Capture (MoCap) data. We have formalized streamed recognizable actions, coming from an online MoCap engine, into a motion graph that is similar to an animation motion graph. This graph is used as an automaton to recognize known actions as well as to add new ones. We have defined and used a spatio-temporal metric for similarity measurements to achieve more accurate feedbacks on classification. The proposed method has the advantage of being linear and incremental, making the recognition process very fast and the addition of a new action straightforward. Furthermore, actions can be recognized with a score even before they are fully completed. Thanks to the use of a skeleton-centric coordinate system, our recognition method has become view-invariant. We have successfully tested our action recognition method on both synthetic and real data. We have also compared our results with four state-of-the-art methods using three well known datasets for human action recognition. In particular, the comparisons have clearly shown the advantage of our method through better recognition rates.     

Early detection of human actions—A hybrid approach

Early detection of human actions is essential in a wide spectrum of applications ranging from video surveillance to health-care. While human action recognition has been extensively studied, little attention is paid to the problem of detecting ongoing human action early, i.e. detecting an action as soon as it begins, but before it finishes. This study aims at training a detector to be capable of recognizing a human action when only partial action sample is seen. To do so, a hybrid technique is proposed in this work which combines the benefits of computer vision as well as fuzzy set theory based on the fuzzy Bandler and Kohout's sub-triangle product (BK subproduct). The novelty lies in the construction of a frame-by-frame membership function for each kind of possible movement. Detection is triggered when a pre-defined threshold is reached in a suitable way. Experimental results on a publicly available dataset demonstrate the benefits and effectiveness of the proposed method.     

Human action recognition using boosted EigenActions

This paper proposes a boosting EigenActions algorithm for human action recognition. A spatio-temporal Information Saliency Map (ISM) is calculated from a video sequence by estimating pixel density function. A continuous human action is segmented into a set of primitive periodic motion cycles from information saliency curve. Each cycle of motion is represented by a Salient Action Unit (SAU), which is used to determine the EigenAction using principle component analysis. A human action classifier is developed using multi-class Adaboost algorithm with Bayesian hypothesis as the weak classifier. Given a human action video sequence, the proposed method effectively locates the SAUs in the video, and recognizes the human actions by categorizing the SAUs. Two publicly available human action databases, namely KTH and Weizmann, are selected for evaluation. The average recognition accuracy are 81.5% and 98.3% for KTH and Weizmann databases, respectively. Comparative results with two recent methods and robustness test results are also reported.     

Human action recognition based on skeleton splitting

Human action recognition, defined as the understanding of the human basic actions from video streams, has a long history in the area of computer vision and pattern recognition because it can be used for various applications. We propose a novel human action recognition methodology by extracting the human skeletal features and separating them into several human body parts such as face, torso, and limbs to efficiently visualize and analyze the motion of human body parts.Our proposed human action recognition system consists of two steps: (i) automatic skeletal feature extraction and splitting by measuring the similarity between neighbor pixels in the space of diffusion tensor fields, and (ii) human action recognition by using multiple kernel based Support Vector Machine. Experimental results on a set of test database show that our proposed method is very efficient and effective to recognize the actions using few parameters.     

Detection of human faces in color image sequences with arbitrary motions for very low bit-rate videophone coding

The problem of human face detection is a focus of interest in image analysis, image databases and video coding. A new multi-resolution method using color and motion information and shape model is developed to detect human faces in videophone QCIF sequences for efficient encoding. The method is based on color segmentation and multiresolution propagation of a geometrical model. A new measure of motion activity is proposed to validate the choice of candidates.     

动作切分和流形度量学习的视频动作识别

目的 为了提高视频中动作识别的准确度,提出基于动作切分和流形度量学习的视频动作识别算法。方法 首先利用基于人物肢体伸展程度分析的动作切分方法对视频中的动作进行切分,将动作识别的对象具体化;然后从动作片段中提取归一化之后的全局时域特征和空域特征、光流特征、帧内的局部旋度特征和散度特征,构造一种7×7的协方差矩阵描述子对提取出的多种特征进行融合;最后结合流形度量学习方法有监督式地寻找更优的距离度量算法提高动作的识别分类效果。结果 对Weizmann公共视频集的切分实验统计结果表明本文提出的视频切分方法具有很好的切分能力,能够作好动作识别前的预处理;在Weizmann公共视频数据集上进行了流形度量学习前后的识别效果对比,结果表明利用流形度量学习方法对动作识别效果提升2.8%;在Weizmann和KTH两个公共视频数据集上的平均识别率分别为95.6%和92.3%,与现有方法的比较表明,本文提出的动作识别方法有更好的识别效果。结论 多次实验结果表明本文算法在预处理过程中动作切分效果理想,描述动作所构造协方差矩阵对动作的表达有良好的多特征融合能力,而且光流信息和旋度、散度信息的加入使得人体各部位的运动方向信息具有了更多细节的描述,有效提高了协方差矩阵的描述能力,结合流形度量学习方法对动作识别的准确性有明显提高。     

一种基于MEM-LBP的动作特征提取及识别方法*

针对动作识别中如何有效地利用人体运动的三维信息的问题,提出一种新的基于深度视频序列的特征提取和识别方法。该方法首先运用运动能量模型（MEM）来表征人体动态特征,即先将整个深度视频序列投影到三个正交的笛卡儿平面上,再把每个投影面的视频系列划分为能量均等的子时间序列,分别计算子序列的深度运动图能量从而得到运动能量模型（MEM）。然后利用局部二值模式（LBP）描述符对运动能量模型编码,进一步提取人体运动的有效信息。最后用 范数协同表示分类器进行动作分类识别。在MSRAction3D、MSRGesture3D数据库上测试所提方法,实验结果表明该方法有较高的识别效果。     

基于人体骨架信息的行为检测系统设计

为了解决现有行为检测系统中依赖惯性传感器、检测结果不够准确的问题,设计了基于人体骨架信息的行为检测系统;系统采用Jetson Nano人工智能计算设备作为主控模块,结合图像采集模块、显示模块和以Atmega328单片机为主的报警模块构成;系统利用图像采集模块采集行为视频信息,通过主控模块中的行为检测器对视频中人体行为进行检测,报警模块通过串口接收检测结果并对危险行为进行预警;同时,利用人体骨架的关节空间运动幅度、肢体关联差异,建立了关节帧间位移矢量和骨骼夹角变化的关节行为模型,再借助长短时记忆网络提取行为特征,并训练实时行为检测器;经实验测试,该系统能够有效检测常见的人体行为并对危险行为类别进行报警提示。     

基于Fisher鉴别字典学习的人体行为识别

人体行为识别是计算机视觉中的一个重要研究领域,具有广阔的应用前景。研究了基于Fisher鉴别的字典学习方法在人体行为识别上的应用。首先对人体行为的视频序列提取了局部时空特征,并通过随机投影法降维;然后把降维后的特征作为待分类的信号进行Fisher鉴别字典学习,从而增强字典和编码系数的鉴别能力;最后同时利用重构误差和稀疏表示系数进行分类。实验结果验证了所提方法在人体行为识别上的有效性与鲁棒性。     

基于新投影策略的人体行为识别方法研究

为解决微小动作识别率低的问题,提出一种结合新投影策略和能量均匀化视频分割的多层深度运动图的人体行为识别方法。首先,提出一种新的投影策略,将深度图像投影到三个正交笛卡尔平面,以保留更多的行为信息;其次,基于整个视频的多层深度运动图图像虽然可反映整体运动信息,但却忽略了很多细节,采用基于能量均匀化的视频分割方法,将视频划分为多个子视频序列,可以更加全面地刻画动作细节信息;最后,为描述多层深度运动图图像纹理细节,采用局部二值模式作为动作特征描述子,结合核极端学习机分类器进行动作识别。实验结果表明：在公开动作识别库MSRAction3D和手势识别库MSRGesture3D上,本文算法准确率分别达94.55%和95.67%,与现存许多算法相比,有更高的识别率。     

Real-time Unsupervised Segmentation of human whole-body motion and its application to humanoid robot acquisition of motion symbols

An interactive loop between motion recognition and motion generation is a fundamental mechanism for humans and humanoid robots. We have been developing an intelligent framework for motion recognition and generation based on symbolizing motion primitives. The motion primitives are encoded into Hidden Markov Models (HMMs), which we call “motion symbols”. However, to determine the motion primitives to use as training data for the HMMs, this framework requires a manual segmentation of human motions. Essentially, a humanoid robot is expected to participate in daily life and must learn many motion symbols to adapt to various situations. For this use, manual segmentation is cumbersome and impractical for humanoid robots. In this study, we propose a novel approach to segmentation, the Real-time Unsupervised Segmentation (RUS) method, which comprises three phases. In the first phase, short human movements are encoded into feature HMMs. Seamless human motion can be converted to a sequence of these feature HMMs. In the second phase, the causality between the feature HMMs is extracted. The causality data make it possible to predict movement from observation. In the third phase, movements having a large prediction uncertainty are designated as the boundaries of motion primitives. In this way, human whole-body motion can be segmented into a sequence of motion primitives. This paper also describes an application of RUS to AUtonomous Symbolization of motion primitives (AUS). Each derived motion primitive is classified into an HMM for a motion symbol, and parameters of the HMMs are optimized by using the motion primitives as training data in competitive learning. The HMMs are gradually optimized in such a way that the HMMs can abstract similar motion primitives. We tested the RUS and AUS frameworks on captured human whole-body motions and demonstrated the validity of the proposed framework.     

实时动作识别方法研究

针对传统RGB视频中动作识别算法时间复杂度高而识别准确率低的问题,提出一种基于深度图像的动作识别方法。该方法首先对深度图像在三投影面系中进行投影,然后对三个投影图分别提取Gabor特征,最后使用这些特征训练极限学习机分类器,从而完成动作分类。在公开数据集MSR Action3D上进行了实验验证,该方法在三组实验上的平均准确率分别为97.80%、99.10%和88.35%,识别单个深度视频的用时小于1 s。实验结果表明,该方法能够对深度图像序列中的人体动作进行有效识别,并基本满足深度序列识别的实时性要求。     

Exploring trace transform for robust human action recognition

Machine based human action recognition has become very popular in the last decade. Automatic unattended surveillance systems, interactive video games, machine learning and robotics are only few of the areas that involve human action recognition. This paper examines the capability of a known transform, the so-called Trace, for human action recognition and proposes two new feature extraction methods based on the specific transform. The first method extracts Trace transforms from binarized silhouettes, representing different stages of a single action period. A final history template composed from the above transforms, represents the whole sequence containing much of the valuable spatio-temporal information contained in a human action. The second, involves Trace for the construction of a set of invariant features that represent the action sequence and can cope with variations usually appeared in video capturing. The specific method takes advantage of the natural specifications of the Trace transform, to produce noise robust features that are invariant to translation, rotation, scaling and are effective, simple and fast to create. Classification experiments performed on two well known and challenging action datasets (KTH and Weizmann) using Radial Basis Function (RBF) Kernel SVM provided very competitive results indicating the potentials of the proposed techniques.     

A dual fast and slow feature interaction in biologically inspired visual recognition of human action

Computational neuroscience studies have examined the human visual system through functional magnetic resonance imaging (fMRI) and identified a model where the mammalian brain pursues two independent pathways for recognizing biological movement tasks. On the one hand, the dorsal stream analyzes the motion information by applying optical flow, which considers the fast features. On the other hand, the ventral stream analyzes the form information with slow features. The proposed approach suggests that the motion perception of the human visual system comprises fast and slow feature interactions to identify biological movements. The form features in the visual system follow the application of the active basis model (ABM) with incremental slow feature analysis (IncSFA). Episodic observation is required to extract the slowest features, whereas the fast features update the processing of motion information in every frame. Applying IncSFA provides an opportunity to abstract human actions and use action prototypes. However, the fast features are obtained from the optical flow division, which gives an opportunity to interact with the system as the final recognition is performed through a combination of the optical flow and ABM-IncSFA information and through the application of kernel extreme learning machine. Applying IncSFA into the ventral stream and involving slow and fast features in the recognition mechanism are the major contributions of this research. The two human action datasets for benchmarking (KTH and Weizmann) and the results highlight the promising performance of this approach in model modification.