View-invariant motion trajectory-based activity classification and recognition

Motion trajectories provide rich spatio-temporal information about an object's activity. The trajectory information can be obtained using a tracking algorithm on data streams available from a range of devices including motion sensors, video cameras, haptic devices, etc. Developing view-invariant activity recognition algorithms based on this high dimensional cue is an extremely challenging task. This paper presents efficient activity recognition algorithms using novel view-invariant representation of trajectories. Towards this end, we derive two Affine-invariant representations for motion trajectories based on curvature scale space (CSS) and centroid distance function (CDF). The properties of these schemes facilitate the design of efficient recognition algorithms based on hidden Markov models (HMMs). In the CSS-based representation, maxima of curvature zero crossings at increasing levels of smoothness are extracted to mark the location and extent of concavities in the curvature. The sequences of these CSS maxima are then modeled by continuous density (HMMs). For the case of CDF, we first segment the trajectory into subtrajectories using CDF-based representation. These subtrajectories are then represented by their Principal Component Analysis (PCA) coefficients. The sequences of these PCA coefficients from subtrajectories are then modeled by continuous density hidden Markov models (HMMs). Different classes of object motions are modeled by one Continuous HMM per class where state PDFs are represented by GMMs. Experiments using a database of around 1750 complex trajectories (obtained from UCI-KDD data archives) subdivided into five different classes are reported.     

Markerless tracking and gesture recognition using polar correlation of camera optical flow

We present a novel, real-time, markerless vision-based tracking system, employing a rigid orthogonal configuration of two pairs of opposing cameras. Our system uses optical flow over sparse features to overcome the limitation of vision-based systems that require markers or a pre-loaded model of the physical environment. We show how opposing cameras enable cancellation of common components of optical flow leading to an efficient tracking algorithm that captures five degrees of freedom including direction of translation and angular velocity. Experiments comparing our device with an electromagnetic tracker show that its average tracking accuracy is 80 % over 185 frames, and it is able to track large range motions even in outdoor settings. We also present how our tracking system can be used for gesture recognition by combining it with a simple linear classifier over a set of 15 gestures. Experimental results show that we are able to achieve 86.7 % gesture recognition accuracy.     

3D text segmentation and recognition using leap motion

In this paper, we present a method of Human-Computer-Interaction (HCI) through 3D air-writing. Our proposed method includes a natural way of interaction without pen and paper. The online texts are drawn on air by 3D gestures using fingertip within the field of view of a Leap motion sensor. The texts consist of single stroke only. Hence gaps between adjacent words are usually absent. This makes the system different as compared to the conventional 2D writing using pen and paper. We have collected a dataset that comprises with 320 Latin sentences. We have used a heuristic to segment 3D words from sentences. Subsequently, we present a methodology to segment continuous 3D strokes into lines of texts by finding large gaps between the end and start of the lines. This is followed by segmentation of the text lines into words. In the next phase, a Hidden Markov Model (HMM) based classifier is used to recognize 3D sequences of segmented words. We have used dynamic as well as simple features for classification. We have recorded an overall accuracy of 80.3 % in word segmentation. Recognition accuracies of 92.73 % and 90.24 % have been recorded when tested with dynamic and simple features, respectively. The results show that the Leap motion device can be a low-cost but useful solution for inputting text naturally as compared to conventional systems. In future, this may be extended such that the system can successfully work on cluttered gestures.     

Model-based recognition using 3D structure from motion

The effectiveness of shape information alone, without size, for recognizing stored 3D models is considered. The geometric constraint filtering method of Grimson and Lozano-Pérez is used to curb the potentially combinatorial expansion of the model search space. Results, typical of those from several models experimented with, are given for the task of recognizing a plug from uncertain surface normal data. They show that, at least for an ‘interesting’ view, shape data is highly effective, even when the sensed surface normals are uncertain in direction to, say, ± 10°. The loss of size information does, however, result in a drop in search efficiency, but this appears relatively small: u factor of ≈ 5 for the example of a plug at the 10° error mark.     

An efficient algorithm for 3D hand gesture recognition using combined neural classifiers

Gestures are the dynamic movements of hands within a certain time interval, which are of practical importance in many areas, such as human–computer interaction, computer vision, and computer graphics. The human hand gesture can provide a free and natural alternative to today’s cumbersome interface devices so as to improve the efficiency and effectiveness of human–computer interaction. This paper presents a neural-based combined classifier for 3D gesture recognition. The combined classifier is based on varying the parameters related to both the design and training of neural network classifier. The boosting algorithm is used to make perturbation of the training set employing the Multi-Layer Perceptron as base classifier. The final decision of the ensemble of classifiers is based on the majority voting rule. Experiments performed on 3D gesture database show the robustness of the proposed technique.     

Extraction of 2D motion trajectories and its application to hand gesture recognition

We present an algorithm for extracting and classifying two-dimensional motion in an image sequence based on motion trajectories. First, a multiscale segmentation is performed to generate homogeneous regions in each frame. Regions between consecutive frames are then matched to obtain two-view correspondences. Affine transformations are computed from each pair of corresponding regions to define pixel matches. Pixels matches over consecutive image pairs are concatenated to obtain pixel-level motion trajectories across the image sequence. Motion patterns are learned from the extracted trajectories using a time-delay neural network. We apply the proposed method to recognize 40 hand gestures of American Sign Language. Experimental results show that motion patterns of hand gestures can be extracted and recognized accurately using motion trajectories.     

Principal motion components for one-shot gesture recognition

This paper introduces principal motion components (PMC), a new method for one-shot gesture recognition. In the considered scenario a single training video is available for each gesture to be recognized, which limits the application of traditional techniques (e.g., HMMs). In PMC, a 2D map of motion energy is obtained per each pair of consecutive frames in a video. Motion maps associated to a video are processed to obtain a PCA model, which is used for recognition under a reconstruction-error approach. The main benefits of the proposed approach are its simplicity, easiness of implementation, competitive performance and efficiency. We report experimental results in one-shot gesture recognition using the ChaLearn Gesture Dataset; a benchmark comprising more than 50,000 gestures, recorded as both RGB and depth video with a Kinect?camera. Results obtained with PMC are competitive with alternative methods proposed for the same data set.     

Generating 3D architectural models based on hand motion and gesture

This paper presents a novel method for rapidly generating 3D architectural models based on hand motion and design gestures captured by a motion capture system. A set of sign language-based gestures, architectural hand signs (AHS), has been developed. AHS is performed on the left hand to define various “components of architecture”, while “location, size and shape” information is defined by the motion of Marker-Pen on the right hand. The hand gestures and motions are recognized by the system and then transferred into 3D curves and surfaces correspondingly. This paper demonstrates the hand gesture-aided architectural modeling method with some case studies.     

Fast optical flow using 3D shortest path techniques

Optical flow or image motion estimation is important in the area of computer vision. This paper presents a fast and reliable optical flow algorithm which produces a dense optical flow map by using fast cross correlation and 3D shortest path techniques. Fast correlation is achieved by using the box-filtering technique which is invariant to the size of the correlation window. The motion for each scanline or each column of the input image is obtained from the correlation coefficient volume by finding the best 3D path using dynamic programming techniques rather than simply choosing the position that gives the maximum cross correlation coefficient. Sub-pixel accuracy is achieved by fitting the local correlation coefficients to a quadratic surface. Typical running time for a 256×256 image is in the order of a few seconds on a 85 MHz computer. A variety of synthetic and real images have been tested, and good results have been obtained.     

Hand gesture recognition using topological features

Multimedia Tools and Applications - Hand Gestures Recognition (HGR) is one of the main areas of research for Human Computer Interaction applications. Most existing approaches are based on local or...     

Static hand gesture recognition using neural networks

This paper presents a novel technique for hand gesture recognition through human–computer interaction based on shape analysis. The main objective of this effort is to explore the utility of a neural network-based approach to the recognition of the hand gestures. A unique multi-layer perception of neural network is built for classification by using back-propagation learning algorithm. The goal of static hand gesture recognition is to classify the given hand gesture data represented by some features into some predefined finite number of gesture classes. The proposed system presents a recognition algorithm to recognize a set of six specific static hand gestures, namely: Open, Close, Cut, Paste, Maximize, and Minimize. The hand gesture image is passed through three stages, preprocessing, feature extraction, and classification. In preprocessing stage some operations are applied to extract the hand gesture from its background and prepare the hand gesture image for the feature extraction stage. In the first method, the hand contour is used as a feature which treats scaling and translation of problems (in some cases). The complex moment algorithm is, however, used to describe the hand gesture and treat the rotation problem in addition to the scaling and translation. The algorithm used in a multi-layer neural network classifier which uses back-propagation learning algorithm. The results show that the first method has a performance of 70.83% recognition, while the second method, proposed in this article, has a better performance of 86.38% recognition rate.     

Articulated body motion tracking using illumination invariant optical flow

We propose a model-based tracking method for articulated objects in monocular video sequences under varying illumination conditions. The tracking method uses estimates of optical flows constructed by projecting model textures into the camera images and comparing the projected textures with the recorded information. An articulated body is modelled in terms of 3D primitives, each possessing a specified texture on its surface. An important step in model-based tracking of 3D objects is the estimation of the pose of the object during the tracking process. The optimal pose is estimated by minimizing errors between the computed optical flow and the projected 2D velocities of the model textures. This estimation uses a least-squares method with kinematic constraints for the articulated object and a perspective camera model. We test our framework with an articulated robot and show results.     

Real-time 3D pointing gesture recognition for mobile robots with cascade HMM and particle filter

In this paper, we present a real-time 3D pointing gesture recognition algorithm for mobile robots, based on a cascade hidden Markov model (HMM) and a particle filter. Among the various human gestures, the pointing gesture is very useful to human-robot interaction (HRI). In fact, it is highly intuitive, does not involve a-priori assumptions, and has no substitute in other modes of interaction. A major issue in pointing gesture recognition is the difficultly of accurate estimation of the pointing direction, caused by the difficulty of hand tracking and the unreliability of the direction estimation.The proposed method involves the use of a stereo camera and 3D particle filters for reliable hand tracking, and a cascade of two HMMs for a robust estimate of the pointing direction. When a subject enters the field of view of the camera, his or her face and two hands are located and tracked using particle filters. The first stage HMM takes the hand position estimate and maps it to a more accurate position by modeling the kinematic characteristics of finger pointing. The resulting 3D coordinates are used as input into the second stage HMM that discriminates pointing gestures from other types. Finally, the pointing direction is estimated for the pointing state.The proposed method can deal with both large and small pointing gestures. The experimental results show gesture recognition and target selection rates of better than 89% and 99% respectively, during human-robot interaction.     

Real-time one-shot learning gesture recognition based on lightweight 3D Inception-ResNet with separable convolutions

Pattern Analysis and Applications - Gesture recognition is a popular research field in computer vision and the application of deep neural networks greatly improves its performance. However, the...     

Feature fusion for 3D hand gesture recognition by learning a shared hidden space

Hand gesture recognition has been intensively applied in various human-computer interaction (HCI) systems. Different hand gesture recognition methods were developed based on particular features, e.g., gesture trajectories and acceleration signals. However, it has been noticed that the limitation of either features can lead to flaws of a HCI system. In this paper, to overcome the limitations but combine the merits of both features, we propose a novel feature fusion approach for 3D hand gesture recognition. In our approach, gesture trajectories are represented by the intersection numbers with randomly generated line segments on their 2D principal planes, acceleration signals are represented by the coefficients of discrete cosine transformation (DCT). Then, a hidden space shared by the two features is learned by using penalized maximum likelihood estimation (MLE). An iterative algorithm, composed of two steps per iteration, is derived to for this penalized MLE, in which the first step is to solve a standard least square problem and the second step is to solve a Sylvester equation. We tested our hand gesture recognition approach on different hand gesture sets. Results confirm the effectiveness of the feature fusion method.     

3D arm movement recognition using syntactic pattern recognition

Gesture-based applications widely range from direct manipulation interfaces to speaking aids for the deaf. The crucial point in recognizing gestures is that it requires great computational power to deal with spatio-temporal patterns. In this paper, a syntactic approach is proposed to provide a simple recognition algorithm. In order to verify the proposed method, we apply it to recognize 3D arm movements involved in the Taiwanese Sign Language. We extract prime patterns from the input patterns. The classification is then accomplished by deciding which one of possible arm movements can produce the sequence of primary patterns. Experiments were conducted to confirm the effectiveness of the method.     

The impact of motion dimensionality and bit cardinality on the design of 3D gesture recognizers

The interactive demands of the upcoming ubiquitous computing era have set off researchers and practitioners toward prototyping new gesture-sensing devices and gadgets. At the same time, the practical needs of developing for such miniaturized prototypes with sometimes very low processing power and memory resources make practitioners in high demand of fast gesture recognizers employing little memory. However, the available work on motion gesture classifiers has mainly focused on delivering high recognition performance with less discussion on execution speed or required memory. This work investigates the performance of today's commonly used 3D motion gesture recognizers under the effect of different gesture dimensionality and bit cardinality representations. Specifically, we show that few sampling points and low bit depths are sufficient for most motion gesture metrics to attain their peak recognition performance in the context of the popular Nearest-Neighbor classification approach. As a practical consequence, 16x faster recognizers working with 32x less memory while delivering the same high levels of recognition performance are being reported. We present recognition results for a large gesture corpus consisting in nearly 20,000 gesture samples. In addition, a toolkit is provided to assist practitioners in optimizing their gesture recognizers in order to increase classification speed and reduce memory consumption for their designs. At a deeper level, our findings suggest that the precision of the human motor control system articulating 3D gestures is needlessly surpassed by the precision of today's motion sensing technology that unfortunately bares a direct connection with the sensors' cost. We hope this work will encourage practitioners to consider improving the performance of their prototypes by careful analysis of motion gesture representation rather than by throwing more processing power and more memory into the design.