Automatic acquisition and initialization of articulated models

Tracking, classification and visual analysis of articulated motion is challenging because of the difficulties involved in separating noise and variabilities caused by appearance, size and viewpoint fluctuations from task-relevant variations. By incorporating powerful domain knowledge, model-based approaches are able to overcome these problem to a great extent and are actively explored by many researchers. However, model acquisition, initialization and adaptation are still relatively under-investigated problems, especially for the case of single-camera systems. In this paper, we address the problem of automatic acquisition and initialization of articulated models from monocular video without any prior knowledge of shape and kinematic structure. The framework is applied in a human-computer interaction context where articulated shape models have to be acquired from unknown users for subsequent limb tracking. Bayesian motion segmentation is used to extract and initialize articulated models from visual data. Image sequences are decomposed into rigid components that can undergo parametric motion. The relative motion of these components is used to obtain joint information. The resulting components are assembled into an articulated kinematic model which is then used for visual tracking, eliminating the need for manual initialization or adaptation. The efficacy of the method is demonstrated on synthetic as well as natural image sequences. The accuracy of the joint estimation stage is verified on ground truth data.Correspondence to: N. Krahnstoever     

Generative tracking of 3D human motion in latent space by sequential clonal selection algorithm

High dimensional pose state space is the main challenge in articulated human pose tracking which makes pose analysis computationally expensive or even infeasible. In this paper, we propose a novel generative approach in the framework of evolutionary computation, by which we try to widen the bottleneck with effective search strategy embedded in the extracted state subspace. Firstly, we use ISOMAP to learn the low-dimensional latent space of pose state in the aim of both reducing dimensionality and extracting the prior knowledge of human motion simultaneously. Then, we propose a manifold reconstruction method to establish smooth mappings between the latent space and original space, which enables us to perform pose analysis in the latent space. In the search strategy, we adopt a new evolutionary approach, clonal selection algorithm (CSA), for pose optimization. We design a CSA based method to estimate human pose from static image, which can be used for initialization of motion tracking. In order to make CSA suitable for motion tracking, we propose a sequential CSA (S-CSA) algorithm by incorporating the temporal continuity information into the traditional CSA. Actually, in a Bayesian inference view, the sequential CSA algorithm is in essence a multilayer importance sampling based particle filter. Our methods are demonstrated in different motion types and different image sequences. Experimental results show that our CSA based pose estimation method can achieve viewpoint invariant 3D pose reconstruction and the S-CSA based motion tracking method can achieve accurate and stable tracking of 3D human motion.     

Tracking People on a Torus

We present a framework for monocular 3D kinematic pose tracking and viewpoint estimation of periodic and quasi-periodic human motions from an uncalibrated camera. The approach we introduce here is based on learning both the visual observation manifold and the kinematic manifold of the motion using a joint representation. We show that the visual manifold of the observed shape of a human performing a periodic motion, observed from different viewpoints, is topologically equivalent to a torus manifold. The approach we introduce here is based on the supervised learning of both the visual and kinematic manifolds. Instead of learning an embedding of the manifold, we learn the geometric deformation between an ideal manifold (conceptual equivalent topological structure) and a twisted version of the manifold (the data). Experimental results show accurate estimation of the 3D body posture and the viewpoint from a single uncalibrated camera.     

Analyzing and capturing articulated hand motion in image sequences

Capturing the human hand motion from video involves the estimation of the rigid global hand pose as well as the nonrigid finger articulation. The complexity induced by the high degrees of freedom of the articulated hand challenges many visual tracking techniques. For example, the particle filtering technique is plagued by the demanding requirement of a huge number of particles and the phenomenon of particle degeneracy. This paper presents a novel approach to tracking the articulated hand in video by learning and integrating natural hand motion priors. To cope with the finger articulation, this paper proposes a powerful sequential Monte Carlo tracking algorithm based on importance sampling techniques, where the importance function is based on an initial manifold model of the articulation configuration space learned from motion-captured data. In addition, this paper presents a divide-and-conquer strategy that decouples the hand poses and finger articulations and integrates them in an iterative framework to reduce the complexity of the problem. Our experiments show that this approach is effective and efficient for tracking the articulated hand. This approach can be extended to track other articulated targets.     

Joint origin identification of articulated robots with marker-based multi-camera optical tracking systems

Marker-based multi-camera optical tracking systems are being used in the robotics field to track robots for validation, verification, and calibration of their kinematic and dynamic models. These tracking systems estimate the pose of tracking bodies attached to objects within a tracking volume. In this work, we explore the case of tracking the origins of joints of articulated robots when the tracking bodies are mounted on limbs or structures relative to the joints. This configuration leads to an unknown relative pose between the tracking body and the joint origin. The identification of this relative pose is essential for an accurate representation of the kinematic model. We propose an approach for the identification of the origin of joints relative to tracking bodies by using state-of-the-art center of rotation (CoR) and axis of rotation (AoR) estimation methods. The applicability and effectiveness of our approach is demonstrated in two successful case studies: (i) the verification of the upper body kinematics of DLR’s humanoid Rollin’ Justin and (ii) the identification of the kinematic parameters of an ST Robot arm relative to its environment for the embodiment of a situated conversational assistant.     

Recognizing multiple human activities and tracking full-body pose in unconstrained environments

Visual observations, such as camera images, are hard to obtain for long-term human motion analysis in unconstrained environments. In this paper, we present a method for human full-body pose tracking and activity recognition from measurements of few body-worn inertial orientation sensors. The sensors make our approach insensitive to illumination and occlusions and permit a person to move freely. Since the data provided by inertial sensors is sparse, noisy and often ambiguous, we use a generative prior model of feasible human poses and movements to constrain the tracking problem. Our model consists of several low-dimensional, activity-specific manifold embeddings that significantly restrict the search space for pose tracking. Using a particle filter, our method continuously explores multiple pose hypotheses in the embedding space. An efficient activity switching mechanism governs the distribution of particles across the activity-specific manifold embeddings. Selecting a pose hypothesis that best explains incoming sensor observations simultaneously allows us to classify the activity a person is performing and to estimate the full-body pose. We also derive an effective measure of predictive confidence that enables detecting anomalous movements. Experiments on a multi-person data set containing several activities show that our method can seamlessly detect activity switches and accurately reconstruct full-body poses from the data of only six wearable inertial sensors.     

基于干扰观测器的机械臂广义模型预测轨迹跟踪控制

为实现对多自由度机械臂关节运动精确轨迹跟踪,提出一种基于非线性干扰观测器的广义模型预测轨迹跟踪控制方法。针对机械臂轨迹跟踪运动学子系统,采用广义预测控制（Generalized Predictive Control,GPC）方法设计期望的虚拟关节角速度。对于机械臂轨迹跟踪动力学子系统,考虑机械臂的参数不确定性和未知外界扰动,利用GPC方法设计关节力矩控制输入,基于非线性干扰观测器方法实时估计和补偿系统模型中的不确定性。在李雅普诺夫稳定性理论框架下证明了机械臂关节角位置和角速度的跟踪误差最终收敛于零的小邻域。数值仿真验证了所提出控制方法的有效性和优越性。     

Multi-activity 3D human motion recognition and tracking in composite motion model with synthesized transition bridges

Recognizing and tracking multiple activities are all extremely challenging machine vision tasks due to diverse motion types included and high-dimensional (HD) state space. To overcome these difficulties, a novel generative model called composite motion model (CMM) is proposed. This model contains a set of independent, low-dimensional (LD), and activity-specific manifold models that effectively constrain the state search space for 3D human motion recognition and tracking. This separate modeling of activity-specific movements can not only allow each manifold model to be optimized in accordance with only its respective movement, but also improve the scalability of the models. For accurate tracking with our CMM, a particle filter (PF) method is thus employed and then the particles can be distributed in all manifold models at each time step. In addition, an efficient activity switching strategy is proposed to dominate the particle distribution on all LD manifolds. To diffuse the particles amongst manifold models and respond quickly to the sudden changes in the activity, a set of visually-reasonable and kinematically-realistic transition bridges are synthesized by using the good properties of LD latent space and HD observation space, which enables the inter-activity motions seem more natural and realistic. Finally, a pose hypothesis that can best interpret the visual observation is selected and then used to recognize the activity that is currently observed. Extensive experiments, via qualitative and quantitative analyses, verify the effectiveness and robustness of our proposed CMM in the tasks of multi-activity 3D human motion recognition and tracking.     

Predicting Articulated Human Motion from Spatial Processes

We present a probabilistic interpretation of inverse kinematics and extend it to sequential data. The resulting model is used to estimate articulated human motion in visual data. The approach allows us to express the prior temporal models in spatial limb coordinates, which is in contrast to most recent work where prior models are derived in terms of joint angles. This approach has several advantages. First of all, it allows us to construct motion models in low dimensional spaces, which makes motion estimation more robust. Secondly, as many types of motion are easily expressed in spatial coordinates, the approach allows us to construct high quality application specific motion models with little effort. Thirdly, the state space is a real vector space, which allows us to use off-the-shelf stochastic processes as motion models, which is rarely possible when working with joint angles. Fourthly, we avoid the problem of accumulated variance, where noise in one joint affects all joints further down the kinematic chains. All this combined allows us to more easily construct high quality motion models. In the evaluation, we show that an activity independent version of our model is superior to the corresponding state-of-the-art model. We also give examples of activity dependent models that would be hard to phrase directly in terms of joint angles.     

Combined Direct and Inverse Kinematic Control for Articulated Figure Motion Editing

A new approach for the animation of articulated figures is presented. We propose a system of articulated motion design which offers a full combination of both direct and inverse kinematic control of the joint parameters. Such an approach allows an animator to specify interactively goal-directed changes to existing sampled joint motions, resulting in a more general and expressive class of possible joint motions. The fundamental idea is to consider any desired-joint space motion as a reference model inserted into the secondary task of an inverse kinematic control scheme. This approach profits from the use of half-space Cartesian main tasks in conjunction with a parallel control of the articulated figure called the coach-trainee metaphor. In addition, a transition function is introduced so as to guarantee the continuity of the control. The resulting combined kinematic control scheme leads to a new methodology of joint-motion editing which is demonstrated through the improvement of a functional model of human walking.     

3D Articulated Models and Multiview Tracking with Physical Forces

We present a method for automatically estimating the motion of an articulated object filmed by two or more fixed cameras. We focus our work on the case where the quality of the images is poor, and where only an approximation of a geometric model of the tracked object is available. Our technique uses physical forces applied to each rigid part of a kinematic 3D model of the object we are tracking. These forces guide the minimization of the differences between the pose of the 3D model and the pose of the real object in the video images. We use a fast recursive algorithm to solve the dynamical equations of motion of any 3D articulated model. We explain the key parts of our algorithms: how relevant information is extracted from the images, how the forces are created, and how the dynamical equations of motion are solved. A study of what kind of information should be extracted in the images and of when our algorithms fail is also presented. Finally we present some results about the tracking of a person. We also show the application of our method to the tracking of a hand in sequences of images, showing that the kind of information to extract from the images depends on their quality and of the configuration of the cameras.     

3D Human Motion Tracking with a Coordinated Mixture of Factor Analyzers

A major challenge in applying Bayesian tracking methods for tracking 3D human body pose is the high dimensionality of the pose state space. It has been observed that the 3D human body pose parameters typically can be assumed to lie on a low-dimensional manifold embedded in the high-dimensional space. The goal of this work is to approximate the low-dimensional manifold so that a low-dimensional state vector can be obtained for efficient and effective Bayesian tracking. To achieve this goal, a globally coordinated mixture of factor analyzers is learned from motion capture data. Each factor analyzer in the mixture is a “locally linear dimensionality reducer” that approximates a part of the manifold. The global parametrization of the manifold is obtained by aligning these locally linear pieces in a global coordinate system. To enable automatic and optimal selection of the number of factor analyzers and the dimensionality of the manifold, a variational Bayesian formulation of the globally coordinated mixture of factor analyzers is proposed. The advantages of the proposed model are demonstrated in a multiple hypothesis tracker for tracking 3D human body pose. Quantitative comparisons on benchmark datasets show that the proposed method produces more accurate 3D pose estimates over time than those obtained from two previously proposed Bayesian tracking methods.     

基于高斯和SCKF 的姿态角辅助三维目标跟踪

首先, 根据目标运动与姿态角的关系, 分析目标在偏航角和俯仰角下的速度变化, 进而推导出姿态角辅助三维目标跟踪模型; 然后, 针对姿态角量测非高斯情况, 在分析均方根容积卡尔曼滤波的基础上, 提出新的高斯和均方根容积卡尔曼滤波算法, 以提高非线性非高斯的处理能力; 最后, 结合不同运动模式下姿态角分量的特点, 建立姿态角分量不同的跟踪模型, 通过模型切换实现对姿态角机动的跟踪. 仿真结果验证了所提出跟踪模型和滤波算法的正确性和有效性.     

Mixtures of Gaussian process models for human pose estimation

Discriminative human pose estimation is the problem of inferring the 3D articulated pose of a human directly from an image feature. This is a challenging problem due to the highly non-linear and multi-modal mapping from the image feature space to the pose space. To address this problem, we propose a model employing a mixture of Gaussian processes where each Gaussian process models a local region of the pose space. By employing the models in this way we are able to overcome the limitations of Gaussian processes applied to human pose estimation — their O(N³) time complexity and their uni-modal predictive distribution. Our model is able to give a multi-modal predictive distribution where each mode is represented by a different Gaussian process prediction. A logistic regression model is used to give a prior over each expert prediction in a similar fashion to previous mixture of expert models. We show that this technique outperforms existing state of the art regression techniques on human pose estimation data sets for ballet dancing, sign language and the HumanEva data set.     

Markerless Multiview Motion Capture with 3D Shape Model Adaptation

In this paper, we address simultaneous markerless motion and shape capture from 3D input meshes of partial views onto a moving subject. We exploit a computer graphics model based on kinematic skinning as template tracking model. This template model consists of vertices, joints and skinning weights learned a priori from registered full‐body scans, representing true human shape and kinematics‐based shape deformations. Two data‐driven priors are used together with a set of constraints and cues for setting up sufficient correspondences. A Gaussian mixture model‐based pose prior of successive joint configurations is learned to soft‐constrain the attainable pose space to plausible human poses. To make the shape adaptation robust to outliers and non‐visible surface regions and to guide the shape adaptation towards realistically appearing human shapes, we use a mesh‐Laplacian‐based shape prior. Both priors are learned/extracted from the training set of the template model learning phase. The output is a model adapted to the captured subject with respect to shape and kinematic skeleton as well as the animation parameters to resemble the observed movements. With example applications, we demonstrate the benefit of such footage. Experimental evaluations on publicly available datasets show the achieved natural appearance and accuracy.     

Human attributes from 3D pose tracking

It is well known that biological motion conveys a wealth of socially meaningful information. From even a brief exposure, biological motion cues enable the recognition of familiar people, and the inference of attributes such as gender, age, mental state, actions and intentions. In this paper we show that from the output of a video-based 3D human tracking algorithm we can infer physical attributes (e.g., gender and weight) and aspects of mental state (e.g., happiness or sadness). In particular, with 3D articulated tracking we avoid the need for view-based models, specific camera viewpoints, and constrained domains. The task is useful for man–machine communication, and it provides a natural benchmark for evaluating the performance of 3D pose tracking methods (vs. conventional Euclidean joint error metrics). We show results on a large corpus of motion capture data and on the output of a simple 3D pose tracker applied to videos of people walking.     

Optimal Metric Projections for Deformable and Articulated Structure-from-Motion

This paper describes novel algorithms for recovering the 3D shape and motion of deformable and articulated objects purely from uncalibrated 2D image measurements using a factorisation approach. Most approaches to deformable and articulated structure from motion require to upgrade an initial affine solution to Euclidean space by imposing metric constraints on the motion matrix. While in the case of rigid structure the metric upgrade step is simple since the constraints can be formulated as linear, deformability in the shape introduces non-linearities. In this paper we propose an alternating bilinear approach to solve for non-rigid 3D shape and motion, associated with a globally optimal projection step of the motion matrices onto the manifold of metric constraints. Our novel optimal projection step combines into a single optimisation the computation of the orthographic projection matrix and the configuration weights that give the closest motion matrix that satisfies the correct block structure with the additional constraint that the projection matrix is guaranteed to have orthonormal rows (i.e. its transpose lies on the Stiefel manifold). This constraint turns out to be non-convex. The key contribution of this work is to introduce an efficient convex relaxation for the non-convex projection step. Efficient in the sense that, for both the cases of deformable and articulated motion, the proposed relaxations turned out to be exact (i.e. tight) in all our numerical experiments. The convex relaxations are semi-definite (SDP) or second-order cone (SOCP) programs which can be readily tackled by popular solvers. An important advantage of these new algorithms is their ability to handle missing data which becomes crucial when dealing with real video sequences with self-occlusions. We show successful results of our algorithms on synthetic and real sequences of both deformable and articulated data. We also show comparative results with state of the art algorithms which reveal that our new methods outperform existing ones.     

Articulated body motion tracking using illumination invariant optical flow

We propose a model-based tracking method for articulated objects in monocular video sequences under varying illumination conditions. The tracking method uses estimates of optical flows constructed by projecting model textures into the camera images and comparing the projected textures with the recorded information. An articulated body is modelled in terms of 3D primitives, each possessing a specified texture on its surface. An important step in model-based tracking of 3D objects is the estimation of the pose of the object during the tracking process. The optimal pose is estimated by minimizing errors between the computed optical flow and the projected 2D velocities of the model textures. This estimation uses a least-squares method with kinematic constraints for the articulated object and a perspective camera model. We test our framework with an articulated robot and show results.     

基于二维关节型人体模型和EM算法的人体跟踪

提出一种跟踪单眼图像序列中的行人,并恢复其运动参数的新方法.在跟踪中采用了基于SPM(Sealed Prismat Model)扩展的二维纸板人模型取代三维人体模型,以获取更快的计算速度.作者使用EM算法在概率框架下进行运动估计,同时,算法也考虑了混合的运动模型和运动约束,以减小解的搜索空间.试验结果证明了该方法的有效性.     

HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion

While research on articulated human motion and pose estimation has progressed rapidly in the last few years, there has been no systematic quantitative evaluation of competing methods to establish the current state of the art. We present data obtained using a hardware system that is able to capture synchronized video and ground-truth 3D motion. The resulting HumanEva datasets contain multiple subjects performing a set of predefined actions with a number of repetitions. On the order of 40,000 frames of synchronized motion capture and multi-view video (resulting in over one quarter million image frames in total) were collected at 60 Hz with an additional 37,000 time instants of pure motion capture data. A standard set of error measures is defined for evaluating both 2D and 3D pose estimation and tracking algorithms. We also describe a baseline algorithm for 3D articulated tracking that uses a relatively standard Bayesian framework with optimization in the form of Sequential Importance Resampling and Annealed Particle Filtering. In the context of this baseline algorithm we explore a variety of likelihood functions, prior models of human motion and the effects of algorithm parameters. Our experiments suggest that image observation models and motion priors play important roles in performance, and that in a multi-view laboratory environment, where initialization is available, Bayesian filtering tends to perform well. The datasets and the software are made available to the research community. This infrastructure will support the development of new articulated motion and pose estimation algorithms, will provide a baseline for the evaluation and comparison of new methods, and will help establish the current state of the art in human pose estimation and tracking.