Region-based pose tracking with occlusions using 3D models

Despite great progress achieved in 3-D pose tracking during the past years, occlusions and self-occlusions are still an open issue. This is particularly true in silhouette-based tracking where even visible parts cannot be tracked as long as they do not affect the object silhouette. Multiple cameras or motion priors can overcome this problem. However, multiple cameras or appropriate training data are not always readily available. We propose a framework in which the pose of 3-D models is found by minimising the 2-D projection error through minimisation of an energy function depending on the pose parameters. This framework makes it possible to handle occlusions and self-occlusions by tracking multiple objects and object parts simultaneously. Therefore, each part is described by its own image region each of which is modeled by one probability density function. This allows to deal with occlusions explicitly, which includes self-occlusions between different parts of the same object as well as occlusions between different objects. The results we present for simulations and real-world scenes demonstrate the improvements achieved in monocular and multi-camera settings. These improvements are substantiated by quantitative evaluations, e.g. based on the HumanEVA benchmark.     

Viewpoint invariant exemplar-based 3D human tracking

This paper proposes a clustered exemplar-based model for performing viewpoint invariant tracking of the 3D motion of a human subject from a single camera. Each exemplar is associated with multiple view visual information of a person and the corresponding 3D skeletal pose. The visual information takes the form of contours obtained from different viewpoints around the subject. The inclusion of multi-view information is important for two reasons: viewpoint invariance; and generalisation to novel motions. Visual tracking of human motion is performed using a particle filter coupled to the dynamics of human movement represented by the exemplar-based model. Dynamics are modelled by clustering 3D skeletal motions with similar movement and encoding the flow both within and between clusters. Results of single view tracking demonstrate that the exemplar-based models incorporating dynamics generalise to viewpoint invariant tracking of novel movements.     

3D head tracking under partial occlusion

A new algorithm for 3D head tracking under partial occlusion from 2D monocular image sequences is proposed. The extended superquadric (ESQ) is used to generate a geometric 3D face model in order to reduce the shape ambiguity during tracking. Optical flow is then regularized by this model to estimate the 3D rigid motion. To deal with occlusion, a new motion segmentation algorithm using motion residual error analysis is developed. The occluded areas are successfully detected and discarded as noise. Furthermore, accumulation error is heavily reduced by a new post-regularization process based on edge flow. This makes the algorithm more stable over long image sequences. The algorithm is applied to both synthetic occlusion sequence and real image sequences. Comparisons with the ground truth indicate that our method is effective and is not sensitive to occlusion during head tracking.     

Human attributes from 3D pose tracking

It is well known that biological motion conveys a wealth of socially meaningful information. From even a brief exposure, biological motion cues enable the recognition of familiar people, and the inference of attributes such as gender, age, mental state, actions and intentions. In this paper we show that from the output of a video-based 3D human tracking algorithm we can infer physical attributes (e.g., gender and weight) and aspects of mental state (e.g., happiness or sadness). In particular, with 3D articulated tracking we avoid the need for view-based models, specific camera viewpoints, and constrained domains. The task is useful for man–machine communication, and it provides a natural benchmark for evaluating the performance of 3D pose tracking methods (vs. conventional Euclidean joint error metrics). We show results on a large corpus of motion capture data and on the output of a simple 3D pose tracker applied to videos of people walking.     

A 2D/3D model-based object tracking framework

This paper presents a robust framework for tracking complex objects in video sequences. Multiple hypothesis tracking (MHT) algorithm reported in (IEEE Trans. Pattern Anal. Mach. Intell. 18(2) (1996)) is modified to accommodate a high level representations (2D edge map, 3D models) of objects for tracking. The framework exploits the advantages of MHT algorithm which is capable of resolving data association/uncertainty and integrates it with object matching techniques to provide a robust behavior while tracking complex objects. To track objects in 2D, a 4D feature is used to represent edge/line segments and are tracked using MHT. In many practical applications 3D models provide more information about the object's pose (i.e., rotation information in the transformation space) which cannot be recovered using 2D edge information. Hence, a 3D model-based object tracking algorithm is also presented. A probabilistic Hausdorff image matching algorithm is incorporated into the framework in order to determine the geometric transformation that best maps the model features onto their corresponding ones in the image plane. 3D model of the object is used to constrain the tracker to operate in a consistent manner. Experimental results on real and synthetic image sequences are presented to demonstrate the efficacy of the proposed framework.     

3D pose tracking using particle filter with back projection-based sampling

This paper explores pose tracking with a 3D object model using a particle filter. In tracking an object with a particle filter, a problem of high dimensional solution space often appears. In a huge space, a search process is tedious and falls foul of the exponential computational complexity. We propose the back projection-based sampling with a depth map from a stereo camera, which reduces the solution space significantly. In addition, we treat a method that constructs the 3D model used in this paper. Results are shown for tracking the pose of several objects.     

A 3D unsplit-advection volume tracking algorithm with planarity-preserving interface reconstruction

A new volume tracking method is introduced for tracking interfaces in three-dimensional (3D) geometries partitioned with orthogonal hexahedra. The method approximates interface geometries as piecewise planar, and advects volumes in a single unsplit step using fully multidimensional fluxes that have their definition based in backward-trajectory remapping. By using multidimensional unsplit advection, the expense of high-order interface reconstruction is incurred only once per timestep. Simple departures from strict backward-trajectory remapping remove any need for consideration of volume computations involving shapes consisting of non-planar ruled surfaces. Second-order accuracy of the method is demonstrated even for vigorous 3D deformations.     

3D hand tracking for human computer interaction

We propose a real-time model-based 3D hand tracker that combines image regions and the signal from an off-the-shelf 3-axis accelerometer placed on the user's hand. The visual regions allow the tracker to cope with occlusions, motion blur and background clutter, while the latter aids with the inherent silhouette-pose ambiguities. The accelerometer and tracker are synchronised by casting the calibration problem as one of principal component analysis. Based on the assumption that, often, the number of possible hand configurations is limited by the activity the hand is engaging in, we use a multiclass pose classifier to distinguish between a number of activity dependent articulated hand configurations. We demonstrate the benefits of our method, both qualitatively and quantitatively, on a variety of video sequences and hand configurations and show a proof-of-concept human computer interface based on our system.     

From rendering to tracking point-based 3D models

This paper adds to the abundant visual tracking literature with two main contributions. First, we illustrate the interest of using Graphic Processing Units (GPU) to support efficient implementations of computer vision algorithms, and secondly, we introduce the use of point-based 3D models as a shape prior for real-time 3D tracking with a monocular camera.     

On-line modeling for real-time 3D target tracking

Model-based object tracking can provide autonomous mobile robotic systems with real-time 6-dof pose information, for example, enabling them to rendezvous with targets from a particular desired direction. Most existing model-based trackers, however, require the geometric model of the target to be known a priori, which may pose a practical problem in real-world environments. This paper presents a novel 3D modeler capable of building an approximate model of a target object on-line. The proposed technique rapidly constructs a 3D tessellated enveloping mesh and uses projective texture mapping to further model the target object’s surface features. Separation of the target object from background clutter is achieved via customizable interest filters. The resulting real-time object-tracking system was tested extensively via simulations and experiments.     

Automatic initialization for 3D soccer player tracking

As a special application of computer vision, automatic sports video analysis has been studied by some researchers. This sports video analysis via computer vision is a moderately challenging problem: it is more difficult than analyzing a video of a few laboratory members acting as in a simple scenario and is easier than analyzing a video of crowded people at a subway station. So the success of an analysis heavily depends on how much one can exploit the prior information on the sport and setting. The most challenging and important part would be the tracking of players (and ball). With a multi-camera system, 3D tracking is feasible which is much more meaningful than 2D tracking for the analysis. As an initial step of 3D player tracking from multi-view soccer videos, this paper deals with automatic initialization of player positions. Initial 3D positions can be estimated by exploiting some conditions of a soccer match. To make it robust, prior knowledge on the features of players is learnt by support vector machines (SVM). Experimental results show that the proposed system is efficient for general soccer sequences.     

Non-rigid 3D shape tracking from multiview video

We present a fast and efficient non-rigid shape tracking method for modeling dynamic 3D objects from multiview video. Starting from an initial mesh representation, the shape of a dynamic object is tracked over time, both in geometry and topology, based on multiview silhouette and 3D scene flow information. The mesh representation of each frame is obtained by deforming the mesh representation of the previous frame towards the optimal surface defined by the time-varying multiview silhouette information with the aid of 3D scene flow vectors. The whole time-varying shape is then represented as a mesh sequence which can efficiently be encoded in terms of restructuring and topological operations, and small-scale vertex displacements along with the initial model. The proposed method has the ability to deal with dynamic objects that may undergo non-rigid transformations and topological changes. The time-varying mesh representations of such non-rigid shapes, which are not necessarily of fixed connectivity, can successfully be tracked thanks to restructuring and topological operations employed in our deformation scheme. We demonstrate the performance of the proposed method both on real and synthetic sequences.     

Visual tracking of a moving target using active contour based SSD algorithm

This paper presents a new image based visual tracking scheme for a mobile robot to trace a moving target using a single camera mounted on the mobile robot. To accurately estimate the position of the target in the next image, it decomposes the effect of the camera motion on the velocity vector of the target in the image frame. Based on the estimated velocity of the target and the image Jacobian, the control inputs of the mobile robot are determined in such a way that the target may appear inside the central area of the image frame. Since the shape of the target in the image frame varies due to rotation and translation of the target, a new shape adaptive Sum-of-Squared Difference (SSD) algorithm is proposed which uses the extended snake algorithm to extract the contour of the target and updates the template in every step of the matching process. The proposed scheme has been implemented using a Nomad Scout Robot II. The experimental results have shown that the proposed scheme follows the target within a negligible error range even when the target is temporarily lost due to various reasons.     

Human tracking using 3D surface colour distributions

A likelihood formulation for detailed human tracking in real-world scenes is presented. In this formulation, the appearance, modelled using feature distributions defined over regions on the surface of an articulated 3D model, is estimated and propagated as part of the state. The benefit of such a formulation over currently used techniques is that it provides a dense, highly discriminatory object-based cue that applies in real world scenes. Multi-dimensional histograms are used to represent the feature distributions and an on-line clustering algorithm, driven by prior knowledge of clothing structure, is derived that enhances appearance estimation and computational efficiency. An investigation of the likelihood model shows its profile to be smooth and broad while region grouping is shown to improve localisation and discrimination. These properties of the likelihood model ease pose estimation by allowing coarse, hierarchical sampling and local optimisation.     

Hyperpatches for 3D model acquisition and tracking

Automatic 3D model acquisition and 3D tracking of simple objects under motion using a single camera is often difficult due to the sparsity of information from which to establish the model. We developed an automatic scheme that first computes a simple Euclidean model of the object and then enriches this model using hyperpatches. These hyperpatches contain information on both the orientation and intensity pattern variation of roughly planar patches on an object. This information allows both the spatial and intensity distortions of the projected patch to be modeled accurately under 3D object motion. Considering human tracking as a specific application, we show that hyperpatches can not only be computed automatically during model acquisition from a monocular image sequence, but that they are also extremely appropriate for the task of visual tracking     

Automatic model-based 3D object recognition by combining feature matching with tracking

We propose a vision-based robust automatic 3D object recognition, which provides object identification and 3D pose information by combining feature matching with tracking. For object identification, we propose a robust visual feature and a probabilistic voting scheme. An initial object pose is estimated using correlations between the model image and the 3D CAD model, which are predefined, and the homography, byproduct of the identification. In tracking, a Lie group formalism is used for robust and fast motion computation. Experimental results show that object recognition by the proposed method improves the recognition range considerably. Sungho Kim received the B.S. degree in Electrical Engineering from Korea University, Korea in 2000 and the M.S. degree in Electrical Engineering and Computer Science from Korea Advanced Institute of Science and Technology, Korea in 2002. He is currently pursuing his Ph.D. at the latter institution, concentrating on 3D object recognition and tracking. In So Kweon received the Ph.D. degree in robotics from Carnegie Mellon University, Pittsburgh, PA, in 1990. Since 1992, he has been a Professor of Electrical Engineering at KAIST. His current research interests include human visual perception, object recognition, real-time tracking, vision-based mobile robot localization, volumetric 3D reconstruction, and camera calibration. He is a member of the IEEE, and Korea Robotics Society (KRS).     

面向模型验证的空间网格面的染色及显示技术

针对工程分析用的网格模型,以图论中的拓扑图为基础提出了一种空间网格面的染色算法.该算法对可平面图几何模型、非可平面图几何模型以及奇异几何模型均可染色.利用染色结果,给出了在AutoCAD平台上编程显示染色几何模型的方法.多项工程应用表明,该染色算法可靠、适应性强、染色效果好,可用于几何分析模型的验证.     

基于多传感器视听融合的三维目标跟踪

目前的跟踪技术主要是用完全基于声音或视觉的传感器,但音频定位具有精度差而覆盖面广的特点,视觉跟踪具有定位精度高而受摄录角度限制的特点,以至于在复杂环境下难以取得理想的跟踪效果。针对这一问题,提出了一种利用从立体视觉和立体音频得到的融合信息对三维物体进行目标跟踪的新方法,介绍了一个包含2个麦克风和立体视觉的简单跟踪系统,由这2个系统提供的定位估计使用一种改进的PSO算法(TRIBES)来融合、综合2种传感器各自的优点。实验表明:与传统的方法相比,这种新技术可以实现更快、更精确的跟踪性能。     

Comparison of stochastic filtering methods for 3D tracking

In the recent years, the 3D visual research has gained momentum with publications appearing for all aspects of 3D including visual tracking. This paper presents a review of the literature published for 3D visual tracking over the past five years. The work particularly focuses on stochastic filtering techniques such as particle filter and Kalman filter. These two filters are extensively used for tracking due to their ability to consider uncertainties in the estimation. The improvement in computational power of computers and increasing interest in robust tracking algorithms lead to increase in the use of stochastic filters in visual tracking in general and 3D visual tracking in particular. Stochastic filters are used for numerous applications in the literature such as robot navigation, computer games and behavior analysis. Kalman filter is a linear estimator which approximates system's dynamics with Gaussian model while particle filter approximates system's dynamics using weighted samples. In this paper, we investigate the implementation of Kalman and particle filters in the published work and we provide comparison between these techniques qualitatively as well as quantitatively. The quantitative analysis is in terms of computational time and accuracy. The quantitative analysis has been implemented using four parameters of the tracked object which are object position, velocity, size of bounding ellipse and orientation angle.     

Real time 3D face and facial feature tracking

Detecting and tracking human faces in video sequences is useful in a number of applications such as gesture recognition and human-machine interaction. In this paper, we show that online appearance models (holistic approaches) can be used for simultaneously tracking the head, the lips, the eyebrows, and the eyelids in monocular video sequences. Unlike previous approaches to eyelid tracking, we show that the online appearance models can be used for this purpose. Neither color information nor intensity edges are used by our proposed approach. More precisely, we show how the classical appearance-based trackers can be upgraded in order to deal with fast eyelid movements. The proposed eyelid tracking is made robust by avoiding eye feature extraction. Experiments on real videos show the usefulness of the proposed tracking schemes as well as their enhancement to our previous approach.