3-D human pose recovery using nonrigid point set registration and body part tracking of depth data

In this paper, we present a novel approach for recovering a 3-D pose from a single human body depth silhouette using nonrigid point set registration and body part tracking. In our method, a human body depth silhouette is presented as a set of 3-D points and matched to another set of 3-D points using point correspondences. To recognize and maintain body part labels, we initialize the first set of points to corresponding human body parts, resulting in a body part-labeled map. Then, we transform the points to a sequential set of points based on point correspondences determined by nonrigid point set registration. After point registration, we utilize the information from tracked body part labels and registered points to create a human skeleton model. A 3-D human pose gets recovered by mapping joint information from the skeleton model to a 3-D synthetic human model. Quantitative and qualitative evaluation results on synthetic and real data show that complex human poses can be recovered more reliably with lower errors compared to other conventional techniques for 3-D pose recovery.     

3-D Scene Data Recovery Using Omnidirectional Multibaseline Stereo

A traditional approach to extracting geometric information from a large scene is to compute multiple 3-D depth maps from stereo pairs or direct range finders, and then to merge the 3-D data. However, the resulting merged depth maps may be subject to merging errors if the relative poses between depth maps are not known exactly. In addition, the 3-D data may also have to be resampled before merging, which adds additional complexity and potential sources of errors.This paper provides a means of directly extracting 3-D data covering a very wide field of view, thus by-passing the need for numerous depth map merging. In our work, cylindrical images are first composited from sequences of images taken while the camera is rotated 360° about a vertical axis. By taking such image panoramas at different camera locations, we can recover 3-D data of the scene using a set of simple techniques: feature tracking, an 8-point structure from motion algorithm, and multibaseline stereo. We also investigate the effect of median filtering on the recovered 3-D point distributions, and show the results of our approach applied to both synthetic and real scenes.     

3-D recovery of human motion by mobile stereo cameras

A method is described which recovers the 3-D shape of deformable objects, particularly human motions, from mobile stereo images. In the proposed technique, camera calibration is not required when taking images. Existing optical 3-D modeling systems must employ calibrated cameras that are set at fixed positions. This inevitably puts constraints on the range of the movement of an object. In the proposed method, multiple mobile cameras take images of a deformable object moving freely, and its 3-D model is reconstructed from the video image streams obtained. The advantages of the proposed method include the fact that the cameras employed are calibration-free, and that the image-taking cameras can move freely. The theory is described, and the performance is shown by an experiment on 3-D human motion modeling in an outdoor environment. The accuracy of the 3-D model obtained is evaluated and a discussion is given. This work was presented in part at the 10th International Symposium on Artificial Life and Robotics, Oita, Japan, February 4–6, 2005     

3-D shape reconstruction in an active stereo vision system using genetic algorithms

The recovery of 3-D shape information (depth) using stereo vision analysis is one of the major areas in computer vision and has given rise to a great deal of literature in the recent past. The widely known stereo vision methods are the passive stereo vision approaches that use two cameras. Obtaining 3-D information involves the identification of the corresponding 2-D points between left and right images. Most existing methods tackle this matching task from singular points, i.e. finding points in both image planes with more or less the same neighborhood characteristics. One key problem we have to solve is that we are on the first instance unable to know a priori whether a point in the first image has a correspondence or not due to surface occlusion or simply because it has been projected out of the scope of the second camera. This makes the matching process very difficult and imposes a need of an a posteriori stage to remove false matching.In this paper we are concerned with the active stereo vision systems which offer an alternative to the passive stereo vision systems. In our system, a light projector that illuminates objects to be analyzed by a pyramid-shaped laser beam replaces one of the two cameras. The projections of laser rays on the objects are detected as spots in the image. In this particular case, only one image needs to be treated, and the stereo matching problem boils down to associating the laser rays and their corresponding real spots in the 2-D image. We have expressed this problem as a minimization of a global function that we propose to perform using Genetic Algorithms (GAs). We have implemented two different algorithms: in the first, GAs are performed after a deterministic search. In the second, data is partitioned into clusters and GAs are independently applied in each cluster. In our second contribution in this paper, we have described an efficient system calibration method. Experimental results are presented to illustrate the feasibility of our approach. The proposed method yields high accuracy 3-D reconstruction even for complex objects. We conclude that GAs can effectively be applied to this matching problem.     

3-D motion estimation using a sequence of noisy stereo images:models, estimation, and uniqueness results

A kinematic model-based approach for the estimation of 3-D motion parameters from a sequence of noisy stereo images is discussed. The approach is based on representing the constant acceleration translational motion and constant precession rotational motion in the form of a bilinear state-space model using standard rectilinear states for translation and quaternions for rotation. Closed-form solutions of the state transition equations are obtained to propagate the quaternions. The measurements are noisy perturbations of 3-D feature points represented in an inertial coordinate system. It is assumed that the 3-D feature points are extracted from the stereo images and matched over the frames. Owing to the nonlinearity in the state model, nonlinear filters are designed for the estimation of motion parameters. Simulation results are included. The Cramer-Rao performance bounds for motion parameter estimates are computed. A constructive proof for the uniqueness of motion parameters is given. It is shown that with uniform sampling in time, three noncollinear feature points in five consecutive binocular image pairs contain all the spatial and temporal information. Both nondegenerate and degenerate motions are analyzed. A deterministic algorithm to recover motion parameters from a stereo image sequence is summarized from the constructive proof     

Tracking 3-D motion from straight lines with trifocal tensors

We present a novel approach to track the position and orientation of a stereo camera using line features in the images. The method combines the strengths of trifocal tensors and Bayesian filtering. The trifocal tensor provides a geometric constraint to lock line features among every three frames. It eliminates the explicit reconstruction of the scene even if the 3-D scene structure is not known. Such a trifocal constraint thus makes the algorithm fast and robust. The twist motion model is applied to further improve its computation efficiency. Another major contribution is that our approach can obtain the 3-D camera motion using as little as 2 line correspondences instead of 13 in the traditional approaches. This makes the approach attractive for realistic applications. The performance of the proposed method has been evaluated using both synthetic and real data with encouraging results. Our algorithm is able to estimate 3-D camera motion in real scenarios accurately having little drifting from an image sequence longer than a 1,000 frames.     

Appearance-Based Structure from Motion Using Linear Classes of 3-D Models

In this paper, we address the problem of recovering 3-D models from sequences of partly calibrated images with unknown correspondence. To that end, we integrate tracking, structure from motion with geometric constraints (specifically in the form of linear class models) in a single framework. The key to making the proposed approach work is the use of appearance-based model matching and refinement which updates the estimated correspondences on each iteration of the algorithm. Another key feature is the matching of a 3-D model directly with the input images without the conventional 2-step approach of stereo data recovery and 3-D model fitting. Initialization of the linear class model to one of the input images (the reference image) is currently partly manual.This synthesis and refine approach, or appearance-based constrained structure from motion (AbCSfm), is especially useful in recovering shapes of objects whose general structureis known but which may have little discernable texture in significant parts of their surfaces. We applied the proposed approach to 3-D face modeling from multiple images to create new 3-D faces for DECface, a synthetic talking head developed at Cambridge Research Laboratory, Digital Equipment Corporation. The DECface model comprises a collection of 3-D triangular and rectangular facets, with nodes as vertices. In recovering the DECface model, we assume that the sequence of images is taken with a camera with unknown focal length and pose. The geometric constraints used are of the form of linear combination of prototypes of 3-D faces of real people. Results of this approach show its good convergence properties and its robustness against cluttered backgrounds.     

Wide baseline image registration with application to 3-D face modeling

Establishing correspondence between features in two images of the same scene taken from different viewing angles is a challenging problem in image processing and computer vision. However, its solution is an important step in many applications like wide baseline stereo, three-dimensional (3-D) model alignment, creation of panoramic views, etc. In this paper, we propose a technique for registration of two images of a face obtained from different viewing angles. We show that prior information about the general characteristics of a face obtained from video sequences of different faces can be used to design a robust correspondence algorithm. The method works by matching two-dimensional (2-D) shapes of the different features of the face (e.g., eyes, nose etc.). A doubly stochastic matrix, representing the probability of match between the features, is derived using the Sinkhorn normalization procedure. The final correspondence is obtained by minimizing the probability of error of a match between the entire constellation of features in the two sets, thus taking into account the global spatial configuration of the features. The method is applied for creating holistic 3-D models of a face from partial representations. Although this paper focuses primarily on faces, the algorithm can also be used for other objects with small modifications.     

基于半监督聚类的3维肢体分割算法

行为分析已经成为计算机视觉研究领域的热点,行为主体的肢体部件分割是行为分析中很重要的一部分同时也是一个难点问题,为了对3维肢体进行有效分割,提出了一种基于半监督聚类的肢体分割算法。该算法首先利用前一帧姿势估计反馈的时域信息来对3维主体进行初始的肢体部件分割;然后根据人体结构信息进一步确定行为主体上各个点与肢体部件之间的关系来得到半监督聚类的初始值;之后基于各个肢体部件的形状信息进行半监督聚类,迭代求解肢体部件分割的最优解;最后利用分割后的各个肢体部件进行行为主体的姿势参数估计。通过对IXMAS数据库中6种行为下的6个行为主体进行的肢体部件的分割实验结果证明,该新算法具有主体适应性、视角适应性以及行为适应性。     

On 3-D scene flow and structure recovery from multiview image sequences

Two novel systems computing dense three-dimensional (3-D) scene flow and structure from multiview image sequences are described in this paper. We do not assume rigidity of the scene motion, thus allowing for nonrigid motion in the scene. The first system, integrated model-based system (IMS), assumes that each small local image region is undergoing 3-D affine motion. Non-linear motion model fitting based on both optical flow constraints and stereo constraints is then carried out on each local region in order to simultaneously estimate 3-D motion correspondences and structure. The second system is based on extended gradient-based system (EGS), a natural extension of two-dimensional (2-D) optical flow computation. In this method, a new hierarchical rule-based stereo matching algorithm is first developed to estimate the initial disparity map. Different available constraints under a multiview camera setup are further investigated and utilized in the proposed motion estimation. We use image segmentation information to adopt and maintain the motion and depth discontinuities. Within the framework for EGS, we present two different formulations for 3-D scene flow and structure computation. One formulation assumes that initial disparity map is accurate, while the other does not. Experimental results on both synthetic and real imagery demonstrate the effectiveness of our 3-D motion and structure recovery schemes. Empirical comparison between IMS and EGS is also reported.     

A functional-based segmentation of human body scans in arbitrary postures.

This paper presents a general framework that aims to address the task of segmenting three-dimensional (3-D) scan data representing the human form into subsets which correspond to functional human body parts. Such a task is challenging due to the articulated and deformable nature of the human body. A salient feature of this framework is that it is able to cope with various body postures and is in addition robust to noise, holes, irregular sampling and rigid transformations. Although whole human body scanners are now capable of routinely capturing the shape of the whole body in machine readable format, they have not yet realized their potential to provide automatic extraction of key body measurements. Automated production of anthropometric databases is a prerequisite to satisfying the needs of certain industrial sectors (e.g., the clothing industry). This implies that in order to extract specific measurements of interest, whole body 3-D scan data must be segmented by machine into subsets corresponding to functional human body parts. However, previously reported attempts at automating the segmentation process suffer from various limitations, such as being restricted to a standard specific posture and being vulnerable to scan data artifacts. Our human body segmentation algorithm advances the state of the art to overcome the above limitations and we present experimental results obtained using both real and synthetic data that confirm the validity, effectiveness, and robustness of our approach.     

Three-dimensional image mosaicking using multiple projection planes for 3-D visualization of roadside standing buildings.

A novel image-mosaicking technique suitable for 3-D visualization of roadside buildings on websites or mobile systems is proposed. Our method was tested on a roadside building scene taken using a side-looking video camera employing a continuous set of vertical-textured planar faces. A vertical plane approximation of the scene geometry for each frame was calculated using sparsely distributed feature points that were assigned 3-D data through bundle adjustments. These vertical planes were concatenated to create an approximate model on which the images could be backprojected as textures and blended together. Additionally, our proposed method includes an expanded crossed-slits projection around far-range areas to reduce the "ghost effect," a phenomenon in which a particular object appears repeatedly in a created image mosaic. The final step was to produce seamless image mosaics using Dijkstra's algorithm to find the optimum seam line to blend overlapping images. We used our algorithm to create efficient image mosaics in 3-D space from a sequence of real images.     

基于二维关节型人体模型和EM算法的人体跟踪

提出一种跟踪单眼图像序列中的行人,并恢复其运动参数的新方法.在跟踪中采用了基于SPM(Sealed Prismat Model)扩展的二维纸板人模型取代三维人体模型,以获取更快的计算速度.作者使用EM算法在概率框架下进行运动估计,同时,算法也考虑了混合的运动模型和运动约束,以减小解的搜索空间.试验结果证明了该方法的有效性.     

3-D face recognition method based on optimum 3-D image measurement technology

Most face recognition systems employ 2-D color or gray-scale images. However, face recognition based on 2-D images is adversely affected by 3-D movement, variable lighting, and the use of cosmetics. 3-D image measurement technology has the potential to overcome these limitations of face recognition based on 2-D images since it can perform geometric analysis. We propose a method that is capable of recognizing a person from a 3-D facial image obtained using a 3-D shape measurement system by employing a technique that optimizes the intensity-modulation pattern projection. This face recognition method is based on the iterative closest point algorithm. It is robust to changes in reflectivity and color. Since the 3-D facial information can be registered, this method can estimate rotations and translations to compensate for different positions or directions. In order to prove the validity of the proposed technique, a verification experiment was conducted which used 105 sample 3-D images obtained from 15 subjects. It achieved a detection rate of 96% when heads were turned at an angle of 20° or less relative to the camera.     

Matching 3-D Models to 2-D Images

We consider the problem of analytically characterizing the set of all 2-D images that a group of 3-D features may produce, and demonstrate that this is a useful thing to do. Our results apply for simple point features and point features with associated orientation vectors when we model projection as a 3-D to 2-D affine transformation. We show how to represent the set of images that a group of 3-D points produces with two lines (1-D subspaces), one in each of two orthogonal, high-dimensional spaces, where a single image group corresponds to one point in each space. The images of groups of oriented point features can be represented by a 2-D hyperbolic surface in a single high-dimensional space. The problem of matching an image to models is essentially reduced to the problem of matching a point to simple geometric structures. Moreover, we show that these are the simplest and lowest dimensional representations possible for these cases.We demonstrate the value of this way of approaching matching by applying our results to a variety of vision problems. In particular, we use this result to build a space-efficient indexing system that performs 3-D to 2-D matching by table lookup. This system is analytically built and accessed, accounts for the effects of sensing error, and is tested on real images. We also derive new results concerning the existence of invariants and non-accidental properties in this domain. Finally, we show that oriented points present unexpected difficulties: indexing requires fundamentally more space with oriented than with simple points, we must use more images in a motion sequence to determine the affine structure of oriented points, and the linear combinations result does not hold for oriented points.     

Integration of a stereo vision system and a laser range finder for 3-D construction

In this article, we investigate the problem of integrating a binocular stereo vision system and a laser range finder to construct a 3-D map of the environment. The proposed scheme is realized by using the alignment parameters obtained in the 2-D map construction of the laser range finder for the 3-D data generated by the stereo vision system. The 2-D map alignment task is formulated as an optimization problem of minimizing the alignment errors between local maps and selected parts of the developing global map. The problem is then solved using the Simplex method. To increase the robustness of the searching process, multiple initial guesses are provided in the Simplex method. The performance of the proposed architecture is verified by experimental results from a mobile vehicle for obstacle avoidance.     

Markerless human body motion capture using Markov random field and dynamic graph cuts

Current vision-based human body motion capture methods always use passive markers that are attached to key locations on the human body. However, such systems may confront subjects with cumbersome markers, making it difficult to convert the marker data into kinematic motion. In this paper, we propose a new algorithm for markerless computer vision-based human body motion capture. We compute volume data (voxels) representation from the images using the method of SFS (shape from silhouettes), and consider the volume data as a MRF (Markov random field). Then we match a predefined human body model with pose parameters to the volume data, and the calculation of this matching is transformed into energy function minimization. We convert the problem of energy function construction into a 3D graph construction, and get the minimal energy by the max-flow theory. Finally, we recover the human pose by Powell algorithm.     

三维人体点云模型多约束肢体分割

针对人体点云模型的肢体分割这一动作识别和虚拟重建领域的重要问题,提出了一种基于分类骨架线、测地距离、特征点和姿态分析的多约束肢体分割算法,通过生成点云模型的分类骨架线,配合测地距离获得人体各部位粗分割点云集,利用测地路径方法实现关键特征点的定位,并利用曲线拟合方式进行定位优化,针对头颈、上肢、下肢和躯干之间关联部位的解剖学特征,构造多种约束条件,对各部位粗分割点云集进行了优化再分割。实验结果表明,所提算法对站姿条件下的不同动作、不同体型、不同精度人体点云模型均能取得与视觉理解相吻合的分割效果。通过该算法得到的肢体各部分点云数据可用于姿态分析等后续研究。     

基于体绘制技术的层状地层三维建模平台的设计和实现

随着地学研究的不断深入,层状地层三维建模已成为研究的重点。基于格式化存储的多层DEM数据文件,将直接体绘制的思想融入到基于图像空间体绘制的经典算法—光线投射算法之中,提出基于多层DEM数据的光线投射算法。基于IDL和.NET的混合环境设计并实现了层状地层的三维可视化的平台。在该平台上,使用改进后的光线投射算法实现了层状地层的三维建模,以及对三维层状地质体的基本操作和基本地质信息的提取。