联合LBS和Snake的3D人体外形和运动跟踪方法

为了解决基于多目视频轮廓信息的3D人体外形和运动跟踪问题,提出一种联合线性混合蒙皮和Snake变形模型的算法框架.首先建立人物对象的蒙皮模型,以每一帧多目同步视频的轮廓作为输入,采用一种基于剪影轮廓的可视外壳重建算法,使得作为3D特征的可视外壳保持了局部细节且更加光滑;并使用关节型迭代最近点算法进行匹配以捕获出每一帧骨架子空间下的人物3D外形及运动;再一次使用当前帧的多目轮廓信息,让Snake内外力共同作用于人物网格模型上的顶点,使之自由地趋近于目标对象.使用带ground-truth的合成数据进行对比实验的结果表明,该方法因同时使用3D误差约束和2D误差约束,提高了跟踪精度.     

Shape from incomplete silhouettes based on the reprojection error

Traditional shape from silhouette methods compute the 3D shape as the intersection of the back-projected silhouettes in the 3D space, the so called visual hull. However, silhouettes that have been obtained with background subtraction techniques often present miss-detection errors (produced by false negatives or occlusions) which produce incomplete 3D shapes. Our approach deals with miss-detections, false alarms, and noise in the silhouettes. We recover the voxel occupancy which describes the 3D shape by minimizing an energy based on an approximation of the error between the shape 2D projections and the silhouettes. Two variants of the projection – and as a result the energy – as a function of the voxel occupancy are proposed. One of these variants outperforms the other. The energy also includes a sparsity measure, a regularization term, and takes into account the visibility of the voxels in each view in order to handle self-occlusions.     

三维模型轮廓线抽取算法

虽然三维模型的轮廓线在图形交流中起着重要的作用,但由于轮廓线是视点依赖的,当物体在空间运动时,用现有算法从复杂的三维模型中抽取轮廓线需耗费大量时间,为了提高抽取轮廓线的速度,提出了两种新的抽取三维模型轮廓线算法,这两种算法是先利用轮廓线的局部极值特性来获得部分轮廓边,然后利用轮廓线的连通性,通过简单的比较运算,即可获得三维模型的外部轮廓线,实验结果表明,该两种算法都可快速获得三维模型的外围轮廓线,最后还将本算法与相关算法进行了详细的比较分析。     

Parametric body shape model of standing children aged 3–11 years

A statistical body shape model (SBSM) for children was developed for generating a child body shape with desired anthropometric parameters. A standardised template mesh was fit to whole-body laser scan data from 137 children aged 3–11 years. The mesh coordinates along with a set of surface landmarks and 27 manually measured anthropometric variables were analysed using principal component (PC) analysis. PC scores were associated with anthropometric predictors such as stature, body mass index (BMI) and ratio of erect sitting height to stature (SHS) using a regression model. When the original scan data were compared with the predictions of the SBSM using each subject's stature, BMI and SHS, the mean absolute error was 10.4 ± 5.8 mm, and 95th percentile error was 24.0 ± 18.5 mm. The model, publicly available online, will have utility for a wide range of applications.

Practitioner Summary: A statistical body shape model for children helps to account for inter-individual variability in body shapes as well as anthropometric dimensions. This parametric modelling approach is useful for reliable prediction of the body shape of a specific child with a few given predictors such as stature, body mass index and age.     

Silhouette smoothing for real-time rendering of mesh surfaces

Coarse piecewise linear approximation of surfaces causes undesirable polygonal appearance of silhouettes. We present an efficient method for smoothing the silhouettes of coarse triangle meshes using efficient 3D curve reconstruction and simple local re-meshing. It does not assume the availability of a fine mesh and generates only moderate amount of additional data at run time. Furthermore, polygonal feature edges are also smoothed in a unified framework. Our method is based on a novel interpolation scheme over silhouette triangles and this ensures that smooth silhouettes are faithfully reconstructed and always change continuously with respect to continuous movement of the view point or objects. We speed up computation with GPU assistance to achieve real-time rendering of coarse meshes with the smoothed silhouettes. Experiments show that this method outperforms previous methods for silhouette smoothing.     

Detail control in line drawings of 3D meshes

We address the problem of rendering a 3D mesh in the style of a line drawing, in which little or no shading is used and instead shape cues are provided by silhouettes and suggestive contours. Our specific goal is to depict shape features at a chosen scale. For example, when mesh triangles project into the image plane at subpixel sizes, both suggestive contours and silhouettes may form dense networks that convey shape poorly. The solution we propose is to convert the input mesh to a multiresolution representation (specifically, a progressive mesh), then view-dependently refine or coarsen the mesh to control the size of its triangles in image space. We thereby control the scale of shape features that are depicted via silhouettes and suggestive contours. We propose a novel refinement criterion that achieves this goal and address the problem of maintaining temporal coherence of silhouette and suggestive contours when extracting them from a changing mesh.     

Child body shape measurement using depth cameras and a statistical body shape model

We present a new method for rapidly measuring child body shapes from noisy, incomplete data captured from low-cost depth cameras. This method fits the data using a statistical body shape model (SBSM) to find a complete avatar in the realistic body shape space. The method also predicts a set of standard anthropometric data for a specific subject without measuring dimensions directly from the fitted model. Since the SBSM was developed using principal component (PC) analysis, we formulate an optimisation problem to fit the model in which the degrees of freedom are defined in PC-score space. The mean unsigned distance between the fitted-model based on depth-camera data and the high-resolution laser scan data was 9.4 mm with a standard deviation (SD) of 5.1 mm. For the torso, the mean distance was 2.9 mm (SD 1.4 mm). The correlations between standard anthropometric dimensions predicted by the SBSM and manually measured dimensions exceeded 0.9.     

基于步态的身份识别

提出了一种简单有效的自动步态识别算法，对于每个序列而言，一种改进的背景减除方法用于检测行人的运动轮廓，然后，这些时变的2D轮廓形状被转换为对应的1D距离信号，同时通过特征空间变换来提取低维步态特征，基于时空相关或归一化欧氏距离度量，标准的模式分类技术用于最终的识别，实验结果表明该算法不仅获得了令人鼓舞的识别性能，而且拥有相对较低的计算代价。     

Shape-From-Silhouette Across Time Part I: Theory and Algorithms

Shape-From-Silhouette (SFS) is a shape reconstruction method which constructs a 3D shape estimate of an object using silhouette images of the object. The output of a SFS algorithm is known as the Visual Hull (VH). Traditionally SFS is either performed on static objects, or separately at each time instant in the case of videos of moving objects. In this paper we develop a theory of performing SFS across time: estimating the shape of a dynamic object (with unknown motion) by combining all of the silhouette images of the object over time. We first introduce a one dimensional element called a Bounding Edge to represent the Visual Hull. We then show that aligning two Visual Hulls using just their silhouettes is in general ambiguous and derive the geometric constraints (in terms of Bounding Edges) that govern the alignment. To break the alignment ambiguity, we combine stereo information with silhouette information and derive a Temporal SFS algorithm which consists of two steps: (1) estimate the motion of the objects over time (Visual Hull Alignment) and (2) combine the silhouette information using the estimated motion (Visual Hull Refinement). The algorithm is first developed for rigid objects and then extended to articulated objects. In the Part II of this paper we apply our temporal SFS algorithm to two human-related applications: (1) the acquisition of detailed human kinematic models and (2) marker-less motion tracking.     

Silhouette coherence for camera calibration under circular motion

We present a new approach to camera calibration as a part of a complete and practical system to recover digital copies of sculpture from uncalibrated image sequences taken under turntable motion. In this paper, we introduce the concept of the silhouette coherence of a set of silhouettes generated by a 3D object. We show how the maximization of the silhouette coherence can be exploited to recover the camera poses and focal length. Silhouette coherence can be considered as a generalization of the well-known epipolar tangency constraint for calculating motion from silhouettes or outlines alone. Further, silhouette coherence exploits all the geometric information encoded in the silhouette (not just at epipolar tangency points) and can be used in many practical situations where point correspondences or outer epipolar tangents are unavailable. We present an algorithm for exploiting silhouette coherence to efficiently and reliably estimate camera motion. We use this algorithm to reconstruct very high quality 3D models from uncalibrated circular motion sequences, even when epipolar tangency points are not available or the silhouettes are truncated. The algorithm has been integrated into a practical system and has been tested on more than 50 uncalibrated sequences to produce high quality photo-realistic models. Three illustrative examples are included in this paper. The algorithm is also evaluated quantitatively by comparing it to a state-of-the-art system that exploits only epipolar tangents     

Experimenting with nonintrusive motion capture in a virtual environment

A growing number of promising applications requires recognizing human posture and motion. Conventional techniques require us to attach foreign objects to the body, which in some applications is disturbing or even impossible. New, nonintrusive motion capture approaches are called for. The well-known shape-from-silhouette technique for understanding 3D shapes could also be effective for human bodies. We present a novel technique for model-based motion capture that uses silhouettes extracted from multiple views. A 3D reconstruction of the performer can be computed from a silhouette with a technique known as volume intersection. We can recover the posture by fitting a model of the human body to the reconstructed volume. The purpose of this work is to test the effectiveness of this approach in a virtual environment by investigating the precision of the posture and motion obtained with various numbers and arrangements of stationary cameras. An average 1% position error has been obtained with five cameras.     

Animated solid model of the lung constructed from unsynchronized MR sequential images

This work discusses a 4D lung reconstruction method from unsynchronized MR sequential images. The lung, differently from the heart, does not have its own muscles, turning impossible to see its real movements. The visualization of the lung in motion is an actual topic of research in medicine. CT (Computerized Tomography) can obtain spatio-temporal images of the heart by synchronizing with electrocardiographic waves. The FOV of the heart is small when compared to the lung’s FOV. The lung’s movement is not periodic and is susceptible to variations in the degree of respiration. Compared to CT, MR (Magnetic Resonance) imaging involves longer acquisition times and it is not possible to obtain instantaneous 3D images of the lung. For each slice, only one temporal sequence of 2D images can be obtained. However, methods using MR are preferable because they do not involve radiation. In this paper, based on unsynchronized MR images of the lung an animated B-Rep solid model of the lung is created. The 3D animation represents the lung’s motion associated to one selected sequence of MR images. The proposed method can be divided in two parts. First, the lung’s silhouettes moving in time are extracted by detecting the presence of a respiratory pattern on 2D spatio-temporal MR images. This approach enables us to determine the lung’s silhouette for every frame, even on frames with obscure edges. The sequence of extracted lung’s silhouettes are unsynchronized sagittal and coronal silhouettes. Using our algorithm it is possible to reconstruct a 3D lung starting from a silhouette of any type (coronal or sagittal) selected from any instant in time. A wire-frame model of the lung is created by composing coronal and sagittal planar silhouettes representing cross-sections. The silhouette composition is severely underconstrained. Many wire-frame models can be created from the observed sequences of silhouettes in time. Finally, a B-Rep solid model is created using a meshing algorithm. Using the B-Rep solid model the volume in time for the right and left lungs were calculated. It was possible to recognize several characteristics of the 3D real right and left lungs in the shaded model.     

Detection and classification of vehicles from omnidirectional videos using multiple silhouettes

To detect and classify vehicles in omnidirectional videos, we propose an approach based on the shape (silhouette) of the moving object obtained by background subtraction. Different from other shape-based classification techniques, we exploit the information available in multiple frames of the video. We investigated two different approaches for this purpose. One is combining silhouettes extracted from a sequence of frames to create an average silhouette, the other is making individual decisions for all frames and use consensus of these decisions. Using multiple frames eliminates most of the wrong decisions which are caused by a poorly extracted silhouette from a single video frame. The vehicle types we classify are motorcycle, car (sedan) and van (minibus). The features extracted from the silhouettes are convexity, elongation, rectangularity and Hu moments. We applied two separate methods of classification. First one is a flowchart-based method that we developed and the second is K-nearest neighbour classification. 60% of the samples in the dataset are used for training. To ensure randomization in the experiments, threefold cross-validation is applied. The results indicate that using multiple silhouettes increases the classification performance.     

Motion capture and human pose reconstruction from a single-view video sequence

We propose a framework to reconstruct the 3D pose of a human for animation from a sequence of single-view video frames. The framework for pose construction starts with background estimation and the performer?s silhouette is extracted using image subtraction for each frame. Then the body silhouettes are automatically labeled using a model-based approach. Finally, the 3D pose is constructed from the labeled human silhouette by assuming orthographic projection. The proposed approach does not require camera calibration. It assumes that the input video has a static background, it has no significant perspective effects, and the performer is in an upright position. The proposed approach requires minimal user interaction.     

Shape from silhouettes based on a centripetal pentahedron model

In this paper we present a novel volumetric shape from silhouette (SfS) algorithm based on a centripetal pentahedron model (pent-model). The pent-model is an object-centered volumetric model composed of a set of pentahedrons cut from the centripetal triangular pyramids, which together partition the 3D space. The SfS algorithm first computes the pyramids by constructing a geodesic sphere. These pyramids are then projected onto the image planes of all cameras. The intersections between the projected pyramids and the silhouettes, which are a set of hexagons, are computed. This process can be performed very efficiently with pre-computed polar silhouette graphs (PSGs) and reduced PSGs. The hexagons are then back-projected into the 3D space, where the intersections are calculated and the pent-model is derived. After that, a mesh surface model can be extracted by marching pentahedrons. Our algorithm has the combined advantages of robustness, speediness and preciseness. Experimental results based on both synthetic images and real photos are presented.     

Fast joint estimation of silhouettes and dense 3D geometry from multiple images

We propose a probabilistic formulation of joint silhouette extraction and 3D reconstruction given a series of calibrated 2D images. Instead of segmenting each image separately in order to construct a 3D surface consistent with the estimated silhouettes, we compute the most probable 3D shape that gives rise to the observed color information. The probabilistic framework, based on Bayesian inference, enables robust 3D reconstruction by optimally taking into account the contribution of all views. We solve the arising maximum a posteriori shape inference in a globally optimal manner by convex relaxation techniques in a spatially continuous representation. For an interactively provided user input in the form of scribbles specifying foreground and background regions, we build corresponding color distributions as multivariate Gaussians and find a volume occupancy that best fits to this data in a variational sense. Compared to classical methods for silhouette-based multiview reconstruction, the proposed approach does not depend on initialization and enjoys significant resilience to violations of the model assumptions due to background clutter, specular reflections, and camera sensor perturbations. In experiments on several real-world data sets, we show that exploiting a silhouette coherency criterion in a multiview setting allows for dramatic improvements of silhouette quality over independent 2D segmentations without any significant increase of computational efforts. This results in more accurate visual hull estimation, needed by a multitude of image-based modeling approaches. We made use of recent advances in parallel computing with a GPU implementation of the proposed method generating reconstructions on volume grids of more than 20 million voxels in up to 4.41 seconds.     

Real-time 3D human pose recovery from a single depth image using principal direction analysis

In this paper, we present a novel approach to recover a 3D human pose in real-time from a single depth image using principal direction analysis (PDA). Human body parts are first recognized from a human depth silhouette via trained random forests (RFs). PDA is applied to each recognized body part, which is presented as a set of points in 3D, to estimate its principal direction. Finally, a 3D human pose is recovered by mapping the principal direction to each body part of a 3D synthetic human model. We perform both quantitative and qualitative evaluations of our proposed 3D human pose recovering methodology. We show that our proposed approach has a low average reconstruction error of 7.07 degrees for four key joint angles and performs more reliably on a sequence of unconstrained poses than conventional methods. In addition, our methodology runs at a speed of 20 FPS on a standard PC, indicating that our system is suitable for real-time applications. Our 3D pose recovery methodology is applicable to applications ranging from human computer interactions to human activity recognition.     

Constructing 3D human model from front and side images

Constructing 3D human model from 2D images provides a cost-effective approach to visualize digital human in virtual environment. This paper presents a systematic approach for constructing 3D human model using the front and side images of a person. The silhouettes of human body are first detected and the feature points on the silhouettes are subsequently identified. The feature points are further used to obtain the body dimensions that are necessary for identifying a template 3D human model. The shape of the template human model can be modified by the free-form deformation method. Moreover, the proposed approach has been applied for constructing the 3D human models of 30 subjects. The comparisons between the constructed 3D models and the 3D scanning models of the 30 subjects indicate that the proposed system is very effective and robust.