Subspace Procrustes Analysis

Procrustes analysis (PA) has been a popular technique to align and build 2-D statistical models of shapes. Given a set of 2-D shapes PA is applied to remove rigid transformations. Later, a non-rigid 2-D model is computed by modeling the residual (e.g., PCA). Although PA has been widely used, it has several limitations for modeling 2-D shapes: occluded landmarks and missing data can result in local minima solutions, and there is no guarantee that the 2-D shapes provide a uniform sampling of the 3-D space of rotations for the object. To address previous issues, this paper proposes subspace PA (SPA). Given several instances of a 3-D object, SPA computes the mean and a 2-D subspace that can model rigid and non-rigid deformations of the 3-D object. We propose a discrete (DSPA) and continuous (CSPA) formulation for SPA, assuming that 3-D samples of an object are provided. DSPA extends the traditional PA, and produces unbiased 2-D models by uniformly sampling different views of the 3-D object. CSPA provides a continuous approach to uniformly sample the space of 3-D rotations, being more efficient in space and time. We illustrate the benefits of SPA in two different applications. First, SPA is used to learn 2-D face and body models from 3-D datasets. Experiments on the FaceWarehouse and CMU motion capture (MoCap) datasets show the benefits of our 2-D models against the state-of-the-art PA approaches and conventional 3-D models. Second, SPA learns an unbiased 2-D model from CMU MoCap dataset and it is used to estimate the human pose on the Leeds Sports dataset.     

Defining the Pose of Any 3D Rigid Object and an Associated Distance

The pose of a rigid object is usually regarded as a rigid transformation, described by a translation and a rotation. However, equating the pose space with the space of rigid transformations is in general abusive, as it does not account for objects with proper symmetries—which are common among man-made objects. In this article, we define pose as a distinguishable static state of an object, and equate a pose to a set of rigid transformations. Based solely on geometric considerations, we propose a frame-invariant metric on the space of possible poses, valid for any physical rigid object, and requiring no arbitrary tuning. This distance can be evaluated efficiently using a representation of poses within a Euclidean space of at most 12 dimensions depending on the object’s symmetries. This makes it possible to efficiently perform neighborhood queries such as radius searches or k-nearest neighbor searches within a large set of poses using off-the-shelf methods. Pose averaging considering this metric can similarly be performed easily, using a projection function from the Euclidean space onto the pose space. The practical value of those theoretical developments is illustrated with an application of pose estimation of instances of a 3D rigid object given an input depth map, via a Mean Shift procedure.     

EM-GPA: Generalized Procrustes analysis with hidden variables for 3D shape modeling

Aligning shapes is essential in many computer vision problems and generalized Procrustes analysis (GPA) is one of the most popular algorithms to align shapes. However, if some of the shape data are missing, GPA cannot be applied. In this paper, we propose EM-GPA, which extends GPA to handle shapes with hidden (missing) variables by using the expectation-maximization (EM) algorithm. For example, 2D shapes can be considered as 3D shapes with missing depth information due to the projection of 3D shapes into the image plane. For a set of 2D shapes, EM-GPA finds scales, rotations and 3D shapes along with their mean and covariance matrix for 3D shape modeling. A distinctive characteristic of EM-GPA is that it does not enforce any rank constraint often appeared in other work and instead uses GPA constraints to resolve the ambiguity in finding scales, rotations, and 3D shapes. The experimental results show that EM-GPA can recover depth information accurately even when the noise level is high and there are a large number of missing variables. By using the images from the FRGC database, we show that EM-GPA can successfully align 2D shapes by taking the missing information into consideration. We also demonstrate that the 3D mean shape and its covariance matrix are accurately estimated. As an application of EM-GPA, we construct a 2D + 3D AAM (active appearance model) using the 3D shapes obtained by EM-GPA, and it gives a similar success rate in model fitting compared to the method using real 3D shapes. EM-GPA is not limited to the case of missing depth information, but it can be easily extended to more general cases.     

Recognition by linear combinations of models

An approach to visual object recognition in which a 3D object is represented by the linear combination of 2D images of the object is proposed. It is shown that for objects with sharp edges as well as with smooth bounding contours, the set of possible images of a given object is embedded in a linear space spanned by a small number of views. For objects with sharp edges, the linear combination representation is exact. For objects with smooth boundaries, it is an approximation that often holds over a wide range of viewing angles. Rigid transformations (with or without scaling) can be distinguished from more general linear transformations of the object by testing certain constraints placed on the coefficients of the linear combinations. Three alternative methods of determining the transformation that matches a model to a given image are proposed     

Robust shape normalization of 3D articulated volumetric models

3D shape normalization is a common task in various computer graphics and pattern recognition applications. It aims to normalize different objects into a canonical coordinate frame with respect to rigid transformations containing translation, rotation and scaling in order to guarantee a unique representation. However, the conventional normalization approaches do not perform well when dealing with 3D articulated objects.To address this issue, we introduce a new method for normalizing a 3D articulated object in the volumetric form. We use techniques from robust statistics to guide the classical normalization computation. The key idea is to estimate the initial normalization by using implicit shape representation, which produces a novel articulation insensitive weight function to reduce the influence of articulated deformation. We also propose and prove the articulation insensitivity of implicit shape representation. The final solution is found by means of iteratively reweighted least squares. Our method is robust to articulated deformation without any explicit shape decomposition. The experimental results and some applications are presented for demonstrating the effectiveness of our method.     

Procrustes Analysis with Deformations: A Closed-Form Solution by Eigenvalue Decomposition

Generalized Procrustes Analysis (GPA) is the problem of bringing multiple shapes into a common reference by estimating transformations. GPA has been extensively studied for the Euclidean and affine transformations. We introduce GPA with deformable transformations, which forms a much wider and difficult problem. We specifically study a class of transformations called the Linear Basis Warps, which contains the affine transformation and most of the usual deformation models, such as the Thin-Plate Spline (TPS). GPA with deformations is a nonconvex underconstrained problem. We resolve the fundamental ambiguities of deformable GPA using two shape constraints requiring the eigenvalues of the shape covariance. These eigenvalues can be computed independently as a prior or posterior. We give a closed-form and optimal solution to deformable GPA based on an eigenvalue decomposition. This solution handles regularization, favoring smooth deformation fields. It requires the transformation model to satisfy a fundamental property of free-translations, which asserts that the model can implement any translation. We show that this property fortunately holds true for most common transformation models, including the affine and TPS models. For the other models, we give another closed-form solution to GPA, which agrees exactly with the first solution for models with free-translation. We give pseudo-code for computing our solution, leading to the proposed DefGPA method, which is fast, globally optimal and widely applicable. We validate our method and compare it to previous work on six diverse 2D and 3D datasets, with special care taken to choose the hyperparameters from cross-validation.

     

A Local/Global Approach to Mesh Parameterization

We present a novel approach to parameterize a mesh with disk topology to the plane in a shape‐preserving manner. Our key contribution is a local/global algorithm, which combines a local mapping of each 3D triangle to the plane, using transformations taken from a restricted set, with a global “stitch” operation of all triangles, involving a sparse linear system. The local transformations can be taken from a variety of families, e.g. similarities or rotations, generating different types of parameterizations. In the first case, the parameterization tries to force each 2D triangle to be an as‐similar‐as‐possible version of its 3D counterpart. This is shown to yield results identical to those of the LSCM algorithm. In the second case, the parameterization tries to force each 2D triangle to be an as‐rigid‐as‐possible version of its 3D counterpart. This approach preserves shape as much as possible. It is simple, effective, and fast, due to pre‐factoring of the linear system involved in the global phase. Experimental results show that our approach provides almost isometric parameterizations and obtains more shape‐preserving results than other state‐of‐the‐art approaches. We present also a more general “hybrid” parameterization model which provides a continuous spectrum of possibilities, controlled by a single parameter. The two cases described above lie at the two ends of the spectrum. We generalize our local/global algorithm to compute these parameterizations. The local phase may also be accelerated by parallelizing the independent computations per triangle.     

Shape from interaction

We present “shape from interaction” (SfI), an approach to the problem of acquiring 3D representations of rigid objects through observing the activity of a human who handles a tool. SfI relies on the fact that two rigid objects cannot share the same physical space. The 3D reconstruction of the unknown object is achieved by tracking the known 3D tool and by carving out the space it occupies as a function of time. Due to this indirection, SfI reconstructs rigid objects regardless of their material and appearance properties and proves particularly useful for the cases of textureless, transparent, translucent, refractive and specular objects for which there exists no practical vision-based 3D reconstruction method. Additionally, object concavities that are not directly observable can also be reconstructed. The 3D tracking of the tool is formulated as an optimization problem that is solved based on visual input acquired by a multicamera system. Experimental results from a prototype implementation of SfI support qualitatively and quantitatively the effectiveness of the proposed approach.     

Nonrigid structure-from-motion: estimating shape and motion with hierarchical Priors

This paper describes methods for recovering time-varying shape and motion of non-rigid 3D objects from uncalibrated 2D point tracks. For example, given a video recording of a talking person, we would like to estimate the 3D shape of the face at each instant, and learn a model of facial deformation. Time-varying shape is modeled as a rigid transformation combined with a non-rigid deformation. Reconstruction is ill-posed if arbitrary deformations are allowed, and thus additional assumptions about deformations are required. We first suggest restricting shapes to lie within a low-dimensional subspace, and describe estimation algorithms. However, this restriction alone is insufficient to constrain reconstruction. To address these problems, we propose a reconstruction method using a Probabilistic Principal Components Analysis (PPCA) shape model, and an estimation algorithm that simultaneously estimates 3D shape and motion for each instant, learns the PPCA model parameters, and robustly fills-in missing data points. We then extend the model to model temporal dynamics in object shape, allowing the algorithm to robustly handle severe cases of missing data.     

Deformation by examples: a density flow approach

In this article, a shape transformation technique is introduced for deforming objects based on a given deformation example. The example consists of two reference shapes representing two different states of an object. The reference shapes are assumed to morph from one state to the other. The evolution between the two reference shapes determines the shape transformation function. Any given objects can then be deformed by the same transformation. A continuous 4D Radial Basis Function is used to construct a density flow field (an extension of the optical flow in computer vision) representing the shape transformation of the example in 3‐space. Objects embedded in the density flow field are deformed by moving vertices of the objects along the density flow vectors. Additional parameters are introduced to control the process of the deformation. This provides explicit control on the shape of the object obtained in the deformation process. Copyright © 2007 John Wiley & Sons, Ltd.     

Reconstruction of non-rigid 3D shapes from stereo-motion

Several non-rigid structure from motion methods have been proposed so far in order to recover both the motion and the non-rigid structure of an object. However, these monocular algorithms fail to give reliable 3D shape estimates when the overall rigid motion of the sequence is small. Aiming to overcome this limitation, in this paper we propose a novel approach for the 3D Euclidean reconstruction of deformable objects observed by an uncalibrated stereo rig. Using a stereo setup drastically improves the 3D model estimation when the observed 3D shape is mostly deforming without undergoing strong rigid motion. Our approach is based on the following steps. Firstly, the stereo system is automatically calibrated and used to compute metric rigid structures from pairs of views. Afterwards, these 3D shapes are aligned to a reference view using a RANSAC method in order to compute the mean shape of the object and to select the subset of points which have remained rigid throughout the sequence. The selected rigid points are then used to compute frame-wise shape registration and to robustly extract the motion parameters from frame to frame. Finally, all this information is used as initial estimates of a non-linear optimization which allows us to refine the initial solution and also to recover the non-rigid 3D model. Exhaustive results on synthetic and real data prove the performance of our proposal estimating motion, non-rigid models and stereo camera parameters even when there is no rigid motion in the original sequence.     

Polyhedra recognition by hypothesis accumulation

A new method is presented for the recognition of polyhedra in range data. The method is based on a hypothesis accumulation scheme which allows parallel implementations. The different objects to be recognized are modeled by a set of local geometrical patterns. Local patterns of the same nature are extracted from the scene. For the recognition of an object, local scene and model patterns having the same geometrical characteristics are matched. For each of the possible matches, the geometric transformations (i.e., rotations and translations) are computed, which allows the overlapping of the model elements with those from the scene. This transformation permits the establishment of a hypothesis on the location of the object in the scene and the determination of a point in the transformation space. The presence of an object similar to a model involves the generation of several compatible hypotheses and creates a compact cluster in the transformation space. The recognition of the object is based on the detection of this cluster. The cluster coordinates give the values of the rotations and the translations to be applied to the model such that it corresponds to the object in the scene. The exact location of this object is given by the transformed model.     

Part I: Modeling image curves using invariant 3-D object curvemodels-a path to 3-D recognition and shape estimation from imagecontours

This paper and its companion are concerned with the problems of 3-D object recognition and shape estimation from image curves using a 3-D object curve model that is invariant to affine transformation onto the image space, and a binocular stereo imaging system. The objects of interest here are the ones that have markings (e.g., characters, letters, special drawings and symbols, etc.) on their surfaces. The 3-D curves on the object are modeled as B-splines, which are characterized by a set of parameters (the control points) from which the 3-D curve can be totally generated. The B-splines are invariant under affine transformations. That means that the affine projected object curve onto the image space is a B-spline whose control points are related to the object control points through the affine transformation. Part I deals with issues relating to the curve modeling process. In particular, the authors address the problems of estimating the control points from the data curve, and of deciding on the “best” order B-spline and the “best” number of control points to be used to model the image or object curve(s). A minimum mean-square error (mmse) estimation technique which is invariant to affine transformations is presented as a noniterative, simple, and fast approach for control point estimation. The “best” B-spline is decided upon using a Bayesian selection rule. Finally, we present a matching algorithm that allocates a sample curve to one of p prototype curves when the sample curve is an a priori unknown affine transformation of one of the prototype curves stored in the data base. The approach is tried on a variety of images of real objects     

Viewpoint-aware object detection and continuous pose estimation

We describe an approach to category-level detection and viewpoint estimation for rigid 3D objects from single 2D images. In contrast to many existing methods, we directly integrate 3D reasoning with an appearance-based voting architecture. Our method relies on a nonparametric representation of a joint distribution of shape and appearance of the object class. Our voting method employs a novel parameterization of joint detection and viewpoint hypothesis space, allowing efficient accumulation of evidence. We combine this with a re-scoring and refinement mechanism, using an ensemble of view-specific support vector machines. We evaluate the performance of our approach in detection and pose estimation of cars on a number of benchmark datasets. Finally we introduce the “Weizmann Cars ViewPoint” (WCVP) dataset, a benchmark for evaluating continuous pose estimation.     

Acquiring a complete 3D model from specular motion under theillumination of circular-shaped light sources

We recover 3D models of objects with specular surfaces. An object is rotated and its continuous images are taken. Circular-shaped light sources that generate conic rays are used to illuminate the rotating object in such a way that highlighted stripes can be observed on most of the specular surfaces. Surface shapes can be computed from the motions of highlights in the continuous images; either specular motion stereo or single specular trace mode can be used. When the lights are properly set, each point on the object can be highlighted during the rotation. The shape for each rotation plane is measured independently using its corresponding epipolar plane image. A 3D shape model is subsequently reconstructed by combining shapes at different rotation planes. Computing a shape is simple and requires only the motion of highlight on each rotation plane. The novelty of this paper is the complete modeling of a general type of specular objects that has not been accomplished before