Recursive non-rigid structure from motion with online learned shape prior

Most existing approaches in structure from motion for deformable objects focus on non-incremental solutions utilizing batch type algorithms. All data is collected before shape and motion reconstruction take place. This methodology is inherently unsuitable for applications that require real-time learning. Ideally the online system is capable of incrementally learning and building accurate shapes using current measurement data and past reconstructed shapes. Estimation of 3D structure and camera position is done online. To rely only on the measurements up until that moment is still a challenging problem.     

摄像机沿光轴运动下的场景三维重建

讨论了采用针孔摄像机进行摄像机沿光轴运动下的场景三维重建的方法．基于摄像机轴向运动的特点和性质,利用该方法找到图像间的缩放因子,进而解决了轴向运动下的特征匹配;采用Sturm的摄像机自标定方法得到摄像机的内外参数;从而实现了摄像机沿光轴运动下的场景三维重建．     

Detailed Real-Time Urban 3D Reconstruction from Video

The paper presents a system for automatic, geo-registered, real-time 3D reconstruction from video of urban scenes. The system collects video streams, as well as GPS and inertia measurements in order to place the reconstructed models in geo-registered coordinates. It is designed using current state of the art real-time modules for all processing steps. It employs commodity graphics hardware and standard CPU’s to achieve real-time performance. We present the main considerations in designing the system and the steps of the processing pipeline. Our system extends existing algorithms to meet the robustness and variability necessary to operate out of the lab. To account for the large dynamic range of outdoor videos the processing pipeline estimates global camera gain changes in the feature tracking stage and efficiently compensates for these in stereo estimation without impacting the real-time performance. The required accuracy for many applications is achieved with a two-step stereo reconstruction process exploiting the redundancy across frames. We show results on real video sequences comprising hundreds of thousands of frames.     

EM-GPA: Generalized Procrustes analysis with hidden variables for 3D shape modeling

Aligning shapes is essential in many computer vision problems and generalized Procrustes analysis (GPA) is one of the most popular algorithms to align shapes. However, if some of the shape data are missing, GPA cannot be applied. In this paper, we propose EM-GPA, which extends GPA to handle shapes with hidden (missing) variables by using the expectation-maximization (EM) algorithm. For example, 2D shapes can be considered as 3D shapes with missing depth information due to the projection of 3D shapes into the image plane. For a set of 2D shapes, EM-GPA finds scales, rotations and 3D shapes along with their mean and covariance matrix for 3D shape modeling. A distinctive characteristic of EM-GPA is that it does not enforce any rank constraint often appeared in other work and instead uses GPA constraints to resolve the ambiguity in finding scales, rotations, and 3D shapes. The experimental results show that EM-GPA can recover depth information accurately even when the noise level is high and there are a large number of missing variables. By using the images from the FRGC database, we show that EM-GPA can successfully align 2D shapes by taking the missing information into consideration. We also demonstrate that the 3D mean shape and its covariance matrix are accurately estimated. As an application of EM-GPA, we construct a 2D + 3D AAM (active appearance model) using the 3D shapes obtained by EM-GPA, and it gives a similar success rate in model fitting compared to the method using real 3D shapes. EM-GPA is not limited to the case of missing depth information, but it can be easily extended to more general cases.     

轨迹空间中基于特征符号搜索算法的非刚体三维运动重建研究

现有研究一般是在正交约束条件下采用最小二乘法来求解三维运动轨迹基重建问题,然而这些算法的性能容易受噪声影响,产生不适定性。为此,通过利用离散线性的方法为三维运动搜索最优轨迹基系数和三维结构,建立了一种基于特征符号搜索算法的三维运动轨迹基重建方法。利用该方法,对一系列单目图像序列进行了重建实验研究,结果表明所提出的方法是可行的,并提高了重建算法的准确性。     

Non-rigid metric reconstruction from perspective cameras

The metric reconstruction of a non-rigid object viewed by a generic camera poses new challenges since current approaches for Structure from Motion assume the rigidity constraint of a shape as an essential condition. In this work, we focus on the estimation of the 3-D Euclidean shape and motion of a non-rigid shape observed by a perspective camera. In such case deformation and perspective effects are difficult to decouple – the parametrization of the 3-D non-rigid body may mistakenly account for the perspective distortion. Our method relies on the fact that it is often a reasonable assumption that some of the points on the object’s surface are deforming throughout the sequence while others remain rigid. Thus, relying on the rigidity constraints of a subset of rigid points, we estimate the perspective to metric upgrade transformation. First, we use an automatic segmentation algorithm to identify the set of rigid points. These are then used to estimate the internal camera calibration parameters and the overall rigid motion. Finally, we formulate the problem of non-rigid shape and motion estimation as a non-linear optimization where the objective function to be minimized is the image reprojection error. The prior information that some of the points in the object are rigid can also be added as a constraint to the non-linear minimization scheme in order to avoid ambiguous configurations. We perform experiments on different synthetic and real data sets which show that even when using a minimal set of rigid points and when varying the intrinsic camera parameters it is possible to obtain reliable metric information.     

Inferring 3D structure from three points in rigid motion

We prove the following: Given four (or more) orthographic views of three points then (a) the views almost surely have no rigid interpretation but (b) if they do then they almost surely have at most thirty-two rigid interpretations. Part (a) means that the measure of false targets, viz., the measure of nonrigid motions that project to views having rigid interpretations, is zero. Part (b) means that rigid interpretations, when they exist, are not unique. Uniqueness of interpretation can be obtained if a point is added, but not if views are added. Our proof relies on an upper semicontinuity theorem for proper mappings of complex algebraic varieties. We note some psychophysical motivations of the theory.     

Inferring 3D structure from image motion: The constraint of Poinsot motion

Monocular observers perceive as three-dimensional (3D) many displays that depict three points rotating rigidly in space but rotating about an axis that is itself tumbling. No theory of structure from motion currently available can account for this ability. We propose a formal theory for this ability based on the constraint of Poinsot motion, i.e., rigid motion with constant angular momentum. In particular, we prove that three (or more) views of three (or more) points are sufficient to decide if the motion of the points conserves angular momentum and, if it does, to compute a unique 3D interpretation. Our proof relies on an upper semicontinuity theorem for finite morphisms of algebraic varieties. We discuss some psychophysical implications of the theory.This work was supported by National Science Foundation grants IRI-8700924 and DIR-9014278 and by Office of Naval Research contract N00014-88-K-0354.     

Photorealistic 3D reconstruction from handheld cameras

One of the major challenges in the fields of computer vision and computer graphics is the construction and representation of life-like virtual 3D scenarios within a computer. The VISIRE project attempts to reconstruct photo-realistic 3D models of large scenarios using as input multiple freehand video sequences, while rendering the technology accessible to the non-expert. VISIRE is application oriented and hence must deal with multiple issues of practical relevance that were commonly overlooked in past experiences. The paper presents both an innovative approach for the integration of previously unrelated experiences, as well as a number of novel contributions, such as: an innovative algorithm to enforce closedness of the trajectories, a new approach to 3D mesh generation from sparse data, novel techniques dealing with partial occlusions and a method for using photo-consistency and visibility constrains to refine the 3D mesh. Tomas Rodriguez was born in Madrid in 1961. Bachelor in Physics and Master in Electronics by the Universidad Complutense de Madrid. He started his career in the private R&D sector in 1987, when he specialized in computer vision and parallel processing systems. In the early nineties he participated in the EVA project; one of the most outstanding traffic monitoring system of the time. For more than 10 years, he has been involved in international research projects within the ambit of EUREKA, ESPRIT, V and VI Framework programmes. During this time, he coordinated eight international projects (CAMELOT, CITRONE, ON-LIVE, SAID, VISIRE, EVENTS, ITALES, HOLONICS) and acted as principal investigator in two additional ones (CITRUS and VICTORIA). Since the early days, he had the opportunity to collaborate with some of the most prestigious research institutions in Europe: Franhoufer Inst., INRIA, CNRS, University of Oxford, University of Lund, DFKI, Siemens C-Lab, Philips Research Labs, etc. Evaluator of R&D projects for the Spanish Ministry for Science and reviewer of international scientific journals, he is currently the R&D manager and coordinator for European projects at Eptron SA. His recent interests include: computer vision, real time software, industrial control, parallel processing, iTV, and mobile technologies, etc.     

基于图像序列的非刚体三维运动恢复

研究图像序列中非刚体的三维运动重建问题。介绍非刚体运动重建的两种重要算法:奇异值分解法和线性迭代法。对迭代算法所用的重构方法进行修改,在迭代过程中应用结果更为精确的重构方法来求解非刚体的模型和旋转矩阵。对真实图像序列的实验结果验证了该算法的有效性和精确性。     

透视投影下三维运动重建

单相机下,三维信息的恢复往往呈现出病态,如何在透视投影约束下恢复高质量的三维人体运动信息是计算机视觉领域一个具有挑战性的课题。提出一个扩展模型,即如何从起点出发,依次扩展到得三维空间中人体关节点的位置,从而恢复出运动信息。相对于已有的算法,提出在搜索空间中用一组最优准则得到连续平滑的扩展起点,并且在扩展过程中利用全局运动平滑假设,有效地消除了深度信息恢复时固有的二义性。最后,将结果运用于动画制作,验证提出的算法的有效性。     

Segmentation from motion of non-rigid objects by neuronal lateral interaction

The problem we are stating is the discrimination of non-rigid objects capable of holding our attention in a scene. Motion allows gradually obtaining all moving objects shapes. We introduce an algorithm that fuses spots obtained by means of neuronal lateral interaction in accumulative computation.     

A hierarchy of cameras for 3D photography

The view-independent visualization of 3D scenes is most often based on rendering accurate 3D models or utilizes image-based rendering techniques. To compute the 3D structure of a scene from a moving vision sensor or to use image-based rendering approaches, we need to be able to estimate the motion of the sensor from the recorded image information with high accuracy, a problem that has been well-studied. In this work, we investigate the relationship between camera design and our ability to perform accurate 3D photography, by examining the influence of camera design on the estimation of the motion and structure of a scene from video data. By relating the differential structure of the time varying plenoptic function to different known and new camera designs, we can establish a hierarchy of cameras based upon the stability and complexity of the computations necessary to estimate structure and motion. At the low end of this hierarchy is the standard planar pinhole camera for which the structure from motion problem is non-linear and ill-posed. At the high end is a camera, which we call the full field of view polydioptric camera, for which the motion estimation problem can be solved independently of the depth of the scene which leads to fast and robust algorithms for 3D Photography. In between are multiple view cameras with a large field of view which we have built, as well as omni-directional sensors.     

Stabilizing the Focal Length Computation for 3-D Reconstruction from Two Uncalibrated Views

In order to reconstruct 3-D shape from two uncalibrated views, one needs to resolve two problems: (i) the computed focal lengths can be imaginary; (ii) the computation fails for fixated images. We present a practical remedy for these by subsampling feature points and fixing the focal length. We first summarize theoretical backgrounds and then do simulations, which reveal a rather surprising fact that when the focal length is actually fixed, not using that knowledge yields better results for non-fixated images. We give an explanation to this seeming paradox and derive a hybrid method switching the computation by judging whether or not the images are fixated. Doing simulations and real image experiments, we demonstrate the effectiveness of our method.     

Motion of disturbances: detection and tracking of multi-body non-rigid motion

We present a new approach to the tracking of very non-rigid patterns of motion, such as water flowing down a stream. The algorithm is based on a “disturbance map”, which is obtained by linearly subtracting the temporal average of the previous frames from the new frame. Every local motion creates a disturbance having the form of a wave, with a “head” at the present position of the motion and a historical “tail” that indicates the previous locations of that motion. These disturbances serve as loci of attraction for “tracking particles” that are scattered throughout the image. The algorithm is very fast and can be performed in real time. We provide excellent tracking results on various complex sequences, using both stabilized and moving cameras, showing a busy ant column, waterfalls, rapids and flowing streams, shoppers in a mall, and cars in a traffic intersection. Received: 24 June 1997 / Accepted: 30 July 1998     

Closed Form Solutions for Reconstruction Via Complex Analysis

We address the problem of control-based recovery of robot pose and environmental lay-out. Panoramic sensors provide a 1D projection of characteristic features of a 2D operation map. Trajectories of these projections contain information about the position of a priori unknown landmarks in the environment. We introduce the notion of spatiotemporal signatures of projection trajectories. These signatures are global measures, characterized by considerably higher robustness with respect to noise and outliers than the commonly applied point correspondence. By modeling the 2D motion plane as the complex plane we show that by means of complex analysis the reconstruction problem can be reduced to a quadratic—or even linear in some cases—equation. The algorithm is tested in simulations and in a real experiment.     

Reconstruction of a Scene with Multiple Linearly Moving Objects

In this paper we describe an algorithm to recover the scene structure, the trajectories of the moving objects and the camera motion simultaneously given a monocular image sequence. The number of the moving objects is automatically detected without prior motion segmentation. Assuming that the objects are moving linearly with constant speeds, we propose a unified geometrical representation of the static scene and the moving objects. This representation enables the embedding of the motion constraints into the scene structure, which leads to a factorization-based algorithm. We also discuss solutions to the degenerate cases which can be automatically detected by the algorithm. Extension of the algorithm to weak perspective projections is presented as well. Experimental results on synthetic and real images show that the algorithm is reliable under noise.     

Reconstruction of 3D Object Meshes from Silhouette Images

In this work we propose a method for computing mesh representations of 3D objects reconstructed from a set of silhouette images. Our method is based on the polygonization of volumetric reconstructions by using a modified version of the dual contouring method. In order to apply dual contouring on volumetric reconstruction from silhouettes we devised a method that is able to determine the discrete topology of the surface in relation to the octree cells. We also developed a new scheme for computing hermitian data representing the intersections of conic volumes with the octree cells and their corresponding normals with subpixel accuracy. Due to the discrete and extremely noisy nature of the data used in the reconstruction we had to devise a different criterion for mesh simplification that applies topological consistency tests only when the geometric error measure is beyond a given tolerance. We present results of the application of the proposed method in the extraction of a mesh corresponding to the surface of objects of a real scene.     

Incorporating non-motion cues into 3D motion segmentation

We address the problem of segmenting an image sequence into rigidly moving 3D objects. An elegant solution to this problem in the case of orthographic projection is the multibody factorization approach in which the measurement matrix is factored into lower rank matrices. Despite progress in factorization algorithms, their performance is still far from satisfactory and in scenes with missing data and noise, most existing algorithms fail.In this paper we propose a method for incorporating 2D non-motion cues (such as spatial coherence) into multibody factorization. We show the similarity of the problem to constrained factor analysis and use the EM algorithm to find the segmentation. We show that adding these cues improves performance in real and synthetic sequences.