Uncalibrated Motion Capture Exploiting Articulated Structure Constraints

We present an algorithm for 3D reconstruction of dynamic articulated structures, such as humans, from uncalibrated multiple views. The reconstruction exploits constraints associated with a dynamic articulated structure, specifically the conservation over time of length between rotational joints. These constraints admit reconstruction of metric structure from at least two different images in each of two uncalibrated parallel projection cameras. As a by product, the calibration of the cameras can also be computed. The algorithm is based on a stratified approach, starting with affine reconstruction from factorization, followed by rectification to metric structure using the articulated structure constraints. The exploitation of these specific constraints admits reconstruction and self-calibration with fewer feature points and views compared to standard self-calibration. The method is extended to pairs of cameras that are zooming, where calibration of the cameras allows compensation for the changing scale factor in a scaled orthographic camera. Results are presented in the form of stick figures and animated 3D reconstructions using pairs of sequences from broadcast television. The technique shows promise as a means of creating 3D animations of dynamic activities such as sports events.     

Reconstruction of a Scene with Multiple Linearly Moving Objects

In this paper we describe an algorithm to recover the scene structure, the trajectories of the moving objects and the camera motion simultaneously given a monocular image sequence. The number of the moving objects is automatically detected without prior motion segmentation. Assuming that the objects are moving linearly with constant speeds, we propose a unified geometrical representation of the static scene and the moving objects. This representation enables the embedding of the motion constraints into the scene structure, which leads to a factorization-based algorithm. We also discuss solutions to the degenerate cases which can be automatically detected by the algorithm. Extension of the algorithm to weak perspective projections is presented as well. Experimental results on synthetic and real images show that the algorithm is reliable under noise.     

Qualitative Scene Interpretation Using Planar Surfaces

Our aim is to provide an autonomous vehicle moving into an indoor environment with a visual system to perform a qualitative 3D structure reconstruction of the surrounding environment by recovering the different planar surfaces present in the observed scene.The method is based on qualitative detection of planar surfaces by using projective invariant constraints without the use of depth estimates. The goal is achieved by analyzing two images acquired by observing the scene from two different points of view. The method can be applied to both stereo images and motion images.Our method recovers planar surfaces by clustering high variance interest points whose cross ratio measurements are preserved in two different perspective projections. Once interest points are extracted from each image, the clustering process requires to grouping corresponding points by preserving the cross ratio measurements.We solve the twofold problem of finding corresponding points and grouping the coplanar ones through a global optimization approach based on matching of high relational graphs and clustering on the corresponding association graph through a relaxation labeling algorithm.Through our experimental tests, we found the method to be very fast to converge to a solution, showing how higher order interactions, instead to giving rise to a more complex problem, help to speed-up the optimization process and to reach at same time good results.     

基于透视和正投影图像的3D运动与结构恢复

论文首次研究了由一幅正投影图像和一幅透视投影图像的特征点对应来进行刚体3D运动重建与结构恢复的问题,给出了有效的线性算法。以往的由运动恢复结构的工作主要集中于一组透视图像或一组正投影(通常为仿射)图像,文中采用了透视模型和正投影模型的组合。数据模拟实验结果显示该方法是比较有效和稳定的。     

Structure from Motion: Beyond the Epipolar Constraint

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images.The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the least varying estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.     

三维场景中四视图联动的研究与实现

JAVA3D是JAVA语言向三维领域扩展的一组API,是一种能够帮助用户实现网络时代三维梦想的强有力工具。首先介绍了Java3D中场景图的概念,并通过实例说明了生成场景图和模拟装配的方法;然后着重研究了三维场景中四视图的联动原理。基于以上方法作者开发了一套三维场景生成工具,通过它,用户可以非编程地快速搭建所需的三维场景。结果表明所提出的方法很有效和实用。     

基于直线光流场的三维运动和结构重建

利用直线间运动对应关系,将像素点光流的概念和定义方法应用于直线,提出了直线光流的概念,建立了求解空间物体运动参数的线性方程组,利用三幅图像21条直线的光流场,可以求得物体运动的12个参数以及空间直线坐标．但是在实际应用当中,要找出这21条直线的光流场是很困难的,因此该文提出了运用解非线性方程组的方法,只需要6条直线的光流．就可以分步求出物体的12个运动参数,并根据求得的12个运动参数和一致的图像坐标系中的直线坐标,求得空间直线的坐标,从而实现了三维场景的重建．     

Scene Reconstruction and Robot Navigation Using Dynamic Fields

In this paper, we present an approach to autonomous robot navigation in an unknown environment. We design and integrate algorithms to reconstruct the scene, locate obstacles and do short-term field-based path planning. The scene reconstruction is done using a region matching flow algorithm to recover image deformation and structure from motion to recover depth. Obstacles are located by comparing the surface normal of the known floor with the surface normal of the scene. Our path planning method is based on electric-like fields and uses current densities that can guarantee fields without local minima and maxima which can provide solutions without the need of heuristics that plague the more traditional potential fields approaches. We implemented a modular distributed software platform (FBN) to test this approach and we ran several experiments to verify the performance with very encouraging results.     

基于形态及受约束结构的三维物体建模方法

文中提出一种基于物体形态及受约束结构的三维物体建模方法,该方法利用具有透视不变性的三维结构来表达物体的各个形态。利用该表达方法可以使机器视觉系统在用单幅灰度图像识别物体时,在模型索引阶段避开求解物体位姿、摄像机参数、特征对应等复杂问题,从而实现先索引后匹配的识别策略,提高识别物体的实时性。文中首先论述了透视不变性和具有透视不变性的受约束结构的基本概念;其次,给出了用受约束结构进行三维物体建模的一般方法和应用实例;最后,指出了这种方法的不足和进一步的研究方向。     

An Efficient and Accurate Method for 3D-Point Reconstruction from Multiple Views

In this paper we consider the problem of finding the position of a point in space given its projections in multiple images taken by cameras with known calibration and pose. Ideally the 3D point can be obtained as the intersection of multiple known rays in space. However, with noise the rays do not meet at a single point generally. Therefore, it is necessary to find a best point of intersection. In this paper we propose a modification of the method (Ma et al., 2001. Journal of Communications in Information and Systems, (1):51–73) based on the multiple-view epipolar constraints. The solution is simple in concept and straightforward to implement. It includes generally two steps: first, image points are corrected through approximating the error model to the first order, and then the 3D point can be reconstructed from the corrected image points using any generic triangulation method. Experiments are conducted both on simulated data and on real data to test the proposed method against previous methods. It is shown that results obtained with the proposed method are consistently more accurate than those of other linear methods. When the measurement error of image points is relatively small, its results are comparable to those of maximum likelihood estimation using Newton-type optimizers; and when processing image-point correspondences cross a small number of views, the proposed method is by far more efficient than the Newton-type optimizers.     

Multi-View Scene Capture by Surfel Sampling: From Video Streams to Non-Rigid 3D Motion,Shape and Reflectance

In this paper we study the problem of recovering the 3D shape, reflectance, and non-rigid motion properties of a dynamic 3D scene. Because these properties are completely unknown and because the scene's shape and motion may be non-smooth, our approach uses multiple views to build a piecewise-continuous geometric and radiometric representation of the scene's trace in space-time. A basic primitive of this representation is the dynamic surfel, which (1) encodes the instantaneous local shape, reflectance, and motion of a small and bounded region in the scene, and (2) enables accurate prediction of the region's dynamic appearance under known illumination conditions. We show that complete surfel-based reconstructions can be created by repeatedly applying an algorithm called Surfel Sampling that combines sampling and parameter estimation to fit a single surfel to a small, bounded region of space-time. Experimental results with the Phong reflectancemodel and complex real scenes (clothing, shiny objects, skin) illustrate our method's ability to explain pixels and pixel variations in terms of their underlying causes—shape, reflectance, motion, illumination, and visibility.     

一种新的几何约束结构及其射影不变量

几何不变量,特别是射影不变量,是基于单视点灰度图像识别三维物体的一条有效途径．但理论研究表明,只有特定的几何约束结构,才具有射影不变量．所以,研究并发现这种几何约束结构就具有十分重要的意义．该文提出了一种新的由相邻3平面上5条直线组成的几何约束结构及其所具有的射影不变量．该结构较Sugimoto提出的几何约束结构简单,可从结构同样复杂的物体中获得更多的几何不变量,有利于提高物体识别的稳定性;同时,由于该结构大量存在于由多面体组合而构成的人造物体及地面建筑物中,因此它非常适合这类物体的识别．实验验证了文中提出的几何约束结构具有不随物体成像视点改变的射影不变量．     

使用多视图L1跟踪器的三维人体运动恢复

近年来从视频中恢复三维人体运动的研究发展很快,其中大部分方法是基于前景轮廓的。提出了一种基于纹理信息的三维人体运动恢复方法,并给出了一个鲁棒、自适应的跟踪器模型。该模型基于L1跟踪器,并将其扩展到多个视图中,使用分层搜索来跟踪人体的各个部位。它可以寻找在模板子空间里重构误差最小的跟踪目标,将每个视图的重构误差作为衡量人体三维姿态与图像拟合的可能性函数。整个算法在退火粒子滤波的框架下进行。为了提高跟踪准度,在纹理模板更新过程中使用了两种方法:用人体的三维模型来检测自遮挡;根据模板系数检测计算错误的跟踪结果。综合这两种检测器,可以防止遮挡后和计算错误的跟踪结果加入到纹理模板中。在HumanEva-Ⅱ测试集上的实验表明,该算法能够得到较好的结果。     

A Unified Theory of Uncalibrated Stereo for Both Perspective and Affine Cameras

This paper addresses the recovery of structure and motion from uncalibrated images of a scene under full perspective or under affine projection. Particular emphasis is placed on the configuration of two views, while the extension to $N$ views is given in Appendix. A unified expression of the fundamental matrix is derived which is valid for any projection model without lens distortion (including full perspective and affine camera). Affine reconstruction is considered as a special projective reconstruction. The theory is elaborated in a way such that everyone having knowledge of linear algebra can understand the discussion without difficulty. A new technique for affine reconstruction is developed, which consists in first estimating the affine epipolar geometry and then performing a triangulation for each point match with respect to an implicit common affine basis.     

面向增强视频的基于结构和运动恢复的摄像机定标

提出了一种高效鲁棒的长序列摄像机定标算法,能稳定处理焦距未知且变化的视频序列,适用于增强视频的应用．该算法从长视频序列中根据特征匹配点提炼出相互之间具有较长基线的关键帧,以保证求解的稳定性．算法先在关键帧序列上渐进式求解,以准确恢复特征匹配点的互维结构信息;利用精确恢复的三维点,求解整个序列的摄像机运动参数．该算法选择最适合初始化的三帧求解,并将解及时从射影空间转换到欧氏空间．实验结果显示了所恢复的摄像机参数和三维点的高度精确性,证明了该方法稳定高效,能够满足增强视频的高端要求．     

Critical Configurations for Projective Reconstruction from Multiple Views

This paper investigates a classical problem in computer vision: Given corresponding points in multiple images, when is there a unique projective reconstruction of the 3D geometry of the scene points and the camera positions? A set of points and cameras is said to be critical when there is more than one way of realizing the resulting image points. For two views, it has been known for almost a century that the critical configurations consist of points and camera lying on a ruled quadric surface. We give a classification of all possible critical configurations for any number of points in three images, and show that in most cases, the ambiguity extends to any number of cameras.The underlying framework for deriving the critical sets is projective geometry. Using a generalization of Pascal's Theorem, we prove that any number of cameras and scene points on an elliptic quartic form a critical set. Another important class of critical configurations consists of cameras and points on rational quartics. The theoretical results are accompanied by many examples and illustrations.     

Projective Reconstruction from Multiple Views with Minimization of 2D Reprojection Error

The problem of projective reconstruction by minimization of the 2D reprojection error in multiple images is considered. Although bundle adjustment techniques can be used to minimize the 2D reprojection error, these methods being based on nonlinear optimization algorithms require a good starting point. Quasi-linear algorithms with better global convergence properties can be used to generate an initial solution before submitting it to bundle adjustment for refinement. In this paper, we propose a factorization-based method to integrate the initial search as well as the bundle adjustment into a single algorithm consisting of a sequence of weighted least-squares problems, in which a control parameter is initially set to a relaxed state to allow the search of a good initial solution, and subsequently tightened up to force the final solution to approach a minimum point of the 2D reprojection error. The proposed algorithm is guaranteed to converge. Our method readily handles images with missing points.     

Appearance-Based Structure from Motion Using Linear Classes of 3-D Models

In this paper, we address the problem of recovering 3-D models from sequences of partly calibrated images with unknown correspondence. To that end, we integrate tracking, structure from motion with geometric constraints (specifically in the form of linear class models) in a single framework. The key to making the proposed approach work is the use of appearance-based model matching and refinement which updates the estimated correspondences on each iteration of the algorithm. Another key feature is the matching of a 3-D model directly with the input images without the conventional 2-step approach of stereo data recovery and 3-D model fitting. Initialization of the linear class model to one of the input images (the reference image) is currently partly manual.This synthesis and refine approach, or appearance-based constrained structure from motion (AbCSfm), is especially useful in recovering shapes of objects whose general structureis known but which may have little discernable texture in significant parts of their surfaces. We applied the proposed approach to 3-D face modeling from multiple images to create new 3-D faces for DECface, a synthetic talking head developed at Cambridge Research Laboratory, Digital Equipment Corporation. The DECface model comprises a collection of 3-D triangular and rectangular facets, with nodes as vertices. In recovering the DECface model, we assume that the sequence of images is taken with a camera with unknown focal length and pose. The geometric constraints used are of the form of linear combination of prototypes of 3-D faces of real people. Results of this approach show its good convergence properties and its robustness against cluttered backgrounds.     

Multiple View Geometry of General Algebraic Curves

We introduce a number of new results in the context of multi-view geometry from general algebraic curves. We start with the recovery of camera geometry from matching curves. We first show how one can compute, without any knowledge on the camera, the homography induced by a single planar curve. Then we continue with the derivation of the extended Kruppa's equations which are responsible for describing the epipolar constraint of two projections of a general algebraic curve. As part of the derivation of those constraints we address the issue of dimension analysis and as a result establish the minimal number of algebraic curves required for a solution of the epipolar geometry as a function of their degree and genus.We then establish new results on the reconstruction of general algebraic curves from multiple views. We address three different representations of curves: (i) the regular point representation in which we show that the reconstruction from two views of a curve of degree d admits two solutions, one of degree d and the other of degree d(d – 1). Moreover using this representation, we address the problem of homography recovery for planar curves, (ii) dual space representation (tangents) for which we derive a lower bound for the number of views necessary for reconstruction as a function of the curve degree and genus, and (iii) a new representation (to computer vision) based on the set of lines meeting the curve which does not require any curve fitting in image space, for which we also derive lower bounds for the number of views necessary for reconstruction as a function of curve degree alone.