Affine shape representation from motion through reference points

This paper shows how an affine representation of spatial configuration is obtained from a pair of projection views. Calibration of cameras and knowledge of the camera's motion are not necessary; however, some preselected reference points and their correspondences are needed. Projective and affine geometry invariants are trickily manipulated to do the affine reconstruction. The method is thus geometrically constructive. When it is compared with the solution proposed in 1989 by J.J. Koenderink and A.J. Van Doorn (Affine Structure from Motion, Technical Report, Utrect University), the method provides a viewpoint-independent affine representation under parallel projections. Further, we investigate the central-projection case in which, with three additional special reference points, the same affine reconstruction can be done. We also discuss some important applications of this viewpoint independence of shape representation.     

Robust structure and motion from outlines of smooth curved surfaces

This paper addresses the problem of estimating the motion of a camera as it observes the outline (or apparent contour) of a solid bounded by a smooth surface in successive image frames. In this context, the surface points that project onto the outline of an object depend on the viewpoint and the only true correspondences between two outlines of the same object are the projections of frontier points where the viewing rays intersect in the tangent plane of the surface. In turn, the epipolar geometry is easily estimated once these correspondences have been identified. Given the apparent contours detected in an image sequence, a robust procedure based on RANSAC and a voting strategy is proposed to simultaneously estimate the camera configurations and a consistent set of frontier point projections by enforcing the redundancy of multiview epipolar geometry. The proposed approach is, in principle, applicable to orthographic, weak-perspective, and affine projection models. Experiments with nine real image sequences are presented for the orthographic projection case, including a quantitative comparison with the ground-truth data for the six data sets for which the latter information is available. Sample visual hulls have been computed from all image sequences for qualitative evaluation.     

A curve fitting problem and its application in modeling objects inmonocular image sequences

Presents a solution to a particular curve (surface) fitting problem and demonstrate its application in modeling objects from monocular image sequences. The curve-fitting algorithm is based on a modified nonparametric regression method, which forms the core contribution of this work. This method is far more effective compared to standard estimation techniques, such as the maximum likelihood estimation method, and can take into account the discontinuities present in the curve. Next, the theoretical results of this 1D curve estimation technique ate extended significantly for an object modeling problem. The input to the algorithm is a monocular image sequence of an object undergoing rigid motion. By using the affine camera projection geometry and a given choice of an image frame pair in the sequence, we adopt the KvD (Koenderink and van Doorn, 1991) model to express the depth at each point on the object as a function of the unknown out-of-plane rotation, and some measurable quantities computed directly from the optical flow. This is repeated for multiple image pairs (keeping one fixed image frame which we formally call the base image and choosing another frame from the sequence). The depth map is next estimated from these equations using the modified nonparametric regression analysis. We conducted experiments on various image sequences to verify the effectiveness of the technique. The results obtained using our curve-fitting technique can be refined further by hierarchical techniques, as well as by nonlinear optimization techniques in structure from motion     

Motion of points and lines in the uncalibrated case

In the present paper we address the problem of computing structure and motion, given a set point and/or line correspondences, in a monocular image sequence, when the camera is not calibrated.Considering point correspondences first, we analyse how to parameterize the retinal correspondences, in function of the chosen geometry: Euclidean, affine or projective geometry. The simplest of these parameterizations is called the FQs-representation and is a composite projective representation. The main result is that considering N+1 views in such a monocular image sequence, the retinal correspondences are parameterized by 11 N–4 parameters in the general projective case. Moreover, 3 other parameters are required to work in the affine case and 5 additional parameters in the Euclidean case. These 8 parameters are calibration parameters and must be calculated considering at least 8 external informations or constraints. The method being constructive, all these representations are made explicit.Then, considering line correspondences, we show how the the same parameterizations can be used when we analyse the motion of lines, in the uncalibrated case. The case of three views is extensively studied and a geometrical interpretation is proposed, introducing the notion of trifocal geometry which generalizes the well known epipolar geometry. It is also discussed how to introduce line correspondences, in a framework based on point correspondences, using the same equations.Finally, considering the F Qs-representation, one implementation is proposed as a motion module, taking retinal correspondences as input, and providing and estimation of the 11 N–4 retinal motion parameters. As discussed in this paper, this module can also estimate the 3D depth of the points up to an affine and projective transformation, defined by the 8 parameters identified in the first section. Experimental results are provided.     

Model-based object pose in 25 lines of code

In this paper, we describe a method for finding the pose of an object from a single image. We assume that we can detect and match in the image four or more noncoplanar feature points of the object, and that we know their relative geometry on the object. The method combines two algorithms; the first algorithm,POS (Pose from Orthography and Scaling) approximates the perspective projection with a scaled orthographic projection and finds the rotation matrix and the translation vector of the object by solving a linear system; the second algorithm,POSIT (POS with ITerations), uses in its iteration loop the approximate pose found by POS in order to compute better scaled orthographic projections of the feature points, then applies POS to these projections instead of the original image projections. POSIT converges to accurate pose measurements in a few iterations. POSIT can be used with many feature points at once for added insensitivity to measurement errors and image noise. Compared to classic approaches making use of Newton's method, POSIT does not require starting from an initial guess, and computes the pose using an order of magnitude fewer floating point operations; it may therefore be a useful alternative for real-time operation. When speed is not an issue, POSIT can be written in 25 lines or less in Mathematica; the code is provided in an Appendix.     

What can two images tell us about a third one?

This paper discusses the problem of predicting image features in an image from image features in two other images and the epipolar geometry between the three images. We adopt the most general camera model of perspective projection and show that a point can be predicted in the third image as a bilinear function of its images in the first two cameras, that the tangents to three corresponding curves are related by a trilinear function, and that the curvature of a curve in the third image is a linear function of the curvatures at the corresponding points in the other two images. Our analysis relies heavily on the use of the fundamental matrix which has been recently introduced (Faugeras et al, 1992) and on the properties of a special plane which we call the trifocal plane. Though the trinocular geometry of points and lines has been very recently addressed, our use of the differential properties of curves for prediction is unique.We thus completely solve the following problem: given two views of an object, predict what a third view would look like. The problem and its solution bear upon several areas of computer vision, stereo, motion analysis, and model-based object recognition. Our answer is quite general since it assumes the general perspective projection model for image formation and requires only the knowledge of the epipolar geometry for the triple of views. We show that in the special case of orthographic projection our results for points reduce to those of Ullman and Basri (Ullman and Basri, 1991). We demonstrate on synthetic as well as on real data the applicability of our theory.     

Simultaneous two-view epipolar geometry estimation and motion segmentation by 4D tensor voting

We address the problem of simultaneous two-view epipolar geometry estimation and motion segmentation from nonstatic scenes. Given a set of noisy image pairs containing matches of n objects, we propose an unconventional, efficient, and robust method, 4D tensor voting, for estimating the unknown n epipolar geometries, and segmenting the static and motion matching pairs into n, independent motions. By considering the 4D isotropic and orthogonal joint image space, only two tensor voting passes are needed, and a very high noise to signal ratio (up to five) can be tolerated. Epipolar geometries corresponding to multiple, rigid motions are extracted in succession. Only two uncalibrated frames are needed, and no simplifying assumption (such as affine camera model or homographic model between images) other than the pin-hole camera model is made. Our novel approach consists of propagating a local geometric smoothness constraint in the 4D joint image space, followed by global consistency enforcement for extracting the fundamental matrices corresponding to independent motions. We have performed extensive experiments to compare our method with some representative algorithms to show that better performance on nonstatic scenes are achieved. Results on challenging data sets are presented.     

Zero crossings on lines of curvature

We investigate how much information can be found about the geometry of an object from an image when the general form of the reflection function is known but its specific form is not. We prove theorems showing that the zero crossings of the second directional derivatives occur near the extrema of the curvature along the principal directions of curvature. We next rederive and extend results of Koenderink and van Doorn showing that most extrema of the image intensity lie on parabolic lines. We prove that the directions of the isophotes (the lines of constant image intensity) always lie along the directions of curvature at parabolic lines and hence are photometric invariants. We prove that isophotes which are brighter (or dimmer) than their neighbours must necessarily be parabolic lines.     

Motion Analysis with the Radon Transform on Log-Polar Images

Image projections provide an effective way of describing image contents or estimate particular kinds of motion. However, most (if not all) of previous literature on projections has been done on Cartesian images. In contrast, the work described in this paper is aimed at exploring how projections can be defined on log-polar images and how they perform in estimating motion. In the proposed algorithm, a set of projection signals is computed in two consecutive frames. Then, 1D affine motion between each pair of corresponding projection signals is estimated. Finally, 2D image affine motion is derived from the set of estimated 1D motion parameters, using a 2D-1D motion mapping model (MMM). A reduced, 5-parameter, affine motion model can be estimated with this MMM. The algorithm is implemented in both, log-polar and Cartesian images. Synthetic motion is used for a careful analysis of the strengths and weaknesses of the algorithm. The comparison of the results with log-polar and Cartesian images reveal that the limitations of the approach are due to the MMM, rather than to the inherent difficulties and distortions introduced by the space-variant nature of log-polar images. Another significant finding is that Cartesian images require much more pixels than log-polar images to get comparable estimation performance.

Generalized Thin-Plate Spline Warps

The Thin-Plate Spline warp has been shown to be a very effective parameterized model of the optic flow field between images of various types of deformable surfaces, such as a paper sheet being bent. Recent work has also used such warps for images of a smooth and rigid surface. Standard Thin-Plate Spline warps are however not rigid, in the sense that they do not comply with the epipolar geometry. They are also intrinsically affine, in the sense of the affine camera model, since they are not able to simply model the effect of perspective projection.     

Self-calibration of an affine camera from multiple views

A key limitation of all existing algorithms for shape and motion from image sequences under orthographic, weak perspective and para-perspective projection is that they require the calibration parameters of the camera. We present in this paper a new approach that allows the shape and motion to be computed from image sequences without having to know the calibration parameters. This approach is derived with the affine camera model, introduced by Mundy and Zisserman (1992), which is a more general class of projections including orthographic, weak perspective and para-perspective projection models. The concept of self-calibration, introduced by Maybank and Faugeras (1992) for the perspective camera and by Hartley (1994) for the rotating camera, is then applied for the affine camera.This paper introduces the 3 intrinsic parameters that the affine camera can have at most. The intrinsic parameters of the affine camera are closely related to the usual intrinsic parameters of the pin-hole perspective camera, but are different in the general case. Based on the invariance of the intrinsic parameters, methods of self-calibration of the affine camera are proposed. It is shown that with at least four views, an affine camera may be self-calibrated up to a scaling factor, leading to Euclidean (similarity) shape réconstruction up to a global scaling factor. Another consequence of the introduction of intrinsic and extrinsic parameters of the affine camera is that all existing algorithms using calibrated affine cameras can be assembled into the same framework and some of them can be easily extented to a batch solution.Experimental results are presented and compared with other methods using calibrated affine cameras.     

Epipolar geometry from profiles under circular motion

Addresses the problem of motion estimation from profiles (apparent contours) of an object rotating on a turntable in front of a single camera. A practical and accurate technique for solving this problem from profiles alone is developed. It is precise enough to reconstruct the shape of the object. No correspondences between points or lines are necessary. Symmetry of the surface of revolution swept out by the rotating object is exploited to obtain the image of the rotation axis and the homography relating epipolar lines in two views robustly and elegantly. These, together with geometric constraints for images of rotating objects, are used to obtain first the image of the horizon, which is the projection of the plane that contains the camera centers, and then the epipoles, thus fully determining the epipolar geometry of the image sequence. The estimation of this geometry by this sequential approach avoids many of the problems found in other algorithms. The search for the epipoles, by far the most critical step, is carried out as a simple 1D optimization. Parameter initialization is trivial and completely automatic at all stages. After the estimation of the epipolar geometry, the Euclidean motion is recovered using the fixed intrinsic parameters of the camera obtained either from a calibration grid or from self-calibration techniques. Finally, the spinning object is reconstructed from its profiles using the motion estimated in the previous stage. Results from real data are presented, demonstrating the efficiency and usefulness of the proposed methods     

3-D interpretation of optical flow by renormalization

This article studies 3-D interpretation of optical flow induced by a general camera motion relative to a surface of general shape. First, we describe, using the image sphere representation, an analytical procedure that yields an exact solution when the data are exact: we solve theepipolar equation written in terms of theessential parameters and thetwisted optical flow. Introducing a simple model of noise, we then show that the solution is statistically biased. In order to remove the statistical bias, we propose an algorithm calledrenormalization, which automatically adjusts to unknown image noise. A brief discussion is also given to thecritical surface that yields ambiguous 3-D interpretations and the use of theimage plane representation.     

Following Cusps

It is known that the deformation of the apparent contours of a surface under perspective projection and viewer motion enable the recovery of the geometry of the surface, for example by utilising the epipolar parametrization. These methods break down with apparent contours that are singular i.e., with cusps . In this paper we study this situation and show how, nevertheless, the surface geometry (including the Gauss curvature and mean curvature of the surface) can be recovered by following the cusps. Indeed the formulae are much simpler in this case and require lower spatio-temporal derivatives than in the general case of nonsingular apparent contours. We also show that following cusps does not by itself provide us with information on viewer motion.     

A Multibody Factorization Method for Independently Moving Objects

The structure-from-motion problem has been extensively studied in the field of computer vision. Yet, the bulk of the existing work assumes that the scene contains only a single moving object. The more realistic case where an unknown number of objects move in the scene has received little attention, especially for its theoretical treatment. In this paper we present a new method for separating and recovering the motion and shape of multiple independently moving objects in a sequence of images. The method does not require prior knowledge of the number of objects, nor is dependent on any grouping of features into an object at the image level. For this purpose, we introduce a mathematical construct of object shapes, called the shape interaction matrix, which is invariant to both the object motions and the selection of coordinate systems. This invariant structure is computable solely from the observed trajectories of image features without grouping them into individual objects. Once the matrix is computed, it allows for segmenting features into objects by the process of transforming it into a canonical form, as well as recovering the shape and motion of each object. The theory works under a broad set of projection models (scaled orthography, paraperspective and affine) but they must be linear, so it excludes projective cameras.     

基于透视和正投影图像的3D运动与结构恢复

论文首次研究了由一幅正投影图像和一幅透视投影图像的特征点对应来进行刚体3D运动重建与结构恢复的问题,给出了有效的线性算法。以往的由运动恢复结构的工作主要集中于一组透视图像或一组正投影(通常为仿射)图像,文中采用了透视模型和正投影模型的组合。数据模拟实验结果显示该方法是比较有效和稳定的。     

3-D Reconstruction of Urban Scenes from Image Sequences

In this paper, we address the problem of the recovery of a realistic textured model of a scene from a sequence of images, without any prior knowledge either about the parameters of the cameras or about their motion. We do not require any knowledge of the absolute coordinates of some control points in the scene to achieve this goal. First, using various computer vision tools, we establish correspondences between the images and recover the epipolar geometry, from which we show how to compute the complete set of perspective projection matrices for all camera positions. Then, we proceed to reconstruct the geometry of the scene. We show how to rely on information of the scene such as parallel lines or known angles in order to reconstruct the geometry of the scene up to, respectively, an unknown affine transformation or an unknown similitude. Alternatively, if this information is not available, we can still recover the Euclidean structure of the scene through the techniques of self-calibration. The scene geometry is modeled as a set of polyhedra. Textures to be mapped on the scene polygons are extracted automatically from the images. We show how several images can be combined through mosaicing in order to automatically remove visual artifacts such as pedestrians or trees from the textures.This vision system has been implemented as a vision server, which provides to a CAD-CAM modeler geometry or texture information extracted from the set of images. The whole system allows efficient and fast production of scene models of high quality for such applications as simulation, virtual, or augmented reality.     

Motion Recovery by Integrating over the Joint Image Manifold

Recovery of epipolar geometry is a fundamental problem in computer vision. The introduction of the “joint image manifold” (JIM) allows to treat the recovery of camera motion and epipolar geometry as the problem of fitting a manifold to the data measured in a stereo pair. The manifold has a singularity and boundary, therefore special care must be taken when fitting it. Four fitting methods are discussed—direct, algebraic, geometric, and the integrated maximum likelihood (IML) based method. The first three methods are the exact analogues of three common methods for recovering epipolar geometry. The more recently introduced IML method seeks the manifold which has the highest “support,” in the sense that the largest measure of its points are close to the data. While computationally more intensive than the other methods, its results are better in some scenarios. Both simulations and experiments suggest that the advantages of IML manifold fitting carry over to the task of recovering epipolar geometry, especially when the extent of the data and/or the motion are small.     

Rigid body segmentation and shape description from dense opticalflow under weak perspective

We present an algorithm for identifying and tracking independently moving rigid objects from optical flow. Some previous attempts at segmentation via optical flow have focused on finding discontinuities in the flow field. While discontinuities do indicate a change in scene depth, they do not in general signal a boundary between two separate objects. The proposed method uses the fact that each independently moving object has a unique epipolar constraint associated with its motion. Thus motion discontinuities based on self-occlusion can be distinguished from those due to separate objects. The use of epipolar geometry allows for the determination of individual motion parameters for each object as well as the recovery of relative depth for each point on the object. The algorithm assumes an affine camera where perspective effects are limited to changes in overall scale. No camera calibration parameters are required. A Kalman filter based approach is used for tracking motion parameters with time