Recovery of the 3-D location and motion of a rigid object through camera image (An Extended Kalman Filter Approach)

This paper deals with the problem of locating a rigid object and estimating its motion in three dimensions. This involves determining the position and orientation of the object at each instant when an image is captured by a camera, and recovering the motion of the object between consecutive frames.In the implementation scheme used here, a sequence of camera images, digitized at the sample instants, is used as the initial input data. Measurements are made of the locations of certain features (e.g., maximum curvature points of an image contour, corners, edges, etc.) on the 2-D images. To measure the feature locations a matching algorithm is used, which produces correspondences between the features in the image and the object.Using the measured feature locations on the image, an algorithm is developed to solve the location and motion problem. The algorithm is an extended Kalman filter modeled for this application.Department of Electrical Engineering and Alberta Center for Machine Intelligence and Robotics, University of Alberta     

Iterative Pose Computation from Line Correspondences

This paper presents a method for estimating the position and orientation of a camera with respect to a known 3-D object from line correspondences. The main idea of the method is to estimate a pose with either a weak perspective or a paraperspective camera model and to improve this pose iteratively. At convergence the result is compatible with a perspective camera model. This iterative improvement of a linear (affine) camera model has already been used for points but has never been extended to lines. Known methods which compute pose from line correspondences deal with a set of nonlinear equations which are solved either in closed-form or using minimization techniques. These methods have to deal with multiple solutions. In contrast our method starts with a solution which is very close to the true solution and converges in very few iterations (typically three to five iterations). The rank analysis of the linear system to be solved at each iteration allows us to characterize geometric configurations which defeat the algorithm.     

Orientation in manhattan: equiprojective classes and sequential estimation

The problem of inferring 3D orientation of a camera from video sequences has been mostly addressed by first computing correspondences of image features. This intermediate step is now seen as the main bottleneck of those approaches. In this paper, we propose a new 3D orientation estimation method for urban (indoor and outdoor) environments, which avoids correspondences between frames. The scene property exploited by our method is that many edges are oriented along three orthogonal directions; this is the recently introduced Manhattan world (MW) assumption. The main contributions of this paper are: the definition of equivalence classes of equiprojective orientations, the introduction of a new small rotation model, formalizing the fact that the camera moves smoothly, and the decoupling of elevation and twist angle estimation from that of the compass angle. We build a probabilistic sequential orientation estimation method, based on an MW likelihood model, with the above-listed contributions allowing a drastic reduction of the search space for each orientation estimate. We demonstrate the performance of our method using real video sequences.     

Two-View Orthographic Epipolar Geometry: Minimal and Optimal Solvers

We will in this paper present methods and algorithms for estimating two-view geometry based on an orthographic camera model. We use a previously neglected nonlinear criterion on rigidity to estimate the calibrated essential matrix. We give efficient algorithms for estimating it minimally (using only three point correspondences), in a least squares sense (using four or more point correspondences), and optimally with respect to the number of inliers. The inlier-optimal algorithm is based on a three-point solver and gives a fourth-order polynomial time algorithm. These methods can be used as building blocks to robustly find inlier correspondences in the presence of high degrees of outliers. We show experimentally that our methods can be used in many instances, where the orthographic camera model isn’t generally used. A case of special interest is situations with repetitive structures, which give high amounts of outliers in the initial feature point matching.     

一种基于主动视觉的三维结构恢复和直接欧氏重建算法

利用三正交平移运动, 提出了一种三维结构恢复和直接欧氏重建新算法. 算法仅需利用主动视觉平台控制相机作一组三正交平移运动, 然后通过图像对应点和平移运动的距离就可以恢复平面结构信息和进行欧氏重建. 并且无需假定相机畸变因子为零. 算法计算过程中无需求解相机的内参数, 也无需进行分层重构, 它是一种直接的欧氏重建算法, 避免了传统算法中的相机标定、仿射重建等两大难题, 并且计算过程完全线性化, 简单实用. 最后用模拟实验和真实图像实验对算法进行验证, 实验结果表明了算法的有效性和准确性.     

Tracking 3-D motion from straight lines with trifocal tensors

We present a novel approach to track the position and orientation of a stereo camera using line features in the images. The method combines the strengths of trifocal tensors and Bayesian filtering. The trifocal tensor provides a geometric constraint to lock line features among every three frames. It eliminates the explicit reconstruction of the scene even if the 3-D scene structure is not known. Such a trifocal constraint thus makes the algorithm fast and robust. The twist motion model is applied to further improve its computation efficiency. Another major contribution is that our approach can obtain the 3-D camera motion using as little as 2 line correspondences instead of 13 in the traditional approaches. This makes the approach attractive for realistic applications. The performance of the proposed method has been evaluated using both synthetic and real data with encouraging results. Our algorithm is able to estimate 3-D camera motion in real scenarios accurately having little drifting from an image sequence longer than a 1,000 frames.     

Egomotion estimation of a range camera using the space envelope

In this paper we present a method to compute the egomotion of a range camera using the space envelope. The space envelope is a geometric model that provides more information than a simple segmentation for correspondences and motion estimation. We describe a novel variation of the maximal matching algorithm that matches surface normals to find correspondences. These correspondences are used to compute rotation and translation estimates of the egomotion. We demonstrate our methods on two image sequences containing 70 images. We also discuss the cases where our methods fail, and additional possible methods for exploiting the space envelope.     

Extracting 3D facial animation parameters from multiview video clips

We propose an accurate and inexpensive procedure that estimates 3D facial motion parameters from mirror-reflected multiview video clips. We place two planar mirrors near a subject's cheeks and use a single camera to simultaneously capture a marker's front and side view images. We also propose a novel closed-form linear algorithm to reconstruct 3D positions from real versus mirrored point correspondences in an uncalibrated environment. Our computer simulations reveal that exploiting mirrors' various reflective properties yields a more robust, accurate, and simpler 3D position estimation approach than general-purpose stereo vision methods that use a linear approach or maximum-likelihood optimization. Our experiments show a root mean square (RMS) error of less than 2 mm in 3D space with only 20-point correspondences. For semiautomatic 3D motion tracking, we use an adaptive Kalman predictor and filter to improve stability and infer the occluded markers' position. Our approach tracks more than 50 markers on a subject's face and lips from 30-frame-per-second video clips. We've applied the facial motion parameters estimated from the proposed method to our facial animation system.     

Linear N-point camera pose determination

The determination of camera position and orientation from known correspondences of 3D reference points and their images is known as pose estimation in computer vision and space resection in photogrammetry. It is well-known that from three corresponding points there are at most four algebraic solutions. Less appears to be known about the cases of four and five corresponding points. We propose a family of linear methods that yield a unique solution to 4- and 5-point pose determination for generic reference points. We first review the 3-point algebraic method. Then we present our two-step, 4-point and one-step, 5-point linear algorithms. The 5-point method can also be extended to handle more than five points. Finally, we demonstrate our methods on both simulated and real images. We show that they do not degenerate for coplanar configurations and even outperform the special linear algorithm for coplanar configurations in practice     

Rainbow Flash Camera: Depth Edge Extraction Using Complementary Colors

We present a novel color multiplexing method for extracting depth edges in a scene. It has been shown that casting shadows from different light positions provides a simple yet robust cue for extracting depth edges. Instead of flashing a single light source at a time as in conventional methods, our method flashes all light sources simultaneously to reduce the number of captured images. We use a ring light source around a camera and arrange colors on the ring such that the colors form a hue circle. Since complementary colors are arranged at any position and its antipole on the ring, shadow regions where a half of the hue circle is occluded are colorized according to the orientations of depth edges, while non-shadow regions where all the hues are mixed have a neutral color in the captured image. Thus the colored shadows in the single image directly provide depth edges and their orientations in an ideal situation. We present an algorithm that extracts depth edges from a single image by analyzing the colored shadows. We also present a more robust depth edge extraction algorithm using an additional image captured by rotating the hue circle with \(180^\circ \) to compensate for scene textures and ambient lights. We compare our approach with conventional methods for various scenes using a camera prototype consisting of a standard camera and 8 color LEDs. We also demonstrate a bin-picking system using the camera prototype mounted on a robot arm.     

Determination of camera location from 2-D to 3-D line and pointcorrespondences

A method for the determination of camera location from two-dimensional (2-D) to three-dimensional (3-D) straight line or point correspondences is presented. With this method, the computations of the rotation matrix and the translation vector of the camera are separable. First, the rotation matrix is found by a linear algorithm using eight or more line correspondences, or by a nonlinear algorithm using three or more line correspondences, where the line correspondences are either given or derived from point correspondences. Then, the translation vector is obtained by solving a set of linear equations based on three or more line correspondences, or two or more point correspondences. Eight 2-D to 3-D line correspondences or six 2-D to 3-D point correspondences are needed for the linear approach; three 2-D to 3-D line or point correspondences for the nonlinear approach. Good results can be obtained in the presence of noise if more than the minimum required number of correspondences are used     

单目同时定位与建图中的地图恢复融合技术

目的 传统的单目视觉SLAM（simultaneous localization and mapping）跟踪失败后需要相机重新回到丢失的位置才能重定位并恢复建图,这极大限制了单目SLAM的应用场景。为解决这一问题,提出一种基于视觉惯性传感器融合的地图恢复融合算法。方法 当系统跟踪失败,仅由惯性传感器提供相机位姿,通过对系统重新初始化并结合惯性传感器提供的丢失部分的相机位姿将丢失前的地图融合到当前的地图中;为解决视觉跟踪丢失期间由惯性测量计算导致的相机位姿误差,提出了一种以关键帧之间的共视关系为依据的跳跃式的匹配搜索策略,快速获得匹配地图点,再通过非线性优化求解匹配点之间的运动估计,进行误差补偿,获得更加准确的相机位姿,并删减融合后重复的点云;最后建立前后两个地图中关键帧之间与地图点之间的联系,用于联合优化后续的跟踪建图过程中相机位姿和地图点位置。结果 利用Euroc数据集及其他数据进行地图精度和地图完整性实验,在精度方面,将本文算法得到的轨迹与ground truth和未丢失情况下得到的轨迹进行对比,结果表明,在SLAM系统跟踪失败的情况下,此方法能有效解决系统无法继续跟踪建图的问题,其精度可达厘米级别。在30 m²的室内环境中,仅有9 cm的误差,而在300 m²工厂环境中误差仅有7 cm。在完整性方面,在相机运动较剧烈的情况下,恢复地图的完整性优于ORB_SLAM的重定位算法,通过本文算法得到的地图关键帧数量比ORB_SLAM多30%。结论 本文提出的算法在单目视觉SLAM系统跟踪失败之后,仍然能够继续跟踪建图,不会丢失相机轨迹。此外,无需相机回到丢失之前的场景中,只需相机观察到部分丢失前场景,即可恢复融合所有地图。本文算法不仅保证了恢复地图的精度,还保证了建图的完整性。与传统的重定位方法相比,本文算法在系统建图较少时跟踪失败的情况下效果更好。     

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.     

基于胶囊内窥镜图像的肠胃道三维重建技术

为了改进胶囊内窥镜观测的准确性和真实性,提出了基于胶囊内窥镜序列图像的胃肠道三维重建的方法.首先利用SIFT算法提取前后两幅序列图像中尽可能多的对应特征点;计算获取各特征点在成像面上的二维坐标;进一步利用8点算法计算胶囊内镜运动变化的旋转矩阵和平移矢量.进而计算得到每个特征点的相对三维坐标和世界三维坐标;然后,采用Delaunay三角剖分算法对各三维点进行网格化,并完成场景的三维重建.实验表明相机与被测点距离在100 mm之内时,得到的深度误差小于1 mm;距离250 mm内时,相对误差在3％之内.说明所提出的算法是可行的.     

Estimating parameters of noncentral catadioptric systems using bundle adjustment

This paper describes a new method to calibrate the intrinsic and extrinsic parameters of a generalized catadioptric camera (central or noncentral). The algorithm has two steps. The first one is the estimation of correspondences between incident lines in space and pixels (black box model calibration) in an arbitrary world reference frame. The second step is the calibration of the intrinsic parameters of the pinhole camera, the coefficients of the mirror expressed by a quadric (quadric mirror shape and the pose of the camera in relation to it), the position of the optical center of the camera in the world reference frame and its relative orientation (pose of the camera in world reference frame). A projection model relaxing Snell’s Law is derived. The deviations from Snell’s Law and the image reprojection errors are minimized by means of bundle adjustment. Information about the apparent contour of the mirror can be used to reduce the uncertainty in the estimation by introducing a new term in the cost function of the second step minimization process. Simulations and real experiments show good accuracy and robustness for this framework. However, the convergence is dependent on the initial guess as expected. A well-behaved algorithm to automatically generate the initial estimate to be used in the bundle adjustment is also presented.     

3D Pose Estimation by Directly Matching Polyhedral Models to Gray Value Gradients

This contribution addresses the problem of pose estimation and tracking of vehicles in image sequences from traffic scenes recorded by a stationary camera. In a new algorithm, the vehicle pose is estimated by directly matching polyhedral vehicle models to image gradients without an edge segment extraction process. The new approach is significantly more robust than approaches that rely on feature extraction since the new approach exploits more information from the image data. We successfully tracked vehicles that were partially occluded by textured objects, e.g., foliage, where a previous approach based on edge segment extraction failed. Moreover, the new pose estimation approach is also used to determine the orientation and position of the road relative to the camera by matching an intersection model directly to image gradients. Results from various experiments with real world traffic scenes are presented.     

Self-identifying patterns for plane-based camera calibration

Determining camera calibration parameters is a time-consuming task despite the availability of calibration algorithms and software. A set of correspondences between points on the calibration target and the camera image(s) must be found, usually a manual or manually guided process. Most calibration tools assume that the correspondences are already found. We present a system which allows a camera to be calibrated merely by passing it in front of a panel of self-identifying patterns. This calibration scheme uses an array of fiducial markers which are detected with a high degree of confidence, each detected marker provides one or four correspondence points. Experiments were performed calibrating several cameras in a short period of time with no manual intervention. This marker-based calibration system was compared to one using the OpenCV chessboard grid finder which also finds correspondences automatically. We show how our new marker-based system more robustly finds the calibration pattern and how it provides more accurate intrinsic camera parameters.     

The Problem of Degeneracy in Structure and Motion Recovery from Uncalibrated Image Sequences

The aim of this work is the recovery of 3D structure and camera projection matrices for each frame of an uncalibrated image sequence. In order to achieve this, correspondences are required throughout the sequence. A significant and successful mechanism for automatically establishing these correspondences is by the use of geometric constraints arising from scene rigidity. However, problems arise with such geometry guided matching if general viewpoint and general structure are assumed whilst frames in the sequence and/or scene structure do not conform to these assumptions. Such cases are termed degenerate.In this paper we describe two important cases of degeneracy and their effects on geometry guided matching. The cases are a motion degeneracy where the camera does not translate between frames, and a structure degeneracy where the viewed scene structure is planar. The effects include the loss of correspondences due to under or over fitting of geometric models estimated from image data, leading to the failure of the tracking method. These degeneracies are not a theoretical curiosity, but commonly occur in real sequences where models are statistically estimated from image points with measurement error.We investigate two strategies for tackling such degeneracies: the first uses a statistical model selection test to identify when degeneracies occur: the second uses multiple motion models to overcome the degeneracies. The strategies are evaluated on real sequences varying in motion, scene type, and length from 13 to 120 frames.     

基于山体轮廓匹配的相机空间定位方法研究

相机空间定位是一个经典问题。本文提出了一种匹配图像轮廓线的相机空间定位方法。这种方法可以在没有标定物的情况下,利用已有的地理信息进行相机的空间定位。实验证明,该方法可以发展成为一种行之有效的计算机辅助空间定位工具。