Conics-based stereo,motion estimation,and pose determination

Stereo vision, motion and structure parameter estimation, and pose determination are three important problems in 3-D computer vision. The first step in all of these problems is to choose and to extract primitives and their features in images. In most of the previous work, people usually use edge points or straight line segments as primitives and their local properties as features. Few methods have been presented in the literature using more compact primitives and their global features. This article presents an approach using conics as primitives. For stereo vision, a closed-form solution is provided for both establishing the correspondence of conics in images and the reconstruction of conics in space. With this method, the correspondence is uniquely determined and the reconstruction is global. It is shown that the method can be extended for higher degree (degree3) planar curves.For motion and structure parameter estimation, it is shown that, in general, two sequential images of at least three conics are needed in order to determine the camera motion. A complicated nonlinear system must be solved in this case. In particular, if we are given two images of a pair of coplanar conics, a closed-form solution of camera motion is presented. In a CAD-based vision system, the object models are available, and this makes it possible to recognize 3-D objects and to determine their poses from a single image.For pose determination, it is shown that if there exist two conics on the surface of an object, the object's pose can be determined by an efficient one-dimensional search. In particular, if two conics are coplanar, a closed-form solution of the object's pose is presented.Uniqueness analysis and some experiments with real or synthesized data are presented in this article.     

Hypercatadioptric line images for 3D orientation and image rectification

In central catadioptric systems 3D lines are projected into conics. In this paper we present a new approach to extract conics in the raw catadioptric image, which correspond to projected straight lines in the scene. Using the internal calibration and two image points we are able to compute analytically these conics which we name hypercatadioptric line images. We obtain the error propagation from the image points to the 3D line projection in function of the calibration parameters. We also perform an exhaustive analysis on the elements that can affect the conic extraction accuracy. Besides that, we exploit the presence of parallel lines in man-made environments to compute the dominant vanishing points (VPs) in the omnidirectional image. In order to obtain the intersection of two of these conics we analyze the self-polar triangle common to this pair. With the information contained in the vanishing points we are able to obtain the 3D orientation of the catadioptric system. This method can be used either in a vertical stabilization system required by autonomous navigation or to rectify images required in applications where the vertical orientation of the catadioptric system is assumed. We use synthetic and real images to test the proposed method. We evaluate the 3D orientation accuracy with a ground truth given by a goniometer and with an inertial measurement unit (IMU). We also test our approach performing vertical and full rectifications in sequences of real images.     

Robust omnidirectional mobile robot topological navigation system using omnidirectional vision

Robust topological navigation strategy for omnidirectional mobile robot using an omnidirectional camera is described. The navigation system is composed of on-line and off-line stages. During the off-line learning stage, the robot performs paths based on motion model about omnidirectional motion structure and records a set of ordered key images from omnidirectional camera. From this sequence a topological map is built based on the probabilistic technique and the loop closure detection algorithm, which can deal with the perceptual aliasing problem in mapping process. Each topological node provides a set of omnidirectional images characterized by geometrical affine and scale invariant keypoints combined with GPU implementation. Given a topological node as a target, the robot navigation mission is a concatenation of topological node subsets. In the on-line navigation stage, the robot hierarchical localizes itself to the most likely node through the robust probability distribution global localization algorithm, and estimates the relative robot pose in topological node with an effective solution to the classical five-point relative pose estimation algorithm. Then the robot is controlled by a vision based control law adapted to omnidirectional cameras to follow the visual path. Experiment results carried out with a real robot in an indoor environment show the performance of the proposed method.     

On the Importance of Being Asymmetric in Stereopsis—Or Why We Should Use Skewed Parallel Cameras

This paper presents a new clever camera sensor, where relative pose determination is not needed, and the sensor is simultaneously capable of using vergence micromovements. Sweeping depth using vergence micromovements promises subpixel depth precision, measuring zero disparity at each time instant. We show that curves preserving zero disparity are exactly conics, nondegenerate or degenerate. Oddly enough, only circles (Vieth-Müller circles) are routinely considered, either theoretically or in practical work, in vergence stereo. Horopters in human vision, cf. Ogle (1932), closely resemble conics.We introduce translational vergence by suggesting the use of a pair of shift-optics CCD cameras. The nonrigidity causes zero disparity curves to become planes, for each fixation. (They are degenerate conics.) We have parallel optical axes, but slanting left and right primary lines of sight. During vergence movements, the primary lines of sight move over time. This has farreaching consequences: Binocular head-eye systems all involve relative camera rotation, to fixate. But, camera rotation is unnecessary. Hence, for relative depth maps, there is no need for measuring camera rotation (relative camera pose) from mechanical sources. Nor are algorithms needed for calculating epipolar lines. The suggested technique removes the need for camera rotations about the optical centers in a binocular head-eye system.     

Robotic Hand Pose Estimation Based on Stereo Vision and GPU-enabled Internal Graphical Simulation

Humanoid robots have complex kinematic chains whose modeling is error prone. If the robot model is not well calibrated, its hand pose cannot be determined precisely from the encoder readings, and this affects reaching and grasping accuracy. In our work, we propose a novel method to simultaneously i) estimate the pose of the robot hand, and ii) calibrate the robot kinematic model. This is achieved by combining stereo vision, proprioception, and a 3D computer graphics model of the robot. Notably, the use of GPU programming allows to perform the estimation and calibration in real time during the execution of arm reaching movements. Proprioceptive information is exploited to generate hypotheses about the visual appearance of the hand in the camera images, using the 3D computer graphics model of the robot that includes both kinematic and texture information. These hypotheses are compared with the actual visual input using particle filtering, to obtain both i) the best estimate of the hand pose and ii) a set of joint offsets to calibrate the kinematics of the robot model. We evaluate two different approaches to estimate the 6D pose of the hand from vision (silhouette segmentation and edges extraction) and show experimentally that the pose estimation error is considerably reduced with respect to the nominal robot model. Moreover, the GPU implementation ensures a performance about 3 times faster than the CPU one, allowing real-time operation.     

Optimization and Rapid Prototyping of Catadioptric Omnidirectional Stereo Sensors

Omnidirectional stereo imaging provides useful depth information for autonomous navigation. In principle, omnidirectional stereo images can be achieved at a low cost, using a single camera and two curved mirrors, but such systems are not widely deployed. Here we describe the optimization and rapid prototyping of a low-cost omnidirectional stereo sensor for a telepresence robot. We consider single-viewpoint and non-single-viewpoint designs. We present a new way of relaxing single-viewpoint constraints, while retaining the ability to vertically rectify images. We also present a method of optimizing the resulting design to minimize depth errors. However, we show that despite these steps, nonsingle-viewpoint designs produce stereo disparities over a more practical range of distances. The lack of a single viewpoint could potentially introduce distortions that affect stereo matching, but these distortions are removed by projecting rays through the mirror geometry. We also describe a new method for rapid prototyping of curved mirrors using 3D printing and vacuum forming.     

Extrinsic calibration of heterogeneous cameras by line images

The extrinsic calibration refers to determining the relative pose of cameras. Most of the approaches for cameras with non-overlapping fields of view (FOV) are based on mirror reflection, object tracking or rigidity constraint of stereo systems whereas cameras with overlapping FOV can be calibrated using structure from motion solutions. We propose an extrinsic calibration method within structure from motion framework for cameras with overlapping FOV and its extension to cameras with partially non-overlapping FOV. Recently, omnidirectional vision has become a popular topic in computer vision as an omnidirectional camera can cover large FOV in one image. Combining the good resolution of perspective cameras and the wide observation angle of omnidirectional cameras has been an attractive trend in multi-camera system. For this reason, we present an approach which is applicable to heterogeneous types of vision sensors. Moreover, this method utilizes images of lines as these features possess several advantageous characteristics over point features, especially in urban environment. The calibration consists of a linear estimation of orientation and position of cameras and optionally bundle adjustment to refine the extrinsic parameters.     

Graph cut based panoramic 3D modeling and ground truth comparison with a mobile platform – The Wägele

Efficient and comfortable acquisition of large 3D scenes is an important topic for many current and future applications in the field of robotics, factory and office visualization, 3DTV and cultural heritage.In this paper we present both an omnidirectional stereo vision approach for 3D modeling based on graph cut techniques and also a new mobile 3D model acquisition platform where it is employed. The platform comprises a panoramic camera and a 2D laser range scanner for self localization by scan matching. 3D models are acquired just by moving the platform around and recording images in regular intervals. Additionally, we concurrently build 3D models using two supplementary laser range scanners. This enables the investigation of the stereo algorithm’s quality by comparing it with the laser scanner based 3D model as ground truth. This offers a more objective point of view on the achieved 3D model quality.     

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.     

Stereo vision based automation for a bin-picking solution

As flexibility becomes an important factor in factory automation, the bin-picking system, where a robot performs pick-and-place tasks for randomly piled parts in a bin through measuring the 3D pose of an object by a 3D vision sensor, has been actively studied. However, conventional bin-picking systems that are employed for particular tasks are limited by such things as the FOV (Field of View), the shape of landmark features, and computation time. This paper proposes a general-purpose stereo vision based bin-picking system. To detect the workpiece to be picked, a geometric pattern matching (GPM) method with respect to the 2D image with a wide FOV is applied. The accurate 3D pose of a selected workpiece among the pick-up candidates is acquired by measuring the 3D positions of three features in the workpiece using the stereo camera. In order to improve the 3D position estimation performance, the GPM method is also used instead of the stereo matching method. The multiple pattern registration and ellipse fitting techniques are additionally applied to increase the reliability. The grasp position of a workpiece without collision is determined using the pose of the object and the bin information. By using these methods a practical bin-picking strategy is established to operate robustly with minimum help from the human workers in the factory. Through experiments on commercial industrial workpieces and industrial robot, we validated that the proposed vision system accurately measures the 3D pose of part and the robot successfully manipulates the workpiece among randomly stacked parts.     

Omnidirectional vision-based ego-pose estimation for an autonomous in-pipe mobile robot

This paper describes the omnidirectional vision-based ego-pose estimation method of an in-pipe mobile robot. An in-pipe mobile robot has been developed for inspecting the inner surface of various pipeline configurations, such as the straight pipeline, the elbow and the multiple-branch. Because the proposed in-pipe mobile robot has four individual drive wheels, it has the ability of flexible motions in various pipelines. The ego-pose estimation is indispensable for the autonomous navigation of the proposed in-pipe robot. An omnidirectional camera and four laser modules mounted on the mobile robot are used for ego-pose estimation. An omnidirectional camera is also used for investigating the inner surface of the pipeline. The pose of the in-pipe mobile robot is estimated from the relationship equation between the pose of a robot and the pixel coordinates of four intersection points where light rays that emerge from four laser modules intersect the inside of the pipeline. This relationship equation is derived from the geometry analysis of an omnidirectional camera and four laser modules. In experiments, the performance of the proposed method is evaluated by comparing the result of our algorithm with the measurement value of a specifically designed sensor, which is a kind of a gyroscope.     

Wide-baseline stereo vision for terrain mapping

Terrain mapping is important for mobile robots to perform localization and navigation. Stereo vision has been used extensively for this purpose in outdoor mapping tasks. However, conventional stereo does not scale well to distant terrain. This paper examines the use of wide-baseline stereo vision in the context of a mobile robot for terrain mapping, and we are particularly interested in the application of this technique to terrain mapping for Mars exploration. In wide-baseline stereo, the images are not captured simultaneously by two cameras, but by a single camera at different positions. The larger baseline allows more accurate depth estimation of distant terrain, but the robot motion between camera positions introduces two new problems. One issue is that the robot estimates the relative positions of the camera at the two locations imprecisely, unlike the precise calibration that is performed in conventional stereo. Furthermore, the wide-baseline results in a larger change in viewpoint than in conventional stereo. Thus, the images are less similar and this makes the stereo matching process more difficult. Our methodology addresses these issues using robust motion estimation and feature matching. We give results using real images of terrain on Earth and Mars and discuss the successes and failures of the technique.     

Recognition and movement in an artificial environment with a bipedal robot

However well we control a walking bipedal robot, the images obtained by a camera are tilted to the left or right, and have small irregularities. This complicates the recognition of an environment by using a camera in a walking robot when the robot cannot move smoothly. The reason for using a bipedal robot is to make the robot as similar as possible to a human in body shape and behavior in order to make collaboration easier. This is difficult to attain with other types of robot such as wheel-driven robots (Sato et al. AROB2008; Fujiwara et al. WMSCI2009). In an artificial environment which mainly consists of vertical and horizontal lines, the tilt angle of camera images must be corrected by using the Hough transformation, which detects lines which are nearly vertical (Okutomi et al. 2004; Forsyth and Ponce 2007). As a result, the robot can successfully recognize the environment with stereo vision using images obtained by correcting the tilted ones.     

Omnidirectional Image Stabilization for Visual Object Recognition

In this paper, we present a pipeline for camera pose and trajectory estimation, and image stabilization and rectification for dense as well as wide baseline omnidirectional images. The proposed pipeline transforms a set of images taken by a single hand-held camera to a set of stabilized and rectified images augmented by the computed camera 3D trajectory and a reconstruction of feature points facilitating visual object recognition. The paper generalizes previous works on camera trajectory estimation done on perspective images to omnidirectional images and introduces a new technique for omnidirectional image rectification that is suited for recognizing people and cars in images. The performance of the pipeline is demonstrated on real image sequences acquired in urban as well as natural environments.     

Cooperative 3D tunnel measurement based on 2D–3D registration of omnidirectional laser light

In this study, we proposed a high-density three-dimensional (3D) tunnel measurement method, which estimates the pose changes of cameras based on a point set registration algorithm regarding 2D and 3D point clouds. To detect small deformations and defects, high-density 3D measurements are necessary for tunnel construction sites. The line-structured light method uses an omnidirectional laser to measure a high-density cross-section point cloud from camera images. To estimate the pose changes of cameras in tunnels, which have few textures and distinctive shapes, cooperative robots are useful because they estimate the pose by aggregating relative poses from the other robots. However, previous studies mounted several sensors for both the 3D measurement and pose estimation, increasing the size of the measurement system. Furthermore, the lack of 3D features makes it difficult to match point clouds obtained from different robots. The proposed measurement system consists of a cross-section measurement unit and a pose estimation unit; one camera was mounted for each unit. To estimate the relative poses of the two cameras, we designed a 2D–3D registration algorithm for the omnidirectional laser light, and implemented hand-truck and unmanned aerial vehicle systems. In the measurement of a tunnel with a width of 8.8 m and a height of 6.4 m, the error of the point cloud measured by the proposed method was 162.8 and 575.3 mm along 27 m, respectively. In a hallway measurement, the proposed method generated less errors in straight line shapes with few distinctive shapes compared with that of the 3D point set registration algorithm with Light Detection and Ranging.     

Omnidirectional texturing based on robust 3D registration through Euclidean reconstruction from two spherical images

We propose a semi-automatic omnidirectional texturing method that maps a spherical image onto a dense 3D model obtained by a range sensor. For accurate texturing, accurate estimation of the extrinsic parameters is inevitable. In order to estimate these parameters, we propose a robust 3D registration-based method between a dense range data set and a sparse spherical image stereo data set. For measuring the distances between the two data sets, we introduce generalized distances taking account of 3D error distributions of the stereo data. To reconstruct 3D models by images, we use two spherical images taken at arbitrary positions in arbitrary poses. Then, we propose a novel rectification method for spherical images that is derived from E matrix and facilitates the estimation of the disparities. The experimental results show that the proposed method can map the spherical image onto the dense 3D models effectively and accurately.     

A 2-point algorithm for 3D reconstruction of horizontal lines from a single omni-directional image

Reconstruction of 3D scenes with abundant straight line features has many applications in computer vision and robot navigation. Most approaches to this problem involve stereo techniques, in which a solution to the correspondence problem between at least two different images is required. In contrast, 3D reconstruction of straight horizontal lines from a single 2D omni-directional image is studied in this paper. The authors show that, for symmetric non-central catadioptric systems, a 3D horizontal line can be estimated using only two points extracted from a single image of the line. One of the two points is the symmetry point of the image curve of horizontal line, and the other is a generic point on the image curve. This paper improves on several prior works, including horizontal line detection in omni-directional image and line reconstruction from four viewing rays, but is simpler than those methods while being more robust. We evaluate how the precision of feature point extraction can affect line reconstruction accuracy, and discuss preliminary experimental results.     

Fully Vision-based Calibration of a Hand-Eye Robot

This article is concerned with calibrating an anthropomorphic two-armed robot equipped with a stereo-camera vision system, that is estimating the different geometric relationships involved in the model of the robot. The calibration procedure that is presented is fully vision-based: the relationships between each camera and the neck and between each arm and the neck are determined using visual measurements. The online calculation of all the relationships involved in the model of the robot is obtained with satisfactory precision and, above all, without expensive calibration mechanisms. For this purpose, two new main algorithms have been developed. The first one implements a non-linear optimization method using quaternions for camera calibration from 2D to 3D point or line correspondences. The second one implements a real-time camera pose estimation method based on the iterative use of a paraperspective camera model.     

THE SPATIAL SEMANTIC HIERARCHY IMPLEMENTED WITH AN OMNIDIRECTIONAL VISION SYSTEM

In this article, we propose a new approach to the map building task: the implementation of the Spatial Semantic Hierarchy (SSH), proposed by B. Kuipers, on a real robot fitted with an omnidirectional camera. The original Kuiper's formulation of the SSH was slightly modified, in order to manage in a more efficient way the knowledge the real robot collects while moving in the environment. The sensory data experienced by the robot are transformed by the different levels of the SSH in order to obtain a compact representation of the environment. This knowledge is stored in the form of a topological map and, eventually, of a metrical map. The aim of this article is to show that a catadioptric omnidirectional camera is a good sensor for the SSH and nicely couples with several elements of the SSH. The panoramic view and rotational invariance of our omnidirectional camera makes the identification and labelling of places a simple matter. A deeper insight is that the tracking and identification of events on an omnidirectional image such as occlusions and alignments can be used for the segmentation of continuous sensory image data into the discrete topological and metric elements of a map. The proposed combination of the SSH and omnidirectional vision provides a powerful general framework for robot maping and offers new insights into the concept of “place.” Some preliminary experiments performed with a real robot in an unmodified office environment are presented.     

基于单相机的全向深度图获取

为了搜寻移动机器人周围最大的可通行区域,采用全向立体视觉系统,提出获取可靠的致密三维深度图方法。视觉系统由1个普通相机和2个双曲面镜组成。当系统标定后,空间点的三维坐标可以通过匹配上下镜面的成像点计算得出。匹配方法分3步：最大FX匹配,特征匹配和歧义去除。定义合适的能量函数通过动态规划来实现剩余点的匹配。实验表明该系统精度高、具有实用价值。