Automatic feature matching in uncalibrated stereo vision through the use of color

One difficult problem in stereo vision is how to establish correspondence between features extracted from a pair of images. The difficulty is due to ambiguities or inconsistencies of available information on images. In this paper, we invetigate stereo correspondence problem in the framework of color stereo vision. We propose the use of a matching consistence (MC) constraint in RGB color space and the generalized epipolar geometry to develop an automatic feature matching algorithm.     

A wavelet-based multiresolution approach to solve the stereo correspondence problem using mutual information.

In this correspondence, we propose a wavelet-based hierarchical approach using mutual information (MI) to solve the correspondence problem in stereo vision. The correspondence problem involves identifying corresponding pixels between images of a given stereo pair. This results in a disparity map, which is required to extract depth information of the relevant scene. Until recently, mostly correlation-based methods have been used to solve the correspondence problem. However, the performance of correlation-based methods degrades significantly when there is a change in illumination between the two images of the stereo pair. Recent studies indicate MI to be a more robust stereo matching metric for images affected by such radiometric distortions. In this short correspondence paper, we compare the performances of MI and correlation-based metrics for different types of illumination changes between stereo images. MI, as a statistical metric, is computationally more expensive. We propose a wavelet-based hierarchical technique to counter the increase in computational cost and show its effectiveness in stereo matching.     

Stereo by intra- and inter-scanline search using dynamic programming

This paper presents a stereo matching algorithm using the dynamic programming technique. The stereo matching problem, that is, obtaining a correspondence between right and left images, can be cast as a search problem. When a pair of stereo images is rectified, pairs of corresponding points can be searched for within the same scanlines. We call this search intra-scanline search. This intra-scanline search can be treated as the problem of finding a matching path on a two-dimensional (2D) search plane whose axes are the right and left scanlines. Vertically connected edges in the images provide consistency constraints across the 2D search planes. Inter-scanline search in a three-dimensional (3D) search space, which is a stack of the 2D search planes, is needed to utilize this constraint. Our stereo matching algorithm uses edge-delimited intervals as elements to be matched, and employs the above mentioned two searches: one is inter-scanline search for possible correspondences of connected edges in right and left images and the other is intra-scanline search for correspondences of edge-delimited intervals on each scanline pair. Dynamic programming is used for both searches which proceed simultaneously: the former supplies the consistency constraint to the latter while the latter supplies the matching score to the former. An interval-based similarity metric is used to compute the score. The algorithm has been tested with different types of images including urban aerial images, synthesized images, and block scenes, and its computational requirement has been discussed.     

Non-Rigid Stereo Factorization

In this paper we address the problem of recovering 3D non-rigid structure from a sequence of images taken with a stereo pair. We have extended existing non-rigid factorization algorithms to the stereo camera case and presented an algorithm to decompose the measurement matrix into the motion of the left and right cameras and the 3D shape, represented as a linear combination of basis-shapes. The added constraints in the stereo camera case are that both cameras are viewing the same structure and that the relative orientation between both cameras is fixed. Our focus in this paper is on the recovery of flexible 3D shape rather than on the correspondence problem. We propose a method to compute reliable 3D models of deformable structure from stereo images. Our experiments with real data show that improved reconstructions can be achieved using this method. The algorithm includes a non-linear optimization step that minimizes image reprojection error and imposes the correct structure to the motion matrix by choosing an appropriate parameterization. We show that 3D shape and motion estimates can be successfully disambiguated after bundle adjustment and demonstrate this on synthetic and real image sequences. While this optimization step is proposed for the stereo camera case, it can be readily applied to the case of non-rigid structure recovery using a monocular video sequence. Electronic supplementary material Electronic supplementary material is available for this article at and accessible for authorised users.     

Rectified catadioptric stereo sensors

It has been shown elsewhere how mirrors can be used to capture stereo images with a single camera, an approach termed catadioptric stereo. We present novel catadioptric sensors that use mirrors to produce rectified stereo images. The scanline correspondence of these images benefits real-time stereo by avoiding the computational cost and image degradation due to resampling when rectification is performed after image capture. First, we develop a theory which determines the number of mirrors that must be used and the constraints on those mirrors that must be satisfied to obtain rectified stereo images with a single camera. Then, we discuss in detail the use of both one and three mirrors. In addition, we show how the mirrors should be placed in order to minimize sensor size for a given baseline, an important design consideration. In order to understand the feasibility of building these sensors, we analyze rectification errors due to misplacement of the camera with respect to the mirrors     

基于形变模型由立体序列图象恢复物体的3D形状

结合立体视觉和形变模型提出了一种新的物体3D形状的恢复方法。采用立体视觉方法导出物体表面的3D坐标;利用光流模型估计物体的3D运动,根据此运动移动形变模型,使其对准物体的表面块;由形变模型将由各幅图象得到的离散的3D点融为一起,得到物体的表面形状。实验结果表明该方法能用于形状复杂的物体恢复。     

3D reconstruction for a scanning electron microscope

An algorithm for the reconstruction of the 3D shape of the surface of a micro-object from a stereo pair of images obtained on a raster electron microscope (REM) has been considered. A model of building an image in REM has been presented. The SIFT algorithm was used for determination of the correspondence points. The correspondence points are used to determine the mutual position of the object in a stereo pair by the RANSAC method. A set of points is created in the 3D space, which is later interpolated to reconstruct the 3D surface.     

Shape Reconstruction of 3D Bilaterally Symmetric Surfaces

The paper presents a new approach for shape recovery based on integrating geometric and photometric information. We consider 3D bilaterally symmetric objects, that is, objects which are symmetric with respect to a plane (e.g., faces), and their reconstruction from a single image. Both the viewpoint and the illumination are not necessarily frontal. Furthermore, no correspondence between symmetric points is required.The basic idea is that an image taken from a general, non frontal viewpoint, under non-frontal illumination can be regarded as a pair of images. Each image of the pair is one half of the object, taken from different viewing positions and with different lighting directions. Thus, one-image-variants of geometric stereo and of photometric stereo can be used. Unlike the separate invocation of these approaches, which require point correspondence between the two images, we show that integrating the photometric and geometric information suffice to yield a dense correspondence between pairs of symmetric points, and as a result, a dense shape recovery of the object. Furthermore, the unknown lighting and viewing parameters, are also recovered in this process.Unknown distant point light source, Lambertian surfaces, unknown constant albedo, and weak perspective projection are assumed. The method has been implemented and tested experimentally on simulated and real data.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.     

Stereo geometry from 3D ego-motion streams

This paper addresses the problem of geometry determination of a stereo rig that undergoes general rigid motions. Neither known reference objects nor stereo correspondence are required. With almost no exception, all existing online solutions attempt to recover stereo geometry by first establishing stereo correspondences. We first describe a mathematical framework that allows us to solve for stereo geometry, i.e., the rotation and translation between the two cameras, using only motion correspondence that is far easier to acquire than stereo correspondence. Second, we show how to recover the rotation and present two linear methods, as well as a nonlinear one to solve for the translation. Third, we perform a stability study for the developed methods in the presence of image noise, camera parameter noise, and ego-motion noise. We also address accuracy issues. Experiments with real image data are presented. The work allows the concept of online calibration to be broadened, as it is no longer true that only single cameras can exploit structure-from-motion strategies; even the extrinsic parameters of a stereo rig of cameras can do so without solving stereo correspondence. The developed framework is applicable for estimating the relative three-dimensional (3D) geometry associated with a wide variety of mounted devices used in vision and robotics, by exploiting their scaled ego-motion streams.     

Reconstructing a 3-D depth map from one or more images

Several algorithms are suggested for recovering depth and orientation maps of a surface from its image intensities. They combine the advantages of stereo vision and shape-from-shading (SFS) methods. These algorithms generate accurate, unambiguous and dense surface depth and orientation maps. Most of the existing SFS algorithms cannot be directly extended to combine stereo images because the recovery of surface depth and that of orientation are separated in these formulations. We first present an SFS algorithm that couples the generation of depth and orientation maps. This formulation also ensures that the reconstructed surface depth and its orientation are consistent. The SFS algorithm for a single image is then extended to utilize stereo images. The correspondence over stereo images is established simultaneously with the generation of surface depth and orientation. An alternative approach is also suggested for combining stereo and SFS techniques. This approach can be used to combine needle maps which are directly available from other sources such as photometric stereo. Finally we present an algorithm to combine sparse depth measurements with an orientation map to reconstruct a surface. The same algorithm can be combined with the above algorithms for solving the SFS problem with sparse depth measurements. Thus various information sources can be used to accurately reconstruct a surface.     

Dense Features for Semi-Dense Stereo Correspondence

We present a new feature based algorithm for stereo correspondence. Most of the previous feature based methods match sparse features like edge pixels, producing only sparse disparity maps. Our algorithm detects and matches dense features between the left and right images of a stereo pair, producing a semi-dense disparity map. Our dense feature is defined with respect to both images of a stereo pair, and it is computed during the stereo matching process, not a preprocessing step. In essence, a dense feature is a connected set of pixels in the left image and a corresponding set of pixels in the right image such that the intensity edges on the boundary of these sets are stronger than their matching error (which is the difference in intensities between corresponding boundary pixels). Our algorithm produces accurate semi-dense disparity maps, leaving featureless regions in the scene unmatched. It is robust, requires little parameter tuning, can handle brightnessdifferences between images, nonlinear errors, and is fast (linear complexity).     

自适应窗口的时间规整立体匹配算法

针对立体视觉中图像对应点的误匹配问题,以时间规整算法(DTW)为基础,提出了自适应窗口的立体匹配算法．根据外极线的约束,在自适应窗口内采用灰度相关技术得到长度不相等的两个灰度段作为相容的匹配序列;利用动态规划法及连续性约束寻找一条最佳的匹配路径．根据回溯得到的匹配路径及其坐标值得到高密度视差图．实验结果表明,该算法具有较高的运行效率和良好的匹配效果．     

Generation of Temporally Consistent Multiple Virtual Camera Views from Stereoscopic Image Sequences

The emergence of a new generation of 3D auto stereoscopic displays is driving the requirement for multi-baseline images. The dominant form of this display technology requires multiple views of the same scene, captured at a single instance in time along a common baseline in order to project stereoscopic images to the viewer. The direct acquisition of multiple views (typically 8 or 16 for the current generation of such displays) is problematic due to the difficulty of configuring, calibrating and controlling multiple cameras simultaneously.This paper describes a technique that alleviates these problems by generating the required views from binocular images. Considering each stereo pair in isolation leads to inconsistency across image sequences. By incorporating a motion-tracking algorithm this problem is significantly reduced. In this paper we describe a novel approach to stereo matching of image sequences for the purpose of generating multiple virtual cameraviews. Results of extensive tests on stereo image sequences will be documented indicating that this approach is promising both in terms of the speed of execution and the quality of the results produced.     

Advances in computational stereo

Extraction of three-dimensional structure of a scene from stereo images is a problem that has been studied by the computer vision community for decades. Early work focused on the fundamentals of image correspondence and stereo geometry. Stereo research has matured significantly throughout the years and many advances in computational stereo continue to be made, allowing stereo to be applied to new and more demanding problems. We review recent advances in computational stereo, focusing primarily on three important topics: correspondence methods, methods for occlusion, and real-time implementations. Throughout, we present tables that summarize and draw distinctions among key ideas and approaches. Where available, we provide comparative analyses and we make suggestions for analyses yet to be done.     

SymStereo: Stereo Matching using Induced Symmetry

Stereo methods always require a matching function for assessing the likelihood of two pixels being in correspondence. Such functions, commonly referred as matching costs, measure the photo-similarity (or dissimilarity) between image regions centered in putative matches. This article proposes a new family of stereo cost functions that measure symmetry instead of photo-similarity for associating pixels across views. We start by observing that, given two stereo views and an arbitrary virtual plane passing in-between the cameras, it is possible to render image signals that are either symmetric or anti-symmetric with respect to the contour where the virtual plane meets the scene. The fact is investigated in detail and used as cornerstone to develop a new stereo framework that relies in symmetry cues for solving the data association problem. Extensive experiments in dense stereo show that our symmetry-based cost functions compare favorably against the best performing photo-similarity matching costs. In addition, we investigate the possibility of accomplishing Stereo Rangefinding that consists in using passive stereo to exclusively recover depth along a pre-defined scan plane. Thorough experiments provide evidence that stereo from induced symmetry is specially well suited for this purpose.     

Forward-backward study of the stereo vision problem

In stereo vision a pair of two-dimensional (2D) stereo images is given and the purpose is to find out the depth (disparity) of regions of the image in relation to the background, so that we can reconstruct the 3D structure of the image from the pair of 2D stereo images given. Using the Bayesian framework we implemented the Forward-Backward algorithm to unfold the disparity (depth) of a pair of stereo images. The results showed are very reasonable, but we point out there is room for improvement concerning the graph structure used.     

快速对称多窗口正交三目立体匹配

提出了一种对称多窗口正交三目立体匹配算法,利用双目立体视觉中有效的对称多窗口（symmetry multi-window,SMW)方法视差估计的不确定度,在正交三目立体匹配中融合通过SMW方法得到的水平和垂直两个图像对的视差计算结果形成新的视差图。在实现三目立体匹配的同时提出了双目SMW立体匹配方法快速算法,使得计算一对立体图像的9个窗口两个方向（从左到右和从右到左）ZNCC的速度和计算一个ZNCC相当,计算时间约只有原来的1／18。通过对实际的三目立体图像组的实验,结果表明相对于单纯的双目SMW方法,在遮断区域、单方向平滑区域都得到了具有鲁棒性的视差估计,而且计算速度令人满意。     

Catadioptric Stereo Using Planar Mirrors

By using mirror reflections of a scene, stereo images can be captured with a single camera (catadioptric stereo). In addition to simplifying data acquisition single camera stereo provides both geometric and radiometric advantages over traditional two camera stereo. In this paper, we discuss the geometry and calibration of catadioptric stereo with two planar mirrors. In particular, we will show that the relative orientation of a catadioptric stereo rig is restricted to the class of planar motions thus reducing the number of external calibration parameters from 6 to 5. Next we derive the epipolar geometry for catadioptric stereo and show that it has 6 degrees of freedom rather than 7 for traditional stereo. Furthermore, we show how focal length can be recovered from a single catadioptric image solely from a set of stereo correspondences. To test the accuracy of the calibration we present a comparison to Tsai camera calibration and we measure the quality of Euclidean reconstruction. In addition, we will describe a real-time system which demonstrates the viability of stereo with mirrors as an alternative to traditional two camera stereo.     

基于矩形子块法的快速全局立体匹配算法

由于计算量大,采用割图法(GC,GraphCuts)来最小化立体匹配的能量函数,计算速度慢,难以满足许多方面应用的要求。为了提高立体匹配能量函数最小化的速度,文章提出了一种新的算法:矩形子块割图法(RSRGC,RectangularSubregioningGraphCuts),即将大图像分割为小矩形子块,然后再在各小矩形子块上分别利用GC最小化能量函数。实验表明,采用该方法可以大大提高立体匹配的速度,而不影响立体匹配的效果。