存在镜面反射时的立体匹配研究

传统的立体匹配方法建立在Lambertian的漫反射模型之上,漫反射模型的立体匹配在一个图像中大部分是有效的,但是在处理图像中包含镜面反射部分时结果会产生严重的匹配错误.为了解决个问题,根据二色反射模型引入一种漫反射和镜面反射的分离方法,匹配图像中存在镜面反射部分时先滤除掉镜面反射再进行匹配,在镜面反射部分也能匹配得到正确的视差.实验结果证明该方法很有效.     

结合同场景立体图对的高质量深度图像重建

目的 越来越多的应用依赖于对场景深度图像准确且快速的观测和分析,如机器人导航以及在电影和游戏中对虚拟场景的设计建模等.飞行时间深度相机等直接的深度测量设备可以实时的获取场景的深度图像,但是由于硬件条件的限制,采集的深度图像分辨率比较低,无法满足实际应用的需要.通过立体匹配算法对左右立体图对之间进行匹配获得视差从而得到深度图像是计算机视觉的一种经典方法,但是由于左右图像之间遮挡以及无纹理区域的影响,立体匹配算法在这些区域无法匹配得到正确的视差,导致立体匹配算法在实际应用中存在一定的局限性.方法 结合飞行时间深度相机等直接的深度测量设备和立体匹配算法的优势,提出一种新的深度图像重建方法.首先结合直接的深度测量设备采集的深度图像来构造自适应局部匹配权值,对左右图像之间的局部窗立体匹配过程进行约束,得到基于立体匹配算法的深度图像;然后基于左右检测原理将采集到的深度图像和匹配得到的深度图像进行有效融合;接着提出一种局部权值滤波算法,来进一步提高深度图像的重建质量.结果 实验结果表明,无论在客观指标还是视觉效果上,本文提出的深度图像重建算法较其他立体匹配算法可以得到更好的结果.其中错误率比较实验表明,本文算法较传统的立体匹配算法在深度重建错误率上可以提升10％左右.峰值信噪比实验结果表明,本文算法在峰值信噪比上可以得到10 dB左右的提升.结论 提出的深度图像重建方法通过结合高分辨率左右立体图对和初始的低分辨率深度图像,可以有效地重建高质量高分辨率的深度图像.     

SymStereo: Stereo Matching using Induced Symmetry

Stereo methods always require a matching function for assessing the likelihood of two pixels being in correspondence. Such functions, commonly referred as matching costs, measure the photo-similarity (or dissimilarity) between image regions centered in putative matches. This article proposes a new family of stereo cost functions that measure symmetry instead of photo-similarity for associating pixels across views. We start by observing that, given two stereo views and an arbitrary virtual plane passing in-between the cameras, it is possible to render image signals that are either symmetric or anti-symmetric with respect to the contour where the virtual plane meets the scene. The fact is investigated in detail and used as cornerstone to develop a new stereo framework that relies in symmetry cues for solving the data association problem. Extensive experiments in dense stereo show that our symmetry-based cost functions compare favorably against the best performing photo-similarity matching costs. In addition, we investigate the possibility of accomplishing Stereo Rangefinding that consists in using passive stereo to exclusively recover depth along a pre-defined scan plane. Thorough experiments provide evidence that stereo from induced symmetry is specially well suited for this purpose.     

A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms

Stereo matching is one of the most active research areas in computer vision. While a large number of algorithms for stereo correspondence have been developed, relatively little work has been done on characterizing their performance. In this paper, we present a taxonomy of dense, two-frame stereo methods. Our taxonomy is designed to assess the different components and design decisions made in individual stereo algorithms. Using this taxonomy, we compare existing stereo methods and present experiments evaluating the performance of many different variants. In order to establish a common software platform and a collection of data sets for easy evaluation, we have designed a stand-alone, flexible C++ implementation that enables the evaluation of individual components and that can easily be extended to include new algorithms. We have also produced several new multi-frame stereo data sets with ground truth and are making both the code and data sets available on the Web. Finally, we include a comparative evaluation of a large set of today's best-performing stereo algorithms.     

根据灰度值信息自适应窗口的半全局匹配

目的 立体匹配算法是立体视觉研究的关键点,算法的匹配精度和速度直接影响3维重建的效果。对于传统立体匹配算法来说,弱纹理区域、视差深度不连续区域和被遮挡区域的匹配精度依旧不理想,为此选择具有全局匹配算法和局部匹配算法部分优点、性能介于两种算法之间、且鲁棒性强的半全局立体匹配算法作为研究内容,提出自适应窗口与半全局立体匹配算法相结合的改进方向。方法 以通过AD（absolute difference）算法求匹配代价的半全局立体匹配算法为基础,首先改变算法匹配代价的计算方式,研究窗口大小对算法性能的影响,然后加入自适应窗口算法,研究自适应窗口对算法性能的影响,最后对改进算法进行算法性能评价与比较。结果 实验结果表明,匹配窗口的选择能够影响匹配算法性能、提高算法的适用范围,自适应窗口的加入能够提高算法匹配精度特别是深度不连续区域的匹配精度,并有效降低算法运行时间,对Cones测试图像集,改进的算法较改进前误匹配率在3个测试区域平均减少2.29%;对于所有测试图像集,算法运行时间较加入自适应窗口前平均减少28.5%。结论 加入自适应窗口的半全局立体匹配算法具有更优的算法性能,能够根据应用场景调节算法匹配精度和匹配速度。     

Combining Shape from Shading and Stereo: A Joint Variational Method for Estimating Depth,Illumination and Albedo

Shape from shading (SfS) and stereo are two fundamentally different strategies for image-based 3-D reconstruction. While approaches for SfS infer the depth solely from pixel intensities, methods for stereo are based on a matching process that establishes correspondences across images. This difference in approaching the reconstruction problem yields complementary advantages that are worthwhile being combined. So far, however, most “joint” approaches are based on an initial stereo mesh that is subsequently refined using shading information. In this paper we follow a completely different approach. We propose a joint variational method that combines both cues within a single minimisation framework. To this end, we fuse a Lambertian SfS approach with a robust stereo model and supplement the resulting energy functional with a detail-preserving anisotropic second-order smoothness term. Moreover, we extend the resulting model in such a way that it jointly estimates depth, albedo and illumination. This in turn makes the approach applicable to objects with non-uniform albedo as well as to scenes with unknown illumination. Experiments for synthetic and real-world images demonstrate the benefits of our combined approach: They not only show that our method is capable of generating very detailed reconstructions, but also that joint approaches are feasible in practice.     

Accurate real-time stereo correspondence using intra- and inter-scanline optimization

This paper deals with a novel stereo algorithm that can generate accurate dense disparity maps in real time. The algorithm employs an effective cross-based variable support aggregation strategy within a scanline optimization framework. Rather than matching intensities directly, the use of adaptive support aggregation allows for precisely handling the weak textured regions as well as depth discontinuities. To improve the disparity results with global reasoning, we reformulate the energy function on a tree structure over the whole 2D image area, as opposed to dynamic programming of individual scanlines. By applying both intra- and inter-scanline optimizations, the algorithm reduces the typical ’streaking’ artifact while maintaining high computational efficiency. The experimental results are evaluated on the Middlebury stereo dataset, showing that our approach is among the best for all real-time approaches. We implement the algorithm on a commodity graphics card with CUDA architecture, running at about 35 fames/s for a typical stereo pair with a resolution of 384×288 and 16 disparity levels.     

A LSS-based registration of stereo thermal–visible videos of multiple people using belief propagation

In this paper, we propose a novel stereo method for registering foreground objects in a pair of thermal and visible videos of close-range scenes. In our stereo matching, we use Local Self-Similarity (LSS) as similarity metric between thermal and visible images. In order to accurately assign disparities to depth discontinuities and occluded Region Of Interest (ROI), we have integrated color and motion cues as soft constraints in an energy minimization framework. The optimal disparity map is approximated for image ROIs using a Belief Propagation (BP) algorithm. We tested our registration method on several challenging close-range indoor video frames of multiple people at different depths, with different clothing, and different poses. We show that our global optimization algorithm significantly outperforms the existing state-of-the art method, especially for disparity assignment of occluded people at different depth in close-range surveillance scenes and for relatively large camera baseline.     

基于mean-shift全局立体匹配方法

针对图像全局立体匹配精度高、计算量大的问题,提出基于mean shift图像分割的全局立体匹配方法。首先,通过mean shift算法对图像进行分割,获取图像同质区域数量和区域的标号。在计算匹配代价时,根据像素所属的分割区域,对像素进行筛选,从而提高匹配代价计算速度;其次,在代价聚合前,将mean shift算法获取的同质区域数K值赋值给K-means聚类算法,对像素再次聚类,提高立体匹配精度和速度;最后通过TRW-S置信传播解决能量最小化问题。实验表明,该算法明显提高了匹配的准确性和速度,与单纯的全局匹配算法相比,具有更大的优势。     

3-D shape reconstruction in an active stereo vision system using genetic algorithms

The recovery of 3-D shape information (depth) using stereo vision analysis is one of the major areas in computer vision and has given rise to a great deal of literature in the recent past. The widely known stereo vision methods are the passive stereo vision approaches that use two cameras. Obtaining 3-D information involves the identification of the corresponding 2-D points between left and right images. Most existing methods tackle this matching task from singular points, i.e. finding points in both image planes with more or less the same neighborhood characteristics. One key problem we have to solve is that we are on the first instance unable to know a priori whether a point in the first image has a correspondence or not due to surface occlusion or simply because it has been projected out of the scope of the second camera. This makes the matching process very difficult and imposes a need of an a posteriori stage to remove false matching.In this paper we are concerned with the active stereo vision systems which offer an alternative to the passive stereo vision systems. In our system, a light projector that illuminates objects to be analyzed by a pyramid-shaped laser beam replaces one of the two cameras. The projections of laser rays on the objects are detected as spots in the image. In this particular case, only one image needs to be treated, and the stereo matching problem boils down to associating the laser rays and their corresponding real spots in the 2-D image. We have expressed this problem as a minimization of a global function that we propose to perform using Genetic Algorithms (GAs). We have implemented two different algorithms: in the first, GAs are performed after a deterministic search. In the second, data is partitioned into clusters and GAs are independently applied in each cluster. In our second contribution in this paper, we have described an efficient system calibration method. Experimental results are presented to illustrate the feasibility of our approach. The proposed method yields high accuracy 3-D reconstruction even for complex objects. We conclude that GAs can effectively be applied to this matching problem.     

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.     

Fog effect for photography using stereo vision

Fog is an important factor in photography with a special aesthetic, emotional, or compositional meaning. We present a fog-simulation method for photo editing using binocular stereo vision. Given a stereo pair, we estimate the depth information by stereo matching followed by a process to refine depth results for the given photo editing purpose. Then, depth-aware fog effects can be applied on the base image, with optional interaction for control purposes. Besides homogeneous fog, we provide three tools to control the density of the fog media. Thus, various kinds of heterogeneous atmospheric effects can also been simulated. Experiments show that the proposed method can achieve more natural-looking results than manually drawn fog, our results are very close to the appearance of fog in the real world.     

Surfaces from stereo: integrating feature matching, disparityestimation, and contour detection

An approach is described that integrates the processes of feature matching, contour detection, and surface interpolation to determine the three-dimensional distance, or depth, of objects from a stereo pair of images. Integration is necessary to ensure that the detected surfaces are smooth. Surface interpolation takes into account detected occluding and ridge contours in the scene; interpolation is performed within regions enclosed by these contours. Planar and quadratic patches are used as local models of the surface. Occluded regions in the image are identified, and are not used for matching and interpolation. A coarse-to-fine algorithm is presented that generates a multiresolution hierarchy of surface maps, one at each level of resolution. Experimental results are given for a variety of stereo images     

Stereo matching using belief propagation

In this paper, we formulate the stereo matching problem as a Markov network and solve it using Bayesian belief propagation. The stereo Markov network consists of three coupled Markov random fields that model the following: a smooth field for depth/disparity, a line process for depth discontinuity, and a binary process for occlusion. After eliminating the line process and the binary process by introducing two robust functions, we apply the belief propagation algorithm to obtain the maximum a posteriori (MAP) estimation in the Markov network. Other low-level visual cues (e.g., image segmentation) can also be easily incorporated in our stereo model to obtain better stereo results. Experiments demonstrate that our methods are comparable to the state-of-the-art stereo algorithms for many test cases.     

Multiflash stereopsis: depth-edge-preserving stereo with small baseline illumination

Traditional stereo matching algorithms are limited in their ability to produce accurate results near depth discontinuities, due to partial occlusions and violation of smoothness constraints. In this paper, we use small baseline multi-flash illumination to produce a rich set of feature maps that enable acquisition of discontinuity preserving point correspondences. First, from a single multi-flash camera, we formulate a qualitative depth map using a gradient domain method that encodes object relative distances. Then, in a multiview setup, we exploit shadows created by light sources to compute an occlusion map. Finally, we demonstrate the usefulness of these feature maps by incorporating them into two different dense stereo correspondence algorithms, the first based on local search and the second based on belief propagation. Experimental results show that our enhanced stereo algorithms are able to extract high quality, discontinuity preserving correspondence maps from scenes that are extremely challenging for conventional stereo methods. We also demonstrate that small baseline illumination can be useful to handle specular reflections in stereo imagery. Different from most existing active illumination techniques, our method is simple, inexpensive, compact, and requires no calibration of light sources.     

Stereo-based bokeh effects for photography

Bokeh, a sought-after photo rendering style of out-of-focus blur, typically aims at an esthetic quality which is not available to low-end consumer-grade cameras due to the lens design. We present a bokeh simulation method using stereo-vision techniques. We refine a depth map obtained by stereo matching, also using some minor user interaction. Overexposed regions are recovered according to depth information. A depth-aware bokeh effect is then applied with user-adjustable apertures sizes or shapes. We also simulate swirly bokeh, also known as cat-eye effect. Our method mainly aims at the visual quality of the bokeh effect rather than (so far) at time efficiency. Experiments show that our results are natural looking and that they can be comparable to bokeh effects achieved with expensive real-world bokeh-capable camera systems.     

Stereo matching algorithm with guided filter and modified dynamic programming

Dense stereo correspondence is a challenging research problem in computer vision field. To address the poor accuracy behavior of stereo matching, we propose a novel stereo matching algorithm based on guided image filter and modified dynamic programming. Firstly, we suggest a combined matching cost by incorporating the absolute difference and improved color census transform (ICCT). Secondly, we use the guided image filter to filter the cost volume, which can aggregate the costs fast and efficiently. Then, in the disparity computing step, we design a modified dynamic programming algorithm, which can weaken the scanning line effect. At last, final disparity maps are gained after post-processing. The experimental results are evaluated on Middlebury Stereo Datasets, showing that our approach can achieve good results both in low texture and depth discontinuity areas with an average error rate of 5.14 % and strong robustness.     

Improving sub-pixel accuracy for long range stereo

Dense stereo algorithms are able to estimate disparities at all pixels including untextured regions. Typically these disparities are evaluated at integer disparity steps. A subsequent sub-pixel interpolation often fails to propagate smoothness constraints on a sub-pixel level.We propose to increase the sub-pixel accuracy in low-textured regions in four possible ways: First, we present an analysis that shows the benefit of evaluating the disparity space at fractional disparities. Second, we introduce a new disparity smoothing algorithm that preserves depth discontinuities and enforces smoothness on a sub-pixel level. Third, we present a novel stereo constraint (gravitational constraint) that assumes sorted disparity values in vertical direction and guides global algorithms to reduce false matches, especially in low-textured regions. Finally, we show how image sequence analysis improves stereo accuracy without explicitly performing tracking. Our goal in this work is to obtain an accurate 3D reconstruction. Large-scale 3D reconstruction will benefit heavily from these sub-pixel refinements.Results based on semi-global matching, obtained with the above mentioned algorithmic extensions are shown for the Middlebury stereo ground truth data sets. The presented improvements, called ImproveSubPix, turn out to be one of the top-performing algorithms when evaluating the set on a sub-pixel level while being computationally efficient. Additional results are presented for urban scenes. The four improvements are independent of the underlying type of stereo algorithm.     

A cost-benefit analysis of a third camera for stereo correspondence

This paper looks at the twin issues of the gain in accuracy of stereo correspondence and the accompanying increase in computational cost due to the use of a third camera for stereo analysis. Trinocular stereo algorithms differ from binocular algorithms essentially in the epipolar constraint used in the local matching stage. The current literature does not provide any insight into the relative merits of binocular and trinocular stereo matching with the matching accuracy being verified aginst the ground truth. Experiments for evaluating the relative performance of binocular and trinocular stereo algorithms were conducted. The stereo images used for the performance evaluation were generated by applying a Lambertian reflectance model to real Digital Elevation Maps (DEMs) available from the U.S. Geological Survey. The matching accuracy of the stereo algorithms was evaluated by comparing the observed stereo disparity against the ground truth derived from the DEMs. It was observed that trinocular local matching reduced the percentage of mismatches having large disparity errors by more than half when compared to binocular matching. On the other hand, trinocular stereopsis increased the computational cost of local matching over binocular by only about one-fourth. We also present a quantization-error analysis of the depth reconstruction process for the nonparallel stereo-imaging geometry used in our experiments.     

Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo

An integrated approach to extract depth, efficiently and accurately, from a sequence of images is presented in this paper. The method combines the ability of the stereo processing to acquire highly accurate depth measurements and the efficiency of spatial and temporal gradient analysis. As a result of this integration, depth measurements of high quality are obtained at a speed approximately ten times greater than that of stereo processing. Without any a priori information of the locations of the points in the scene, the correspondence problem in stereo processing is computationally expensive. In our approach, we use spatial and temporal gradient (STG) analysis, which has been shown to provide depth with great efficiency, but limited accuracy, to guide the matching process of stereo. The camera motion used in the approach can be either lateral or axial. Extensive experiments on real scenes have shown the ability of the integrated approach to acquire depth with a mean error of less than 3%.