Matching two perspective views

A computational approach to image matching is described. It uses multiple attributes associated with each image point to yield a generally overdetermined system of constraints, taking into account possible structural discontinuities and occlusions. In the algorithm implemented, intensity, edgeness, and cornerness attributes are used in conjunction with the constraints arising from intraregional smoothness, field continuity and discontinuity, and occlusions to compute dense displacement fields and occlusion maps along the pixel grids. The intensity, edgeness, and cornerness are invariant under rigid motion in the image plane. In order to cope with large disparities, a multiresolution multigrid structure is employed. Coarser level edgeness and cornerness measures are obtained by blurring the finer level measures. The algorithm has been tested on real-world scenes with depth discontinuities and occlusions. A special case of two-view matching is stereo matching, where the motion between two images is known. The algorithm can be easily specialized to perform stereo matching using the epipolar constraint     

Dense disparity estimation with a divide-and-conquer disparityspace image technique

A new divide-and-conquer technique for disparity estimation is proposed in this paper. This technique performs feature matching following the high confidence first principle, starting with the strongest feature point in the stereo pair of scanlines. Once the first matching pair is established, the ordering constraint in disparity estimation allows the original intra-scanline matching problem to be divided into two smaller subproblems. Each subproblem can then be solved recursively until there is no reliable feature point within the subintervals. This technique is very efficient for dense disparity map estimation for stereo images with rich features. For general scenes, this technique can be paired up with the disparity-space image (DSI) technique to compute dense disparity maps with integrated occlusion detection. In this approach, the divide-and-conquer part of the algorithm handles the matching of stronger features and the DSI-based technique handles the matching of pixels in between feature points and the detection of occlusions. An extension to the standard disparity-space technique is also presented to compliment the divide-and-conquer algorithm. Experiments demonstrate the effectiveness of the proposed divide-and-conquer DSI algorithm     

Stereo correspondence with occlusion handling in a symmetric patch-based graph-cuts model

A novel patch-based correspondence model is presented in this paper. Many segment-based correspondence approaches have been proposed in recent years. Untextured pixels and boundaries of discontinuities are imposed with hard constraints by the discontinuity assumption that large disparity variation only happens at the boundaries of segments in the above approaches. Significant improvements on performance of untextured and discontinuity area have been reported. But, the performance near occlusion is not satisfactory because a segmented region in one image may be only partially visible in the other one. To solve this problem, we utilize the observation that the shared edge of a visible area and an occluded area corresponds to the discontinuity in the other image. So, the proposed model conducts color segmentation on both images first and then a segment in one image is further cut into smaller patches corresponding to the boundaries of segments in the other when it is assigned with a disparity. Different visibility of patches in one segment is allowed. The uniqueness constraint in a segment level is used to compute the occlusions. An energy minimization framework using graph-cuts is proposed to find a global optimal configuration including both disparities and occlusions. Besides, some measurements are taken to make our segment-based algorithm suffer less from violation of the discontinuity assumption. Experimental results have shown superior performance of the proposed approach, especially on occlusions, untextured areas, and near discontinuities     

A sub-pixel stereo matching algorithm and its applications in fabric imaging

In this paper, we describe a sub-pixel stereo matching algorithm where disparities are iteratively refined within a regularization framework. We choose normalized cross-correlation as the matching metric, and perform disparity refinement based on correlation gradients, which is distinguished from intensity gradient-based methods. We propose a discontinuity-preserving regularization technique which utilizes local coherence in the disparity space image, instead of estimating discontinuities in the intensity images. A concise numerical solution is derived by parameterizing the disparity space with dense bicubic B-splines. Experimental results show that the proposed algorithm performs better than correlation fitting methods without regularization. The algorithm has been implemented for applications in fabric imaging. We have shown its potentials in wrinkle evaluation, drape measurement, and pilling assessment.     

Hierarchical adaptive stereo matching algorithm for obstacle detection with dynamic programming

An adaptive weighted stereo matching algorithm with multilevel and bidirectional dynamic programming based on ground control points （GCPs） is presented. To decrease time complexity without losing matching precision, using a multilevel search scheme, the coarse matching is processed in typical disparity space image, while the fine matching is processed in disparity-offset space image. In the upper level, GCPs are obtained by enhanced volumetric iterative algorithm enforcing the mutual constraint and the threshold constraint. Under the supervision of the highly reliable GCPs, bidirectional dynamic programming framework is employed to solve the inconsistency in the optimization path. In the lower level, to reduce running time, disparity-offset space is proposed to efficiently achieve the dense disparity image. In addition, an adaptive dual support-weight strategy is presented to aggregate matching cost, which considers photometric and geometric information. Further, post-processing algorithm can ameliorate disparity results in areas with depth discontinuities and related by occlusions using dual threshold algorithm, where missing stereo information is substituted from surrounding regions. To demonstrate the effectiveness of the algorithm, we present the two groups of experimental results for four widely used standard stereo data sets, including discussion on performance and comparison with other methods, which show that the algorithm has not only a fast speed, but also significantly improves the efficiency of holistic optimization.     

一种基于光流和能量的图像匹配算法

结合光流与图像信息,提出一种获取稠密视差的图像匹配算法.首先对于基线较大的左右图像,在多分辨率框架下采用由粗到精的策略计算光流,从而实现大偏移量时的光流获取.其次为了避免光流在图像边界上的不可靠性,通过光流计算所得的光流场作为初始视差图,采用基于能量的方法依据对应的图像梯度场对光流场内部进行平滑并保持边缘的不连续性,最终得到精准稠密的视差图.实验验证,该方法是一种行之有效的图像匹配算法.     

A parallel stereo algorithm that produces dense depth maps and preserves image features

To compute reliable dense depth maps, a stereo algorithm must preserve depth discontinuities and avoid gross errors. In this paper, we show how simple and parallel techniques can be combined to achieve this goal and deal with complex real world scenes. Our algorithm relies on correlation followed by interpolation. During the correlation phase the two images play a symmetric role and we use a validity criterion for the matches that eliminate gross errors: at places where the images cannot be correlated reliably, due to lack of texture of occlusions for example, the algorithm does not produce wrong matches but a very sparse disparity map as opposed to a dense one when the correlation is successful. To generate a dense depth map, the information is then propagated across the featureless areas, but not across discontinuities, by an interpolation scheme that takes image grey levels into account to preserve image features. We show that our algorithm performs very well on difficult images such as faces and cluttered ground level scenes. Because all the algorithms described here are parallel and very regular they could be implemented in hardware and lead to extremely fast stereo systems.This research was supported in part under the Centre National d'Etudes Spatiales VAP contract and in part under a Defence Advanced Research Projects Agency contract at SRI     

Dense Features for Semi-Dense Stereo Correspondence

We present a new feature based algorithm for stereo correspondence. Most of the previous feature based methods match sparse features like edge pixels, producing only sparse disparity maps. Our algorithm detects and matches dense features between the left and right images of a stereo pair, producing a semi-dense disparity map. Our dense feature is defined with respect to both images of a stereo pair, and it is computed during the stereo matching process, not a preprocessing step. In essence, a dense feature is a connected set of pixels in the left image and a corresponding set of pixels in the right image such that the intensity edges on the boundary of these sets are stronger than their matching error (which is the difference in intensities between corresponding boundary pixels). Our algorithm produces accurate semi-dense disparity maps, leaving featureless regions in the scene unmatched. It is robust, requires little parameter tuning, can handle brightnessdifferences between images, nonlinear errors, and is fast (linear complexity).     

基于信任度传播的体视算法

针对信任度传播算法计算量大及误匹配率高的问题,提出一种高效的计算稠密视差图的全局优化算法。首先,根据像素匹配代价的特点、视差不连续亮度变化的特征,定义具有适应性的数据约束和平滑约束,并对平滑约束进行分层调节后执行消息的传输。其次,讨论消息传输迭代过程中的冗余计算问题,通过检测消息的收敛性减少运行时间。最后,分析信任度传播算法中的误匹配问题,通过匹配的对称性检测遮挡,并提出重建数据项后,利用贪婪迭代法优化所得视差图,将图像中可靠像素的视差向不可靠像素扩散。实验结果表明,该算法能以较快的速度计算出更理想的视差图。     

Discrete signal matching using coarse-to-fine wavelet basis functions

In this paper, we propose a matching algorithm to estimate accurate and dense displacements between two signals. The displacements are characterized by a coarse-to-fine wavelet representation, which is a linear combination of hierarchical basis functions developed by Cai and Wang (SIAM Numer. Anal. 33(3) (1996) 937). The coarser-scale basis function has larger support while the finer-scale basis function has smaller support. During the iterative minimization process, the basis functions are utilized as large-to-small windows in selecting global-to-local regions for signal matching. The estimated wavelet coefficients are then used to reconstruct the signal in a coarse-to-fine manner. Two sets of synthetic examples, one with small displacement and the other with large displacements, have been employed to illustrate the advantages of the wavelet-based method. This method has been applied to estimate the dense disparity between stereo images in several real examples and the results demonstrate its effectiveness.     

一种利用动态规划和左右一致性的立体匹配算法

立体匹配是计算机视觉领域研究的一个重要课题,为了得到准确、稠密的视差图,提出了一种利用动态规划和左右一致性的立体匹配算法。该算法首先分别以左、右图像为基元图像,计算各自的视差空间图像,在视差空间图像上利用动态规划,计算得到左视差图和右视差图;然后通过使用左右视差图之间的一致性关系,消除误匹配点,得到较为准确的部分视差图;最后利用视差图的顺序约束关系,给出未匹配视差点的搜索空间计算方法,并利用一种简单有效的方法来计算这些点的视差值。在一些标准立体图像对上所做的实验结果表明,该算法效果良好。     

Stochastic stereo matching over scale

A stochastic optimization approach to stereo matching is presented. Unlike conventional correlation matching and feature matching, the method provides a dense array of disparities, eliminating the need for interpolation. First, the stereo-matching problem is defined in terms of finding a disparity map that satisfies two competing constraints: (1) matched points should have similar image intensity, and (2) the disparity map should vary as slowly as possible. These constraints are interpreted as specifying the potential energy of a system of oscillators. Ground states are approximated by a new variant of simulated annealing, which has two important features. First, the microcanonical ensemble is simulated using a new algorithm that is more efficient and more easily implemented than the familiar Metropolis algorithm (which simulates the canonical ensemble). Secondly, it uses a hierarchical, coarse-to-fine control structure employing Gaussian or Laplacian pyramids of the stereo images. In this way, quickly computed results at low resolutions are used to initialize the system at higher resolutions.Support for this work was provided by the Defense Advanced Research Projects Agency under contracts DCA 76-85-C-0004 and MDA 903-83-C-0084.     

Occlusions and binocular stereo

Binocular stereo is the process of obtaining depth information from a pair of cameras. In the past, stereo algorithms have had problems at occlusions and have tended to fail there (though sometimes post-processing has been added to mitigate the worst effects). We show that, on the contrary, occlusions can help stereo computation by providing cues for depth discontinuities.We describe a theory for stereo based on the Bayesian approach, using adaptive windows and a prior weak smoothness constraint, which incorporates occlusion. Our model assumes that a disparity discontinuity, along the epipolar line, in one eyealways corresponds to an occluded region in the other eye thus, leading to anocclusion constraint. This constraint restricts the space of possible disparity values, thereby simplifying the computations. An estimation of the disparity at occluded features is also discussed in light of psychophysical experiments. Using dynamic programming we can find the optimal solution to our system and the experimental results are good and support the assumptions made by the model.     

Shape and the Stereo Correspondence Problem

We examine the implications of shape on the process of finding dense correspondence and half-occlusions for a stereo pair of images. The desired property of the disparity map is that it should be a piecewise continuous function which is consistent with the images and which has the minimum number of discontinuities. To zeroth order, piecewise continuity becomes piecewise constancy. Using this approximation, we first discuss an approach for dealing with such a fronto-parallel shapeless world, and the problems involved therein. We then introduce horizontal and vertical slant to create a first order approximation to piecewise continuity. In particular, we emphasize the following geometric fact: a horizontally slanted surface (i.e., having depth variation in the direction of the separation of the two cameras) will appear horizontally stretched in one image as compared to the other image. Thus, while corresponding two images, N pixels on a scanline in one image may correspond to a different number of pixels M in the other image. This leads to three important modifications to existing stereo algorithms: (a) due to unequal sampling, existing intensity matching metrics must be modified, (b) unequal numbers of pixels in the two images must be allowed to correspond to each other, and (c) the uniqueness constraint, which is often used for detecting occlusions, must be changed to an interval uniqueness constraint. We also discuss the asymmetry between vertical and horizontal slant, and the central role of non-horizontal edges in the context of vertical slant. Using experiments, we discuss cases where existing algorithms fail, and how the incorporation of these new constraints provides correct results.     

一种基于特征约束的立体匹配算法

立体匹配一直是计算机视觉领域的一个中心研究问题,为了得到适用于基于图象绘制技术的视图合成高密度视差图,提出了基于边缘特征约束的立体西欧算法,该方法首先利用基于特征技术来得到边缘特征点的准确视差图,然后在边缘特征点视差图的约束下,对非边缘特征点采用区域相关算法进行匹配,这样既缩小了匹配搜索空间,又保证了匹配的可靠性,边缘特征点和边缘特征点的匹配采用双向匹配技术又进一步保证了匹配的可靠性,实验结果表明,该算法效果良好,有实用价值。     

Coarse-to-fine stereo vision with accurate 3D boundaries

This paper presents methods for efficient recovery of accurate binocular disparity estimates in the vicinity of 3D surface discontinuities. Of particular concern are methods that impact coarse-to-fine, local block-based matching as it forms the basis of the fastest and the most resource efficient stereo computation procedures. A novel coarse-to-fine refinement procedure that adapts match window support across scale to ameliorate corruption of disparity estimates near boundaries is presented. Extensions are included to account for half-occlusions and colour uniformity. Empirical results show that incorporation of these advances in the standard coarse-to-fine, block matching framework reduces disparity errors by more than a factor of two, while performing little extra computation, preserving low complexity and the parallel/pipeline nature of the framework. Moreover, the proposed advances prove to be beneficial for CTF global matchers as well.     

一种用于立体图像匹配的改进稀疏匹配算法

立体匹配有着广泛的应用前景,是计算机视觉领域的研究热点。立体匹配是立体视觉中最为关键和困难的一步,它的目标是计算标识匹配像素位置的视差图。文中提出的立体匹配算法基于置信传播（Belief Propagation,BP）。左图像首先经过非均匀采样,得到一个内容自适应的网格近似表示。算法的关键是使用基于置信传播的立体匹配算法,匹配稀疏的左图像和右图像得到稀疏视差图。通过左图像得到网格,稀疏视差图可以经过简单的插值得到稠密视差图。实验结果表明,该方法与现有稀疏立体匹配技术相比在视差图质量上平均有40％的提高。     

融合边缘保持与改进代价聚合的立体匹配算法

目的 立体匹配是计算机双目视觉的重要研究方向,主要分为全局匹配算法与局部匹配算法两类。传统的局部立体匹配算法计算复杂度低,可以满足实时性的需要,但是未能充分利用图像的边缘纹理信息,因此在非遮挡、视差不连续区域的匹配精度欠佳。为此,提出了融合边缘保持与改进代价聚合的立体匹配。方法 首先利用图像的边缘空间信息构建权重矩阵,与灰度差绝对值和梯度代价进行加权融合,形成新的代价计算方式,同时将边缘区域像素点的权重信息与引导滤波的正则化项相结合,并在多分辨率尺度的框架下进行代价聚合。所得结果经过视差计算,得到初始视差图,再通过左右一致性检测、加权中值滤波等视差优化步骤获得最终的视差图。结果 在Middlebury立体匹配平台上进行实验,结果表明,融合边缘权重信息对边缘处像素点的代价量进行了更加有效地区分,能够提升算法在各区域的匹配精度。其中,未加入视差优化步骤的21组扩展图像对的平均误匹配率较改进前减少3.48%,峰值信噪比提升3.57 dB,在标准4幅图中venus上经过视差优化后非遮挡区域的误匹配率仅为0.18%。结论 融合边缘保持的多尺度立体匹配算法有效提升了图像在边缘纹理处的匹配精度,进一步降低了非遮挡区域与视差不连续区域的误匹配率。     

提取轮廓图背景控制点的立体匹配算法

针对动态规划匹配算法的误匹配与狭窄遮挡物问题,提出一种在轮廓图中提取背景控制点的立体匹配算法,在立体图像对的轮廓图中选择背景控制点,利用动态规划在视差空间图像中搜索最优路径,根据视差约束以及狭窄遮挡物的判定公式完善视差图。仿真实验结果表明,该算法能够降低在视差不连续区域匹配上的误匹配率。