An FPGA stereo matching unit based on fuzzy logic

Stereo matching is one of the most used algorithms in real-time image processing applications such as positioning systems for mobile robots, three-dimensional building mapping and recognition, detection and three-dimensional reconstruction of objects. In order to improve the performance, stereo matching algorithms often have been implemented in dedicated hardware such as FPGA or GPU devices. In this paper an FPGA stereo matching unit based on fuzzy logic is described. The proposed algorithm consists of three stages. First, three similarity parameters inherent to each pixel contained in the input stereo pair are computed. Then, the similarity parameters are sent to a fuzzy inference system which determines a fuzzy-similarity value. Finally, the disparity value is defined as the index which maximizes the fuzzy-similarity values (zero up to d_max). Dense disparity maps are computed at a rate of 76 frames per second for input stereo pairs of 1280 × 1024 pixel resolution and a maximum expected disparity equal to 15. The developed FPGA architecture provides reduction of the hardware resource demand compared to other FPGA-based stereo matching algorithms: near to 72.35% for logic units and near to 32.24% for bits of memory. In addition, the developed FPGA architecture increases the processing speed: near to 34.90% pixels per second and outperforms the accuracy of most of real-time stereo matching algorithms in the state of the art.     

根据灰度值信息自适应窗口的半全局匹配

目的 立体匹配算法是立体视觉研究的关键点,算法的匹配精度和速度直接影响3维重建的效果。对于传统立体匹配算法来说,弱纹理区域、视差深度不连续区域和被遮挡区域的匹配精度依旧不理想,为此选择具有全局匹配算法和局部匹配算法部分优点、性能介于两种算法之间、且鲁棒性强的半全局立体匹配算法作为研究内容,提出自适应窗口与半全局立体匹配算法相结合的改进方向。方法 以通过AD（absolute difference）算法求匹配代价的半全局立体匹配算法为基础,首先改变算法匹配代价的计算方式,研究窗口大小对算法性能的影响,然后加入自适应窗口算法,研究自适应窗口对算法性能的影响,最后对改进算法进行算法性能评价与比较。结果 实验结果表明,匹配窗口的选择能够影响匹配算法性能、提高算法的适用范围,自适应窗口的加入能够提高算法匹配精度特别是深度不连续区域的匹配精度,并有效降低算法运行时间,对Cones测试图像集,改进的算法较改进前误匹配率在3个测试区域平均减少2.29%;对于所有测试图像集,算法运行时间较加入自适应窗口前平均减少28.5%。结论 加入自适应窗口的半全局立体匹配算法具有更优的算法性能,能够根据应用场景调节算法匹配精度和匹配速度。     

A fast stereo matching algorithm suitable for embedded real-time systems

In this paper, the challenge of fast stereo matching for embedded systems is tackled. Limited resources, e.g. memory and processing power, and most importantly real-time capability on embedded systems for robotic applications, do not permit the use of most sophisticated stereo matching approaches. The strengths and weaknesses of different matching approaches have been analyzed and a well-suited solution has been found in a Census-based stereo matching algorithm. The novelty of the algorithm used is the explicit adaption and optimization of the well-known Census transform in respect to embedded real-time systems in software. The most important change in comparison with the classic Census transform is the usage of a sparse Census mask which halves the processing time with nearly unchanged matching quality. This is due the fact that large sparse Census masks perform better than small dense masks with the same processing effort. The evidence of this assumption is given by the results of experiments with different mask sizes. Another contribution of this work is the presentation of a complete stereo matching system with its correlation-based core algorithm, the detailed analysis and evaluation of the results, and the optimized high speed realization on different embedded and PC platforms. The algorithm handles difficult areas for stereo matching, such as areas with low texture, very well in comparison to state-of-the-art real-time methods. It can successfully eliminate false positives to provide reliable 3D data. The system is robust, easy to parameterize and offers high flexibility. It also achieves high performance on several, including resource-limited, systems without losing the good quality of stereo matching. A detailed performance analysis of the algorithm is given for optimized reference implementations on various commercial of the shelf (COTS) platforms, e.g. a PC, a DSP and a GPU, reaching a frame rate of up to 75 fps for 640 × 480 images and 50 disparities. The matching quality and processing time is compared to other algorithms on the Middlebury stereo evaluation website reaching a middle quality and top performance rank. Additional evaluation is done by comparing the results with a very fast and well-known sum of absolute differences algorithm using several Middlebury datasets and real-world scenarios.     

Hierarchical stereo matching with image bit-plane slicing

We propose a new stereo matching framework based on image bit-plane slicing. A pair of image sequences with various intensity quantization levels constructed by taking different bit-rate of the images is used for hierarchical stereo matching. The basic idea is to use the low bit-rate image pairs to compute rough disparity maps. The hierarchical matching strategy is then carried out iteratively to update the low confident disparities with the information provided by extra image bit-planes. It is shown that, depending on the stereo matching algorithms, even the image pairs with low intensity quantization are able to produce fairly good disparity results. Consequently, variate bit-rate matching is performed only regionally in the images for each iteration, and the average image bit-rate for disparity computation is reduced. Our method provides a hierarchical matching framework and can be combined with the existing stereo matching algorithms. Experiments on Middlebury datasets show that the proposed technique gives good results compared to the conventional full bit-rate matching.     

Stereo rectification of pushbroom satellite images by robustly estimating the fundamental matrix

ABSTRACT

Stereo rectification is one of the most important steps for stereo matching and subsequently for digital surface model generation from satellite stereo images. This study proposes a new framework to rectify two pushbroom images along the epipolar geometry in order to omit the vertical parallax between two images. Here, we assume the interior and relative parameters between the two pushbroom images are not known and the images can be taken at different dates. Traditional stereo rectification methods of pushbroom images require metadata such as rational polynomial coefficients (RPCs), parameters of physical sensor model or ground control points (GCPs). In this study, we develop an image-based framework for stereo rectification, which works without the need for such data. In the proposed framework, the correspondences are densely extracted by a tilling strategy, and then the fundamental matrix is robustly estimated by two geometric constraints. Both affine and projective fundamental matrices could be used for stereo rectification from pushbroom stereo images. The results on IRS P5, World view III, GeoEye and IKONOS stereo pairs as well as on multi-date stereo images demonstrate that the pushbroom images are rectified with sub-pixel accuracy.     

A real-time global stereo-matching on FPGA

An improved global stereo matching algorithm is implemented on a single FPGA for real-time applications. Stereo matching is widely used in stereo vision systems, i.e. objects detection and autonomous vehicles. Global algorithms have much more accurate results than local algorithms, but global algorithms are not implemented on FPGA since they rely over high-end hardware resources. In this implementation the stereo pairs are divided into blocks, the hardware resources are reduced by processing one block once. The hardware implementation is based on a Xilinx Kintex 7 FPGA. Experiment results show that the proposed implementation has an accurate result for the Middlebury benchmarks and 30 frames per second (fps) @1920 × 1680 is achieved.     

Development of dense models of the Earth’s surface based on the semiglobal matching method for the case of multiple overlapping images

A method is developed for constructing a three-dimensional digital surface model based on the use of aerial images with multiple overlapping. The specific feature of this study is the use of the multiview matching method instead of stereo matching. The method is based on adapting the energy aggregation algorithm, which was proposed in the semiglobal matching (SGM) method, to the object space, as well as using the one-to-many scheme of cost calculation. The reconstructed scene is represented as a voxel grid. A high-performance implementation of the digital surface model construction at all stages is proposed based on the massive parallelization of computations on a graphics processing unit.     

Object oriented framework for real-time image processing on GPU

General purpose computation on graphics processing unit (GPGPU) provides a significant gain in terms of the processing time compared with CPU. Images are particularly good subjects for massive parallel implementations on GPU. Thus, the processing time can be improved for computer vision and image/video processing algorithms. However, GPGPU has a fairly complex integration process in a framework and they evolve very rapidly. In this paper, we present a framework that provides all the desired primitives related to GPGPU-based image processing algorithms, which makes it easy and straightforward for the user to exploit. The proposed framework is object-oriented, and it utilizes design patterns. The user can benefit from all the advantages of object-oriented programming, such as code reusability/extensibility, flexibility, information hiding, and complexity hiding. This makes it possible to rapidly integrate new technologies and functionality as they appear.     

融合边缘保持与改进代价聚合的立体匹配算法

目的 立体匹配是计算机双目视觉的重要研究方向,主要分为全局匹配算法与局部匹配算法两类。传统的局部立体匹配算法计算复杂度低,可以满足实时性的需要,但是未能充分利用图像的边缘纹理信息,因此在非遮挡、视差不连续区域的匹配精度欠佳。为此,提出了融合边缘保持与改进代价聚合的立体匹配。方法 首先利用图像的边缘空间信息构建权重矩阵,与灰度差绝对值和梯度代价进行加权融合,形成新的代价计算方式,同时将边缘区域像素点的权重信息与引导滤波的正则化项相结合,并在多分辨率尺度的框架下进行代价聚合。所得结果经过视差计算,得到初始视差图,再通过左右一致性检测、加权中值滤波等视差优化步骤获得最终的视差图。结果 在Middlebury立体匹配平台上进行实验,结果表明,融合边缘权重信息对边缘处像素点的代价量进行了更加有效地区分,能够提升算法在各区域的匹配精度。其中,未加入视差优化步骤的21组扩展图像对的平均误匹配率较改进前减少3.48%,峰值信噪比提升3.57 dB,在标准4幅图中venus上经过视差优化后非遮挡区域的误匹配率仅为0.18%。结论 融合边缘保持的多尺度立体匹配算法有效提升了图像在边缘纹理处的匹配精度,进一步降低了非遮挡区域与视差不连续区域的误匹配率。     

Probabilistic voxel mapping using an adaptive confidence measure of stereo matching

In this paper, we describe a probabilistic voxel mapping algorithm using an adaptive confidence measure of stereo matching. Most of the 3D mapping algorithms based on stereo matching usually generate a map formed by point cloud. There are many reconstruction errors. The reconstruction errors are due to stereo reconstruction error factors such as calibration errors, stereo matching errors, and triangulation errors. A point cloud map with reconstruction errors cannot accurately represent structures of environments and needs large memory capacity. To solve these problems, we focused on the confidence of stereo matching and probabilistic representation. For evaluation of stereo matching, we propose an adaptive confidence measure that is suitable for outdoor environments. The confidence of stereo matching can be reflected in the probability of restoring structures. For probabilistic representation, we propose a probabilistic voxel mapping algorithm. The proposed probabilistic voxel map is a more reliable representation of environments than the commonly used voxel map that just contains the occupancy information. We test the proposed confidence measure and probabilistic voxel mapping algorithm in outdoor environments.     

多维注意力特征聚合立体匹配算法

现有基于深度学习的立体匹配算法在学习推理过程中缺乏有效信息交互, 而特征提取和代价聚合两个子模块的特征维度存在差异, 导致注意力方法在立体匹配网络中应用较少、方式单一. 针对上述问题, 本文提出了一种多维注意力特征聚合立体匹配算法. 设计2D注意力残差模块, 通过在原始残差网络中引入无降维自适应2D注意力残差单元, 局部跨通道交互并提取显著信息, 为匹配代价计算提供丰富有效的特征. 构建3D注意力沙漏聚合模块, 以堆叠沙漏结构为骨干设计3D注意力沙漏单元, 捕获多尺度几何上下文信息, 进一步扩展多维注意力机制, 自适应聚合和重新校准来自不同网络深度的代价体. 在三大标准数据集上进行评估, 并与相关算法对比, 实验结果表明所提算法具有更高的预测视差精度, 且在无遮挡的显著对象上效果更佳.     

A Performance Study on Different Cost Aggregation Approaches Used in Real-Time Stereo Matching

Many vision applications require high-accuracy dense disparity maps in real-time and online. Due to time constraint, most real-time stereo applications rely on local winner-takes-all optimization in the disparity computation process. These local approaches are generally outperformed by offline global optimization based algorithms. However, recent research shows that, through carefully selecting and aggregating the matching costs of neighboring pixels, the disparity maps produced by a local approach can be more accurate than those generated by many global optimization techniques. We are therefore motivated to investigate whether these cost aggregation approaches can be adopted in real-time stereo applications and, if so, how well they perform under the real-time constraint. The evaluation is conducted on a real-time stereo platform, which utilizes the processing power of programmable graphics hardware. Six recent cost aggregation approaches are implemented and optimized for graphics hardware so that real-time speed can be achieved. The performances of these aggregation approaches in terms of both processing speed and result quality are reported.     

An automatized method to parameterize embedded stereo matching algorithms

Many applications rely on 3D information as a depth map. Stereo Matching algorithms reconstruct a depth map from a pair of stereoscopic images. Stereo Matching algorithms are computationally intensive, that is why implementing efficient stereo matching algorithms on embedded systems is very challenging for real-time applications.Indeed, like many vision algorithms, stereo matching algorithms have to set a lot of parameters and thresholds to work efficiently. When optimizing a stereo-matching algorithm, or changing algorithms parts, all those parameters have to be set manually. Finding the most efficient solution for a stereo-matching algorithm on a specific platform then becomes troublesome.This paper proposes an automatized method to find the optimal parameters of a dense stereo matching algorithm by learning from ground truth on a database in order to compare it with respect to any other alternative.Finally, for the C6678 platform, a map of the best compromise between quality and execution time is obtained, with execution times that are between 42 ms and 382 ms and output errors that are between 6% and 9.8%.     

Evaluation of stereo correspondence algorithms and their implementation on FPGA

The accuracy of stereo vision has been considerably improved in the last decade, but real-time stereo matching is still a challenge for embedded systems where the limited resources do not permit fast operation of sophisticated approaches. This work presents an evaluation of area-based algorithms used for calculating distance in stereoscopic vision systems, their hardware architectures for implementation on FPGA and the cost of their accuracies in terms of FPGA hardware resources. The results show the trade-off between the quality of such maps and the hardware resources which each solution demands, so they serve as a guide for implementing stereo correspondence algorithms in real-time processing systems.     

Joint Image Denoising and Disparity Estimation via Stereo Structure PCA and Noise-Tolerant Cost

Stereo cameras are now commonly available on cars and mobile phones. However, the captured images may suffer from low image quality under noisy conditions, producing inaccurate disparity. In this paper, we aim at jointly restoring a clean image pair and estimating the corresponding disparity. To this end, we propose a new joint framework that iteratively optimizes these two different tasks in a multiscale fashion. First, structure information between the stereo pair is utilized to denoise the images using a non-local means strategy. Second, a new noise-tolerant cost function is proposed for noisy stereo matching. These two terms are integrated into a multiscale framework in which cross-scale information is leveraged to further improve both denoising and stereo matching. Extensive experiments on datasets captured from indoor, outdoor, and low-light conditions show that the proposed method achieves superior performance than the state-of-the-art image denoising and disparity estimation methods. While it outperforms multi-image denoising methods by about 2 dB on average, it achieves a 50% error reduction over radiometric-change-robust stereo matching on the challenging KITTI dataset.     

Stereo Matching with Color-Weighted Correlation, Hierarchical Belief Propagation, and Occlusion Handling

In this paper, we formulate a stereo matching algorithm with careful handling of disparity, discontinuity, and occlusion. The algorithm works with a global matching stereo model based on an energy-minimization framework. The global energy contains two terms, the data term and the smoothness term. The data term is first approximated by a color-weighted correlation, then refined in occluded and low-texture areas in a repeated application of a hierarchical loopy belief propagation algorithm. The experimental results are evaluated on the Middlebury data sets, showing that our algorithm is the top performer among all the algorithms listed there.     

Real-time stereo using approximated joint bilateral filtering and dynamic programming

We present a stereo algorithm that is capable of estimating scene depth information with high accuracy and in real time. The key idea is to employ an adaptive cost-volume filtering stage in a dynamic programming optimization framework. The per-pixel matching costs are aggregated via a separable implementation of the bilateral filtering technique. Our separable approximation offers comparable edge-preserving filtering capability and leads to a significant reduction in computational complexity compared to the traditional 2D filter. This cost aggregation step resolves the disparity inconsistency between scanlines, which are the typical problem for conventional dynamic programming based stereo approaches. Our algorithm is driven by two design goals: real-time performance and high accuracy depth estimation. For computational efficiency, we utilize the vector processing capability and parallelism in commodity graphics hardware to speed up this aggregation process over two orders of magnitude. Over 90 million disparity evaluations per second [the number of disparity evaluations per seconds (MDE/s) corresponds to the product of the number of pixels and the disparity range and the obtained frame rate and, therefore, captures the performance of a stereo algorithm in a single number] are achieved in our current implementation. In terms of quality, quantitative evaluation using data sets with ground truth disparities shows that our approach is one of the state-of-the-art real-time stereo algorithms.     

一种新的基于特征点的立体匹配算法

目前,立体匹配是计算机视觉领域中最活跃的研究主题之一。为了快速并更精确的对特征点进行立体匹配,本文提出了一种新的基于特征点的立体匹配算法。该方法独立于特征点的检测算法,先以扫描线作为匹配单元,然后以鲁棒函数为匹配代价函数,最后用顺序约束对每一匹配单元的视差图进行检测与校正。实验证明,该方法的匹配精度高于传统的基于NCC(norm alized cross-correlation)的立体匹配算法,并且运行时间快,可以应用于纯软件的基于特征点的立体视觉系统中。     

Generalised residual images’ effect on illumination artifact removal for correspondence algorithms

The intensity (grey value) consistency of image pixels in a sequence or stereo camera setup is of central importance to numerous computer vision applications. Most stereo matching and optical flow algorithms minimise an energy function composed of a data term and a regularity or smoothing term. To date, well performing methods rely on the intensity consistency of the image pixel values to model the data term. Such a simple model fails if the illumination is (even slightly) different between the input images. Amongst other situations, this may happen due to background illumination change over the sequence, different reflectivity of a surface, vignetting, or shading effects.In this paper, we investigate the removal of illumination artifacts and show that generalised residual images substantially improve the accuracy of correspondence algorithms. In particular, we motivate the concept of residual images and show two evaluation approaches using either ground truth correspondence fields (for stereo matching and optical flow algorithms) or errors based on a predicted view (for stereo matching algorithms).