局部双目视差回归的目标距离估计

目的 双目视觉是目标距离估计问题的一个很好的解决方案。现有的双目目标距离估计方法存在估计精度较低或数据准备较繁琐的问题,为此需要一个可以兼顾精度和数据准备便利性的双目目标距离估计算法。方法 提出一个基于R-CNN（region convolutional neural network）结构的网络,该网络可以实现同时进行目标检测与目标距离估计。双目图像输入网络后,通过主干网络提取特征,通过双目候选框提取网络以同时得到左右图像中相同目标的包围框,将成对的目标框内的局部特征输入目标视差估计分支以估计目标的距离。为了同时得到左右图像中相同目标的包围框,使用双目候选框提取网络代替原有的候选框提取网络,并提出了双目包围框分支以同时进行双目包围框的回归;为了提升视差估计的精度,借鉴双目视差图估计网络的结构,提出了一个基于组相关和3维卷积的视差估计分支。结果 在KITTI（Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上进行验证实验,与同类算法比较,本文算法平均相对误差值约为3.2%,远小于基于双目视差图估计算法（11.3%）,与基于3维目标检测的算法接近（约为3.9%）。另外,提出的视差估计分支改进对精度有明显的提升效果,平均相对误差值从5.1%下降到3.2%。通过在另外采集并标注的行人监控数据集上进行类似实验,实验结果平均相对误差值约为4.6%,表明本文方法可以有效应用于监控场景。结论 提出的双目目标距离估计网络结合了目标检测与双目视差估计的优势,具有较高的精度。该网络可以有效运用于车载相机及监控场景,并有希望运用于其他安装有双目相机的场景。     

基于分层视差估计的立体图象编码

基于立体视频数据压缩的目的，提出了一种基于分层视差估计/补偿的立体图象编码方案。该方案是采用JPEG标准独立编码参数图象，并利用视差估计/补偿技术编码目标图象，应用分层马尔可夫随机场（MRF）模型。率失真（RD）模型以及交叠块匹配的混合视差估计/补偿算法，可得到光滑准确的视差场，与通常的变尺寸块匹配（VSBM）相比，该算法得到的视差补偿图象的峰值信噪比（PSNR）可提高2.5dB左右；双向半像素精度的视差估计/补偿的性能要比单向整像素搜索提高3dB，实验结果表明，该立体图象编码方案能有效地压缩立体图象数据，并可推广到立体序列图象的编码压缩中。     

Optimizing stereo‐to‐multiview conversion for autostereoscopic displays

We present a novel stereo‐to‐multiview video conversion method for glasses‐free multiview displays. Different from previous stereo‐to‐multiview approaches, our mapping algorithm utilizes the limited depth range of autostereoscopic displays optimally and strives to preserve the scene's artistic composition and perceived depth even under strong depth compression. We first present an investigation of how perceived image quality relates to spatial frequency and disparity. The outcome of this study is utilized in a two‐step mapping algorithm, where we (i) compress the scene depth using a non‐linear global function to the depth range of an autostereoscopic display and (ii) enhance the depth gradients of salient objects to restore the perceived depth and salient scene structure. Finally, an adapted image domain warping algorithm is proposed to generate the multiview output, which enables overall disparity range extension.     

Joint Image Denoising and Disparity Estimation via Stereo Structure PCA and Noise-Tolerant Cost

Stereo cameras are now commonly available on cars and mobile phones. However, the captured images may suffer from low image quality under noisy conditions, producing inaccurate disparity. In this paper, we aim at jointly restoring a clean image pair and estimating the corresponding disparity. To this end, we propose a new joint framework that iteratively optimizes these two different tasks in a multiscale fashion. First, structure information between the stereo pair is utilized to denoise the images using a non-local means strategy. Second, a new noise-tolerant cost function is proposed for noisy stereo matching. These two terms are integrated into a multiscale framework in which cross-scale information is leveraged to further improve both denoising and stereo matching. Extensive experiments on datasets captured from indoor, outdoor, and low-light conditions show that the proposed method achieves superior performance than the state-of-the-art image denoising and disparity estimation methods. While it outperforms multi-image denoising methods by about 2 dB on average, it achieves a 50% error reduction over radiometric-change-robust stereo matching on the challenging KITTI dataset.     

Adaptive disparity computation using local and non-local cost aggregations

A new method is proposed to adaptively compute the disparity of stereo matching by choosing one of the alternative disparities from local and non-local disparity maps. The initial two disparity maps can be obtained from state-of-the-art local and non-local stereo algorithms. Then, the more reasonable disparity is selected. We propose two strategies to select the disparity. One is based on the magnitude of the gradient in the left image, which is simple and fast. The other utilizes the fusion move to combine the two proposal labelings (disparity maps) in a theoretically sound manner, which is more accurate. Finally, we propose a texture-based sub-pixel refinement to refine the disparity map. Experimental results using Middlebury datasets demonstrate that the two proposed selection strategies both perform better than individual local or non-local algorithms. Moreover, the proposed method is compatible with many local and non-local algorithms that are widely used in stereo matching.     

Utilization of stereo disparity and optical flow information for the computer analysis of human interactions

We propose a system that simultaneously utilizes the stereo disparity and optical flow information of real-time stereo grayscale multiresolution images for the recognition of objects and gestures in human interactions. For real-time calculation of the disparity and optical flow information of a stereo image, the system first creates pyramid images using a Gaussian filter. The system then determines the disparity and optical flow of a low-density image and extracts attention regions in a high-density image. The three foremost regions are recognized using higher-order local autocorrelation features and linear discriminant analysis. As the recognition method is view based, the system can process the face and hand recognitions simultaneously in real time. The recognition features are independent of parallel translations, so the system can use unstable extractions from stereo depth information. We demonstrate that the system can discriminate the users, monitor the basic movements of the user, smoothly learn an object presented by users, and can communicate with users by hand signs learned in advance. Received: 31 January 2000 / Accepted: 1 May 2001 Correspondence to: I. Yoda (e-mail: yoda@ieee.org, Tel.: +81-298-615941, Fax: +81-298-613313)     

A LSS-based registration of stereo thermal–visible videos of multiple people using belief propagation

In this paper, we propose a novel stereo method for registering foreground objects in a pair of thermal and visible videos of close-range scenes. In our stereo matching, we use Local Self-Similarity (LSS) as similarity metric between thermal and visible images. In order to accurately assign disparities to depth discontinuities and occluded Region Of Interest (ROI), we have integrated color and motion cues as soft constraints in an energy minimization framework. The optimal disparity map is approximated for image ROIs using a Belief Propagation (BP) algorithm. We tested our registration method on several challenging close-range indoor video frames of multiple people at different depths, with different clothing, and different poses. We show that our global optimization algorithm significantly outperforms the existing state-of-the art method, especially for disparity assignment of occluded people at different depth in close-range surveillance scenes and for relatively large camera baseline.     

Multiple paths extraction in images using a constrained expanded trellis

Single shortest path extraction algorithms have been used in a number of areas such as network flow and image analysis. In image analysis, shortest path techniques can be used for object boundary detection, crack detection, or stereo disparity estimation. Sometimes one needs to find multiple paths as opposed to a single path in a network or an image where the paths must satisfy certain constraints. In this paper, we propose a new algorithm to extract multiple paths simultaneously within an image using a constrained expanded trellis (CET) for feature extraction and object segmentation. We also give a number of application examples for our multiple paths extraction algorithm.     

Stereo matching algorithm with guided filter and modified dynamic programming

Dense stereo correspondence is a challenging research problem in computer vision field. To address the poor accuracy behavior of stereo matching, we propose a novel stereo matching algorithm based on guided image filter and modified dynamic programming. Firstly, we suggest a combined matching cost by incorporating the absolute difference and improved color census transform (ICCT). Secondly, we use the guided image filter to filter the cost volume, which can aggregate the costs fast and efficiently. Then, in the disparity computing step, we design a modified dynamic programming algorithm, which can weaken the scanning line effect. At last, final disparity maps are gained after post-processing. The experimental results are evaluated on Middlebury Stereo Datasets, showing that our approach can achieve good results both in low texture and depth discontinuity areas with an average error rate of 5.14 % and strong robustness.     

A sub-pixel stereo matching algorithm and its applications in fabric imaging

In this paper, we describe a sub-pixel stereo matching algorithm where disparities are iteratively refined within a regularization framework. We choose normalized cross-correlation as the matching metric, and perform disparity refinement based on correlation gradients, which is distinguished from intensity gradient-based methods. We propose a discontinuity-preserving regularization technique which utilizes local coherence in the disparity space image, instead of estimating discontinuities in the intensity images. A concise numerical solution is derived by parameterizing the disparity space with dense bicubic B-splines. Experimental results show that the proposed algorithm performs better than correlation fitting methods without regularization. The algorithm has been implemented for applications in fabric imaging. We have shown its potentials in wrinkle evaluation, drape measurement, and pilling assessment.     

Estimating optimal parameters for MRF stereo from a single image pair

This paper presents a novel approach for estimating the parameters for MRF-based stereo algorithms. This approach is based on a new formulation of stereo as a maximum a posterior (MAP) problem in which both a disparity map and MRF parameters are estimated from the stereo pair itself. We present an iterative algorithm for the MAP estimation that alternates between estimating the parameters while fixing the disparity map and estimating the disparity map while fixing the parameters. The estimated parameters include robust truncation thresholds for both data and neighborhood terms, as well as a regularization weight. The regularization weight can be either a constant for the whole image or spatially-varying, depending on local intensity gradients. In the latter case, the weights for intensity gradients are also estimated. Our approach works as a wrapper for existing stereo algorithms based on graph cuts or belief propagation, automatically tuning their parameters to improve performance without requiring the stereo code to be modified. Experiments demonstrate that our approach moves a baseline belief propagation stereo algorithm up six slots in the Middlebury rankings     

Progressive Linear Search for Stereo Matching and Its Application to Interframe Interpolation

In this paper we present a stereo matching strategy that represents disparity as a linear piecewise function. The function is obtained by recursively subdividing intervals in corresponding scanline pairs. Each subdivision step delineates new intervals by explicitly searching for breaks of disparity. In contrast to most approaches, we do not assume a constant disparity within a region, but we define disparity values by a linear model. A disparity model provides strong constraints in the estimation problem giving spatial coherence. Parametric models are estimated by minimizing the similarity error via the Hough transform. A regularization cost is included during the subdivision process by considering disparity values between consecutive intervals. Experiments on synthetic and real images show that our adaptive matching strategy is capable of obtaining good detail with a small number of spurious points even if scanlines are processed independently and without using any postprocessing smoothing. We have successfully applied our matching results to create realistic image sequences using pixel-based interpolation. Occluded regions are identified by overlapping intervals and they are displayed by using a back-to-front strategy.     

基于区域间协同优化的立体匹配算法

提出了一种基于分割区域间协同优化的立体匹配算法. 该算法以图像区域为匹配基元, 利用区域的彩色特征以及相邻区域间应满足的平滑和遮挡关系定义了区域的匹配能量函数, 并引入区域之间的合作竞争机制, 通过协同优化使所定义的匹配能量极小化, 从而得到比较理想的视差结果. 算法首先对参考图像进行分割, 利用相关法得到各分割区域的初始匹配; 然后用平面模型对各区域的视差进行拟合, 得到各区域的视差平面参数; 最后, 基于协同优化的思想, 采用局部优化的方法对各区域的视差平面参数进行迭代优化, 直至得到比较合理的视差图为止. 采用Middlebury test set进行的实验结果表明, 该方法在性能上可以和目前最好的立体匹配算法相媲美, 得到的视差结果接近于真实视差.     

Hierarchical stereo matching with image bit-plane slicing

We propose a new stereo matching framework based on image bit-plane slicing. A pair of image sequences with various intensity quantization levels constructed by taking different bit-rate of the images is used for hierarchical stereo matching. The basic idea is to use the low bit-rate image pairs to compute rough disparity maps. The hierarchical matching strategy is then carried out iteratively to update the low confident disparities with the information provided by extra image bit-planes. It is shown that, depending on the stereo matching algorithms, even the image pairs with low intensity quantization are able to produce fairly good disparity results. Consequently, variate bit-rate matching is performed only regionally in the images for each iteration, and the average image bit-rate for disparity computation is reduced. Our method provides a hierarchical matching framework and can be combined with the existing stereo matching algorithms. Experiments on Middlebury datasets show that the proposed technique gives good results compared to the conventional full bit-rate matching.     

Content-aware disparity adjustment for different stereo displays

In this paper, we present an effective disparity mapping method for binocular stereoscopic image. It is inspired by the observation that its displayed depth would change, when a stereoscopic image is displayed on different size screens. The phenomenon may bring an uncomfortable experience for viewers. To make a comfortable stereoscopic image for viewers, moreover to adapt a stereoscopic image to a target display screen, we propose a content-aware disparity adjustment method. Firstly, the disparity mapping is established to control and retarget the depth of a stereoscopic scene. Then, the relationship between the disparity editing and image content editing is established to guide the proposed warping model. At last, to implement the disparity mapping operator, we propose a content-aware stereoscopic mesh warping model, which can simultaneously avoid the salient region distortion and adjust disparity to a target range by establishing the relationship. Experimental results show that the proposed method can effectively adjust disparity of stereoscopic image, which not only avoids the salient region distortion and adjusts disparity to a target range.     

Binocular stereo algorithm based on the disparity-gradient limit and using optimization theory

A new algorithm for stereo matching is presented, based on the idea of imposing a limit on disparity gradients allowed in the matched image. The matching problem will be expressed as one of maximizing a certain function, subject to constraints. Standard methods from optimization theory may then be used to find a solution.     

Stereo matching using weighted dynamic programming on a single-direction four-connected tree

In recent years, stereo matching based on dynamic programming (DP) has been widely studied and various tree structures are proposed to improve the matching accuracy. However, previous DP-based algorithms do not incorporate all the smoothness functions determined by the edges between the adjacent pixels in the image, which will usually lead to lower matching accuracies. In this paper, we propose a novel stereo matching algorithm based on weighted dynamic programming on a single-direction four-connected (SDFC) tree. The SDFC tree structure is a new tree structure which includes all the edges in the image and the disparity of a pixel can be affected by all the edges in the image. However, in the SDFC tree, conventional DP-based algorithms will make the pixels that are far away from the root node provide higher energy than the nearby pixels, which will decrease the matching accuracy. So, the weighted dynamic programming approach is proposed to optimize the energy function on the new tree structure, and all the pixels in the SDFC tree are treated equivalently. Dynamic programming in the SDFC tree of every pixel in the image separately is very time-consuming, so a fast DP optimization method is designed for the SDFC tree, which reduces the computational complexity of the proposed weighted DP algorithm to 12 times of conventional DP based algorithm. Experiments show that our algorithm not only produces quite smooth and reasonable disparity maps which are close to the state-of-the-art results, but also can be implemented quite efficiently. Performance evaluations on the Middlebury data set show that our method ranks top in all the DP-based stereo matching algorithms, even better than the algorithms that apply segmentation techniques. Experimental results in an unmanned ground vehicle (UGV) test bed show that our algorithm gets very good matching results in different outdoor conditions, even on the asphaltic road which is considered to be textureless. This illustrates the robustness of our algorithm.     

基于双目视觉的人脸三维重建

基于双目立体匹配算法PatchMatch算法,提出了一种获取人脸三维点云的算法。该算法对局部立体匹配算法PatchMatch进行了优化。该方法既不需要昂贵的设备,也不需要通用的人脸三维模型,而是结合了人脸的拓扑结构信息以及立体视觉局部优化算法。此方法采用非接触式的双目视觉采集技术获取左右视角的人脸图像,利用回归树集合(ensemble of regression trees,ERT)算法对人脸图像进行关键点定位,恢复人脸稀疏的视差估计,运用线性插值方法初步估计脸部的稠密视差值,并结合局部立体匹配算法对得到的视差结果进行平滑处理,重建人脸的三维点云信息。实验结果表明,这种算法能够还原出光滑的稠密人脸三维点云信息,在人脸Bosphorus数据库上取得了更加准确的人脸重建结果。     

Sampling the disparity space image

A central issue in stereo algorithm design is the choice of matching cost. Many algorithms simply use squared or absolute intensity differences based on integer disparity steps. In this paper, we address potential problems with such approaches. We begin with a careful analysis of the properties of the continuous disparity space image (DSI) and propose several new matching cost variants based on symmetrically matching interpolated image signals. Using stereo images with ground truth, we empirically evaluate the performance of the different cost variants and show that proper sampling can yield improved matching performance.     

A Single Image Representation Model for Efficient Stereoscopic Image Creation

Computer graphics is one of the most efficient ways to create a stereoscopic image. The process of stereoscopic CG generation is, however, still very inefficient compared to that of monoscopic CG generation. Despite that stereo images are very similar to each other, they are rendered and manipulated independently. Additional requirements for disparity control specific to stereo images lead to even greater inefficiency. This paper proposes a method to reduce the inefficiency accompanied in the creation of a stereoscopic image. The system automatically generates an optimized single image representation of the entire visible area from both cameras. The single image can be easily manipulated with conventional techniques, as it is spatially smooth and maintains the original shapes of scene objects. In addition, a stereo image pair can be easily generated with an arbitrary disparity setting. These convenient and efficient features are achieved by the automatic generation of a stereo camera pair, robust occlusion detection with a pair of Z‐buffers, an optimization method for spatial smoothness, and stereo image pair generation with a non‐linear disparity adjustment. Experiments show that our technique dramatically improves the efficiency of stereoscopic image creation while preserving the quality of the results.