Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo

An integrated approach to extract depth, efficiently and accurately, from a sequence of images is presented in this paper. The method combines the ability of the stereo processing to acquire highly accurate depth measurements and the efficiency of spatial and temporal gradient analysis. As a result of this integration, depth measurements of high quality are obtained at a speed approximately ten times greater than that of stereo processing. Without any a priori information of the locations of the points in the scene, the correspondence problem in stereo processing is computationally expensive. In our approach, we use spatial and temporal gradient (STG) analysis, which has been shown to provide depth with great efficiency, but limited accuracy, to guide the matching process of stereo. The camera motion used in the approach can be either lateral or axial. Extensive experiments on real scenes have shown the ability of the integrated approach to acquire depth with a mean error of less than 3%.     

全向立体成像系统场景深度估计

基于单相机双抛物面反射镜设计了一种共轴结构的折反射全向立体成像装置,给出了针对相应展开柱面全景立体图像对深度估计的应点匹配方法,最后通过3D Max构造相应虚拟装置和虚拟场景进行了仿真实验,初步证明了该结构设计和对应深度估计方法的有效性。     

Building facade detection,segmentation, and parameter estimation for mobile robot stereo vision

Building facade detection is an important problem in computer vision, with applications in mobile robotics and semantic scene understanding. In particular, mobile platform localization and guidance in urban environments can be enabled with accurate models of the various building facades in a scene. Toward that end, we present a system for detection, segmentation, and parameter estimation of building facades in stereo imagery. The proposed method incorporates multilevel appearance and disparity features in a binary discriminative model, and generates a set of candidate planes by sampling and clustering points from the image with Random Sample Consensus (RANSAC), using local normal estimates derived from Principal Component Analysis (PCA) to inform the planar models. These two models are incorporated into a two-layer Markov Random Field (MRF): an appearance- and disparity-based discriminative classifier at the mid-level, and a geometric model to segment the building pixels into facades at the high-level. By using object-specific stereo features, our discriminative classifier is able to achieve substantially higher accuracy than standard boosting or modeling with only appearance-based features. Furthermore, the results of our MRF classification indicate a strong improvement in accuracy for the binary building detection problem and the labeled planar surface models provide a good approximation to the ground truth planes.     

A local stereo matching algorithm based on weighted guided image filtering for improving the generation of depth range images

Stereo matching is a challenging problem and highly accurate depth image is important in different applications. The main problem is to estimate the correspondence between two pixels in a stereo pair. To solve this problem, in the last decade, several cost aggregation methods aimed at improving the quality of stereo matching algorithms have been introduced. We propose a new cost aggregation method based on weighted guided image filtering (WGIF) for local stereo matching. The proposed algorithm solves multi-label problems in three steps. First, the cost volume is constructed using pixel-wise matching cost computation functions. Then, each slice of the cost volume is independently filtered using the WGIF, which substitutes for the smoothness term in the energy function. Finally, the disparity of any pixel is simply computed. The WGIF uses local weights based on a variance window of pixels in a guidance image for cost volume filtering. Experimental results using Middlebury stereo benchmark verify that the proposed method is effective due to a high quality cost volume filter.     

A real-time digitizer for stereo image processing

This paper discusses the design and implementation of a stereo image digitizer for use in digital image processing applications. The system operates in real time with respect to the standard television refresh rate and it possesses several signal sampling, routing, and transformation options which can be controlled from an external digital computer. Results obtained with a prototype unit are illustrated and compared using precision test patterns and natural scenes.     

A portable stereo vision system for whole body surface imaging

This paper presents a whole body surface imaging system based on stereo vision technology. We have adopted a compact and economical configuration which involves only four stereo units to image the frontal and rear sides of the body. The success of the system depends on a stereo matching process that can effectively segment the body from the background in addition to recovering sufficient geometric details. For this purpose, we have developed a novel sub-pixel, dense stereo matching algorithm which includes two major phases. In the first phase, the foreground is accurately segmented with the help of a predefined virtual interface in the disparity space image, and a coarse disparity map is generated with block matching. In the second phase, local least squares matching is performed in combination with global optimization within a regularization framework, so as to ensure both accuracy and reliability. Our experimental results show that the system can realistically capture smooth and complete whole body shapes with high accuracy.     

K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps

Both time-of-flight （ToF） cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are： （1） An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo; （2） A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.     

A parallel stereo algorithm that produces dense depth maps and preserves image features

To compute reliable dense depth maps, a stereo algorithm must preserve depth discontinuities and avoid gross errors. In this paper, we show how simple and parallel techniques can be combined to achieve this goal and deal with complex real world scenes. Our algorithm relies on correlation followed by interpolation. During the correlation phase the two images play a symmetric role and we use a validity criterion for the matches that eliminate gross errors: at places where the images cannot be correlated reliably, due to lack of texture of occlusions for example, the algorithm does not produce wrong matches but a very sparse disparity map as opposed to a dense one when the correlation is successful. To generate a dense depth map, the information is then propagated across the featureless areas, but not across discontinuities, by an interpolation scheme that takes image grey levels into account to preserve image features. We show that our algorithm performs very well on difficult images such as faces and cluttered ground level scenes. Because all the algorithms described here are parallel and very regular they could be implemented in hardware and lead to extremely fast stereo systems.This research was supported in part under the Centre National d'Etudes Spatiales VAP contract and in part under a Defence Advanced Research Projects Agency contract at SRI     

Evaluation of correspondence errors for stereo

The computation of a scalar correspondence error is the fundamental step in most stereo correspondence algorithms. The quality of the results obtained by the reconstruction algorithm directly depends on the characteristics of such error. We have developed a procedure to evaluate different methods proposed for the computation of the correspondence error. The evaluation is based on exploring the shape of the error surface generated and testing it for uniqueness, isolation and compatibility. The scheme presented makes it possible to recognise the known characteristics of the tested methods for the computation of a correspondence error from the results of the evaluations. Our results show that, for the tested scenes, the evaluation scheme allows us to identify the most appropriate method to compute the correspondence error.     

Model-based interpretation of stereo imagery of textured surfaces

We present a scheme for reliable and accurate surface reconstruction from stereoscopic images containing only fine texture and no stable high-level features. Partial shape information is used to improve surface computation: first by fitting an approximate, global, parametric model, and then by refining this model via local correspondence processes. This scheme eliminates the window size selection problem in existing area-based stereo correspondence schemes. These ideas are integrated in a practical vision system that is being used by environmental scientists to study wind erosion of bulk material such as coal ore being transported in open rail cars. Received: 14 August 1995 / Accepted: 27 May 1997     

Generating an image that affords slant perception from stereo,without pictorial cues

This paper describes an algorithm for generating a planar image that when tilted provides stereo cues to slant, without contamination from pictorial gradients. As the stimuli derived from this image are ultimately intended for use in studies of slant perception under magnification, a further requirement is that the generated image be suitable for high-definition printing or display on a monitor. A first stage generates an image consisting of overlapping edges with sufficient density that when zoomed, edges that nearly span the original scale are replaced with newly emergent content that leaves the visible edge statistics unchanged. A second stage reduces intensity clumping while preserving edges by enforcing a broad dynamic range across the image. Spectral analyses demonstrate that the low-frequency content of the resulting image, which would correspond to the pictorial cue of texture gradient changes under slant, (a) has a power fall-off deviating from 1/f noise (to which the visual system is particularly sensitive), and (b) does not offer systematic cues under changes in scale or slant. Two behavioral experiments tested whether the algorithm generates stimuli that offer cues to slant under stereo viewing only, and not when disparities are eliminated. With a particular adjustment of dynamic range (and nearly so with the other version that was tested), participants viewing without stereo cues were essentially unable to discriminate slanted from flat (frontal) stimuli, and when slant was reported, they failed to discriminate its direction. In contrast, non-stereo viewing of a control stimulus with pictorial cues, as well as stereoscopic observation, consistently allowed participants to perceive slant correctly. Experiment 2 further showed that these results generalized across a population of different stimuli from the same generation process and demonstrated that the process did not substitute biased slant cues.     

Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery

Augmented reality (AR) has received increasing attention in minimally invasive surgery (MIS) applications. The goal of applying AR techniques to MIS is to enhance a surgeon's perception of the spatial relationship by overlaying invisible structures (e.g. tumor or vessels) onto the in vivo endoscopic video acquired during the surgery. One of primary issues of AR visualization is to provide correct depth perception for visible and invisible structures. In this paper, we present a video-based AR system consisting of functional modules for real-time 3D surface capture, reconstruction, and registration with pre-operative segmented CT model. The real-time 3D registration allows precise overlay of invisible structures onto 2D video for AR visualization. The AR overlay result is displayed on a multi-view autostereoscopic lenticular LCD. To study and compare the efficacy of AR visualization techniques, we investigated five different AR visualization modes. Both simulated and in vivo experiments were carried out and autostereoscopic AR visualization results were given. Evaluation and comparison for depth perception between five AR visualization modes are presented. Finally, we conclude the characteristics of these visualization modes. The novelty of our work lies in successful implementation of an end-to-end 3D autostereoscopic AR system from real-time reconstruction and registration with our multi-channel 3D endoscope, and systematic evaluation and comparison of five different visualization modes for depth perception.     

The effect of depth map up-sampling on the overall quality of stereopairs

The quality of depth maps affects the quality of generated 3D content. Practically, the depth maps often have lower resolution than that of color images, thus, Depth map Up-sampling (DU) is needed in various 3D applications. DU can yield specific artifacts which can degrade the quality of depth maps as well as constructed stereoscopic (color plus depth map) images. This paper investigates the effect of DU on 3D perception. The depth maps were up-sampled using seven approaches and the quality of stereoscopic images obtained from up-sampled depth maps was estimated through subjective and objective tests. The objective quality prediction was performed using a depth map quality assessment framework. The method is able to predict the quality of stereoscopic images through evaluation of their corresponding up-sampled depth maps using 2D Image Quality Metrics (IQMs). In order to improve the quality estimation, the framework selects the 2D IQMs with highest correlation to subjective test. Furthermore, motivated by previous researches on multiple metrics combination, a new metric fusion method is proposed. Experimental results show that the combined metric delivers higher performance than single metrics in 3D quality prediction.     

A sub-pixel stereo matching algorithm and its applications in fabric imaging

In this paper, we describe a sub-pixel stereo matching algorithm where disparities are iteratively refined within a regularization framework. We choose normalized cross-correlation as the matching metric, and perform disparity refinement based on correlation gradients, which is distinguished from intensity gradient-based methods. We propose a discontinuity-preserving regularization technique which utilizes local coherence in the disparity space image, instead of estimating discontinuities in the intensity images. A concise numerical solution is derived by parameterizing the disparity space with dense bicubic B-splines. Experimental results show that the proposed algorithm performs better than correlation fitting methods without regularization. The algorithm has been implemented for applications in fabric imaging. We have shown its potentials in wrinkle evaluation, drape measurement, and pilling assessment.     

Image processing and synthesis for extended depth of field of optical microscopes

An optimal microscope is useful for observing various kinds of samples. However, precise observation of an extended region in depth is difficult, because of the narrow range of depth of focus. To overcome this problem, this paper proposes a method of obtaining a pan-focused stereoscopic image. By using image processing, in-focus areas are extracted from multiple images focused on slightly different depths, and a stereoscopic image is composed with these in-focus areas. The proposed method is applied to the observation of three-dimensional distributions of soil particles, a common problem in the field of civil engineering, and the usefulness of the proposed method is demonstrated.     

提高频率域可分级视频信源编码效率的研究

在ＭＰＥＧ－２视频编码标准的框架基础上，本文引入了一种提高频率域可分级视频编码效率的方法。通过在８×８ＤＣＴ块中合理地分割系数及尽量减少在低分辨率层中所必需传送的块结束码（ＥＯＢ）的数量，使为获得频率域可分级性所付出的代价明显减小。计算机模拟实验证实了采用该方法可提高编码效率     

A hybrid learning of multimodal cues for light field depth estimation

Learning based light field depth estimation methods have been proven to be successful, however, most of them focus on the depth cues extraction from a single representation of light field. In our opinion, existing methods are difficult to simultaneously learn various cues with different modalities, which are implied in different representations of light field. In this paper, we formulate the light field depth estimation as a pixel-wise classification task, and introduce a hybrid learning architecture to combine multimodal cues from multiple representations of light field. Three convolutional pathways are applied to predict the disparity label of each pixel from three modality representations of light field. Performing discrete disparity label classification instead of continuous disparity value regression allows us to easily fuse the prediction results of three pathways in the form of probability distribution. Evaluation experiments are carried out on the HCI 4D Light Field Benchmark. Our method, named as “FusionNet” on the benchmark website, ranks first among the published methods on high accuracy metrics of Bad Pixel Ratio 0.03 and 0.01.     

Fast stereo matching using adaptive guided filtering

Dense disparity map is required by many great 3D applications. In this paper, a novel stereo matching algorithm is presented. The main contributions of this work are three-fold. Firstly, a new cost-volume filtering method is proposed. A novel concept named “two-level local adaptation” is introduced to guide the proposed filtering approach. Secondly, a novel post-processing method is proposed to handle both occlusions and textureless regions. Thirdly, a parallel algorithm is proposed to efficiently calculate an integral image on GPU, and it accelerates the whole cost-volume filtering process. The overall stereo matching algorithm generates the state-of-the-art results. At the time of submission, it ranks the 10th among about 152 algorithms on the Middlebury stereo evaluation benchmark, and takes the 1st place in all local methods. By implementing the entire algorithm on the NVIDIA Tesla C2050 GPU, it can achieve over 30 million disparity estimates per second (MDE/s).     

基于立体折反射全向成像的柱面全景深度估算

针对立体视觉原理的新型立体折反射全向成像系统结构设计和面向立体柱面全景像对的局域灰度相关对应点快速匹配算法,从捕获的全向市体影像中提取有效深度信息,用于辅助全向视频分析处理中的对象检测和跟踪.采用单相机和两个不同参数的抛物面型反射镜构造了一种共轴结构的折反射全向立体成像装置,捕获的存在一定视差的原始全向立体像对被投影展开为立体柱面全景像对,而后通过特定对应点匹配算法提取稠密的深度信息.对应点匹配算法采用局部区域灰度相关的算了,并充分利用了双向匹配和柱面全景的外极线约束来提高匹配的速度和准确度.仿真实验有效恢复了场景深度信息,证明了整套装置结构设计及深度估计方法的有效性.     

Inter-view local texture analysis based stereo image reversible data hiding

To enhance security of three-dimensional images, an inter-view local texture analysis (ILTA) based stereo image reversible data hiding method is presented. Due to low accuracy of existing predictors, two novel predictors are proposed to improve the prediction precision. In the first predictor, a texture analysis model is built by using ILTA, in which the texture similarity between a pair of matched pixels in the stereo image is used to classify pixels into horizontal texture, vertical texture, smooth and complex types. Thus, the accurate prediction is adaptively computed by considering the pixel type. Moreover, an intra-view based predictor as the second predictor is also described to predict pixels by optimal weights finding (OWF). Since ILTA and OWF predictors are combined to predict pixels in the stereo image, sharp prediction error histograms of two views are both constructed, and then multi-level histogram shifting is used to embed secret data reversibly for obtaining low image distortion and high embedding capacity. Experimental results demonstrates that ILTA and OWF predictors can obtain precise predicted values, and the proposed data hiding method outperforms some state-of-the-art data hiding methods in terms of embedding capacity and quality of stego stereo image.