Shape from Shading and Photometric Stereo Using Surface Approximation by Legendre Polynomials

In this paper, a new iterative shape from shading (SFS) algorithm is proposed. In the proposed algorithm, the given 3D surface is approximated by Legendre polynomials and the relationships between the given surface and its derivatives are represented in matrix forms using a polynomial coefficient vector. Then the relative depth and its derivatives are iteratively computed by updating the coefficient vector. Also the proposed SFS algorithm is extended to a photometric stereo case. In the proposed photometric stereo algorithm, the reflectance map is linearized and the cost function expressed in quadratic matrix form is minimized. The relative depth and its derivatives are also obtained by updating them iteratively. Performance of the proposed SFS and photometric stereo algorithms is evaluated in terms of three different error measures: the brightness error, orientation error, and height error. In addition, a performance comparison of the proposed and conventional SFS algorithms is shown.     

Realtime Object-aware Monocular Depth Estimation in Onboard Systems

This paper proposes the object depth estimation in real-time, using only a monocular camera in an onboard computer with a low-cost GPU. Our algorithm estimates scene depth from a sparse feature-based visual odometry algorithm and detects/tracks objects’ bounding box by utilizing the existing object detection algorithm in parallel. Both algorithms share their results, i.e., feature, motion, and bounding boxes, to handle static and dynamic objects in the scene. We validate the scene depth accuracy of sparse features with KITTI and its ground-truth depth map made from LiDAR observations quantitatively, and the depth of detected object with the Hyundai driving datasets and satellite maps qualitatively. We compare the depth map of our algorithm with the result of (un-) supervised monocular depth estimation algorithms. The validation shows that our performance is comparable to that of monocular depth estimation algorithms which train depth indirectly (or directly) from stereo image pairs (or depth image), and better than that of algorithms trained with monocular images only, in terms of the error and the accuracy. Also, we confirm that our computational load is much lighter than the learning-based methods, while showing comparable performance.

     

A parallel stereo algorithm that produces dense depth maps and preserves image features

To compute reliable dense depth maps, a stereo algorithm must preserve depth discontinuities and avoid gross errors. In this paper, we show how simple and parallel techniques can be combined to achieve this goal and deal with complex real world scenes. Our algorithm relies on correlation followed by interpolation. During the correlation phase the two images play a symmetric role and we use a validity criterion for the matches that eliminate gross errors: at places where the images cannot be correlated reliably, due to lack of texture of occlusions for example, the algorithm does not produce wrong matches but a very sparse disparity map as opposed to a dense one when the correlation is successful. To generate a dense depth map, the information is then propagated across the featureless areas, but not across discontinuities, by an interpolation scheme that takes image grey levels into account to preserve image features. We show that our algorithm performs very well on difficult images such as faces and cluttered ground level scenes. Because all the algorithms described here are parallel and very regular they could be implemented in hardware and lead to extremely fast stereo systems.This research was supported in part under the Centre National d'Etudes Spatiales VAP contract and in part under a Defence Advanced Research Projects Agency contract at SRI     

一种用于立体图像匹配的改进稀疏匹配算法

立体匹配有着广泛的应用前景,是计算机视觉领域的研究热点。立体匹配是立体视觉中最为关键和困难的一步,它的目标是计算标识匹配像素位置的视差图。文中提出的立体匹配算法基于置信传播（Belief Propagation,BP）。左图像首先经过非均匀采样,得到一个内容自适应的网格近似表示。算法的关键是使用基于置信传播的立体匹配算法,匹配稀疏的左图像和右图像得到稀疏视差图。通过左图像得到网格,稀疏视差图可以经过简单的插值得到稠密视差图。实验结果表明,该方法与现有稀疏立体匹配技术相比在视差图质量上平均有40％的提高。     

3D Scene Reconstruction from Multiple Spherical Stereo Pairs

We propose a 3D environment modelling method using multiple pairs of high-resolution spherical images. Spherical images of a scene are captured using a rotating line scan camera. Reconstruction is based on stereo image pairs with a vertical displacement between camera views. A 3D mesh model for each pair of spherical images is reconstructed by stereo matching. For accurate surface reconstruction, we propose a PDE-based disparity estimation method which produces continuous depth fields with sharp depth discontinuities even in occluded and highly textured regions. A full environment model is constructed by fusion of partial reconstruction from spherical stereo pairs at multiple widely spaced locations. To avoid camera calibration steps for all camera locations, we calculate 3D rigid transforms between capture points using feature matching and register all meshes into a unified coordinate system. Finally a complete 3D model of the environment is generated by selecting the most reliable observations among overlapped surface measurements considering surface visibility, orientation and distance from the camera. We analyse the characteristics and behaviour of errors for spherical stereo imaging. Performance of the proposed algorithm is evaluated against ground-truth from the Middlebury stereo test bed and LIDAR scans. Results are also compared with conventional structure-from-motion algorithms. The final composite model is rendered from a wide range of viewpoints with high quality textures.     

结合同场景立体图对的高质量深度图像重建

目的 越来越多的应用依赖于对场景深度图像准确且快速的观测和分析,如机器人导航以及在电影和游戏中对虚拟场景的设计建模等.飞行时间深度相机等直接的深度测量设备可以实时的获取场景的深度图像,但是由于硬件条件的限制,采集的深度图像分辨率比较低,无法满足实际应用的需要.通过立体匹配算法对左右立体图对之间进行匹配获得视差从而得到深度图像是计算机视觉的一种经典方法,但是由于左右图像之间遮挡以及无纹理区域的影响,立体匹配算法在这些区域无法匹配得到正确的视差,导致立体匹配算法在实际应用中存在一定的局限性.方法 结合飞行时间深度相机等直接的深度测量设备和立体匹配算法的优势,提出一种新的深度图像重建方法.首先结合直接的深度测量设备采集的深度图像来构造自适应局部匹配权值,对左右图像之间的局部窗立体匹配过程进行约束,得到基于立体匹配算法的深度图像;然后基于左右检测原理将采集到的深度图像和匹配得到的深度图像进行有效融合;接着提出一种局部权值滤波算法,来进一步提高深度图像的重建质量.结果 实验结果表明,无论在客观指标还是视觉效果上,本文提出的深度图像重建算法较其他立体匹配算法可以得到更好的结果.其中错误率比较实验表明,本文算法较传统的立体匹配算法在深度重建错误率上可以提升10％左右.峰值信噪比实验结果表明,本文算法在峰值信噪比上可以得到10 dB左右的提升.结论 提出的深度图像重建方法通过结合高分辨率左右立体图对和初始的低分辨率深度图像,可以有效地重建高质量高分辨率的深度图像.     

Dense Features for Semi-Dense Stereo Correspondence

We present a new feature based algorithm for stereo correspondence. Most of the previous feature based methods match sparse features like edge pixels, producing only sparse disparity maps. Our algorithm detects and matches dense features between the left and right images of a stereo pair, producing a semi-dense disparity map. Our dense feature is defined with respect to both images of a stereo pair, and it is computed during the stereo matching process, not a preprocessing step. In essence, a dense feature is a connected set of pixels in the left image and a corresponding set of pixels in the right image such that the intensity edges on the boundary of these sets are stronger than their matching error (which is the difference in intensities between corresponding boundary pixels). Our algorithm produces accurate semi-dense disparity maps, leaving featureless regions in the scene unmatched. It is robust, requires little parameter tuning, can handle brightnessdifferences between images, nonlinear errors, and is fast (linear complexity).     

由单幅二维灰度图像重构物体表面形状*

鉴于目前很少有论文讨论完整的由单幅二维灰度图像重构物体表面形状的算法,包括它的控制参数的估计及算法的实现,介绍了一种完整的SFS算法.它在考虑自遮掩影响的情况下,有效地估计了SFS算法中涉及的各种控制参数,并引入亮度约束、灰度梯度约束和可积性约束,计算出表面高度和表面向量,实现三维重构.与传统的算法相比,本算法无论是在速度还是在精度方面都达到了比较高的水平,具有一定的应用前景.最后还指出了在MATLAB中实现需要注意的问题.     

多方法相融合的复杂物体深度信息的恢复

在恢复图象深度信息的方法之中,利用立体视觉的偏差来精确地定位物体的深度,是行之有效的,但只能适用于可匹配的特征点,如何建立左右图象中对应点的匹配是该方法的主要障碍;Ｓｈａｐｅ Ｆｒｏｍ Ｓｈａｄｉｎｇ方法是利用单幅图象的灰度信息获取物体表面的形状信息（表面的方向）,而不能获得其深度信息,其约束条件是表面的光滑性。在此用神经网络方法将二者融合起来,形成优势互补,用来获取物体的深度信息,通过对合成图象     

三维图像重构的参数估计与算法实现

论文介绍了一种SFS算法的参数估计及其实现。它在考虑自遮掩影响的情况下,有效地估计了SFS算法中涉及的各种控制参数,并引入亮度约束、灰度梯度约束和可积性约束,计算出表面高度和表面向量,实现三维重构。最后还指出了在Matlab中实现需要注意的问题。     

Forward-backward study of the stereo vision problem

In stereo vision a pair of two-dimensional (2D) stereo images is given and the purpose is to find out the depth (disparity) of regions of the image in relation to the background, so that we can reconstruct the 3D structure of the image from the pair of 2D stereo images given. Using the Bayesian framework we implemented the Forward-Backward algorithm to unfold the disparity (depth) of a pair of stereo images. The results showed are very reasonable, but we point out there is room for improvement concerning the graph structure used.     

Stereo reconstruction from multiperspective panoramas

A new approach to computing a panoramic (360 degrees) depth map is presented in this paper. Our approach uses a large collection of images taken by a camera whose motion has been constrained to planar concentric circles. We resample regular perspective images to produce a set of multiperspective panoramas and then compute depth maps directly from these resampled panoramas. Our panoramas sample uniformly in three dimensions: rotation angle, inverse radial distance, and vertical elevation. The use of multiperspective panoramas eliminates the limited overlap present in the original input images and, thus, problems as in conventional multibaseline stereo can be avoided. Our approach differs from stereo matching of single-perspective panoramic images taken from different locations, where the epipolar constraints are sine curves. For our multiperspective panoramas, the epipolar geometry, to the first order approximation, consists of horizontal lines. Therefore, any traditional stereo algorithm can be applied to multiperspective panoramas with little modification. In this paper, we describe two reconstruction algorithms. The first is a cylinder sweep algorithm that uses a small number of resampled multiperspective panoramas to obtain dense 3D reconstruction. The second algorithm, in contrast, uses a large number of multiperspective panoramas and takes advantage of the approximate horizontal epipolar geometry inherent in multiperspective panoramas. It comprises a novel and efficient 1D multibaseline matching technique, followed by tensor voting to extract the depth surface. Experiments show that our algorithms are capable of producing comparable high quality depth maps which can be used for applications such as view interpolation.     

A two-level generative model for cloth representation and shape from shading

In this paper, we present a two-level generative model for representing the images and surface depth maps of drapery and clothes. The upper level consists of a number of folds which will generate the high contrast (ridge) areas with a dictionary of shading primitives (for 2D images) and fold primitives (for 3D depth maps). These primitives are represented in parametric forms and are learned in a supervised learning phase using 3D surfaces of clothes acquired through photometric stereo. The lower level consists of the remaining flat areas which fill between the folds with a smoothness prior (Markov random field). We show that the classical ill-posed problem-shape from shading (SFS) can be much improved by this two-level model for its reduced dimensionality and incorporation of middle-level visual knowledge, i.e., the dictionary of primitives. Given an input image, we first infer the folds and compute a sketch graph using a sketch pursuit algorithm as in the primal sketch (Guo et al., 2003). The 3D folds are estimated by parameter fitting using the fold dictionary and they form the "skeleton" of the drapery/cloth surfaces. Then, the lower level is computed by conventional SFS method using the fold areas as boundary conditions. The two levels interact at the final stage by optimizing a joint Bayesian posterior probability on the depth map. We show a number of experiments which demonstrate more robust results in comparison with state-of-the-art work. In a broader scope, our representation can be viewed as a two-level inhomogeneous MRF model which is applicable to general shape-from-X problems. Our study is an attempt to revisit Marr's idea (Marr and Freeman, 1982) of computing the 2frac12D sketch from primal sketch. In a companion paper (Barbu and Zhu, 2005), we study shape from stereo based on a similar two-level generative sketch representation.     

Stereoscopic Scene Flow Computation for 3D Motion Understanding

Building upon recent developments in optical flow and stereo matching estimation, we propose a variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences. The proposed algorithm takes into account image pairs from two consecutive times and computes both depth and a 3D motion vector associated with each point in the image. In contrast to previous works, we partially decouple the depth estimation from the motion estimation, which has many practical advantages. The variational formulation is quite flexible and can handle both sparse or dense disparity maps. The proposed method is very efficient; with the depth map being computed on an FPGA, and the scene flow computed on the GPU, the proposed algorithm runs at frame rates of 20 frames per second on QVGA images (320×240 pixels). Furthermore, we present solutions to two important problems in scene flow estimation: violations of intensity consistency between input images, and the uncertainty measures for the scene flow result.     

Stereo retinex

The retinex algorithm for lightness and color constancy is extended to include 3-dimensional spatial information reconstructed from a stereo image. A key aspect of traditional retinex is that, within each color channel, it makes local spatial comparisons of intensity. In particular, intensity ratios are computed between neighboring spatial locations, retinex assumes that a large ratio indicates a change in surface reflectance, not a change in incident illumination; however, this assumption is often violated in 3-dimensional scenes, where an abrupt change in surface orientation can lead to a significant change in illumination. In this paper, retinex is modified to use the 3-dimensional edge information derived from stereo images. The edge map is used so that spatial comparisons are only made between locations lying on approximately the same plane in 3-dimensions. Experiments on real images show this method works well, however, they also reveal that it can lead to isolated regions, which, as a result of being isolated, are incorrectly determined to be grey. To overcome this problem, stereo retinex is extended to allow information that is orthogonal to the space of possible illuminants to propagate across changes in surface orientation. This is accomplished by transforming the original RGB image data into a color space based on coordinates of luminance, illumination and reflectance. This coordinate system allows stereo retinex to propagate reflectance information across changes in surface orientation, while at the same time inhibiting the propagation of potentially invalid illumination information. The stereo retinex algorithm builds upon the multi-resolution implementation of retinex known as McCann99. Experiments on synthetic and real images show that stereo retinex performs significantly better than unmodified McCann99 retinex when evaluated in terms of the accuracy with which correct surface object colors are estimated.     

Depth map inpainting via sparse distortion model

The depth map captured from a real scene by the Kinect motion sensor is always influenced by noise and other environmental factors. As a result, some depth information is missing from the map. This distortion of the depth map directly deteriorates the quality of the virtual viewpoints rendered in 3D video systems. We propose a depth map inpainting algorithm based on a sparse distortion model. First, we train the sparse distortion model using the distortion and real depth maps to obtain two learning dictionaries: one for distortion and one for real depth maps. Second, the sparse coefficients of the distortion and the real depth maps are calculated by orthogonal matching pursuit. We obtain the approximate features of the distortion from the relationship between the learning dictionary and the sparse coefficients of the distortion map. The noisy images are filtered by the joint space structure filter, and the extraction factor is obtained from the resulting image by the extraction factor judgment method. Finally, we combine the learning dictionary and sparse coefficients from the real depth map with the extraction factor to repair the distortion in the depth map. A quality evaluation method is proposed for the original real depth maps with missing pixels. The proposed method achieves better results than comparable methods in terms of depth inpainting and the subjective quality of the rendered virtual viewpoints.     

3-D Scene Data Recovery Using Omnidirectional Multibaseline Stereo

A traditional approach to extracting geometric information from a large scene is to compute multiple 3-D depth maps from stereo pairs or direct range finders, and then to merge the 3-D data. However, the resulting merged depth maps may be subject to merging errors if the relative poses between depth maps are not known exactly. In addition, the 3-D data may also have to be resampled before merging, which adds additional complexity and potential sources of errors.This paper provides a means of directly extracting 3-D data covering a very wide field of view, thus by-passing the need for numerous depth map merging. In our work, cylindrical images are first composited from sequences of images taken while the camera is rotated 360° about a vertical axis. By taking such image panoramas at different camera locations, we can recover 3-D data of the scene using a set of simple techniques: feature tracking, an 8-point structure from motion algorithm, and multibaseline stereo. We also investigate the effect of median filtering on the recovered 3-D point distributions, and show the results of our approach applied to both synthetic and real scenes.     

Overcoming shadows in 3-source photometric stereo

Light occlusions are one of the most significant difficulties of photometric stereo methods. When three or more images are available without occlusion, the local surface orientation is overdetermined so that shape can be computed and the shadowed pixels can be discarded. In this paper, we look at the challenging case when only two images are available without occlusion, leading to a one degree of freedom ambiguity per pixel in the local orientation. We show that, in the presence of noise, integrability alone cannot resolve this ambiguity and reconstruct the geometry in the shadowed regions. As the problem is ill-posed in the presence of noise, we describe two regularization schemes that improve the numerical performance of the algorithm while preserving the data. Finally, the paper describes how this theory applies in the framework of color photometric stereo where one is restricted to only three images and light occlusions are common. Experiments on synthetic and real image sequences are presented.