Vision and inertial sensor cooperation using gravity as a vertical reference

This paper explores the combination of inertial sensor data with vision. Visual and inertial sensing are two sensory modalities that can be explored to give robust solutions on image segmentation and recovery of 3D structure from images, increasing the capabilities of autonomous robots and enlarging the application potential of vision systems. In biological systems, the information provided by the vestibular system is fused at a very early processing stage with vision, playing a key role on the execution of visual movements such as gaze holding and tracking, and the visual cues aid the spatial orientation and body equilibrium. In this paper, we set a framework for using inertial sensor data in vision systems, and describe some results obtained. The unit sphere projection camera model is used, providing a simple model for inertial data integration. Using the vertical reference provided by the inertial sensors, the image horizon line can be determined. Using just one vanishing point and the vertical, we can recover the camera's focal distance and provide an external bearing for the system's navigation frame of reference. Knowing the geometry of a stereo rig and its pose from the inertial sensors, the collineations of level planes can be recovered, providing enough restrictions to segment and reconstruct vertical features and leveled planar patches.     

栅格法视觉传感集成及机器人实时避障

研究移动机器人在室内环境下集成双目视觉和激光测距仪信息进行障碍物实时检测。由双目视觉系统检测环境获取视差信息,通过直接对视差信息进行地平面拟合的方法快速检测障碍物;拟合过程中采用了随机采样一致性估计算法去除干扰点的影响,提高了障碍物检测的鲁棒性。用栅格地图表示基于机器人坐标系的地平面障碍物信息并对栅格信息进行提取,最后把双目视觉与激光测距得到的栅格信息进行集成。实验表明,通过传感信息集成,移动机器人既得到了充分的障碍物信息,又保证了检测的实时性、准确性。     

Error analysis in stereo determination of 3-d point positions

The relationship between the geometry of a stereo camera setup and the accuracy in obtaining three-dimensional position information is of great practical importance in many imaging applications. Assuming a point in a scene has been correctly identified in each image, its three-dimensional position can be recovered via a simple geometrical method known as triangulation. The probability that position estimates from triangulation are within some specified error tolerance is derived. An ideal pinhole camera model is used and the error is modeled as known spatial image plane quantization. A point's measured position maps to a small volume in 3-D determined by the finite resolution of the stereo setup. With the assumption that the point's actual position is uniformly distributed inside this volume, closed form expressions for the probability distribution of error in position along each coordinate direction (horizontal, vertical, and range) are derived. Following this, the probability that range error dominates over errors in the point's horizontal or vertical position is determined. It is hoped that the results presented will have an impact upon both sensor design and error modeling of position measuring systems for computer vision and related applications.     

基于自适应聚合的立体视觉合作算法

提出了一种恢复高质量稠密视差图的立体视觉合作算法.该算法采用基于形态学相似性的自适应加权方法,迭代地进行局部邻域的自适应聚合和抑制放大,实现高效率和高质量稠密视差图计算.将该算法推广到三目摄像机立体匹配系统中,通过重建摄像机坐标系实现图像校正,并根据连续性假设和唯一性假设,建立视差空间中的支持关系和三目摄像机之间的抑制关系.实验结果表明,三目立体合作算法能够得到精确的场景视差映射,并可以实现多基线方向的遮挡检测.该算法特别适用于由多个廉价摄像机组成的立体视觉系统,在几乎不增加软件和硬件资源的情况下,就可以得到高质量的稠密视差图.     

基于视差平面分割的移动机器人障碍物地图构建方法

作为自主移动机器人地表障碍物探测（GPOD）技术的一部分,提出了一种利用双目摄像机的视差图像 获取信息来构建机器人前方障碍物栅格地图的方法. 该方法融合了3 维立体视觉技术以及2 维图像处理技术,前者 依据视差图的直方图信息对视差图像进行自适应平面分割,把每个平面看作是3 维场景中的实物切片进而提取障碍 物3 维信息,后者通过计算各平面上的障碍物信息曲线来提取障碍物信息,把立体视觉数据从视差图像空间变换到 2 维的障碍物地图空间. 给出了该方法构建障碍物地图的整体过程,试验结果证明了该算法的有效性和精确性.     

基于双目视觉的玻璃料滴体积测量方法

针对目前玻璃料滴常规称量方法测量效率低且受环境影响较大的问题, 提出一种基于双目视觉的非接触式测量方法. 搭建双目视觉系统, 对采集的图像进行滤波去噪和特征轮廓提取, 基于融合料滴图像梯度信息的Census变换立体匹配算法得到边缘信息完整的视差图. 分别分析发生相机平面方向偏转和相机景深方向偏转的料滴对水平切片法精度的影响, 首先采用最小外接矩形算法对发生相机平面方向偏转的料滴进行校正, 然后利用视差信息修正发生相机景深方向偏转的料滴, 最后基于水平切片法累加水平切片获得料滴体积及质量. 实验结果验证, 该方法对发生空间偏转的料滴也能达到精度标准, 能够满足玻璃瓶生产的需求.     

Stereo reconstruction from multiperspective panoramas

A new approach to computing a panoramic (360 degrees) depth map is presented in this paper. Our approach uses a large collection of images taken by a camera whose motion has been constrained to planar concentric circles. We resample regular perspective images to produce a set of multiperspective panoramas and then compute depth maps directly from these resampled panoramas. Our panoramas sample uniformly in three dimensions: rotation angle, inverse radial distance, and vertical elevation. The use of multiperspective panoramas eliminates the limited overlap present in the original input images and, thus, problems as in conventional multibaseline stereo can be avoided. Our approach differs from stereo matching of single-perspective panoramic images taken from different locations, where the epipolar constraints are sine curves. For our multiperspective panoramas, the epipolar geometry, to the first order approximation, consists of horizontal lines. Therefore, any traditional stereo algorithm can be applied to multiperspective panoramas with little modification. In this paper, we describe two reconstruction algorithms. The first is a cylinder sweep algorithm that uses a small number of resampled multiperspective panoramas to obtain dense 3D reconstruction. The second algorithm, in contrast, uses a large number of multiperspective panoramas and takes advantage of the approximate horizontal epipolar geometry inherent in multiperspective panoramas. It comprises a novel and efficient 1D multibaseline matching technique, followed by tensor voting to extract the depth surface. Experiments show that our algorithms are capable of producing comparable high quality depth maps which can be used for applications such as view interpolation.     

3-D Scene Data Recovery Using Omnidirectional Multibaseline Stereo

A traditional approach to extracting geometric information from a large scene is to compute multiple 3-D depth maps from stereo pairs or direct range finders, and then to merge the 3-D data. However, the resulting merged depth maps may be subject to merging errors if the relative poses between depth maps are not known exactly. In addition, the 3-D data may also have to be resampled before merging, which adds additional complexity and potential sources of errors.This paper provides a means of directly extracting 3-D data covering a very wide field of view, thus by-passing the need for numerous depth map merging. In our work, cylindrical images are first composited from sequences of images taken while the camera is rotated 360° about a vertical axis. By taking such image panoramas at different camera locations, we can recover 3-D data of the scene using a set of simple techniques: feature tracking, an 8-point structure from motion algorithm, and multibaseline stereo. We also investigate the effect of median filtering on the recovered 3-D point distributions, and show the results of our approach applied to both synthetic and real scenes.     

基于单相机的全向深度图获取

为了搜寻移动机器人周围最大的可通行区域,采用全向立体视觉系统,提出获取可靠的致密三维深度图方法。视觉系统由1个普通相机和2个双曲面镜组成。当系统标定后,空间点的三维坐标可以通过匹配上下镜面的成像点计算得出。匹配方法分3步：最大FX匹配,特征匹配和歧义去除。定义合适的能量函数通过动态规划来实现剩余点的匹配。实验表明该系统精度高、具有实用价值。     

Using vanishing points for camera calibration

In this article a new method for the calibration of a vision system which consists of two (or more) cameras is presented. The proposed method, which uses simple properties of vanishing points, is divided into two steps. In the first step, the intrinsic parameters of each camera, that is, the focal length and the location of the intersection between the optical axis and the image plane, are recovered from a single image of a cube. In the second step, the extrinsic parameters of a pair of cameras, that is, the rotation matrix and the translation vector which describe the rigid motion between the coordinate systems fixed in the two cameras are estimated from an image stereo pair of a suitable planar pattern. Firstly, by matching the corresponding vanishing points in the two images the rotation matrix can be computed, then the translation vector is estimated by means of a simple triangulation. The robustness of the method against noise is discussed, and the conditions for optimal estimation of the rotation matrix are derived. Extensive experimentation shows that the precision that can be achieved with the proposed method is sufficient to efficiently perform machine vision tasks that require camera calibration, like depth from stereo and motion from image sequence.     

Reconstructing a 3-D depth map from one or more images

Several algorithms are suggested for recovering depth and orientation maps of a surface from its image intensities. They combine the advantages of stereo vision and shape-from-shading (SFS) methods. These algorithms generate accurate, unambiguous and dense surface depth and orientation maps. Most of the existing SFS algorithms cannot be directly extended to combine stereo images because the recovery of surface depth and that of orientation are separated in these formulations. We first present an SFS algorithm that couples the generation of depth and orientation maps. This formulation also ensures that the reconstructed surface depth and its orientation are consistent. The SFS algorithm for a single image is then extended to utilize stereo images. The correspondence over stereo images is established simultaneously with the generation of surface depth and orientation. An alternative approach is also suggested for combining stereo and SFS techniques. This approach can be used to combine needle maps which are directly available from other sources such as photometric stereo. Finally we present an algorithm to combine sparse depth measurements with an orientation map to reconstruct a surface. The same algorithm can be combined with the above algorithms for solving the SFS problem with sparse depth measurements. Thus various information sources can be used to accurately reconstruct a surface.     

基于地面视差分布的栅格地图建立方法

针对使用立体视觉建立环境地图方法存在信息不完整的问题,提出一种基于地面视差分布的栅格地图建立方法。利用地面视差分布在视差图中进行障碍物和地面点的检测,通过统一但参数值不同的投影模型将障碍物像素和地面点像素投影到栅格地图中,同时考虑立体视觉的量化和匹配误差、地面视差和栅格占据概率的空间分布。通过在非结构化环境中的实验表明,该方法可以实时地建立信息完整且准确的栅格地图。     

结合同场景立体图对的高质量深度图像重建

目的 越来越多的应用依赖于对场景深度图像准确且快速的观测和分析,如机器人导航以及在电影和游戏中对虚拟场景的设计建模等.飞行时间深度相机等直接的深度测量设备可以实时的获取场景的深度图像,但是由于硬件条件的限制,采集的深度图像分辨率比较低,无法满足实际应用的需要.通过立体匹配算法对左右立体图对之间进行匹配获得视差从而得到深度图像是计算机视觉的一种经典方法,但是由于左右图像之间遮挡以及无纹理区域的影响,立体匹配算法在这些区域无法匹配得到正确的视差,导致立体匹配算法在实际应用中存在一定的局限性.方法 结合飞行时间深度相机等直接的深度测量设备和立体匹配算法的优势,提出一种新的深度图像重建方法.首先结合直接的深度测量设备采集的深度图像来构造自适应局部匹配权值,对左右图像之间的局部窗立体匹配过程进行约束,得到基于立体匹配算法的深度图像;然后基于左右检测原理将采集到的深度图像和匹配得到的深度图像进行有效融合;接着提出一种局部权值滤波算法,来进一步提高深度图像的重建质量.结果 实验结果表明,无论在客观指标还是视觉效果上,本文提出的深度图像重建算法较其他立体匹配算法可以得到更好的结果.其中错误率比较实验表明,本文算法较传统的立体匹配算法在深度重建错误率上可以提升10％左右.峰值信噪比实验结果表明,本文算法在峰值信噪比上可以得到10 dB左右的提升.结论 提出的深度图像重建方法通过结合高分辨率左右立体图对和初始的低分辨率深度图像,可以有效地重建高质量高分辨率的深度图像.     

五自由度立体视觉平台中摄像机的标定

文章从立体视觉与机器人控制集成的角度出发,建立了一个主动立体视觉跟踪和定位系统,用于柔性装配线中装配零件的运动跟踪和装配工位的精确定位。该系统采用基于平面正方形的摄像机标定方法,该方法要求摄像机在三个(或三个以上)不同的位置摄取一个标准的正方形的图像,经过图像处理找到正方形的四个顶点在图像上的坐标,通过计算就可以线性求解摄像机全部内外参数。该方法简单易懂,非常适合于摄像机内外参数频繁标定的场合,计算速度快、所需设备简单、实用性好。文章的最后,给出了使用文中提出方法进行标定的实验结果,说明了该方法的有效性。     

Multi-view stereo in the Deep Learning Era: A comprehensive review

Multi-view stereo infers the 3D geometry from a set of images captured from several known positions and viewpoints. It is one of the most important components of 3D reconstruction. Recently, deep learning has been increasingly used to solve several 3D vision problems due to the predominating performance, including the multi-view stereo problem. This paper presents a comprehensive review, covering recent deep learning methods for multi-view stereo. These methods are mainly categorized into depth map based and volumetric based methods according to the 3D representation form, and representative methods are reviewed in detail. Specifically, the plane sweep based methods leveraging depth maps are presented following the stage of approaches, i.e. feature extraction, cost volume construction, cost volume regularization, depth map regression and post-processing. This review also summarizes several widely used datasets and their corresponding metrics for evaluation. Finally, several insightful observations and challenges are put forward enlightening future research directions.     

Fast Relative Depth Computation for an Active Stereo Vision System

Real-time computation of exact depth is not feasible in an active vision setup. Instead, reliable relative depth information which can be rapidly computed is preferred. In this paper, a stereo cue for computing relative depth obtained from an active stereo vision system is proposed. The proposed stereo cue can be computed purely from the coordinates of points in the stereo pair. The computational cost required is very low. No camera calibration or prior knowledge of the parameters of the stereo vision system is required. We show that the relationship between the relative depth cue and the actual depth in the three-dimensional (3D) space is monotonic. Such a relation is maintained even when the focal length and the vergence angle are changed, so long as the focal lengths of the two cameras are similar. Therefore, real-time implementation in an active vision setup can be realized. Stability analysis shows that the proposed method will be stable in practical situations, unless the stereo camera diverges. Experimental results are presented to highlight the properties and advantages of the proposed method.     

Outdoor Localization Using Stereo Vision Under Various Illumination Conditions

We present a mobile robot localization method using only a stereo camera. Vision-based localization in outdoor environments is a challenging issue because of extreme changes in illumination. To cope with varying illumination conditions, we use two-dimensional occupancy grid maps generated from three-dimensional point clouds obtained by a stereo camera. Furthermore, we incorporate salient line segments extracted from the ground into the grid maps. The grid maps are not significantly affected by illumination conditions because occupancy information and salient line segments can be robustly obtained. On the grid maps, a robot's poses are estimated using a particle filter that combines visual odometry and map matching. We use edge-point-based stereo simultaneous localization and mapping to obtain simultaneously occupancy information and robot ego-motion estimation. We tested our method under various illumination and weather conditions, including sunny and rainy days. The experimental results showed the effectiveness and robustness of the proposed method. Our method enables localization under extremely poor illumination conditions, which are challenging for even existing state-of-the-art methods.     

单目同时定位与建图中的地图恢复融合技术

目的 传统的单目视觉SLAM（simultaneous localization and mapping）跟踪失败后需要相机重新回到丢失的位置才能重定位并恢复建图,这极大限制了单目SLAM的应用场景。为解决这一问题,提出一种基于视觉惯性传感器融合的地图恢复融合算法。方法 当系统跟踪失败,仅由惯性传感器提供相机位姿,通过对系统重新初始化并结合惯性传感器提供的丢失部分的相机位姿将丢失前的地图融合到当前的地图中;为解决视觉跟踪丢失期间由惯性测量计算导致的相机位姿误差,提出了一种以关键帧之间的共视关系为依据的跳跃式的匹配搜索策略,快速获得匹配地图点,再通过非线性优化求解匹配点之间的运动估计,进行误差补偿,获得更加准确的相机位姿,并删减融合后重复的点云;最后建立前后两个地图中关键帧之间与地图点之间的联系,用于联合优化后续的跟踪建图过程中相机位姿和地图点位置。结果 利用Euroc数据集及其他数据进行地图精度和地图完整性实验,在精度方面,将本文算法得到的轨迹与ground truth和未丢失情况下得到的轨迹进行对比,结果表明,在SLAM系统跟踪失败的情况下,此方法能有效解决系统无法继续跟踪建图的问题,其精度可达厘米级别。在30 m²的室内环境中,仅有9 cm的误差,而在300 m²工厂环境中误差仅有7 cm。在完整性方面,在相机运动较剧烈的情况下,恢复地图的完整性优于ORB_SLAM的重定位算法,通过本文算法得到的地图关键帧数量比ORB_SLAM多30%。结论 本文提出的算法在单目视觉SLAM系统跟踪失败之后,仍然能够继续跟踪建图,不会丢失相机轨迹。此外,无需相机回到丢失之前的场景中,只需相机观察到部分丢失前场景,即可恢复融合所有地图。本文算法不仅保证了恢复地图的精度,还保证了建图的完整性。与传统的重定位方法相比,本文算法在系统建图较少时跟踪失败的情况下效果更好。     

On-Board Dual-Stereo-Vision for the Navigation of an Autonomous MAV

We present a quadrotor Micro Aerial Vehicle (MAV) equipped with four cameras, which are arranged in two stereo configurations. The MAV is able to perform stereo matching for each camera pair on-board and in real-time, using an efficient sparse stereo method. In case of the camera pair that is facing forward, the stereo matching results are used for a reduced stereo SLAM system. The other camera pair, which is facing downwards, is used for ground plane detection and tracking. Hence, we are able to obtain a full 6DoF pose estimate from each camera pair, which we fuse with inertial measurements in an extended Kalman filter. Special care is taken to compensate various drift errors. In an evaluation we show that using two instead of one camera pair significantly increases the pose estimation accuracy and robustness.     

Multiflash stereopsis: depth-edge-preserving stereo with small baseline illumination

Traditional stereo matching algorithms are limited in their ability to produce accurate results near depth discontinuities, due to partial occlusions and violation of smoothness constraints. In this paper, we use small baseline multi-flash illumination to produce a rich set of feature maps that enable acquisition of discontinuity preserving point correspondences. First, from a single multi-flash camera, we formulate a qualitative depth map using a gradient domain method that encodes object relative distances. Then, in a multiview setup, we exploit shadows created by light sources to compute an occlusion map. Finally, we demonstrate the usefulness of these feature maps by incorporating them into two different dense stereo correspondence algorithms, the first based on local search and the second based on belief propagation. Experimental results show that our enhanced stereo algorithms are able to extract high quality, discontinuity preserving correspondence maps from scenes that are extremely challenging for conventional stereo methods. We also demonstrate that small baseline illumination can be useful to handle specular reflections in stereo imagery. Different from most existing active illumination techniques, our method is simple, inexpensive, compact, and requires no calibration of light sources.