多目标背景下双目视觉深度信息优化研究及其实现

针对机器人领域应用视觉进行目标物体抓取问题,提出了一种针对多目标背景下,新的深度优化处理方法.通过设定一个阈值块,以遍历成块的深度信息用类似聚类的方法,提出目标物体的具体坐标,传递给机器人手臂,完成准确的抓取操作.依次介绍了双目视觉原理、摄像机标定、双目矫正和双目匹配等内容,以及呈现出原始的深度信息图以及优化后的深度信息图,比较它们的差距.最后在实验中给出了证明:此种深度信息优化方法能够有效的提高机器人抓取目标物体的成功率.最后,还在文章最后给出了下一步的研究方向.     

Differential Geometric Inference in Surface Stereo

Many traditional two-view stereo algorithms explicitly or implicitly use the frontal parallel plane assumption when exploiting contextual information since, e.g., the smoothness prior biases toward constant disparity (depth) over a neighborhood. This introduces systematic errors to the matching process for slanted or curved surfaces. These errors are nonnegligible for detailed geometric modeling of natural objects such as a human face. We show how to use contextual information geometrically to avoid such errors. A differential geometric study of smooth surfaces allows contextual information to be encoded in Cartan's moving frame model over local quadratic approximations, providing a framework of geometric consistency for both depth and surface normals; the accuracy of our reconstructions argues for the sufficiency of the approximation. In effect, Cartan's model provides the additional constraint necessary to move beyond the frontal parallel plane assumption in stereo reconstruction. It also suggests how geometry can extend surfaces to account for unmatched points due to partial occlusion.     

局部双目视差回归的目标距离估计

目的 双目视觉是目标距离估计问题的一个很好的解决方案。现有的双目目标距离估计方法存在估计精度较低或数据准备较繁琐的问题,为此需要一个可以兼顾精度和数据准备便利性的双目目标距离估计算法。方法 提出一个基于R-CNN（region convolutional neural network）结构的网络,该网络可以实现同时进行目标检测与目标距离估计。双目图像输入网络后,通过主干网络提取特征,通过双目候选框提取网络以同时得到左右图像中相同目标的包围框,将成对的目标框内的局部特征输入目标视差估计分支以估计目标的距离。为了同时得到左右图像中相同目标的包围框,使用双目候选框提取网络代替原有的候选框提取网络,并提出了双目包围框分支以同时进行双目包围框的回归;为了提升视差估计的精度,借鉴双目视差图估计网络的结构,提出了一个基于组相关和3维卷积的视差估计分支。结果 在KITTI（Karlsruhe Institute of Technology and Toyota Technological Institute）数据集上进行验证实验,与同类算法比较,本文算法平均相对误差值约为3.2%,远小于基于双目视差图估计算法（11.3%）,与基于3维目标检测的算法接近（约为3.9%）。另外,提出的视差估计分支改进对精度有明显的提升效果,平均相对误差值从5.1%下降到3.2%。通过在另外采集并标注的行人监控数据集上进行类似实验,实验结果平均相对误差值约为4.6%,表明本文方法可以有效应用于监控场景。结论 提出的双目目标距离估计网络结合了目标检测与双目视差估计的优势,具有较高的精度。该网络可以有效运用于车载相机及监控场景,并有希望运用于其他安装有双目相机的场景。     

Shape estimation using polarization and shading from two views

This paper presents a novel method for 3D surface reconstruction that uses polarization and shading information from two views. The method relies on polarization data acquired using a standard digital camera and a linear polarizer. Fresnel theory is used to process the raw images and to obtain initial estimates of surface normals, assuming that the reflection type is diffuse. Based on this idea, the paper presents two novel contributions to the problem of surface reconstruction. The first is a technique to enhance the surface normal estimates by incorporating shading information into the method. This is done using robust statistics to estimate how the measured pixel brightnesses depend on the surface orientation. This gives an estimate of the object material reflectance function, which is used to refine the estimates of the surface normals. The second contribution is to use the refined estimates to establish correspondence between two views of an object. To do this, a set of patches are extracted from each view and are aligned by minimizing an energy functional based on the surface normal estimates and local topographic properties. The optimum alignment parameters for different patch pairs are then used to establish stereo correspondence. This process results in an unambiguous field of surface normals, which can be integrated to recover the surface depth. Our technique is most suited to smooth, non-metallic surfaces. It complements existing stereo algorithms since it does not require salient surface features to obtain correspondences. An extensive set of experiments, yielding reconstructed objects and reflectance functions, are presented and compared to ground truth.     

Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo

An integrated approach to extract depth, efficiently and accurately, from a sequence of images is presented in this paper. The method combines the ability of the stereo processing to acquire highly accurate depth measurements and the efficiency of spatial and temporal gradient analysis. As a result of this integration, depth measurements of high quality are obtained at a speed approximately ten times greater than that of stereo processing. Without any a priori information of the locations of the points in the scene, the correspondence problem in stereo processing is computationally expensive. In our approach, we use spatial and temporal gradient (STG) analysis, which has been shown to provide depth with great efficiency, but limited accuracy, to guide the matching process of stereo. The camera motion used in the approach can be either lateral or axial. Extensive experiments on real scenes have shown the ability of the integrated approach to acquire depth with a mean error of less than 3%.     

Depth incorporating with color improves salient object detection

Detecting salient objects in challenging images attracts increasing attention as many applications require more robust method to deal with complex images from the Internet. Prior methods produce poor saliency maps in challenging cases mainly due to the complex patterns in the background and internal color edges in the foreground. The former problem may introduce noises into saliency maps and the later forms the difficulty in determining object boundaries. Observing that depth map can supply layering information and more reliable boundary, we improve salient object detection by integrating two features: color information and depth information which are calculated from stereo images. The two features collaborate in a two-stage framework. In the object location stage, depth mainly helps to produce a noise-filtered salient patch, which indicates the location of the object. In the object boundary inference stage, boundary information is encoded in a graph using both depth and color information, and then we employ the random walk to infer more reliable boundaries and obtain the final saliency map. We also build a data set containing 100+ stereo pairs to test the effectiveness of our method. Experiments show that our depth-plus-color based method significantly improves salient object detection compared with previous color-based methods.     

基于立体折反射全向成像的柱面全景深度估算

针对立体视觉原理的新型立体折反射全向成像系统结构设计和面向立体柱面全景像对的局域灰度相关对应点快速匹配算法,从捕获的全向市体影像中提取有效深度信息,用于辅助全向视频分析处理中的对象检测和跟踪.采用单相机和两个不同参数的抛物面型反射镜构造了一种共轴结构的折反射全向立体成像装置,捕获的存在一定视差的原始全向立体像对被投影展开为立体柱面全景像对,而后通过特定对应点匹配算法提取稠密的深度信息.对应点匹配算法采用局部区域灰度相关的算了,并充分利用了双向匹配和柱面全景的外极线约束来提高匹配的速度和准确度.仿真实验有效恢复了场景深度信息,证明了整套装置结构设计及深度估计方法的有效性.     

基于场景对象注意与深度图融合的深度估计

现有单目深度估计算法主要从单幅图像中获取立体信息,存在相邻深度边缘细节模糊、明显的对象缺失问题。提出一种基于场景对象注意机制与加权深度图融合的单目深度估计算法。通过特征矩阵相乘的方式计算特征图任意两个位置之间的相似特征向量,以快速捕获长距离依赖关系,增强用于估计相似深度区域的上下文信息,从而解决自然场景中对象深度信息不完整的问题。基于多尺度特征图融合的优点,设计加权深度图融合模块,为具有不同深度信息的多视觉粒度的深度图赋予不同的权值并进行融合,融合后的深度图包含深度信息和丰富的场景对象信息,有效地解决细节模糊问题。在KITTI数据集上的实验结果表明,该算法对目标图像预估时σ<1.25的准确率为0.879,绝对相对误差、平方相对误差和对数均方根误差分别为0.110、0.765和0.185,预测得到的深度图具有更加完整的场景对象轮廓和精确的深度信息。     

K-nearest neighborhood based integration of time-of-flight cameras and passive stereo for high-accuracy depth maps

Both time-of-flight （ToF） cameras and passive stereo can provide the depth information for their corresponding captured real scenes, but they have innate limitations. ToF cameras and passive stereo are intrinsically complementary for certain tasks. It is desirable to appropriately leverage all the available information by ToF cameras and passive stereo. Although some fusion methods have been presented recently, they fail to consider ToF reliability detection and ToF based improvement of passive stereo. As a result, this study proposes an approach to integrating ToF cameras and passive stereo to obtain high-accuracy depth maps. The main contributions are： （1） An energy cost function is devised to use data from ToF cameras to boost the stereo matching of passive stereo; （2） A fusion method is used to combine the depth information from both ToF cameras and passive stereo to obtain high-accuracy depth maps. Experiments show that the proposed approach achieves improved results with high accuracy and robustness.     

Stereo-camera-based object detection using fuzzy color histograms and a fuzzy classifier with depth and shape estimations

This paper proposes a new method of detecting an object containing multiple colors with non-homogeneous distributions in complex backgrounds and subsequently estimating the depth and shape of the object using a stereo camera. To extract features for object detection, this paper proposes fuzzy color histograms (FCHs) based on the self-splitting clustering (SSC) of the hue-saturation (HS) color space. For each scanning window in a pyramid of scaled images, the FCH is obtained by accumulating the fuzzy degrees of all of the pixels belonging to each cluster. The FCH is fed to a fuzzy classifier to detect an object in the left image captured by the stereo camera. To find the matched object region in the right image, the left and right images are first segmented using the SSC-partitioned HS space. The depth of the object is then found by performing stereo matching on the segmented images. To find the shape of the object, a disparity map is built using the estimated object depth to automatically determine the stereo matching window size and disparity search range. Finally, the shape of the object is segmented from the disparity map. The experimental results of the detection of different objects with depth and shape estimations are used to verify the performance of the proposed method. Comparisons with different detection and disparity map construction methods are performed to demonstrate the advantage of the proposed method.     

Object-centered surface reconstruction: Combining multi-image stereo and shading

Our goal is to reconstruct both the shape and reflectance properties of surfaces from multiple images. We argue that an object-centered representation is most appropriate for this purpose because it naturally accommodates multiple sources of data, multiple images (including motion sequences of a rigid object), and self-occlusions. We then present a specific object-centered reconstruction method and its implementation. The method begins with an initial estimate of surface shape provided, for example, by triangulating the result of conventional stereo. The surface shape and reflectance properties are then iteratively adjusted to minimize an objective function that combines information from multiple input images. The objective function is a weighted sum of stereo, shading, and smoothness components, where the weight varies over the surface. For example, the stereo component is weighted more strongly where the surface projects onto highly textured areas in the images, and less strongly otherwise. Thus, each component has its greatest influence where its accuracy is likely to be greatest. Experimental results on both synthetic and real images are presented.     

立体视频对象分割及其三维重建算法研究*

为更加有效分析立体视频对象,本文提出了一种基于离散冗余小波变换的立体视频对象分割算法,首先采用离散冗余小波变换提取特征点结合DT网格技术的视差估计方法,获得了可靠的视差场,再利用视差信息对立体视频中静止对象进行分割。对于立体视频序列中的运动对象,采用离散冗余小波提取运动区域的方法进行分割。实验结果表明,本算法对有重叠的多视频对象具有较好的分割效果,可同时分割静止物体和运动物体,具有较好的精确性和鲁棒性。对于分割出的立体视频对象,结合深度信息对其进行三维重建,得到较好的三维效果。     

基于格雷码和多步相移法的双目立体视觉三维测量技术研究

传统的被动式双目立体视觉三维测量技术,具有操作简单,使用灵活方便,相机标定技术成熟的优点,但是对于特征点稀疏图像,寻找匹配点困难,匹配精度低。编码结构光测量方式通过向待测物体投射特定的编码图案,获取编码图像进行解码求解物体的三维信息,具有着测量精度高,速度快的优点,但是存在着投影仪标定精度低,实现难度大的缺点。提出了将双目立体视觉和编码结构光相结合的三维测量方法,在完成双目校正的基础上,向待测物体投射格雷码图案和多步相移图案,给予被测物体容易识别和可控制的特征信息,最后求取物体的三维信息。而且通过实验论证了投射多步相移图案比起4步相移图案,测量精度更高,能够更好的体现物体细节。     

Range image denoising using a constrained local Gaussian model for 3D object query service in the smart space

A position and direction is a fundamental information for U-Business as an anywhere service. A mobile device camera image can increase an accuracy of the positioning, and a range image provides significant information in an occlusion scene. U-Business service queries the information with the range image for a precision position or a target object. We present a method for smoothing heavy noisy surfaces acquired by mobile 3D imaging devices to obtain the stable curvature. The smoothing is performed in a way that finds centers of probability distributions, which maximizes the likelihood of observed points with smooth constraints. The smooth constraints are derived from the unit tangent vector equality. This provides a way of obtaining smooth surfaces and stable curvatures. We achieve the smoothing by solving the regularized linear system. The unit tangent vector equality involves consideration of geometric symmetry, and it minimizes the variation of differential values that are a factor of curvatures. The proposed algorithm has two apparent advantages. The first thing is that the surfaces in a scene with various signals-to-noise ratio are smoothed, and then they can earn suitable curvatures. The second is that the proposed method works on heavy noisy surfaces, for example, a stereo camera image. Experiments on range images demonstrate that the proposed method yields the smooth surfaces from the input with various signals-to-noise ratio and the stable curvatures obtained from the smooth surfaces.     

基于双目立体视觉的西瓜子三维形态检测

为提高西瓜子自动筛选系统的性能,提出一种用于度量和处理西瓜子弯翘度的方法.把双目平行立体视觉与骨骼线相结合,利用双目立体视觉技术重建物体表面的三维信息.将该信息进行正交投影,分别得到西瓜子在前视图和侧视图的轮廓.利用曲率尺度空间与骨骼线相结合的方法对侧视图的瓜子轮廓进行特征提取.实验结果表明,该方法对弯翘瓜子有较高的识别精度.     

Volumetric stereo and silhouette fusion for image-based modeling

This paper presents a volumetric stereo and silhouette fusion algorithm for acquiring high quality models from multiple calibrated photographs. Our method is based on computing and merging depth maps. Different from previous methods of this category, the silhouette information is also applied in our algorithm to recover the shape information on the textureless and occluded areas. The proposed algorithm starts by computing visual hull using a volumetric method in which a novel projection test method is proposed for visual hull octree construction. Then, the depth map of each image is estimated by an expansion-based approach that returns a 3D point cloud with outliers and redundant information. After generating an oriented point cloud from stereo by rejecting outlier, reducing scale, and estimating surface normal for the depth maps, another oriented point cloud from silhouette is added by carving the visual hull octree structure using the point cloud from stereo to restore the textureless and occluded surfaces. Finally, Poisson Surface Reconstruction approach is applied to convert the oriented point cloud both from stereo and silhouette into a complete and accurate triangulated mesh model. The proposed approach has been implemented and the performance of the approach is demonstrated on several real data sets, along with qualitative comparisons with the state-of-the-art image-based modeling techniques according to the Middlebury benchmark.     

Positioning quadric surfaces in all active stereo imaging system

The location of surfaces using stereo imaging techniques is an important area of research for robot guidance and machine inspection applications, The underlying geometry of finite focal length stereo pinhole cameras is investigated. This is the model used in both active and passive stereo imaging systems. It is shown that the points of intersecting views from the pinhole models result in conic sections. This information is used to locate quadric surfaces in the inspection space. When the projected fringe pattern is encoded to a quadric surface, the underlying intersection mapping can be used to estimate the position of the surface. Subsets of most smooth objects can be fitted to a conic section. For inspection of smooth surfaces with projection moire techniques, this fringe interpretation method would allow for correct placement of the object     

图像亮度线索下的单目深度信息提取

目的 深度信息的获取是3维重建、虚拟现实等应用的关键技术,基于单目视觉的深度信息获取是非接触式3维测量技术中成本最低、也是技术难度最大的手段。传统的单目方法多基于线性透视、纹理梯度、运动视差、聚焦散焦等深度线索来对深度信息进行求取,计算量大,对相机精度要求高,应用场景受限,本文基于固定光强的点光源在场景中的移动所带来的物体表面亮度的变化,提出一种简单快捷的单目深度提取方法。方法 首先根据体表面反射模型,得到光源照射下的物体表面的辐亮度,然后结合光度立体学推导物体表面辐亮度与摄像机图像亮度之间的关系,在得到此关系式后,设计实验,依据点光源移动所带来的图像亮度的变化对深度信息进行求解。结果 该算法在简单场景和一些日常场景下均取得了较好的恢复效果,深度估计值与实际深度值之间的误差小于10%。结论 本文方法通过光源移动带来的图像亮度变化估计深度信息,避免了复杂的相机标定过程,计算复杂度小,是一种全新的场景深度信息获取方法。     

基于图像的自动驾驶3D目标检测综述——基准、制约因素和误差分析

从高分辨率图像中获取周边目标的精准3D位置和尺寸信息是实现自动驾驶控制和行为决策的基础,因此基于图像的3D目标检测是自动驾驶领域中的研究热点。已有学者对该领域方法论及成果进行了比较详细的综述,但对于导致现有方法检测精度不尽如意的制约因素未能进行深入系统的分析。考虑自动驾驶领域在工程应用方面的要求高,且现有方法以数据驱动类型为主,本文从常用数据集和评价基准、数据影响、方法论的制约因素和误差等角度,对学术界和产业界在3D目标检测方面的研究成果及行业应用进行较为系统的阐述。首先,从学术界探索成果以及自动驾驶行业的应用角度进行概要介绍。然后,从数据采集设备、数据精度和标注信息3方面详细分析总结了KITTI等4个通用数据集,并对这些数据集提出的主要评价指标进行对比分析。接着,从数据和方法论方面分析制约算法性能的主要因素及由此造成的误差影响。在数据方面,制约因素主要是数据精度、样本差异、标注数据量和标注规范;在方法论方面,制约因素主要包括先验几何关系、深度预测误差和数据模态等。最后,对国内外研究现状进行总结,并在数据集、评价指标和目标深度预测等方面提出了未来需要重点关注的研究方向。     

基于轮廓提取与深度筛选的双目三维重构技术

针对基于深度图融合三维重构方法获取的三维重构模型,易受到深度信息误差影响的情况,提出一种基于轮廓提取与深度筛选的双目立体视觉三维重构方法。采用标准棋盘校准双目三维重构系统,利用Canny算子对目标物体进行边界检测,综合采用形态学腐蚀与膨胀方法提取指定方向上的连续边界,用连续边界提取目标物体。在此基础上,对目标物体深度信息进行筛选、拟合插值以获取连续深度信息。结果表明,相对于常规三维重构算法,由本算法三维重构的目标物体表面完整度更高,且目标物体周围的背景环境噪声被去除。