A volumetric fusion technique for surface reconstruction from silhouettes and range data

Optical triangulation, an active reconstruction technique, is known to be an accurate method but has several shortcomings due to occlusion and laser reflectance properties of the object surface, that often lead to holes and inaccuracies on the recovered surface. Shape from silhouette, on the other hand, as a passive reconstruction technique, yields robust, hole-free reconstruction of the visual hull of the object. In this paper, a hybrid surface reconstruction method that fuses geometrical information acquired from silhouette images and optical triangulation is presented. Our motivation is to recover the geometry from silhouettes on those parts of the surface which the range data fail to capture. A volumetric octree representation is first obtained from the silhouette images and then carved by range points to amend the missing cavity information. An isolevel value on each surface cube of the carved octree structure is accumulated using local surface triangulations obtained separately from range data and silhouettes. The marching cubes algorithm is then applied for triangulation of the volumetric representation. The performance of the proposed technique is demonstrated on several real objects.     

Robust dense reconstruction by range merging based on confidence estimation

Although the stereo matching problem has been extensively studied during the past decades, automatically computing a dense 3D reconstruction from several multiple views is still a difficult task owing to the problems of textureless regions, outliers, detail loss, and various other factors. In this paper, these difficult problems are handled effectively by a robust model that outputs an accurate and dense reconstruction as the final result from an input of multiple images captured by a normal camera. First, the positions of the camera and sparse 3D points are estimated by a structure-from-motion algorithm and we compute the range map with a confidence estimation for each image in our approach. Then all the range maps are integrated into a fine point cloud data set. In the final step we use a Poisson reconstruction algorithm to finish the reconstruction. The major contributions of the work lie in the following points: effective range-computation and confidence-estimation methods are proposed to handle the problems of textureless regions, outliers and detail loss. Then, the range maps are merged into the point cloud data in terms of a confidence-estimation. Finally, Poisson reconstruction algorithm completes the dense mesh. In addition, texture mapping is also implemented as a post-processing work for obtaining good visual effects. Experimental results are presented to demonstrate the effectiveness of the proposed approach.     

Incorporating object-centered sampling and Delaunay tetrahedrization for visual hull reconstruction

In this paper we present a novel shape from silhouette algorithm. For an object to be modeled, the algorithm first computes a cloud of points located on a pencil of rays and distributed evenly on the visual hull surface, inside and outside the visual hull. Then Delaunay tetrahedrization is applied to the point cloud to partition its convex hull into a set of tetrahedrons. Finally, outlier tetrahedrons are removed by tetrahedron peeling, and a mesh model of the visual hull is extracted. The algorithm is robust, free from discretization artifacts, and produces a mesh model composed of well-shaped triangles.     

Instance segmentation based 6D pose estimation of industrial objects using point clouds for robotic bin-picking

3D object pose estimation for robotic grasping and manipulation is a crucial task in the manufacturing industry. In cluttered and occluded scenes, the 6D pose estimation of the low-textured or textureless industrial object is a challenging problem due to the lack of color information. Thus, point cloud that is hardly affected by the lighting conditions is gaining popularity as an alternative solution for pose estimation. This article proposes a deep learning-based pose estimation using point cloud as input, which consists of instance segmentation and instance point cloud pose estimation. The instance segmentation divides the scene point cloud into multiple instance point clouds, and each instance point cloud pose is accurately predicted by fusing the depth and normal feature maps. In order to reduce the time consumption of the dataset acquisition and annotation, a physically-simulated engine is constructed to generate the synthetic dataset. Finally, several experiments are conducted on the public, synthetic and real datasets to verify the effectiveness of the pose estimation network. The experimental results show that the point cloud based pose estimation network can effectively and robustly predict the poses of objects in cluttered and occluded scenes.     

采用变形模型的图像序列可视外壳计算方法

针对基于图像序列变形模型三维重建中至关重要的初始模型计算问题,提出一种基于图像序列计算可视外壳的方法.该方法基于Snake活动轮廓变形模型,将真实物体可视外壳的计算问题转化为初始曲面通过内外力作用驱动收敛于目标物体的问题.以真实物体图像序列中提取的物体轮廓为输入,根据轮廓信息和光滑度信息计算内力及轮廓力大小,驱动球体初始曲面变形收敛于可视外壳;在变形过程中加入删除短边、分裂长边、对角线翻转等网格优化操作,以避免发生网格错乱.实验结果表明,文中方法有效地克服了传统的基于体元素细分产生网格算法网格质量不高的缺点,且参数可调,易于实现,占用内存少,生成的曲面更加光滑,细节恢复效果理想.     

采用窄带图切割的多目重构方法

针对目前多目重构研究中物体表面获取和凹陷区域恢复的难题,提出采用窄带图切割的可见外壳和多目立体匹配相融合的方法.在可见外壳的拓扑约束下,通过最小化能量泛函使得重构表面与多目立体匹配得到的稠密点云形状一致;并利用窄带图切割计算能量泛函的全局最优解.实验结果表明,该方法能重构封闭表面和凹陷区域;窄带图切割在获得最优解的同时提高了计算效率,与全图切割相比速度提高了3倍以上.     

一种鲁棒的可视外壳生成算法*

针对各种传统可视外壳生成算法中数据冗余、精确度低、健壮性不足等问题, 提出了一种新的可视外壳生成算法,即采用加权线段求交、线段集合中心线性过滤、多边形边界检测等方法重建物体模型。与传统方法相比,本算法能够更稳定地计算线段交集,表面边界提取更加准确,重建结果精确逼近真实物体。实验表明, 通过该算法计算的物体可视外壳能够更好地逼近真实模型,精度高。     

GPU-based parallel construction of compact visual hull meshes

Building a visual hull model from multiple two-dimensional images provides an effective way of understanding the three-dimensional geometries inherent in the images. In this paper, we present a GPU accelerated algorithm for volumetric visual hull reconstruction that aims to harness the full compute power of the many-core processor. From a set of binary silhouette images with respective camera parameters, our parallel algorithm directly outputs the triangular mesh of the resulting visual hull in the indexed face set format for a compact mesh representation. Unlike previous approaches, the presented method extracts a smooth silhouette contour on the fly from each binary image, which markedly reduces the bumpy artifacts on the visual hull surface due to a simple binary in/out classification. In addition, it applies several optimization techniques that allow an efficient CUDA implementation. We also demonstrate that the compact mesh construction scheme can easily be modified for also producing a time- and space-efficient GPU implementation of the marching cubes algorithm.     

Reduced Depth and Visual Hulls of Complex 3D Scenes

Depth and visual hulls are useful for quick reconstruction and rendering of a 3D object based on a number of reference views. However, for many scenes, especially multi‐object, these hulls may contain significant artifacts known as phantom geometry. In depth hulls the phantom geometry appears behind the scene objects in regions occluded from all the reference views. In visual hulls the phantom geometry may also appear in front of the objects because there is not enough information to unambiguously imply the object positions. In this work we identify which parts of the depth and visual hull might constitute phantom geometry. We define the notion of reduced depth hull and reduced visual hull as the parts of the corresponding hull that are phantom‐free. We analyze the role of the depth information in identification of the phantom geometry. Based on this, we provide an algorithm for rendering the reduced depth hull at interactive frame‐rates and suggest an approach for rendering the reduced visual hull. The rendering algorithms take advantage of modern GPU programming techniques. Our techniques bypass explicit reconstruction of the hulls, rendering the reduced depth or visual hull directly from the reference views.     

基于双目视觉的人脸三维重建

基于双目立体匹配算法PatchMatch算法,提出了一种获取人脸三维点云的算法。该算法对局部立体匹配算法PatchMatch进行了优化。该方法既不需要昂贵的设备,也不需要通用的人脸三维模型,而是结合了人脸的拓扑结构信息以及立体视觉局部优化算法。此方法采用非接触式的双目视觉采集技术获取左右视角的人脸图像,利用回归树集合(ensemble of regression trees,ERT)算法对人脸图像进行关键点定位,恢复人脸稀疏的视差估计,运用线性插值方法初步估计脸部的稠密视差值,并结合局部立体匹配算法对得到的视差结果进行平滑处理,重建人脸的三维点云信息。实验结果表明,这种算法能够还原出光滑的稠密人脸三维点云信息,在人脸Bosphorus数据库上取得了更加准确的人脸重建结果。     

视差互信息引导下的立体航空影像与LiDAR点云自动配准

目的 从视差图反映影像景物深度变化并与LiDAR系统距离量测信息"同源"这一认识出发,提出一种基于视差互信息的立体航空影像与LiDAR点云自动配准方法.方法 本文方法分为3个阶段:第一、通过半全局匹配SGM(semi-gdabal matching)生成立体航空影像密集视差图;第二、利用航空影像内参数及初始配准参数(外方位元素)对LiDAR点云进行"针孔"透视成像,生成与待配准的立体航空影像空间分辨率、几何形变相接近且具有相同幅面大小的模拟灰度影像-LiDAR深度影像,以互信息作为相似性测度依据估计航空影像视差图与LiDAR深度影像的几何映射关系,进而以之为基础实现LiDAR点云影像概略相关;第三、以LiDAR点云影像概略相关获得的近似同名像点为观测值,以视差互信息为权重,实施摄影测量空间后方交会计算获得优化的影像外方位元素,生成新的LiDAR深度影像并重复上述过程,直至满足给定的迭代计算条件.结果 选取重叠度约60%、幅面大小7 216×5 428像素、空间分辨率约0.5 m的立体航空像对与平均点间距约1.5 m、水平精度约25 cm的LiDAR"点云"进行空间配准实验,配准精度接近1个像素.结论 实验结果表明,本文方法自动化程度高且配准精度适中,理论上适用于不同场景类型、相机内参数已知立体航空影像,具有良好的应用价值.     

一种可视外壳的快速拓扑生成算法

针对各种传统可视外壳生成算法中数据冗余及壮健性不足等问题，提出了一种新的从阴影图像中快速重构物体可视外壳的壮健的算法，即首先利用物体表面的拓扑结构直接生成外壳，然后使用改进的SurfaceNet算法光滑三维表面，从而在保留经典的体求交方法壮健性的基础上，克服了对于物体内部点的冗余计算和存储问题，不仅使得计算的时间复杂度降低到仅线性依赖于外壳上结点的数目，而且降低了像片数对算法复杂度产生的影响，实验结果表明，在算法复杂度和壮健性上优于诸如八叉树等传统可视外壳算法。     

Multi-frame stereo matching with edges,planes, and superpixels

We present a multi-frame narrow-baseline stereo matching algorithm based on extracting and matching edges across multiple frames. Edge matching allows us to focus on the important features at the very beginning, and deal with occlusion boundaries as well as untextured regions. Given the initial sparse matches, we fit overlapping local planes to form a coarse, over-complete representation of the scene. After breaking up the reference image in our sequence into superpixels, we perform a Markov random field optimization to assign each superpixel to one of the plane hypotheses. Finally, we refine our continuous depth map estimate using a piecewise-continuous variational optimization. Our approach successfully deals with depth discontinuities, occlusions, and large textureless regions, while also producing detailed and accurate depth maps. We show that our method out-performs competing methods on high-resolution multi-frame stereo benchmarks and is well-suited for view interpolation applications.     

Reconstruction of 3D Object Meshes from Silhouette Images

In this work we propose a method for computing mesh representations of 3D objects reconstructed from a set of silhouette images. Our method is based on the polygonization of volumetric reconstructions by using a modified version of the dual contouring method. In order to apply dual contouring on volumetric reconstruction from silhouettes we devised a method that is able to determine the discrete topology of the surface in relation to the octree cells. We also developed a new scheme for computing hermitian data representing the intersections of conic volumes with the octree cells and their corresponding normals with subpixel accuracy. Due to the discrete and extremely noisy nature of the data used in the reconstruction we had to devise a different criterion for mesh simplification that applies topological consistency tests only when the geometric error measure is beyond a given tolerance. We present results of the application of the proposed method in the extraction of a mesh corresponding to the surface of objects of a real scene.     

基于光学图像的多粒度随动环境感知算法

针对快速三维建模中的室内外随动环境感知问题,提出一种基于光学图像的多粒度随动环境感知算法.该算法根据多种光学图像生成拟合真实三维环境的多粒度点云模型,然后通过概率八叉树压缩并统一表示已生成的多粒度三维模型.进而伴随相机轨迹每个时间节点,通过卡尔曼滤波动态融合多粒度点云模型的概率八叉树表示.最终生成唯一的时态融合概率八叉树三维模型,简称TFPOM,使TFPOM能够在较少的噪声影响下以任意粒度动态拟合真实环境.该算法配合剪枝和归并策略能够适应多粒度融合和多粒度表示的环境建模要求,有效压缩环境模型存储空间,实现鲁棒的随动环境感知,便于基于环境模型的视觉导航,增强现实等应用.实验结果表明,该算法能够在以可穿戴设备为代表的内含多种异构光学图像传感器,低计算效能的平台上实时得到充分拟合真实动态环境的多粒度TFPOM,基于该模型的视觉导航具有较小的轨迹误差.     

Octree generation from object silhouettes in perspective views

Octrees are useful for object representation when fast access to coarse spatial occupancy information is necessary. This paper presents an efficient algorithm for generating octrees from multiple perspective views of an object. The algorithm first obtains a polygonal approximation of the object silhouette. This polygon is then decomposed into convex components. For each convex component, a pyramid is formed treating the view point as its apex and the convex components as a cross section. The octree representation of each of these pyramids is obtained by performing intersection detection of the object with the cubes corresponding to octree nodes. The intersection detection step is made efficient by decomposing it into a coarse-to-fine sequence of intersection tests. The octree for one silhouette is obtained by taking the union of octrees obtained for each component. An intersection of octrees corresponding to different viewing directions gives the final octree of the object. An implementation of the algorithm is given. The accuracy of the octree representation of the objects is evaluated. The ratio of the actual volume of the object to the volume of the object reconstructed from the octree representation is used as a performance index of the algorithm.     

A robust multi-scale integration method to obtain the depth from gradient maps

We describe a robust method for the recovery of the depth map (or height map) from a gradient map (or normal map) of a scene, such as would be obtained by photometric stereo or interferometry. Our method allows for uncertain or missing samples, which are often present in experimentally measured gradient maps, and also for sharp discontinuities in the scene’s depth, e.g. along object silhouette edges. By using a multi-scale approach, our integration algorithm achieves linear time and memory costs. A key feature of our method is the allowance for a given weight map that flags unreliable or missing gradient samples. We also describe several integration methods from the literature that are commonly used for this task. Based on theoretical analysis and tests with various synthetic and measured gradient maps, we argue that our algorithm is as accurate as the best existing methods, handling incomplete data and discontinuities, and is more efficient in time and memory usage, especially for large gradient maps.     

Reconstructing a 3-D depth map from one or more images

Several algorithms are suggested for recovering depth and orientation maps of a surface from its image intensities. They combine the advantages of stereo vision and shape-from-shading (SFS) methods. These algorithms generate accurate, unambiguous and dense surface depth and orientation maps. Most of the existing SFS algorithms cannot be directly extended to combine stereo images because the recovery of surface depth and that of orientation are separated in these formulations. We first present an SFS algorithm that couples the generation of depth and orientation maps. This formulation also ensures that the reconstructed surface depth and its orientation are consistent. The SFS algorithm for a single image is then extended to utilize stereo images. The correspondence over stereo images is established simultaneously with the generation of surface depth and orientation. An alternative approach is also suggested for combining stereo and SFS techniques. This approach can be used to combine needle maps which are directly available from other sources such as photometric stereo. Finally we present an algorithm to combine sparse depth measurements with an orientation map to reconstruct a surface. The same algorithm can be combined with the above algorithms for solving the SFS problem with sparse depth measurements. Thus various information sources can be used to accurately reconstruct a surface.     

Poisson Surface Reconstruction with Envelope Constraints

Reconstructing surfaces from scanned 3D points has been an important research area for several decades. One common approach that has proven efficient and robust to noise is implicit surface reconstruction, i.e. fitting to the points a 3D scalar function (such as an indicator function or signed-distance field) and then extracting an isosurface. Though many techniques fall within this category, existing methods either impose no boundary constraints or impose Dirichlet/Neumann conditions on the surface of a bounding box containing the scanned data. In this work, we demonstrate the benefit of supporting Dirichlet constraints on a general boundary. To this end, we adapt the Screened Poisson Reconstruction algorithm to input a constraint envelope in addition to the oriented point cloud. We impose Dirichlet boundary conditions, forcing the reconstructed implicit function to be zero outside this constraint surface. Using a visual hull and/or depth hull derived from RGB-D scans to define the constraint envelope, we obtain substantially improved surface reconstructions in regions of missing data.     

Multiview photometric stereo

This paper addresses the problem of obtaining complete, detailed reconstructions of textureless shiny objects. We present an algorithm which uses silhouettes of the object, as well as images obtained under changing illumination conditions. In contrast with previous photometric stereo techniques, ours is not limited to a single viewpoint but produces accurate reconstructions in full 3D. A number of images of the object are obtained from multiple viewpoints, under varying lighting conditions. Starting from the silhouettes, the algorithm recovers camera motion and constructs the object's visual hull. This is then used to recover the illumination and initialise a multi-view photometric stereo scheme to obtain a closed surface reconstruction. There are two main contributions in this paper: Firstly we describe a robust technique to estimate light directions and intensities and secondly, we introduce a novel formulation of photometric stereo which combines multiple viewpoints and hence allows closed surface reconstructions. The algorithm has been implemented as a practical model acquisition system. Here, a quantitative evaluation of the algorithm on synthetic data is presented together with complete reconstructions of challenging real objects. Finally, we show experimentally how even in the case of highly textured objects, this technique can greatly improve on correspondence-based multi-view stereo results.