结合视觉惯性模组的室内三维布局鲁棒重建方法

针对使用传统单目相机的全自动三维重建方法结果精确度差和整体结构理解缺失等问题,提出一种结合视觉惯性里程计和由运动到结构的全自动室内三维布局重建系统.首先利用视觉里程计获得关键帧图像序列和对应空间位置姿态,并利用运动恢复结构算法计算精确相机位姿;然后利用多图视立体几何算法生成高质量稠密点云;最后基于曼哈顿世界假设,针对典型的现代建筑室内场景,设计一种基于规则的自底向上的布局重建方法,得到最终房间外轮廓布局.使用浙江大学CAD&CG实验室场景现场扫描数据集和人工合成的稠密点云数据集作为实验数据,在Ubuntu 16.04和PCL 1.9环境下进行实验.结果表明,文中方法对三维点云噪声容忍度高,能够有效地重建出室内场景的三维外轮廓布局.     

A new point matching algorithm for non-rigid registration

Feature-based methods for non-rigid registration frequently encounter the correspondence problem. Regardless of whether points, lines, curves or surface parameterizations are used, feature-based non-rigid matching requires us to automatically solve for correspondences between two sets of features. In addition, there could be many features in either set that have no counterparts in the other. This outlier rejection problem further complicates an already difficult correspondence problem. We formulate feature-based non-rigid registration as a non-rigid point matching problem. After a careful review of the problem and an in-depth examination of two types of methods previously designed for rigid robust point matching (RPM), we propose a new general framework for non-rigid point matching. We consider it a general framework because it does not depend on any particular form of spatial mapping. We have also developed an algorithm—the TPS–RPM algorithm—with the thin-plate spline (TPS) as the parameterization of the non-rigid spatial mapping and the softassign for the correspondence. The performance of the TPS–RPM algorithm is demonstrated and validated in a series of carefully designed synthetic experiments. In each of these experiments, an empirical comparison with the popular iterated closest point (ICP) algorithm is also provided. Finally, we apply the algorithm to the problem of non-rigid registration of cortical anatomical structures which is required in brain mapping. While these results are somewhat preliminary, they clearly demonstrate the applicability of our approach to real world tasks involving feature-based non-rigid registration.     

Point set registration: coherent point drift

Point set registration is a key component in many computer vision tasks. The goal of point set registration is to assign correspondences between two sets of points and to recover the transformation that maps one point set to the other. Multiple factors, including an unknown nonrigid spatial transformation, large dimensionality of point set, noise, and outliers, make the point set registration a challenging problem. We introduce a probabilistic method, called the Coherent Point Drift (CPD) algorithm, for both rigid and nonrigid point set registration. We consider the alignment of two point sets as a probability density estimation problem. We fit the Gaussian mixture model (GMM) centroids (representing the first point set) to the data (the second point set) by maximizing the likelihood. We force the GMM centroids to move coherently as a group to preserve the topological structure of the point sets. In the rigid case, we impose the coherence constraint by reparameterization of GMM centroid locations with rigid parameters and derive a closed form solution of the maximization step of the EM algorithm in arbitrary dimensions. In the nonrigid case, we impose the coherence constraint by regularizing the displacement field and using the variational calculus to derive the optimal transformation. We also introduce a fast algorithm that reduces the method computation complexity to linear. We test the CPD algorithm for both rigid and nonrigid transformations in the presence of noise, outliers, and missing points, where CPD shows accurate results and outperforms current state-of-the-art methods.     

Branch-and-Bound Methods for Euclidean Registration Problems

In this paper, we propose a practical and efficient method for finding the globally optimal solution to the problem of determining the pose of an object. We present a framework that allows us to use point-to-point, point-to-line, and point-to-plane correspondences for solving various types of pose and registration problems involving euclidean (or similarity) transformations. Traditional methods such as the iterative closest point algorithm or bundle adjustment methods for camera pose may get trapped in local minima due to the nonconvexity of the corresponding optimization problem. Our approach of solving the mathematical optimization problems guarantees global optimality. The optimization scheme is based on ideas from global optimization theory, in particular convex underestimators in combination with branch-and-bound methods. We provide a provably optimal algorithm and demonstrate good performance on both synthetic and real data. We also give examples of where traditional methods fail due to the local minima problem.     

3-D human pose recovery using nonrigid point set registration and body part tracking of depth data

In this paper, we present a novel approach for recovering a 3-D pose from a single human body depth silhouette using nonrigid point set registration and body part tracking. In our method, a human body depth silhouette is presented as a set of 3-D points and matched to another set of 3-D points using point correspondences. To recognize and maintain body part labels, we initialize the first set of points to corresponding human body parts, resulting in a body part-labeled map. Then, we transform the points to a sequential set of points based on point correspondences determined by nonrigid point set registration. After point registration, we utilize the information from tracked body part labels and registered points to create a human skeleton model. A 3-D human pose gets recovered by mapping joint information from the skeleton model to a 3-D synthetic human model. Quantitative and qualitative evaluation results on synthetic and real data show that complex human poses can be recovered more reliably with lower errors compared to other conventional techniques for 3-D pose recovery.     

Multilinear Factorizations for Multi-Camera Rigid Structure from Motion Problems

Camera networks have gained increased importance in recent years. Existing approaches mostly use point correspondences between different camera views to calibrate such systems. However, it is often difficult or even impossible to establish such correspondences. But even without feature point correspondences between different camera views, if the cameras are temporally synchronized then the data from the cameras are strongly linked together by the motion correspondence: all the cameras observe the same motion. The present article therefore develops the necessary theory to use this motion correspondence for general rigid as well as planar rigid motions. Given multiple static affine cameras which observe a rigidly moving object and track feature points located on this object, what can be said about the resulting point trajectories? Are there any useful algebraic constraints hidden in the data? Is a 3D reconstruction of the scene possible even if there are no point correspondences between the different cameras? And if so, how many points are sufficient? Is there an algorithm which warrants finding the correct solution to this highly non-convex problem? This article addresses these questions and thereby introduces the concept of low-dimensional motion subspaces. The constraints provided by these motion subspaces enable an algorithm which ensures finding the correct solution to this non-convex reconstruction problem. The algorithm is based on multilinear analysis, matrix and tensor factorizations. Our new approach can handle extreme configurations, e.g. a camera in a camera network tracking only one single point. Results on synthetic as well as on real data sequences act as a proof of concept for the presented insights.     

基于最大权团的曲面粗匹配算法

提出一种将曲面匹配问题转化为图论中的最大权团搜索问题、将最优的点对应关系用最大权团表示的曲面粗匹配算法,该算法分为点匹配、点对应图构造和最大权团生成等3个阶段.点匹配使用高曲率点和均匀采样点作为候选点,通过自旋图进行匹配计算,构造初始点对应集合;点对应图构造使用距离约束、法矢约束和唯一性约束构造图的边,并使用自旋图相关系数为顶点赋权值;最大权团生成使用基于分支限界的团搜索算法,从对应点图中提取出代表最优对应的最大权团.实验结果表明,文中算法稳定、有效、可扩展,能够进行部分曲面匹配,并且适用于欠特征曲面.     

基于动态区域剔除的RGB-D视觉SLAM算法

针对动态场景下视觉SLAM(simultaneous localization and mapping)算法易受运动特征点影响,从而导致位姿估计准确度低、鲁棒性差的问题,提出了一种基于动态区域剔除的RGB-D视觉SLAM算法。首先借助语义信息,识别出属于移动对象的特征点,并借助相机的深度信息利用多视图几何检测特征点在此时是否保持静止;然后使用从静态对象提取的特征点和从可移动对象导出的静态特征点来微调相机姿态估计,以此实现系统在动态场景中准确而鲁棒的运行;最后利用TUM数据集中的动态室内场景进行了实验验证。实验表明,在室内动态环境中,所提算法能够有效提高相机的位姿估计精度,实现动态环境中的地图更新,在提升系统鲁棒性的同时也提高了地图构建的准确性。     

SoftPOSIT: Simultaneous Pose and Correspondence Determination

The problem of pose estimation arises in many areas of computer vision, including object recognition, object tracking, site inspection and updating, and autonomous navigation when scene models are available. We present a new algorithm, called SoftPOSIT, for determining the pose of a 3D object from a single 2D image when correspondences between object points and image points are not known. The algorithm combines the iterative softassign algorithm (Gold and Rangarajan, 1996; Gold et al., 1998) for computing correspondences and the iterative POSIT algorithm (DeMenthon and Davis, 1995) for computing object pose under a full-perspective camera model. Our algorithm, unlike most previous algorithms for pose determination, does not have to hypothesize small sets of matches and then verify the remaining image points. Instead, all possible matches are treated identically throughout the search for an optimal pose. The performance of the algorithm is extensively evaluated in Monte Carlo simulations on synthetic data under a variety of levels of clutter, occlusion, and image noise. These tests show that the algorithm performs well in a variety of difficult scenarios, and empirical evidence suggests that the algorithm has an asymptotic run-time complexity that is better than previous methods by a factor of the number of image points. The algorithm is being applied to a number of practical autonomous vehicle navigation problems including the registration of 3D architectural models of a city to images, and the docking of small robots onto larger robots.     

Global Correspondence Optimization for Non‐Rigid Registration of Depth Scans

We present a registration algorithm for pairs of deforming and partial range scans that addresses the challenges of non‐rigid registration within a single non‐linear optimization. Our algorithm simultaneously solves for correspondences between points on source and target scans, confidence weights that measure the reliability of each correspondence and identify non‐overlapping areas, and a warping field that brings the source scan into alignment with the target geometry. The optimization maximizes the region of overlap and the spatial coherence of the deformation while minimizing registration error. All optimization parameters are chosen automatically; hand‐tuning is not necessary. Our method is not restricted to part‐in‐whole matching, but addresses the general problem of partial matching, and requires no explicit prior correspondences or feature points. We evaluate the performance and robustness of our method using scan data acquired by a structured light scanner and compare our method with existing non‐rigid registration algorithms.     

基于共面二点一线特征的单目视觉定位

研究了根据点、线混合特征进行单目视觉定位问题,在给定物体坐标系中共面的两个特征点和一条特征直线的条件下,根据它们在像平面上的对应计算相机与物体之间的位姿参数。根据三个特征之间的几何位置关系,分两种情况给出问题求解的具体过程,最终将问题转换成求解一个二次方程问题,真实的工件定位实验验证了方法的有效性。该结果为应用单目视觉进行工件定位提供了一种新方法。     

Dynamic SfM: Detecting Scene Changes from Image Pairs

Detecting changes in scenes is important in many scene understanding tasks. In this paper, we pursue this goal simply from a pair of image recordings. Specifically, our goal is to infer what the objects are, how they are structured, and how they moved between the images. The problem is challenging as large changes make point‐level correspondence establishment difficult, which in turn breaks the assumptions of standard Structure‐from‐Motion (SfM). We propose a novel algorithm for dynamic SfM wherein we first generate a pool of potential corresponding points by hypothesizing over possible movements, and then use a continuous optimization formulation to obtain a low complexity solution that best explains the scene recordings, i.e., the input image pairs. We test the algorithm on a variety of examples to recover the multiple object structures and their changes.     

基于ORB特征的单目视觉定位算法研究

针对大范围室外场景和具有重复、高频纹理特征(例如水泥地、草坪)的场景,提出了一种鲁棒性强、定位精度高、速度更快的视觉定位算法。采用8级图像金字塔的ORB (Oriented FAST and Rotated BRIEF)特征描述子提取图像特征点,通过K近邻(KNN)匹配相邻图像序列的特征点对,依次解算基础矩阵F和本质矩阵E,采用自适应法利用单应矩阵和本质矩阵进行位姿估计,最后解算两帧图像间相机刚体运动的旋转R和平移t,利用三角测量法则求解出匹配点的三维坐标,重建相机运动轨迹。为了提高算法性能,提出采用最小化基于点特征的非线性重投影误差优化三维点。通过调用OpenCV在C++中实现,对所采集的数据集进行测试,测试结果表明,该方法比传统的3D位姿估计更优,实时可行。由于其基于单目而实现,因此无法得到尺度信息。     

Non‐Rigid Registration Under Isometric Deformations

We present a robust and efficient algorithm for the pairwise non‐rigid registration of partially overlapping 3D surfaces. Our approach treats non‐rigid registration as an optimization problem and solves it by alternating between correspondence and deformation optimization. Assuming approximately isometric deformations, robust correspondences are generated using a pruning mechanism based on geodesic consistency. We iteratively learn an appropriate deformation discretization from the current set of correspondences and use it to update the correspondences in the next iteration. Our algorithm is able to register partially similar point clouds that undergo large deformations, in just a few seconds. We demonstrate the potential of our algorithm in various applications such as example based articulated segmentation, and shape interpolation.     

一种应用于大角度变换点云的配准方法

针对传统配准法不能很好解决大角度变换点云的配准这一问题,提出一种基于精 确对应特征点对及其 K 邻域点云的配准方法。首先分别计算两组点云的 FPFH 值,根据特征值 建立点云间的对应关系;然后通过 RANSAC 滤除其中错误的匹配点对,得到相对精确的特征点 对集合;之后通过 KD-tree 搜索的方式分别找出特征点对 R 半径邻域内的点,应用 ICP 算法得 到两部分点云的最优收敛;最后将计算得到的相对位置关系应用到原始点云上得到配准结果。 通过对斯坦福大学点云库中 Dragon、Happy Buddha 模型以及 Kinect 采集的石膏像数据进行配 准和比较,实验表明该方法能够有效解决大角度变换点云的配准问题,是一种具有高精度和高 鲁棒性的三维点云配准方法。     

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.     

A Tracking Registration Method for Augmented Reality Based on Multi-modal Template Matching and Point Clouds

In order to overcome the defects where the surface of the object lacks sufficient texture features and the algorithm cannot meet the real-time requirements of augmented reality, a markerless augmented reality tracking registration method based on multimodal template matching and point clouds is proposed. The method first adapts the linear parallel multi-modal LineMod template matching method with scale invariance to identify the texture-less target and obtain the reference image as the key frame that is most similar to the current perspective. Then, we can obtain the initial pose of the camera and solve the problem of re-initialization because of tracking registration interruption. A point cloud-based method is used to calculate the precise pose of the camera in real time. In order to solve the problem that the traditional iterative closest point (ICP) algorithm cannot meet the real-time requirements of the system, Kd-tree (k-dimensional tree) is used under the graphics processing unit (GPU) to replace the part of finding the nearest points in the original ICP algorithm to improve the speed of tracking registration. At the same time, the random sample consensus (RANSAC) algorithm is used to remove the error point pairs to improve the accuracy of the algorithm. The results show that the proposed tracking registration method has good real-time performance and robustness.

     

Reconstructing camera projection matrices from multiple pairwise overlapping views

In this work we address the problem of projective reconstruction from multiple views with missing data. Factorization based algorithms require point correspondences across all the views. In many applications this is an unrealistic assumption. Current methods that solve the problem of projective reconstruction with missing data require correspondence information across triplets of images. We propose a projective reconstruction method that yields a consistent camera set given the fundamental matrices between pairs of views without directly using the image correspondences. The algorithm is based on breaking the reconstruction problem into small steps. In each step, we eliminate as much uncertainty as possible.     

Lines and Points in Three Views and the Trifocal Tensor

This paper discusses the basic role of the trifocal tensor in scene reconstruction from three views. This 3× 3× 3 tensor plays a role in the analysis of scenes from three views analogous to the role played by the fundamental matrix in the two-view case. In particular, the trifocal tensor may be computed by a linear algorithm from a set of 13 line correspondences in three views. It is further shown in this paper, that the trifocal tensor is essentially identical to a set of coefficients introduced by Shashua to effect point transfer in the three view case. This observation means that the 13-line algorithm may be extended to allow for the computation of the trifocal tensor given any mixture of sufficiently many line and point correspondences. From the trifocal tensor the camera matrices of the images may be computed, and the scene may be reconstructed. For unrelated uncalibrated cameras, this reconstruction will be unique up to projectivity. Thus, projective reconstruction of a set of lines and points may be carried out linearly from three views.     

Registration of partially overlapping surfaces by rejection of false point correspondences

We present a new algorithm for the registration of three-dimensional partially overlapping surfaces. It is based on an efficient scheme for the rejection of false point correspondences (correspondence outliers) and does not require initial pose estimation or feature extraction. An initial list of corresponding points is first derived using the regional properties of vertices on both surfaces. From these point correspondences, pairs of corresponding rigid triplets are formed. The normal vectors at the vertices of each corresponding triplet are used to compute the candidate rotations. By clustering the candidate rotation axes and candidate rotation angles separately, a large number of false correspondences are eliminated and an approximate rotation is decided, from which an approximate translation is also obtained. Finally, the optimal transformation parameters are determined by further refining the estimated parameters in an iterative manner. Mathematical analysis and experimental results show that the registration process is fast and accurate even when the objects are regularly shaped and contain many regionally similar surface patches.