Dense 3-D Reconstruction of an Outdoor Scene by Hundreds-Baseline Stereo Using a Hand-Held Video Camera

Three-dimensional (3-D) models of outdoor scenes are widely used for object recognition, navigation, mixed reality, and so on. Because such models are often made manually with high costs, automatic 3-D reconstruction has been widely investigated. In related work, a dense 3-D model is generated by using a stereo method. However, such approaches cannot use several hundreds images together for dense depth estimation because it is difficult to accurately calibrate a large number of cameras. In this paper, we propose a dense 3-D reconstruction method that first estimates extrinsic camera parameters of a hand-held video camera, and then reconstructs a dense 3-D model of a scene. In the first process, extrinsic camera parameters are estimated by tracking a small number of predefined markers of known 3-D positions and natural features automatically. Then, several hundreds dense depth maps obtained by multi-baseline stereo are combined together in a voxel space.So, we can acquire a dense 3-D model of the outdoor scene accurately by using several hundreds input images captured by a hand-held video camera.     

复杂机电产品线缆虚实融合装配体系构建及其关键技术

针对传统的基于实物样机的线缆模拟装配实验方法造成的效率低、精确度低、装配 质量和可靠性难以得到保证等问题,提出一种面向复杂机电产品的线缆虚实融合装配方法。首先 在分析了增强现实技术应用于线缆装配的优势基础上,构建了基于增强现实的线缆虚实融合装配 体系;其次,对装配体系中涉及到的关键技术进行了介绍,重点对线缆三维注册技术、线缆虚实 遮挡技术、线缆虚实光照技术以及相机标定与跟踪技术等关键技术进行了阐述;最后介绍了线缆 虚实融合装配研究内容,给出了基于增强现实的线缆虚实融合装配系统运行流程。     

增强现实中的虚实注册技术研究

增强现实是把计算机产生的虚拟物体或其他信息合成到用户看到的真实世界中的一种技术 .实现虚拟信息与真实景象的正确匹配或有机结合是每个增强现实实用系统都必须完成的一项重要任务 ,且它的具体实现需依赖于计算机硬件和软件系统的相互配合及协调 .为了有效地降低增强现实应用系统中的虚实注册误差 ,在对基于位置跟踪和相机定标技术的传统虚实注册技术进行分析的基础上 ,首先研究了仿射变换技术理论在增强现实中的应用 ,并提出了在全局仿射坐标系下 ,获取像素相对深度参数的一种策略 ;然后在对 AR中的虚实注册问题进行研究分析的基础上 ,进一步讨论了图象分析技术在虚实注册问题中的应用问题 ,并提出了一种基于数值背景表达的快速虚实注册策略 ,以提高虚实注册过程中 ,场景特征点提取的精度和速度 ;最后还分别用该仿射理论进行了动态和静态的虚实注册实验 ,实验结果表明 ,该方法能有效地降低增强现实应用系统的虚实注册误差 .     

Real-time and accurate extrinsic camera parameter estimation using feature landmark database for augmented reality

In the field of augmented reality (AR), many kinds of vision-based extrinsic camera parameter estimation methods have been proposed to achieve geometric registration between real and virtual worlds. Previously, a feature landmark-based camera parameter estimation method was proposed. This is an effective method for implementing outdoor AR applications because a feature landmark database can be automatically constructed using the structure-from-motion (SfM) technique. However, the previous method cannot work in real time because it entails a high computational cost or matching landmarks in a database with image features in an input image. In addition, the accuracy of estimated camera parameters is insufficient for applications that need to overlay CG objects at a position close to the user's viewpoint. This is because it is difficult to compensate for visual pattern change of close landmarks when only the sparse depth information obtained by the SfM is available. In this paper, we achieve fast and accurate feature landmark-based camera parameter estimation by adopting the following approaches. First, the number of matching candidates is reduced to achieve fast camera parameter estimation by tentative camera parameter estimation and by assigning priorities to landmarks. Second, image templates of landmarks are adequately compensated for by considering the local 3-D structure of a landmark using the dense depth information obtained by a laser range sensor. To demonstrate the effectiveness of the proposed method, we developed some AR applications using the proposed method.     

基于视觉的增强现实运动跟踪算法

增强现实系统不仅具有虚拟现实的特点同时具有虚实结合的新特性，为实现虚拟物体与真实物体间的完善结合，必须实时地动态跟踪摄像与真实物体间的相对位置和方向，建立观测模,墼是而通过动态三维显示技术迅速地将虚拟物体添加到真实物体之上，然而目前大多数增强现实系统的注册对象均匀静物体，运动物体的注册跟踪尚很少有人涉足。该算法通过标志点的光流场估计真实环境中运动物体的运动参数，根据透视投影原理和刚体的运动特性确定摄像机与运动物体间的相对位置和方向，实现增强现实系统的运动目标跟踪注册。该算法构架简单、实时性强，易于实现，扩展了增强现实系统的应用范围。     

基于Kinect的民族弓弦乐器虚拟演奏系统

利用Kinect相机结合增强现实技术和手势识别方法设计并实现了一个弓弦乐器虚拟演奏系统——以二胡为例.将Kinect获取的现实场景和虚拟乐器融合在一起绘制成增强现实场景.通过Kinect得到的深度数据和贝叶斯肤色模型将用户的左手分割出来,并再次绘制在增强图像上形成新的图像,从而解决虚拟演奏场景中的虚实遮挡问题.利用基于反向动力学和马尔可夫模型的三维虚拟手势拟合方法,对演奏过程中的左手手势进行识别,并结合右手的运动状态完成乐器的虚拟演奏.     

Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

Scalable and Efficient Video Coding Using 3-D Modeling

In this paper, we present a three-dimensional (3D) model-based video coding scheme for streaming static scene video in a compact way but also enabling time and spatial scalability according to network or terminal capability and providing 3D functionalities. The proposed format is based on encoding the sequence of reconstructed models using second-generation wavelets, and efficiently multiplexing the resulting geometric, topological, texture, and camera motion binary representations. The wavelets decomposition can be adaptive in order to fit to images and scene contents. To ensure time scalability, this representation is based on a common connectivity for all 3D models, which also allows straightforward morphing between successive models ensuring visual continuity at no additional cost. The method proves to be better than previous methods for video encoding of static scenes, even better than state-of-the-art video coders such as H264 (also known as MPEG AVC). Another application of our approach are smoothing camera path for suppression of jitter from hand-held acquisition and the fast transmission and real-time visualization of virtual environments obtained by video capture, for virtual or augmented reality and interactive walk-through in photo-realistic 3D environments around the original camera path     

3D human modeling from a single depth image dealing with self-occlusion

This paper presents a 2D to 3D conversion scheme to generate a 3D human model using a single depth image with several color images. In building a complete 3D model, no prior knowledge such as a pre-computed scene structure and photometric and geometric calibrations is required since the depth camera can directly acquire the calibrated geometric and color information in real time. The proposed method deals with a self-occlusion problem which often occurs in images captured by a monocular camera. When an image is obtained from a fixed view, it may not have data for a certain part of an object due to occlusion. The proposed method consists of following steps to resolve this problem. First, the noise in a depth image is reduced by using a series of image processing techniques. Second, a 3D mesh surface is constructed using the proposed depth image-based modeling method. Third, the occlusion problem is resolved by removing the unwanted triangles in the occlusion region and filling the corresponding hole. Finally, textures are extracted and mapped to the 3D surface of the model to provide photo-realistic appearance. Comparison results with the related work demonstrate the efficiency of our method in terms of visual quality and computation time. It can be utilized in creating 3D human models in many 3D applications.     

增强现实中三维跟踪注册技术概述

三维跟踪注册技术是增强现实领域中的重要关键技术，通过对现实场景中的图像或物体进行跟踪与定位，将虚拟物体按照正确的空间透视关系叠加到真实场景当中。综述了增强现实中的三维跟踪注册技术（基于计算机视觉的跟踪注册技术、基于硬件传感器的跟踪注册技术以及混合跟踪注册技术），详细阐述了不同跟踪注册技术的优势与不足，并介绍了不同跟踪注册技术的当前应用现状，随后探讨了跟踪注册技术的发展趋势与存在的难题，对三维跟踪注册技术的进一步研究进行了展望。     

Multithreaded Hybrid Feature Tracking for Markerless Augmented Reality

We describe a novel markerless camera tracking approach and user interaction methodology for augmented reality (AR) on unprepared tabletop environments. We propose a real-time system architecture that combines two types of feature tracking. Distinctive image features of the scene are detected and tracked frame-to-frame by computing optical flow. In order to achieve real-time performance, multiple operations are processed in a synchronized multi-threaded manner: capturing a video frame, tracking features using optical flow, detecting distinctive invariant features, and rendering an output frame. We also introduce user interaction methodology for establishing a global coordinate system and for placing virtual objects in the AR environment by tracking a user's outstretched hand and estimating a camera pose relative to it. We evaluate the speed and accuracy of our hybrid feature tracking approach, and demonstrate a proof-of-concept application for enabling AR in unprepared tabletop environments, using bare hands for interaction.     

Augmented reality camera tracking with homographies

To realistically integrate 3D graphics into an unprepared environment, camera position must be estimated by tracking natural image features. We apply our technique to cases where feature positions in adjacent frames of an image sequence are related by a homography, or projective transformation. We describe this transformation's computation and demonstrate several applications. First, we use an augmented notice board to explain how a homography, between two images of a planar scene, completely determines the relative camera positions. Second, we show that the homography can also recover pure camera rotations, and we use this to develop an outdoor AR tracking system. Third, we use the system to measure head rotation and form a simple low-cost virtual reality (VR) tracking solution.     

基于视觉的增强现实技术研究综述

增强现实技术是当前计算机视觉领域中的热点问题之一,本文系统综述了基于视觉的增强现实 技术的研究现状．首先,对基于视觉的增强现实技术的研究现状进行了概述;其次,详细地阐述了该领域目 前出现的新技术,包括三维注册技术、摄像机标定技术、跟踪技术、虚实光照的一致性以及AR 实空间建模技 术,继而论述了该领域在研究中存在的问题,并对该领域中需要进一步解决的问题进行了讨论;最后,对增 强现实技术的进一步研究进行了展望．     

The constrained SLAM framework for non-instrumented augmented reality

This paper addresses the challenging issue of marker less tracking for Augmented Reality. It proposes a real-time camera localization in a partially known environment, i.e. for which a geometric 3D model of one static object in the scene is available. We propose to take benefit from this geometric model to improve the localization of keyframe-based SLAM by constraining the local bundle adjustment process with this additional information. We demonstrate the advantages of this solution, called contrained SLAM, on both synthetic and real data and present very convincing augmentation of 3D objects in real-time. Using this tracker, we also propose an interactive augmented reality system for training application. This system, based on a Optical See-Through Head Mounted Display, allows to augment the users vision field with virtual information accurately co-registered with the real world. To keep greatly benefit of the potential of this hand free device, the system combines the tracker module with a simple user-interaction vision-based module to provide overlaid information in response to user requests.     

Robust camera pose tracking for augmented reality using particle filtering framework

In this paper, we present new solutions for the problem of estimating the camera pose using particle filtering framework. The proposed approach is suitable for real-time augmented reality (AR) applications in which the camera is held by the user. This work demonstrates that particle filtering improve the robustness of the tracking comparing to existing approaches, such as those based on the Kalman filter. We propose a tracking framework for both points and lines features, the particle filter is used to compute the posterior density for the camera 3D motion parameters. We also analyze the sensitivity of our technique when outliers are present in the match data. Outliers arise frequently due to incorrect correspondences which occur because of either image noise or occlusion. Results from real data in an augmented reality setup are then presented, demonstrating the efficiency and robustness of the proposed method.     

基于CV／CAD的三维物体几何建模

在虚拟现实等技术领域中,都涉及到由现实世界中的实际景物建立对应的计算机描述的虚拟景物的问题,为此提出了利用计算机视觉与CAD几何建模技术相结合的三维珠体建模途径,首先通过编码光栅方法获取三维物体的深度图象,并采用数学形态学的方法加以分割,然后利用代数曲面拟合手段对分割后的三维曲面片进行重建,并使用CAD几何建模工具由重建的曲面片构成物体的几何模型,该文给出了初步的实验结果,证明所提出的技术途径基本可行。     

基于增强现实技术的AR赛车游戏

AR赛车是一款融入了增强现实技术的3D赛车游戏。在赛车游戏中运用增强现实技术．使得游戏有了全新的控制方式和场景效果。在游戏控制方式上,玩家不再需要用键盘来控制赛车,而是通过在摄像头前操纵有特定标识图案的纸片来控制赛车;在场景效果上,AR赛车不再像传统游戏那样始终处于虚拟场景中,而是采用真实场景与虚拟物体相结合。这些特点将会给玩家带来全新的游戏体验。     

基于增强现实技术的景观规划系统

增强现实是把计算机产生的虚拟物体或其他信息合成到用户看到的真实世界中的一种技术，利用增强现实技术，设计人员能够有效地进行建筑布局的规划和设计，本文介绍了一个典型的增强现实系统-景观规划系统的实现，针对合成虚拟景物和真实场景过程中需要解决的几何一致性和光照一致性问题，采用了局部三维重建和交互的相机定标技术来实现几何一致性；对于光照一致性，则采用了光照交互指定和自动恢复相结合的方法，同时还考虑了虚拟物体投射的阴影，针对不同场景的实际应用，依据系统生成的结果合成图象，设计人员能够比较有效地完成对景观规划结果的直观评估。     

RGB-Fusion: Monocular 3D reconstruction with learned depth prediction

Generating large-scale and high-quality 3D scene reconstruction from monocular images is an essential technical foundation in augmented reality and robotics. However, the apparent shortcomings (e.g., scale ambiguity, dense depth estimation in texture-less areas) make applying monocular 3D reconstruction to real-world practice challenging. In this work, we combine the advantage of deep learning and multi-view geometry to propose RGB-Fusion, which effectively solves the inherent limitations of traditional monocular reconstruction. To eliminate the confinements of tracking accuracy imposed by the prediction deficiency of neural networks, we propose integrating the PnP (Perspective-n-Point) algorithm into the tracking module. We employ 3D ICP (Iterative Closest Point) matching and 2D feature matching to construct separate error terms and jointly optimize them, reducing the dependence on the accuracy of depth prediction and improving pose estimation accuracy. The approximate pose predicted by the neural network is employed as the initial optimization value to avoid the trapping of local minimums. We formulate a depth map refinement strategy based on the uncertainty of the depth value, which can naturally lead to a refined depth map. Through our method, low-uncertainty elements can significantly update the current depth value while avoiding high-uncertainty elements from adversely affecting depth estimation accuracy. Numerical qualitative and quantitative evaluation results of tracking, depth prediction, and 3D reconstruction show that RGB-Fusion exceeds most monocular 3D reconstruction systems.