增强现实中三维跟踪注册技术概述

三维跟踪注册技术是增强现实领域中的重要关键技术，通过对现实场景中的图像或物体进行跟踪与定位，将虚拟物体按照正确的空间透视关系叠加到真实场景当中。综述了增强现实中的三维跟踪注册技术（基于计算机视觉的跟踪注册技术、基于硬件传感器的跟踪注册技术以及混合跟踪注册技术），详细阐述了不同跟踪注册技术的优势与不足，并介绍了不同跟踪注册技术的当前应用现状，随后探讨了跟踪注册技术的发展趋势与存在的难题，对三维跟踪注册技术的进一步研究进行了展望。     

Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

利用深度图像的虚实物体碰撞交互研究与实现

对传统增强现实系统中虚拟物体与真实物体难以进行碰撞交互的问题,提出一种对深度图像中的场景进行分割,并基于分割结果构建代理几何体的方法来实现虚、实物体的碰撞交互。采用Kinect等深度获取设备获取当前真实场景的彩色图像信息和深度图像信息;通过深度图像的法向聚类及平面拟合技术来识别出场景中的主平面区域;对除去主平面区域的其他聚类点云区域进行融合处理,得到场景中的其他主要物体区域;为识别出的主平面构建虚拟平面作为该平面的代理几何体,为分割出的物体构建包围盒来作为其代理几何体。将这些代理几何体叠加到真实物体上,并对之赋予物理属性,即可模拟实现虚拟物体与真实物体的碰撞交互。实验结果表明,该方法可有效分割简单场景,从而实现虚实交互。     

基于视觉的增强现实运动跟踪算法

增强现实系统不仅具有虚拟现实的特点同时具有虚实结合的新特性，为实现虚拟物体与真实物体间的完善结合，必须实时地动态跟踪摄像与真实物体间的相对位置和方向，建立观测模,墼是而通过动态三维显示技术迅速地将虚拟物体添加到真实物体之上，然而目前大多数增强现实系统的注册对象均匀静物体，运动物体的注册跟踪尚很少有人涉足。该算法通过标志点的光流场估计真实环境中运动物体的运动参数，根据透视投影原理和刚体的运动特性确定摄像机与运动物体间的相对位置和方向，实现增强现实系统的运动目标跟踪注册。该算法构架简单、实时性强，易于实现，扩展了增强现实系统的应用范围。     

水彩画风格实时增强现实技术

在传统的AR系统中,虚拟物体和真实场景在视觉上存在较为明显的差异,达不到虚拟物体和真实场景无缝结合的要求。将增强现实技术与NPR有机地结合起来,减小这种视觉差异,研究并实现了水彩画风格的增强现实系统。     

Image-driven virtual simulation of arthroscopy

In recent years, minimally invasive arthroscopic surgery has replaced a number of conventional open orthopedic surgery procedures on joints. While this achieves a number of advantages for the patient, the surgeons have to learn very different skills, since the surgery is performed with special miniature pencil-like tools and cameras inserted through little incisions while observing the surgical field on video monitor. Therefore, virtual reality simulation becomes an alternative to traditional surgical training based on hundreds years old apprentice–master model that involves either real patients or increasingly difficult to procure cadavers. Normally, 3D simulation of the virtual surgical field requires significant efforts from the software developers but yet remains not always photorealistic. In contrast to this, for photorealistic visualization and haptic interaction with the surgical field we propose to use real arthroscopic images augmented with 3D object models. The proposed technique allows for feeling the joint cavity displayed on video monitor as real 3D objects rather than their images while various surgical procedures, such as menisectomy, are simulated in real time. In the preprocessing stage of the proposed approach, the arthroscopic images are stitched into panoramas and augmented with implicitly defined object models representing deformable menisci. In the simulation loop, depth information from the mixed scene is used for haptic rendering. The scene depth map and visual display are reevaluated only when the scene is modified.     

Automatic camera control in virtual environments augmented using multiple sparse videos

Automated virtual camera control has been widely used in animation and interactive virtual environments. We have developed a multiple sparse camera based free view video system prototype that allows users to control the position and orientation of a virtual camera, enabling the observation of a real scene in three dimensions (3D) from any desired viewpoint. Automatic camera control can be activated to follow selected objects by the user. Our method combines a simple geometric model of the scene composed of planes (virtual environment), augmented with visual information from the cameras and pre-computed tracking information of moving targets to generate novel perspective corrected 3D views of the virtual camera and moving objects. To achieve real-time rendering performance, view-dependent textured mapped billboards are used to render the moving objects at their correct locations and foreground masks are used to remove the moving objects from the projected video streams. The current prototype runs on a PC with a common graphics card and can generate virtual 2D views from three cameras of resolution 768×576 with several moving objects at about 11 fps.     

基于投影技术的增强现实注册方法

注册技术是实现增强现实应用系统的关键问题之一。实现了一种利用投影技术改进了的注册方法。这种注册方法主要包含4个步骤：（1）检测标识和特征点的位置;（2）利用投影技术跟踪摄像机;（3）计算模型视图矩阵;（4）绘制虚拟物体。在初始阶段,指定4个点建立世界坐标系,虚拟物体将会被叠加到这个坐标系上;在注册过程中,利用投影技术跟踪特征点的位置,并计算用于增强的模型视图转换。这种注册方法可以为一般的室内增强现实系统提供所需的精确度和实时性。     

Toward spontaneous interaction with the Perceptive Workbench

Until now, we have interacted with computers mostly by using wire-based devices. Typically, the wires limit the distance of movement and inhibit freedom of orientation. In addition, most interactions are indirect. The user moves a device as an analog for the action created in the display space. We envision an untethered interface that accepts gestures directly and can accept any objects we choose as interactors. We discuss methods for producing more seamless interaction between the physical and virtual environments through the Perceptive Workbench. We applied the system to an augmented reality game and a terrain navigating system. The Perceptive Workbench can reconstruct 3D virtual representations of previously unseen real-world objects placed on its surface. In addition, the Perceptive Workbench identifies and tracks such objects as they are manipulated on the desk's surface and allows the user to interact with the augmented environment through 2D and 3D gestures     

Tri-view morphing

This paper presents an efficient image-based approach to navigate a scene based on only three wide-baseline uncalibrated images without the explicit use of a 3D model. After automatically recovering corresponding points between each pair of images, an accurate trifocal plane is extracted from the trifocal tensor of these three images. Next, based on a small number of feature marks using a friendly GUI, the correct dense disparity maps are obtained by using our trinocular-stereo algorithm. Employing the barycentric warping scheme with the computed disparity, we can generate an arbitrary novel view within a triangle spanned by three camera centers. Furthermore, after self-calibration of the cameras, 3D objects can be correctly augmented into the virtual environment synthesized by the tri-view morphing algorithm. Three applications of the tri-view morphing algorithm are demonstrated. The first one is 4D video synthesis, which can be used to fill in the gap between a few sparsely located video cameras to synthetically generate a video from a virtual moving camera. This synthetic camera can be used to view the dynamic scene from a novel view instead of the original static camera views. The second application is multiple view morphing, where we can seamlessly fly through the scene over a 2D space constructed by more than three cameras. The last one is dynamic scene synthesis using three still images, where several rigid objects may move in any orientation or direction. After segmenting three reference frames into several layers, the novel views in the dynamic scene can be generated by applying our algorithm. Finally, the experiments are presented to illustrate that a series of photo-realistic virtual views can be generated to fly through a virtual environment covered by several static cameras.     

Interactive 3D content insertion in images for multimedia applications

This article addresses the problem of creating interactive mixed reality applications where virtual objects interact in images of real world scenarios. This is relevant to create games and architectural or space planning applications that interact with visual elements in the images such as walls, floors and empty spaces. These scenarios are intended to be captured by the users with regular cameras or using previously taken photographs. Introducing virtual objects in photographs presents several challenges, such as pose estimation and the creation of a visually correct interaction between virtual objects and the boundaries of the scene. The two main research questions addressed in this article include, the study of the feasibility of creating interactive augmented reality (AR) applications where virtual objects interact in a real world scenario using the image detected high-level features and, also, verifying if untrained users are capable and motivated enough to perform AR initialization steps. The proposed system detects the scene automatically from an image with additional features obtained using basic annotations from the user. This operation is significantly simple to accommodate the needs of non-expert users. The system analyzes one or more photos captured by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create mixed and augmented reality applications where the user interactively introduces virtual objects that blend with the picture in real time and respond to the physical environment. To validate the solution several system tests are described and compared using available external image datasets.     

视触觉融合的增强现实三维注册方法

增强现实技术是近年来人机交互领域的研究热点。在增强现实环境下加入触觉感知,可使用户在真实场景中看到并感知到虚拟对象。为了实现增强现实环境下与虚拟对象之间更加自然的交互,提出一种视触觉融合的三维注册方法。基于图像视觉技术获得三维注册矩阵;借助空间转换关系求解出触觉空间与图像空间的转换关系;结合两者与摄像头空间的关系实现视触觉融合的增强现实交互场景。为验证该方法的有效性,设计了一个基于视触觉增强现实的组装机器人项目。用户可触摸并移动真实环境中的机器人零件,还能在触摸时感受到反馈力,使交互更具真实感。     

Augmented reality navigation systems

The augmented reality (AR) research community has been developing a manifold of ideas and concepts to improve the depiction of virtual objects in a real scene. In contrast, current AR applications require the use of unwieldy equipment which discourages their use. In order to essentially ease the perception of digital information and to naturally interact with the pervasive computing landscape, the required AR equipment has to be seamlessly integrated into the user’s natural environment. Considering this basic principle, this paper proposes the car as an AR apparatus and presents an innovative visualization paradigm for navigation systems that is anticipated to enhance user interaction.     

基于环境映照自动对齐的高质量虚实融合技术

要实现高品质的增强现实效果需要解决虚拟物体与现实场景的光照一致性问题.虽然采用HDR技术能获取场景的环境映照,但需要解决所获取的光照环境信息与真实场景的对齐问题.为此提出一种基于特征自动匹配的环境映照对齐方法.首先采用Affine-SIFT算法和随机抽样一致性算法对环境映照和拍摄场景进行特征匹配并优化匹配结果,然后利用基于运动推断结构的摄像机定标算法求得匹配对的三维位置,从而计算出环境映照与真实场景的对应关系,实现了二者的自动对齐.基于该技术搭建的高真实感实时虚实融合系统采用基于关键帧的相机跟踪技术,可以实时地将虚拟物体注册到拍摄的视频场景里,并允许对其进行实时的交互编辑.在渲染时有效地利用了自动对齐后的光照环境信息,采用重要性采样算法和阴影映射技术实现了实时的高质量渲染.实验结果表明,所搭建的增强现实系统很好地解决了实时虚实融合中的几何一致性和光照一致性问题.     

A hypermedia authoring tool for augmented and virtual reality

Most existing hypermedia authoring systems are intended for use on desktop computers. These systems are typically designed for the creation of 2D documents and therefore employ 2D authoring mechanisms. In contrast, authoring systems for nontraditional multimedia/hypermedia experiences for 3D virtual or augmented worlds focus mainly on creating separate media objects and embedding them within the user's surroundings. As a result, linking these media objects to create 3D hypermedia is a tedious manual task. To address this issue, we present an authoring tool for creating and editing linked 3D hypermedia narratives that are interwoven with a wearable computer user's surrounding environment. Our system is designed for use by authors who are not programmers, and allows them to preview their results on a desktop workstation, as well as with an augmented or virtual reality system.     

边界光场

提出一种新的基于图像的绘制方法——边界光场．该方法基于3D全光函数的思想，并使之与场景几何相结合．该方法克服了已有的IBR漫游系统的一些缺陷，利用自适应的的全光采样模式，根据场景复杂度或用户要求组织采样数据，降低了场景数据量；由于场景几何的参与，纠正了较大的深度变形；新的采样数据组织模式去除了对漫游范围的限制．文中方法可有效地应用于虚拟或真实场景漫游系统中．     

基于Kinect的民族弓弦乐器虚拟演奏系统

利用Kinect相机结合增强现实技术和手势识别方法设计并实现了一个弓弦乐器虚拟演奏系统——以二胡为例.将Kinect获取的现实场景和虚拟乐器融合在一起绘制成增强现实场景.通过Kinect得到的深度数据和贝叶斯肤色模型将用户的左手分割出来,并再次绘制在增强图像上形成新的图像,从而解决虚拟演奏场景中的虚实遮挡问题.利用基于反向动力学和马尔可夫模型的三维虚拟手势拟合方法,对演奏过程中的左手手势进行识别,并结合右手的运动状态完成乐器的虚拟演奏.     

Visual immersive haptic mathematics

In the modern urban society, human brain is not being sufficiently trained to deal with problems which require 3D perception. As a result, when teaching subjects richly infused with mathematics it is usually a challenge for the learners to follow the instructor and visualize how mathematical concepts reflect in 3D geometry and colors. We have proposed an approach that would allow for defining complex geometry, visual appearance and tangible physical properties of the virtual objects using language of mathematical functions. It allows the learners to get immersed within the 3D scene and explore the shapes which are being modeled visually and haptically. We illustrate this concept using our function-based extension of X3D and VRML. Besides definition of objects with mathematical functions straight in the scene file, standard X3D and VRML objects can be converted to tangible ones as well as augmented with function-defined visual appearances. Since the function-defined models are small in size, it is possible to perform their collaborative interactive modifications with concurrent synchronous visualization at each client computer with any required level of detail.     

增强现实技术在虚拟演播室系统中的应用

虚拟演播室是虚拟现实技术和视频合成技术相结合的产物,其场景是计算机生成的三维场景,由于人们对虚拟场景和复杂度的无限要求,使得场景的实时显示十分困难,使用基于图像的绘制技术构造虚拟空间能够较好地解决这个问题。在虚拟演播室中,演员需要与三维运动虚拟物体进行交互,运用增强现实技术,可以将三维虚拟物体与基于图象绘制的虚拟场景融合在一起。     

复杂机电产品线缆虚实融合装配体系构建及其关键技术

针对传统的基于实物样机的线缆模拟装配实验方法造成的效率低、精确度低、装配 质量和可靠性难以得到保证等问题,提出一种面向复杂机电产品的线缆虚实融合装配方法。首先 在分析了增强现实技术应用于线缆装配的优势基础上,构建了基于增强现实的线缆虚实融合装配 体系;其次,对装配体系中涉及到的关键技术进行了介绍,重点对线缆三维注册技术、线缆虚实 遮挡技术、线缆虚实光照技术以及相机标定与跟踪技术等关键技术进行了阐述;最后介绍了线缆 虚实融合装配研究内容,给出了基于增强现实的线缆虚实融合装配系统运行流程。