混合跟踪技术在增强现实系统中的应用

跟踪技术是增强现实的一个重要研究方向,它致力于解决虚拟信息与真实环境的精确配准问题,构建稳定、高效的增强现实系统。通过分析增强现实系统的跟踪需求,指出单一跟踪存在的不足,说明采用混合跟踪技术的必要性,论述了基于视觉一惯性的混合跟踪技术及其在增强现实中的应用,实现了一种基于VisTracker视觉-惯性混合跟踪器的增强现实实验系统,并给出调试结果。     

Use of inertial sensors to support video tracking

One of the biggest obstacles to building effective augmented reality (AR) systems is the lack of accurate sensors that report the location of the user in an environment during arbitrary long periods of movements. In this paper, we present an effective hybrid approach that integrates inertial and vision‐based technologies. This work is motivated by the need to explicitly take into account the relatively poor accuracy of inertial sensors and thus to define an efficient strategy for the collaborative process between the vision‐based system and the sensor. The contributions of this papers are threefold: (i) our collaborative strategy fully integrates the sensitivity error of the sensor: the sensitivity is practically studied and is propagated into the collaborative process, especially in the matching stage (ii) we propose an original online synchronization process between the vision‐based system and the sensor. This process allows us to use the sensor only when needed. (iii) an effective AR system using this hybrid tracking is demonstrated through an e‐commerce application in unprepared environments. Copyright © 2007 John Wiley & Sons, Ltd.     

紧耦合多传感器混合跟踪算法

在增强现实应用中实现对运动目标的准确跟踪是一个具有挑战性的任务。基于混合跟踪通过对多传感器信息的融合通常比单一传感器跟踪算法更为优越的特性,提出了一种新的紧耦合混合跟踪算法实现视觉与惯性传感器信息的实时融合。该算法基于多频率的测量数据同步,通过强跟踪滤波器引入时变衰减因子自适应调整滤波预测误差协方差,实现对运动目标位置数据的准确估计。通过标示物被遮挡状态下的跟踪实验结果表明,该方法能有效改善基于扩展卡尔曼滤波器的混合跟踪算法对运动目标位置信息预测估计的准确性,提高跟踪快速移动目标的稳定性,适用于大范围移动条件下的增强现实系统。     

Orientation tracking for outdoor augmented reality registration

Our work stems from a program focused on developing tracking technologies for wide-area augmented realities in unprepared outdoor environments. Other participants in the Defense Advanced Research Projects Agency (Darpa) funded Geospatial Registration of Information for Dismounted Soldiers (Grids) program included University of North Carolina at Chapel Hill and Raytheon. We describe a hybrid orientation tracking system combining inertial sensors and computer vision. We exploit the complementary nature of these two sensing technologies to compensate for their respective weaknesses. Our multiple-sensor fusion is novel in augmented reality tracking systems, and the results demonstrate its utility     

A Flexible Software Architecture for Hybrid Tracking

Fusion of vision-based and inertial pose estimation has many high-potential applications in navigation, robotics, and augmented reality. Our research aims at the development of a fully mobile, completely self-contained tracking system, that is able to estimate sensor motion from known 3D scene structure. This requires a highly modular and scalable software architecture for algorithm design and testing. As the main contribution of this paper, we discuss the design of our hybrid tracker and emphasize important features: scalability, code reusability, and testing facilities. In addition, we present a mobile augmented reality application, and several first experiments with a fully mobile vision-inertial sensor head. Our hybrid tracking system is not only capable of real-time performance, but can also be used for offline analysis of tracker performance, comparison with ground truth, and evaluation of several pose estimation and information fusion algorithms. © 2004 Wiley Periodicals, Inc.     

增强现实游戏的场景重建和运动物体跟踪技术

针对增强现实游戏应用,介绍了一种基于自然特征的实时跟踪算法,它可以在一个未知但纹理丰富的环境中,不需要任何预先的学习,对相机和若干出现在相机视野里的刚体进行并行跟踪,获得相机和各个刚体的姿态和运动信息,可以在刚体移出相机视野以及相机剧烈运动导致跟踪丢失的情况下进行自动恢复。基于静态环境中并行相机跟踪和三维重建的算法框架进行扩展和改进,引入了用于交互的运动刚体。结果表明,利用现有的多核处理技术,在含有运动物体的环境中进行相机和运动物体的并行跟踪是可行的,为增强现实环境中同时支持显示和交互提出了廉价易行的解决方案。     

Markerless tracking system for augmented reality in the automotive industry

This paper presents a complete natural feature based tracking system that supports the creation of augmented reality applications focused on the automotive sector. The proposed pipeline encompasses scene modeling, system calibration and tracking steps. An augmented reality application was built on top of the system for indicating the location of 3D coordinates in a given environment which can be applied to many different applications in cars, such as a maintenance assistant, an intelligent manual, and many others. An analysis of the system was performed during the Volkswagen/ISMAR Tracking Challenge 2014, which aimed to evaluate state-of-the-art tracking approaches on the basis of requirements encountered in automotive industrial settings. A similar competition environment was also created by the authors in order to allow further studies. Evaluation results showed that the system allowed users to correctly identify points in tasks that involved tracking a rotating vehicle, tracking data on a complete vehicle and tracking with high accuracy. This evaluation allowed also to understand the applicability limits of texture based approaches in the textureless automotive environment, a problem not addressed frequently in the literature. To the best of the authors’ knowledge, this is the first work addressing the analysis of a complete tracking system for augmented reality focused on the automotive sector which could be tested and validated in a major benchmark like the Volkswagen/ISMAR Tracking Challenge, providing useful insights on the development of such expert and intelligent systems.     

增强现实混合跟踪技术综述

混合跟踪是增强现实(Augmented reality, AR)领域最近二十年发展迅速的重要关键技术, 是增强现实姿态跟踪系统同时实现高精度和强鲁棒性的有效途径. 本文全面完整地论述了增强现实混合跟踪技术,详细阐述了混合跟踪器所涉及的混合跟踪、标定和时间同步等重要方法, 介绍了不同类型混合跟踪器的当前应用现状,然后探讨了混合跟踪技术的发展趋势及其存在的难题,最后展望了混合跟踪技术的应用前景.     

Smartphone-based pedestrian tracking in indoor corridor environments

As the use of smartphones spreads rapidly, user localization becomes an important issue for providing diverse location-based services (LBS). While tracking users in outdoor environments is easily done with GPS, the solution for indoor tracking is not trivial. One common technique for indoor user tracking is to employ inertial sensors, but such a system needs to be capable of handling noisy sensors that would normally lead to cumulative locating errors. To reduce such error, additional infrastructure has often been deployed to adjust for these cumulative location errors. As well, previous work has used highly accurate sensors or sensors that are strapped to the body. This paper presents a stand-alone pedestrian tracking system, using only a magnetometer and an accelerometer in a smartphone in indoor corridor environments that are normally laid out in a perpendicular design. Our system provides reasonably accurate pedestrian locations without additional infrastructure or sensors. The experiment results show that the location error is less than approximately 7 m, which is considered adequate for indoor LBS applications.     

Accurate overlaying for mobile augmented reality

Mobile augmented reality requires accurate alignment of virtual information with objects visible in the real world. We describe a system for mobile communications to be developed to meet these strict alignment criteria using a combination of computer vision, inertial tracking and low-latency rendering techniques. A prototype low-power and low-latency renderer using an off-the-shelf 3D card is discussed.     

结合图像识别的移动增强现实系统设计与应用

目的 在移动互联网时代下,移动增强现实应用得到越来越快的发展。然而户外场景中存在许多相似结构的建筑,且手机的存储和计算能力有限,因此应用多集中于室内小范围环境,对于室外大规模复杂场景的适应性较弱。对此,建立一套基于云端图像识别的移动增强现实系统。方法 为解决相似特征的误匹配问题,算法中将重力信息加入到SURF和BRISK特征描述中去,构建Gravity-SURF和Gravity-BRISK特征描述。云端系统对增强信息进行有效管理,采用基于Gravity-SURF特征的VLAD方法对大规模图像进行识别;在智能终端上的应用中呈现识别图像的增强信息,并利用识别图像的Gravity-BRISK特征和光流结合的方法对相机进行跟踪,采用Unity3D渲染引擎实时绘制3维模型。结果 在包含重力信息的4 000幅户外图像的数据库中进行实验。采用结合重力信息的特征描述算法,能够增强具有相似特征的描述符的区分性,并提高匹配正确率。图像识别算法的识别率能达到88%以上,识别时间在420 ms左右;光流跟踪的RMS误差小于1.2像素,帧率能达到23 帧/s。结论 本文针对室外大规模复杂场景建立的基于图像识别的移动增强现实系统,能方便对不同应用的增强现实数据进行管理。系统被应用到谷歌眼镜和新闻领域上,不局限于单一的应用领域。结果表明,识别算法和跟踪注册算法能够满足系统的精度和实时性要求。     

基于视频的人体上半身运动跟踪系统

文章介绍了一个可以实时捕捉人体上半身运动的系统。该系统通过结合使用人脸检测、肤色检测和概率跟踪等技术,利用单个普通摄像头在一般的室内环境下跟踪人体的脸部和手臂关节位置。作为一个实时跟踪系统,该系统实现了跟踪的自动初始化以及自动从跟踪错误中恢复,并且在一定程度上处理光照变化、身体自遮挡等问题。该系统可以用于“虚拟现实”、“视频会议”、“基于摄像头的游戏”等一些领域中。     

Robust tracking for augmented reality using retroreflective markers

In this paper, an optical tracking system is introduced for the use in augmented reality (AR). Only a few AR-systems currently exist which are using stereo-vision techniques to estimate the viewing pose. The presented system uses binocular images and retroreflective markers in order to speed up the tracking process and make it more precise and robust. The camera calibration as well as the pose estimation technique is presented. This new optical tracking system, which is based on standard PC hardware, is even suitable to make it portable. In addition, the system is evaluated with regard to its pixel accuracy and depth measurements. This paper shows that the computer vision techniques, which will be presented, are a good choice in order to create a flexible, accurate and easy to use tracking system.     

An EyeTap video-based featureless projective motion estimation assisted by gyroscopic tracking for wearable computer mediated reality

In this paper we present a computationally economical method of recovering the projective motion of head mounted cameras or EyeTap devices, for use in wearable computer-mediated reality. The tracking system combines featureless vision and inertial methods in a closed loop system to achieve accurate robust head tracking using inexpensive sensors. The combination of inertial and vision techniques provides the high accuracy visual registration needed for fitting computer graphics onto real images and the robustness to large interframe camera motion due to fast head rotations. Operating on a 1.2 GHz Pentium III wearable computer with graphics accelerated hardware, the system is able to register live video images with less than 2 pixels of error (0.3 degrees) at 12 frames per second. Fast image registration is achieved by offloading computer vision computation onto the graphics hardware, which is readily available on many wearable computer systems. As an application of this tracking approach, we present a system which allows wearable computer users to share views of their current environments that have been stabilised to another viewer's head position.

The integration of optical and magnetic tracking for multi-user augmented reality

Multi-user augmented reality systems are especially dependent on precise registration. In this paper, we present a hybrid tracking system that combines optical and magnetic tracking. The magnetic tracking is used to give a robust estimate of position and orientation which then can be refined in realtime by optical tracking. The system is more precise than a magnetic tracker and both faster and more reliable than an optical tracker.     

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.     

Heads up and camera down: a vision-based tracking modality for mobile mixed reality

Anywhere Augmentation pursues the goal of lowering the initial investment of time and money necessary to participate in mixed reality work, bridging the gap between researchers in the field and regular computer users. Our paper contributes to this goal by introducing the GroundCam, a cheap tracking modality with no significant setup necessary. By itself, the GroundCam provides high frequency, high resolution relative position information similar to an inertial navigation system, but with significantly less drift. We present the design and implementation of the GroundCam, analyze the impact of several design and run-time factors on tracking accuracy, and consider the implications of extending our GroundCam to different hardware configurations. Motivated by the performance analysis, we developed a hybrid tracker that couples the GroundCam with a wide area tracking modality via a complementary Kalman filter, resulting in a powerful base for indoor and outdoor mobile mixed reality work. To conclude, the performance of the hybrid tracker and its utility within mixed reality applications is discussed.     

适用于户外增强现实系统的混合跟踪定位算法

单一传感器无法解决户外增强现实系统中的跟踪定位问题.为了提高视觉跟踪定位算法的精度和鲁棒性,提出一种基于惯性跟踪器与视觉测量相结合的混合跟踪定位算法.该算法在扩展卡尔曼滤波框架下,通过融合来自视觉与惯性传感器的信息进行摄像机运动轨迹估计,并利用视觉测量信息对惯性传感器的零点偏差进行实时校正;同时采用SCAAT方法解决惯性传感器与视觉测量间的时间采样不同步问题.实验结果表明,该算法能够有效地提高运动估计的精度和稳定性.     

First Person Indoor/Outdoor Augmented Reality Application: ARQuake

This paper presents a first person outdoor/indoor augmented reality application ARQuake that we have developed. ARQuake is an extension of the desktop game Quake, and as such we are investigating how to convert a desktop first person application into an outdoor/indoor mobile augmented reality application. We present an architecture for a low cost, moderately accurate six degrees of freedom tracking system based on GPS, digital compass, and fiducial vision-based tracking. Usability issues such as monster selection, colour, input devices, and multi-person collaboration are discussed.     

增强现实中的跟踪技术

增强现实是一种正在发展的新技术,它将由计算机产生的虚拟场景或信息准确叠加到真实环境中,可以增强用户对真实世界的感知与交互能力;对用户视场及视点的跟踪是增强现实中实现虚、实场景配准的关键技术之一;通过对增强现实中跟踪技术性能要求的讨论,介绍了基于磁场、声学、惯性、光学传感等多种跟踪技术,分析了各种方法的跟踪性能与局限性,认为以基于视觉跟踪为主的混合跟踪技术将是增强现实系统的主流跟踪技术,重点论述了基于视觉的跟踪与基于视觉一惯性的混合跟踪技术及其有待解决的问题,并针对户外增强现实中的跟踪技术做了专门讨论。