Robust detection and tracking of annotations for outdoor augmented reality browsing

A common goal of outdoor augmented reality (AR) is the presentation of annotations that are registered to anchor points in the real world. We present an enhanced approach for registering and tracking such anchor points, which is suitable for current generation mobile phones and can also successfully deal with the wide variety of viewing conditions encountered in real life outdoor use. The approach is based on on-the-fly generation of panoramic images by sweeping the camera over the scene. The panoramas are then used for stable orientation tracking, while the user is performing only rotational movements. This basic approach is improved by several new techniques for the re-detection and tracking of anchor points. For the re-detection, specifically after temporal variations, we first compute a panoramic image with extended dynamic range, which can better represent varying illumination conditions. The panorama is then searched for known anchor points, while orientation tracking continues uninterrupted. We then use information from an internal orientation sensor to prime an active search scheme for the anchor points, which improves matching results. Finally, global consistency is enhanced by statistical estimation of a global rotation that minimizes the overall position error of anchor points when transforming them from the source panorama in which they were created, to the current view represented by a new panorama. Once the anchor points are redetected, we track the user's movement using a novel 3-degree-of-freedom orientation tracking approach that combines vision tracking with the absolute orientation from inertial and magnetic sensors. We tested our system using an AR campus guide as an example application and provide detailed results for our approach using an off-the-shelf smartphone. Results show that the re-detection rate is improved by a factor of 2 compared to previous work and reaches almost 90% for a wide variety of test cases while still keeping the ability to run at interactive frame rates.     

Natural feature tracking for augmented reality

Natural scene features stabilize and extend the tracking range of augmented reality (AR) pose-tracking systems. We develop robust computer vision methods to detect and track natural features in video images. Point and region features are automatically and adaptively selected for properties that lead to robust tracking. A multistage tracking algorithm produces accurate motion estimates, and the entire system operates in a closed-loop that stabilizes its performance and accuracy. We present demonstrations of the benefits of using tracked natural features for AR applications that illustrate direct scene annotation, pose stabilization, and extendible tracking range. Our system represents a step toward integrating vision with graphics to produce robust wide-area augmented realities     

Markerless tracking system for augmented reality in the automotive industry

This paper presents a complete natural feature based tracking system that supports the creation of augmented reality applications focused on the automotive sector. The proposed pipeline encompasses scene modeling, system calibration and tracking steps. An augmented reality application was built on top of the system for indicating the location of 3D coordinates in a given environment which can be applied to many different applications in cars, such as a maintenance assistant, an intelligent manual, and many others. An analysis of the system was performed during the Volkswagen/ISMAR Tracking Challenge 2014, which aimed to evaluate state-of-the-art tracking approaches on the basis of requirements encountered in automotive industrial settings. A similar competition environment was also created by the authors in order to allow further studies. Evaluation results showed that the system allowed users to correctly identify points in tasks that involved tracking a rotating vehicle, tracking data on a complete vehicle and tracking with high accuracy. This evaluation allowed also to understand the applicability limits of texture based approaches in the textureless automotive environment, a problem not addressed frequently in the literature. To the best of the authors’ knowledge, this is the first work addressing the analysis of a complete tracking system for augmented reality focused on the automotive sector which could be tested and validated in a major benchmark like the Volkswagen/ISMAR Tracking Challenge, providing useful insights on the development of such expert and intelligent systems.     

Orientation tracking for outdoor augmented reality registration

Our work stems from a program focused on developing tracking technologies for wide-area augmented realities in unprepared outdoor environments. Other participants in the Defense Advanced Research Projects Agency (Darpa) funded Geospatial Registration of Information for Dismounted Soldiers (Grids) program included University of North Carolina at Chapel Hill and Raytheon. We describe a hybrid orientation tracking system combining inertial sensors and computer vision. We exploit the complementary nature of these two sensing technologies to compensate for their respective weaknesses. Our multiple-sensor fusion is novel in augmented reality tracking systems, and the results demonstrate its utility     

Hybrid tracking for outdoor augmented reality applications

We've developed a fully mobile, wearable AR system that combines a vision-based tracker (primarily software algorithms) that uses natural landmarks, with an inertial tracker (custom hardware and firmware) based on silicon micromachined accelerometers and gyroscopes. Unlike other vision-based and hybrid systems, both components recover the full 6 DOF pose. Fusing the two tracking subsystems gives us the benefits of both technologies, while the sensors' complementary nature helps overcome sensor-specific deficiencies. Our system is tailored to affordable, lightweight, energy-efficient mobile AR applications for urban environments, especially the historic centers of European cities.     

Robust tracking for augmented reality using retroreflective markers

In this paper, an optical tracking system is introduced for the use in augmented reality (AR). Only a few AR-systems currently exist which are using stereo-vision techniques to estimate the viewing pose. The presented system uses binocular images and retroreflective markers in order to speed up the tracking process and make it more precise and robust. The camera calibration as well as the pose estimation technique is presented. This new optical tracking system, which is based on standard PC hardware, is even suitable to make it portable. In addition, the system is evaluated with regard to its pixel accuracy and depth measurements. This paper shows that the computer vision techniques, which will be presented, are a good choice in order to create a flexible, accurate and easy to use tracking system.     

基于混合跟踪的增强现实系统设计与实现

为了提高增强现实系统在可变环境中的跟踪性能,设计了一种基于混合跟踪的增强现实系统,可根据使用者所处状态与环境动态地加载不同传感器的数据流,调用相应的跟踪算法,进行数据融合,得到准确的空间位置信息,并对系统硬件资源实现动态管理,具备一定的可扩展性和可移植性.最后给出了应用实例及分析.     

Real-time optical markerless tracking for augmented reality applications

Augmented reality (AR) technology consists in adding computer-generated information (2D/3D) to a real video sequence in such a manner that the real and virtual objects appear coexisting in the same world. To get a realistic illusion, the real and virtual objects must be properly aligned with respect to each other, which requires a robust real-time tracking strategy—one of the bottlenecks of AR applications. In this paper, we describe the limitations and advantages of different optical tracking technologies, and we present our customized implementation of both recursive tracking and tracking by detection approaches. The second approach requires the implementation of a classifier and we propose the use of a Random Forest classifier. We evaluated both approaches in the context of an AR application for design review. Some conclusions regarding the performance of each approach are given.     

Light direction estimation and hand touchable interaction for augmented reality

Virtual Reality - Augmented reality is a technology that combines a virtual world with the real world. How to improve the realism of augmented reality is an important topic. One focus of this paper...     

Gesture-based interaction via finger tracking for mobile augmented reality

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.     

增强现实中基于自然特征的实时跟踪方法

在增强现实系统的复杂场景中,对目标的实时跟踪受到场景中诸多因素的制约,导致实时跟踪方法效率低且不准确,为此提出一种基于自然特征的实时跟踪方法。设计了一种螺旋分割模型,对捕获的图像进行螺旋分割,利用SURF算法在分割子块中提取特征点,并进行匹配。在对目标进行跟踪定位时,利用前一帧来预测当前帧目标出现的位置,以减少SURF算法的扫描区域,加速系统运算效率。实验中分别对场景光线强弱、视点和仿射变化以及目标被部分遮挡等不同情况进行测试,该方法均表现出较高的跟踪效率。     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Using visual and auditory cues to locate out-of-view objects in head-mounted augmented reality

When looking for an object in a complex visual scene, Augmented Reality (AR) can assist search with visual cues persistently pointing in the target’s direction. The effectiveness of these visual cues can be reduced if they are placed at a different visual depth plane to the target they are indicating. To overcome this visual-depth problem, we test the effectiveness of adding simultaneous spatialized auditory cues that are fixed at the target’s location. In an experiment we manipulated which cue(s) were available (visual-only vs. visual + auditory), and which disparity plane relative to the target the visual cue was displayed on. Results show that participants were slower at finding targets when the visual cue was placed on a different disparity plane to the target. However, this slowdown in search performance could be substantially reduced with auditory cueing. These results demonstrate the importance of AR cross-modal cueing under conditions of visual uncertainty and show that designers should consider augmenting visual cues with auditory ones.     

Visual inspection with augmented reality head-mounted display: An Australian usability case study

Augmented reality (AR) is an Industry 4.0 technology. For more than a decade, advancements in AR technology and their applications have been expected to revolutionise the manufacturing industry and deliver quality and productivity gains. However, due to factors such as equipment costs, skills shortages and technological limitations of AR devices, operational deployment beyond prototypes has been constrained. Real-world, usability studies can explore barriers to implementation and improve system design. This paper details a mixed method usability case study of an AR head-mounted display (HMD) to perform a short, simple visual inspection task. Twenty-two participants from South Australian manufacturing businesses inspected a pump and pipe skid while working at height. Overall, workload demands for the task were considered acceptable and just below the “low” workload threshold (NASA Task Load Index, mean = 29.3) and the system usability was rated “average” (system usability scale, mean = 68.5). The results suggest the task did not place too high a burden on users and was an appropriate initial exposure to AR HMDs, but further refinement to the interface would be desirable before implementation to minimise frustration and promote learning. Users were enthusiastic and open-minded about the AR HMD although results indicate that even with recent advancements in AR HMD technology, interactions between the task, technology and environment continue to cause human and technical challenges—some of which are relatively straightforward to address but others are dependent on larger-scale efforts.     

The integration of optical and magnetic tracking for multi-user augmented reality

Multi-user augmented reality systems are especially dependent on precise registration. In this paper, we present a hybrid tracking system that combines optical and magnetic tracking. The magnetic tracking is used to give a robust estimate of position and orientation which then can be refined in realtime by optical tracking. The system is more precise than a magnetic tracker and both faster and more reliable than an optical tracker.     

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.     

Real‐time recognition and tracking for augmented reality books

An augmented reality book (AR book) is an application in which such multimedia elements as virtual 3D objects, movie clips, or sound clips are augmented to a conventional book using augmented reality technology. It can provide better understanding about the contents and visual impressions for users. For AR books, this paper presents a markerless tracking method, which recognizes and tracks a large number of pages in real‐time, even on PCs with low computation power. For fast recognition with respect to a large number of pages, we propose a generic randomized forest that is an extension of a randomized forest. In addition, we define the spatial locality of the subregions in an image to resolve the problem of a dropping recognition rate under a complex background. For tracking with minimal jittering, we also propose the adaptive keyframe‐based tracking method, which automatically updates the current frame as a keyframe when it describes the page better than the existing one. Copyright © 2011 John Wiley & Sons, Ltd.     

Rapidly constructed appearance models for tracking in augmented reality applications

This paper presents a method for rapidly generating crude, appearance-based edge models consisting of a set of planes. The appearance of each plane is modeled using a set of keyframes containing a list of edgels. These models are generated from a short video sequence with a few annotated frames indicating the location of the object of interest. The data requires 3–5?min to collect using a handheld device instrumented with a camera. The models produced are used with an existing edge tracking algorithm modified to select the appropriate edge keyframe and detect occlusion. A framestore is also created containing several views of the object represented as sets of point features. The framestore is used to provide an initial, rough pose estimate for initializing contour tracking. The presented system is used to create an augmented reality application to guide a user through a machine tool setup and a printer maintenance task. The models are shown to be an accurate representation of the object. Additionally, the performance of various aspects of the model making and tracking algorithms are evaluated.     

平板图案的跟踪及其在增强现实中的应用

介绍一种实时的基于单摄像机的平板图案跟踪算法。该算法可以在增强现实应用中用于摄像机定位和虚拟物体的放置。针对跟踪中由运动模糊和目标较小所引起的跟踪失败情况做出了讨论,提出结合使用LK跟踪算法和基于匈牙利算法的角点匹配来解决这些问题。还提出使用基于约束的非线性优化来降低噪声对摄像机方位计算的影响。实验结果表明,该算法能够在复杂情况下,可靠地跟踪目标图案,获取准确平滑的摄像机定标结果。