Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

Hybrid tracking for outdoor augmented reality applications

We've developed a fully mobile, wearable AR system that combines a vision-based tracker (primarily software algorithms) that uses natural landmarks, with an inertial tracker (custom hardware and firmware) based on silicon micromachined accelerometers and gyroscopes. Unlike other vision-based and hybrid systems, both components recover the full 6 DOF pose. Fusing the two tracking subsystems gives us the benefits of both technologies, while the sensors' complementary nature helps overcome sensor-specific deficiencies. Our system is tailored to affordable, lightweight, energy-efficient mobile AR applications for urban environments, especially the historic centers of European cities.     

Editorial for the special issue on markerless real-time tracking for augmented reality image synthesis

Augmented reality is a growing field, with many diverse applications ranging from TV and film production, to industrial maintenance, medicine, education, entertainment and games. The central idea is to add virtual objects into a real scene, either by displaying them in a see-through head-mounted display, or by superimposing them on an image of the scene captured by a camera. Depending on the application, the added objects might be virtual characters in a TV or film production, instructions for repairing a car engine, or a reconstruction of an archaeological site. For the effect to be believable, the virtual objects must appear rigidly fixed to the real world, which requires the accurate measurement in real-time of the position of the camera or the user’s head. Present technology cannot achieve this without resorting to systems that require a significant infrastructure in the operating environment, severely restricting the range of possible applications.     

Rapidly constructed appearance models for tracking in augmented reality applications

This paper presents a method for rapidly generating crude, appearance-based edge models consisting of a set of planes. The appearance of each plane is modeled using a set of keyframes containing a list of edgels. These models are generated from a short video sequence with a few annotated frames indicating the location of the object of interest. The data requires 3–5?min to collect using a handheld device instrumented with a camera. The models produced are used with an existing edge tracking algorithm modified to select the appropriate edge keyframe and detect occlusion. A framestore is also created containing several views of the object represented as sets of point features. The framestore is used to provide an initial, rough pose estimate for initializing contour tracking. The presented system is used to create an augmented reality application to guide a user through a machine tool setup and a printer maintenance task. The models are shown to be an accurate representation of the object. Additionally, the performance of various aspects of the model making and tracking algorithms are evaluated.     

The integration of optical and magnetic tracking for multi-user augmented reality

Multi-user augmented reality systems are especially dependent on precise registration. In this paper, we present a hybrid tracking system that combines optical and magnetic tracking. The magnetic tracking is used to give a robust estimate of position and orientation which then can be refined in realtime by optical tracking. The system is more precise than a magnetic tracker and both faster and more reliable than an optical tracker.     

Real-time coherent stylization for augmented reality

This paper presents a new stylized augmented reality (AR) framework which can generate line drawing and abstracted shading styles. In comparison with the state-of-art work, our framework can significantly improve both the visual immersion of a single frame and the temporal coherence of augmented video streams in real time. In our framework, we first render virtual objects over the input camera images and then uniformly process the combined contents with stylization techniques. For generating line drawing stylization, we first propose a specially designed shading method to render the virtual objects, and then use an adapted Flow-based anisotropic Difference-of-Gaussion (FDoG) filter to yield the high-quality line drawing effect. For generating the abstracted stylization, a focus-guided diffusion filter and a soft color quantization operator are sequentially applied to the augmented image, and then the processed result is combined with the detected edges to produce the final abstraction effect. The presented algorithms are all sympathetic to highly parallel processing, allowing a real-time performance on contemporary graphics hardware.     

Natural feature tracking for augmented reality

Natural scene features stabilize and extend the tracking range of augmented reality (AR) pose-tracking systems. We develop robust computer vision methods to detect and track natural features in video images. Point and region features are automatically and adaptively selected for properties that lead to robust tracking. A multistage tracking algorithm produces accurate motion estimates, and the entire system operates in a closed-loop that stabilizes its performance and accuracy. We present demonstrations of the benefits of using tracked natural features for AR applications that illustrate direct scene annotation, pose stabilization, and extendible tracking range. Our system represents a step toward integrating vision with graphics to produce robust wide-area augmented realities     

Orientation tracking for outdoor augmented reality registration

Our work stems from a program focused on developing tracking technologies for wide-area augmented realities in unprepared outdoor environments. Other participants in the Defense Advanced Research Projects Agency (Darpa) funded Geospatial Registration of Information for Dismounted Soldiers (Grids) program included University of North Carolina at Chapel Hill and Raytheon. We describe a hybrid orientation tracking system combining inertial sensors and computer vision. We exploit the complementary nature of these two sensing technologies to compensate for their respective weaknesses. Our multiple-sensor fusion is novel in augmented reality tracking systems, and the results demonstrate its utility     

Real-time infinite horizon tracking with data fusion for augmented reality in a maritime operations context

In this paper, we propose a method for real-time horizon tracking (i.e., separation line between the sky and the sea) in a maritime operations context. We present the fusion of an image processing algorithm with the data obtained from the inertial measurement unit (IMU). The initial aim is to filter out environmental conditions using inertial information in order to combine a video stream with onboard electronic charts. This is achieved by the detection of the horizon with an image processing algorithm in an area defined by the IMU. We then present an evaluation of the algorithm with regard to the rate of detection of the horizon and the impact of the image resolution on the computational time. The purpose of developing this method is to create an augmented reality maritime operations application. We combine the video stream with electronic charts in a single display. We use the position of the horizon in the image to split the display into different areas. Then, we use transparency to display the video, the electronic charts or both.     

基于ORB自然特征的AR实时系统实现

针对当前基于自然特征的增强现实效率低,提出一种新的基于ORB自然特征的实时注册方法。提取视频帧与基准图像中的ORB特征点,使用汉明距离匹配,利用RANSAC算法筛选得到最佳匹配点对,确定摄像机的位姿。将三维虚拟物体叠加到真实场景中,达到虚实结合的效果。实验表明,在不同尺度角度、一定环境光变化、复杂背景和基准图像部分遮挡的情况下,该AR系统都具有良好的性能,跟踪定位准确度高,速度基本达到实时要求。     

Robust tracking for augmented reality using retroreflective markers

In this paper, an optical tracking system is introduced for the use in augmented reality (AR). Only a few AR-systems currently exist which are using stereo-vision techniques to estimate the viewing pose. The presented system uses binocular images and retroreflective markers in order to speed up the tracking process and make it more precise and robust. The camera calibration as well as the pose estimation technique is presented. This new optical tracking system, which is based on standard PC hardware, is even suitable to make it portable. In addition, the system is evaluated with regard to its pixel accuracy and depth measurements. This paper shows that the computer vision techniques, which will be presented, are a good choice in order to create a flexible, accurate and easy to use tracking system.     

Human-centric design personalization of 3D glasses frame in markerless augmented reality

This paper presents a virtual try-on system based on augmented reality for design personalization of facial accessory products. The system offers several novel functions that support real-time evaluation and modification of eyeglasses frame. 3D glasses model is embedded within video stream of the person who is wearing the glasses. Machine learning algorithms are developed for instantaneous tracking of facial features without use of markers. The tracking result enables continuously positioning of the glasses model on the user’s face while it is moving during the try-on process. In addition to color and texture, the user can instantly modify the glasses shape through simple semantic parameters. These functions not only facilitate evaluating products highly interactive with human users, but also engage them in the design process. This work has thus implemented the concept of human-centric design personalization.     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Gesture-based interaction via finger tracking for mobile augmented reality

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.     

增强现实中基于自然特征的实时跟踪方法

在增强现实系统的复杂场景中,对目标的实时跟踪受到场景中诸多因素的制约,导致实时跟踪方法效率低且不准确,为此提出一种基于自然特征的实时跟踪方法。设计了一种螺旋分割模型,对捕获的图像进行螺旋分割,利用SURF算法在分割子块中提取特征点,并进行匹配。在对目标进行跟踪定位时,利用前一帧来预测当前帧目标出现的位置,以减少SURF算法的扫描区域,加速系统运算效率。实验中分别对场景光线强弱、视点和仿射变化以及目标被部分遮挡等不同情况进行测试,该方法均表现出较高的跟踪效率。     

Markerless tracking system for augmented reality in the automotive industry

This paper presents a complete natural feature based tracking system that supports the creation of augmented reality applications focused on the automotive sector. The proposed pipeline encompasses scene modeling, system calibration and tracking steps. An augmented reality application was built on top of the system for indicating the location of 3D coordinates in a given environment which can be applied to many different applications in cars, such as a maintenance assistant, an intelligent manual, and many others. An analysis of the system was performed during the Volkswagen/ISMAR Tracking Challenge 2014, which aimed to evaluate state-of-the-art tracking approaches on the basis of requirements encountered in automotive industrial settings. A similar competition environment was also created by the authors in order to allow further studies. Evaluation results showed that the system allowed users to correctly identify points in tasks that involved tracking a rotating vehicle, tracking data on a complete vehicle and tracking with high accuracy. This evaluation allowed also to understand the applicability limits of texture based approaches in the textureless automotive environment, a problem not addressed frequently in the literature. To the best of the authors’ knowledge, this is the first work addressing the analysis of a complete tracking system for augmented reality focused on the automotive sector which could be tested and validated in a major benchmark like the Volkswagen/ISMAR Tracking Challenge, providing useful insights on the development of such expert and intelligent systems.     

Real-time finite element structural analysis in augmented reality

Conventional finite element analysis (FEA) is usually carried out in offsite and virtual environments, i.e., computer-generated graphics, which does not promote a user’s perception and interaction, and limits its applications. With the purpose of enhancing structural analysis with augmented reality (AR) technologies, the paper presents a system which integrates sensor measurement and real-time FEA simulation into an AR-based environment. By incorporating scientific visualization technologies, this system superimposes FEA results directly on real-world objects, and provides intuitive interfaces for enhanced data exploration. A wireless sensor network has been integrated into the system to acquire spatially distributed loads, and a method to register the sensors onsite has been developed. Real-time FEA methods are employed to generate fast solutions in response to load variations. As a case study, this system is applied to monitor the stresses of a step ladder under actual loading conditions. The relationships among accuracy, mesh resolution and frame rate are investigated.     

基于增强现实的自适应跟踪注册方法

针对增强现实跟踪注册过程中遮挡及尺度变换的问题,提出一种基于Camshift和ORB算法的自适应跟踪注册方法.将高斯背景建模和Camshift算法融入到跟踪过程中,实现对目标尺度变化的自适应跟踪;在特征匹配中,将图像与Gauss函数进行卷积和差分运算,得到稳定的尺度空间,使ORB算法提取的特征点具有尺度不变性.实验结果...     

Robust camera pose tracking for augmented reality using particle filtering framework

In this paper, we present new solutions for the problem of estimating the camera pose using particle filtering framework. The proposed approach is suitable for real-time augmented reality (AR) applications in which the camera is held by the user. This work demonstrates that particle filtering improve the robustness of the tracking comparing to existing approaches, such as those based on the Kalman filter. We propose a tracking framework for both points and lines features, the particle filter is used to compute the posterior density for the camera 3D motion parameters. We also analyze the sensitivity of our technique when outliers are present in the match data. Outliers arise frequently due to incorrect correspondences which occur because of either image noise or occlusion. Results from real data in an augmented reality setup are then presented, demonstrating the efficiency and robustness of the proposed method.