Binocular mobile augmented reality based on stereo camera tracking

Mobile augmented reality (AR) applications have become feasible with the evolution of mobile hardware. For example, the advent of the smartphone allowed implementing real-time mobile AR, which triggered the release of various applications. Recently, rapid development of display technology, especially for stereoscopic displays, has encouraged researches to implement more immersive and realistic AR. In this paper, we present a framework of binocular augmented reality based on stereo camera tracking. Our framework was implemented on a smartphone and supports autostereoscopic display and video see-through display in which a smartphone can be docked. We modified edge-based 3-D object tracking in order to estimate poses of left and right cameras jointly; this guarantees consistent registration across left and right views. Then, virtual contents were overlaid onto camera images using estimated poses, and the augmented stereo images were distorted to be shown through a video see-through display. The feasibility of the proposed framework is shown by experiments and demonstrations.     

Robust and real-time pose tracking for augmented reality on mobile devices

This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.     

Gesture-based interactive augmented reality content authoring system using HMD

This paper proposes an augmented reality content authoring system that enables ordinary users who do not have programming capabilities to easily apply interactive features to virtual objects on a marker via gestures. The purpose of this system is to simplify augmented reality (AR) technology usage for ordinary users, especially parents and preschool children who are unfamiliar with AR technology. The system provides an immersive AR environment with a head-mounted display and recognizes users’ gestures via an RGB-D camera. Users can freely create the AR content that they will be using without any special programming ability simply by connecting virtual objects stored in a database to the system. Following recognition of the marker via the system’s RGB-D camera worn by the user, he/she can apply various interactive features to the marker-based AR content using simple gestures. Interactive features applied to AR content can enlarge, shrink, rotate, and transfer virtual objects with hand gestures. In addition to this gesture-interactive feature, the proposed system also allows for tangible interaction using markers. The AR content that the user edits is stored in a database, and is retrieved whenever the markers are recognized. The results of comparative experiments conducted indicate that the proposed system is easier to use and has a higher interaction satisfaction level than AR environments such as fixed-monitor and touch-based interaction on mobile screens.     

Natural feature tracking for augmented reality

Natural scene features stabilize and extend the tracking range of augmented reality (AR) pose-tracking systems. We develop robust computer vision methods to detect and track natural features in video images. Point and region features are automatically and adaptively selected for properties that lead to robust tracking. A multistage tracking algorithm produces accurate motion estimates, and the entire system operates in a closed-loop that stabilizes its performance and accuracy. We present demonstrations of the benefits of using tracked natural features for AR applications that illustrate direct scene annotation, pose stabilization, and extendible tracking range. Our system represents a step toward integrating vision with graphics to produce robust wide-area augmented realities     

Information filtering for mobile augmented reality

The use of augmented reality (AR) techniques can revolutionize the way people interact with unfamiliar environments. By tracking the user's position and orientation, complicated spatial information can be registered against the real world. My colleagues and I are researching the problem of developing mobile AR systems to be worn by individual users operating in large, complicated environments such as cities. However, an urban environment is extremely complicated. It is populated by large numbers of buildings, each of which can have numerous facts stored about it. Therefore, it is easy for a user to experience information overload. This problem is illustrated. To minimize problems of information overload, we have begun to develop algorithms for information filtering. These tools automatically restrict the amount of information displayed     

Accurate overlaying for mobile augmented reality

Mobile augmented reality requires accurate alignment of virtual information with objects visible in the real world. We describe a system for mobile communications to be developed to meet these strict alignment criteria using a combination of computer vision, inertial tracking and low-latency rendering techniques. A prototype low-power and low-latency renderer using an off-the-shelf 3D card is discussed.     

Webizing mobile augmented reality content

This paper presents a content structure for building mobile augmented reality (AR) applications in HTML5 to achieve a clean separation of the mobile AR content and the application logic for scaling as on the Web. We propose that the content structure contains the physical world as well as virtual assets for mobile AR applications as document object model (DOM) elements and that their behaviour and user interactions are controlled through DOM events by representing objects and places with a uniform resource identifier. Our content structure enables mobile AR applications to be seamlessly developed as normal HTML documents under the current Web eco-system.     

移动增强现实框架系统

在已有的移动增强现实( MAR)系统中存在开发困难,虚拟现实( AR)效果观察缺乏通用性的问题,提出了一种MAR系统的通用框架,该框架包括三个部分：为智能移动设备用户所使用的通用MAR观察器、为MAR应用提供网络服务的MAR服务器,以及为开发人员提供了MAR应用定制器。该框架结合高通的Vuforia SDK针对平面图像实现了MAR的应用定制和通用AR效果浏览。框架为开发人员提供了无代码开发功能,为移动用户提供了遍览一类AR应用的通用观察器。实验结果表明该MAR框架系统降低了MAR应用的开发门槛与时间费用,大大提高了开发效率。     

Orientation tracking for outdoor augmented reality registration

Our work stems from a program focused on developing tracking technologies for wide-area augmented realities in unprepared outdoor environments. Other participants in the Defense Advanced Research Projects Agency (Darpa) funded Geospatial Registration of Information for Dismounted Soldiers (Grids) program included University of North Carolina at Chapel Hill and Raytheon. We describe a hybrid orientation tracking system combining inertial sensors and computer vision. We exploit the complementary nature of these two sensing technologies to compensate for their respective weaknesses. Our multiple-sensor fusion is novel in augmented reality tracking systems, and the results demonstrate its utility     

Hybrid tracking for outdoor augmented reality applications

We've developed a fully mobile, wearable AR system that combines a vision-based tracker (primarily software algorithms) that uses natural landmarks, with an inertial tracker (custom hardware and firmware) based on silicon micromachined accelerometers and gyroscopes. Unlike other vision-based and hybrid systems, both components recover the full 6 DOF pose. Fusing the two tracking subsystems gives us the benefits of both technologies, while the sensors' complementary nature helps overcome sensor-specific deficiencies. Our system is tailored to affordable, lightweight, energy-efficient mobile AR applications for urban environments, especially the historic centers of European cities.     

Robust tracking for augmented reality using retroreflective markers

In this paper, an optical tracking system is introduced for the use in augmented reality (AR). Only a few AR-systems currently exist which are using stereo-vision techniques to estimate the viewing pose. The presented system uses binocular images and retroreflective markers in order to speed up the tracking process and make it more precise and robust. The camera calibration as well as the pose estimation technique is presented. This new optical tracking system, which is based on standard PC hardware, is even suitable to make it portable. In addition, the system is evaluated with regard to its pixel accuracy and depth measurements. This paper shows that the computer vision techniques, which will be presented, are a good choice in order to create a flexible, accurate and easy to use tracking system.     

Real-time optical markerless tracking for augmented reality applications

Augmented reality (AR) technology consists in adding computer-generated information (2D/3D) to a real video sequence in such a manner that the real and virtual objects appear coexisting in the same world. To get a realistic illusion, the real and virtual objects must be properly aligned with respect to each other, which requires a robust real-time tracking strategy—one of the bottlenecks of AR applications. In this paper, we describe the limitations and advantages of different optical tracking technologies, and we present our customized implementation of both recursive tracking and tracking by detection approaches. The second approach requires the implementation of a classifier and we propose the use of a Random Forest classifier. We evaluated both approaches in the context of an AR application for design review. Some conclusions regarding the performance of each approach are given.     

The effect of tracking technique on the quality of user experience for augmented reality mobile navigation

Augmented reality (AR) technology is changing the way we interact with the world by making the journey seamless and interactive. This is done by layering digital enhancements over an existing reality or real life scenario. However, employing AR technologies in wayfinding and navigation does not automatically bring positive experiences. We argue that tracking techniques are important in mobile AR navigation and fundamentally affect the quality of user experience. In this paper, we propose two different tourist AR applications based on different tracking techniques. In addition to the analysis of the applications, we conducted a user evaluation to study the effect of different AR tracking techniques on the performance of users and the quality of user experience.     

增强现实中基于自然特征的实时跟踪方法

在增强现实系统的复杂场景中,对目标的实时跟踪受到场景中诸多因素的制约,导致实时跟踪方法效率低且不准确,为此提出一种基于自然特征的实时跟踪方法。设计了一种螺旋分割模型,对捕获的图像进行螺旋分割,利用SURF算法在分割子块中提取特征点,并进行匹配。在对目标进行跟踪定位时,利用前一帧来预测当前帧目标出现的位置,以减少SURF算法的扫描区域,加速系统运算效率。实验中分别对场景光线强弱、视点和仿射变化以及目标被部分遮挡等不同情况进行测试,该方法均表现出较高的跟踪效率。     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Speech-enabled augmented reality supporting mobile industrial maintenance

The SEAR (speech-enabled AR) framework uses flexible and scalable vision-based localization techniques to offer maintenance technicians a seamless multimodal user interface. The user interface juxtaposes a graphical AR view with a context-sensitive speech dialogue.     

Markerless tracking system for augmented reality in the automotive industry

This paper presents a complete natural feature based tracking system that supports the creation of augmented reality applications focused on the automotive sector. The proposed pipeline encompasses scene modeling, system calibration and tracking steps. An augmented reality application was built on top of the system for indicating the location of 3D coordinates in a given environment which can be applied to many different applications in cars, such as a maintenance assistant, an intelligent manual, and many others. An analysis of the system was performed during the Volkswagen/ISMAR Tracking Challenge 2014, which aimed to evaluate state-of-the-art tracking approaches on the basis of requirements encountered in automotive industrial settings. A similar competition environment was also created by the authors in order to allow further studies. Evaluation results showed that the system allowed users to correctly identify points in tasks that involved tracking a rotating vehicle, tracking data on a complete vehicle and tracking with high accuracy. This evaluation allowed also to understand the applicability limits of texture based approaches in the textureless automotive environment, a problem not addressed frequently in the literature. To the best of the authors’ knowledge, this is the first work addressing the analysis of a complete tracking system for augmented reality focused on the automotive sector which could be tested and validated in a major benchmark like the Volkswagen/ISMAR Tracking Challenge, providing useful insights on the development of such expert and intelligent systems.     

面向移动增强现实的室外阴影实时检测技术

针对目前移动视点下视频阴影检测算法存在的误检测率高和边缘连续性差的问题,提出了一种基于边跟踪、边检测框架的实时阴影检测算法.首先对前后2帧重叠的阴影部分进行2次光流跟踪,并筛选掉前后向跟踪误差较大的点,通过Canny边缘置信保证跟踪边缘的准确性;然后通过基于光流的区域划分法得到待检测的新增区域;其次,针对纹理边缘误检测...