Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Multi-sensor based real-time 6-DoF pose tracking for wearable augmented reality

Wearable augmented reality (WAR) combines a live view of a real scene with computer-generated graphic on resource-limited platforms. One of the crucial technologies for WAR is a real-time 6-DoF pose tracking, facilitating registration of virtual components within in a real scene. Generally, artificial markers are typically applied to provide pose tracking for WAR applications. However, these marker-based methods suffer from marker occlusions or large viewpoint changes. Thus, a multi-sensor based tracking approach is applied in this paper, and it can perform real-time 6-DoF pose tracking with real-time scale estimation for WAR on a consumer smartphone. By combining a wide-angle monocular camera and an inertial sensor, a more robust 6-DoF motion tracking is demonstrated with the mutual compensations of the heterogeneous sensors. Moreover, with the help of the depth sensor, the scale initialization of the monocular tracking is addressed, where the initial scale is propagated within the subsequent sensor-fusion process, alleviating the scale drift in traditional monocular tracking approaches. In addition, a sliding-window based Kalman filter framework is used to provide a low jitter pose tracking for WAR. Finally, experiments are carried out to demonstrate the feasibility and robustness of the proposed tracking method for WAR applications.     

Robust detection and tracking of annotations for outdoor augmented reality browsing

A common goal of outdoor augmented reality (AR) is the presentation of annotations that are registered to anchor points in the real world. We present an enhanced approach for registering and tracking such anchor points, which is suitable for current generation mobile phones and can also successfully deal with the wide variety of viewing conditions encountered in real life outdoor use. The approach is based on on-the-fly generation of panoramic images by sweeping the camera over the scene. The panoramas are then used for stable orientation tracking, while the user is performing only rotational movements. This basic approach is improved by several new techniques for the re-detection and tracking of anchor points. For the re-detection, specifically after temporal variations, we first compute a panoramic image with extended dynamic range, which can better represent varying illumination conditions. The panorama is then searched for known anchor points, while orientation tracking continues uninterrupted. We then use information from an internal orientation sensor to prime an active search scheme for the anchor points, which improves matching results. Finally, global consistency is enhanced by statistical estimation of a global rotation that minimizes the overall position error of anchor points when transforming them from the source panorama in which they were created, to the current view represented by a new panorama. Once the anchor points are redetected, we track the user's movement using a novel 3-degree-of-freedom orientation tracking approach that combines vision tracking with the absolute orientation from inertial and magnetic sensors. We tested our system using an AR campus guide as an example application and provide detailed results for our approach using an off-the-shelf smartphone. Results show that the re-detection rate is improved by a factor of 2 compared to previous work and reaches almost 90% for a wide variety of test cases while still keeping the ability to run at interactive frame rates.     

Augmenting reality by using uncalibrated optical tracking

The paper describes the use of an optical tracking method for superimposing computer-generated images oriented and positioned in 3D to augment a real scene. In a medical scenario, a physician may view the images created by an ultrasound device directly related to the patient. Typically, the position of the images or the position of the physician changes almost always and the mapping of images has to be recalculated. The optical tracking method in combination with artificial landmarks, is used to get the information about how to transform the superimposed images After segmenting and tracking the landmarks the perspective transformation of the ultrasonic image can be determined.     

The integration of optical and magnetic tracking for multi-user augmented reality

Multi-user augmented reality systems are especially dependent on precise registration. In this paper, we present a hybrid tracking system that combines optical and magnetic tracking. The magnetic tracking is used to give a robust estimate of position and orientation which then can be refined in realtime by optical tracking. The system is more precise than a magnetic tracker and both faster and more reliable than an optical tracker.     

增强现实中基于视觉与磁力跟踪器的三维注册方法

讨论了如何实现了一种基于三个标志点的视觉式三维注册方法，对电磁式三维注册方法进行了必要的说明，给出了实验平台，同时对实验结果进行了必要的分析说明。实验表明，此方法比较有效。     

利用增广拉格朗日乘子的鲁棒跟踪算子

针对视频目标鲁棒跟踪问题,提出了一种基于稀疏表示的生成式算法。首先提取特征构建目标和背景模板,并利用随机抽样获得足够多的候选目标状态;然后利用多任务反向稀疏表示算法得到稀疏系数矢量构造相似度测量图,这里引入了增广拉格朗日乘子(ALM)算法解决L₁-min难题;最后从相似度图中使用加性池运算提取判别信息选择与目标模板相似度最高并与背景模板相似度最小的候选目标状态作为跟踪结果,该算法是在贝叶斯滤波框架下实现的。为了适应跟踪过程中目标外观由于光照变化、遮挡、复杂背景以及运动模糊等场景引起的变化,制定了简单却有效的更新机制,对目标和背景模板进行更新。对仿真结果的定性和定量评估均表明与其他跟踪算法相比,所提算法的跟踪准确性和稳定性有了一定的提高,能有效地解决光照和尺度变化、遮挡、复杂背景等场景的跟踪难题。     

Tracking in unprepared environments for augmented reality systems

Many augmented reality applications require accurate tracking. Existing tracking techniques require prepared environments to ensure accurate results. This paper motivates the need to pursue augmented reality tracking techniques that work in unprepared environments, where users are not allowed to modify the real environment, such as in outdoor applications. Accurate tracking in such situations is difficult, requiring hybrid approaches. This paper summarizes two 3DOF results: a real-time system with a compass — inertial hybrid, and a non-real-time system fusing optical and inertial inputs. We then describe the preliminary results of 5- and 6-DOF tracking methods run in simulation. Future work and limitations are described.     

Real-time and accurate extrinsic camera parameter estimation using feature landmark database for augmented reality

In the field of augmented reality (AR), many kinds of vision-based extrinsic camera parameter estimation methods have been proposed to achieve geometric registration between real and virtual worlds. Previously, a feature landmark-based camera parameter estimation method was proposed. This is an effective method for implementing outdoor AR applications because a feature landmark database can be automatically constructed using the structure-from-motion (SfM) technique. However, the previous method cannot work in real time because it entails a high computational cost or matching landmarks in a database with image features in an input image. In addition, the accuracy of estimated camera parameters is insufficient for applications that need to overlay CG objects at a position close to the user's viewpoint. This is because it is difficult to compensate for visual pattern change of close landmarks when only the sparse depth information obtained by the SfM is available. In this paper, we achieve fast and accurate feature landmark-based camera parameter estimation by adopting the following approaches. First, the number of matching candidates is reduced to achieve fast camera parameter estimation by tentative camera parameter estimation and by assigning priorities to landmarks. Second, image templates of landmarks are adequately compensated for by considering the local 3-D structure of a landmark using the dense depth information obtained by a laser range sensor. To demonstrate the effectiveness of the proposed method, we developed some AR applications using the proposed method.     

增强现实中基于自然特征的实时跟踪方法

在增强现实系统的复杂场景中,对目标的实时跟踪受到场景中诸多因素的制约,导致实时跟踪方法效率低且不准确,为此提出一种基于自然特征的实时跟踪方法。设计了一种螺旋分割模型,对捕获的图像进行螺旋分割,利用SURF算法在分割子块中提取特征点,并进行匹配。在对目标进行跟踪定位时,利用前一帧来预测当前帧目标出现的位置,以减少SURF算法的扫描区域,加速系统运算效率。实验中分别对场景光线强弱、视点和仿射变化以及目标被部分遮挡等不同情况进行测试,该方法均表现出较高的跟踪效率。     

Real‐time recognition and tracking for augmented reality books

An augmented reality book (AR book) is an application in which such multimedia elements as virtual 3D objects, movie clips, or sound clips are augmented to a conventional book using augmented reality technology. It can provide better understanding about the contents and visual impressions for users. For AR books, this paper presents a markerless tracking method, which recognizes and tracks a large number of pages in real‐time, even on PCs with low computation power. For fast recognition with respect to a large number of pages, we propose a generic randomized forest that is an extension of a randomized forest. In addition, we define the spatial locality of the subregions in an image to resolve the problem of a dropping recognition rate under a complex background. For tracking with minimal jittering, we also propose the adaptive keyframe‐based tracking method, which automatically updates the current frame as a keyframe when it describes the page better than the existing one. Copyright © 2011 John Wiley & Sons, Ltd.     

Visual-model-based,real-time 3D pose tracking for autonomous navigation: methodology and experiments

This paper presents a novel 3D-model-based computer-vision method for tracking the full six degree-of-freedom (dof) pose (position and orientation) of a rigid body, in real-time. The methodology has been targeted for autonomous navigation tasks, such as interception of or rendezvous with mobile targets. Tracking an object’s complete six-dof pose makes the proposed algorithm useful even when targets are not restricted to planar motion (e.g., flying or rough-terrain navigation). Tracking is achieved via a combination of textured model projection and optical flow. The main contribution of our work is the novel combination of optical flow with z-buffer depth information that is produced during model projection. This allows us to achieve six-dof tracking with a single camera. A localized illumination normalization filter also has been developed in order to improve robustness to shading. Real-time operation is achieved using GPU-based filters and a new data-reduction algorithm based on colour-gradient redundancy, which was developed within the framework of our project. Colour-gradient redundancy is an important property of colour images, namely, that the gradients of all colour channels are generally aligned. Exploiting this property provides a threefold increase in speed. A processing rate of approximately 80 to 100 fps has been obtained in our work when utilizing synthetic and real target-motion sequences. Sub-pixel accuracies were obtained in tests performed under different lighting conditions.

An EyeTap video-based featureless projective motion estimation assisted by gyroscopic tracking for wearable computer mediated reality

In this paper we present a computationally economical method of recovering the projective motion of head mounted cameras or EyeTap devices, for use in wearable computer-mediated reality. The tracking system combines featureless vision and inertial methods in a closed loop system to achieve accurate robust head tracking using inexpensive sensors. The combination of inertial and vision techniques provides the high accuracy visual registration needed for fitting computer graphics onto real images and the robustness to large interframe camera motion due to fast head rotations. Operating on a 1.2 GHz Pentium III wearable computer with graphics accelerated hardware, the system is able to register live video images with less than 2 pixels of error (0.3 degrees) at 12 frames per second. Fast image registration is achieved by offloading computer vision computation onto the graphics hardware, which is readily available on many wearable computer systems. As an application of this tracking approach, we present a system which allows wearable computer users to share views of their current environments that have been stabilised to another viewer's head position.

Using augmented reality to build digital twin for reconfigurable additive manufacturing system

The extrusion-based additive manufacturing (AM) processes are those where one or multiple tools (usually nozzles) are driven along predefined paths to deposit fabrication materials. They are usually inherently slow because solid contours have to be filled with mere single deposition lines of material. An intuitive way to improve the fabrication speed is to introduce multiple independent actuators for concurrent deposition of materials without collision among them. In this paper, a methodology of using augmented reality (AR) technique is presented to conveniently communicate the layout information between a reconfigurable AM system made of robotic arms and its corresponding digital twin for toolpath planning and simulation. A prototype system made of two desktops AM robotic arms is developed, and transformation matrices are derived to determine the spatial relation between different items in the system, including camera, markers, robotic arms and part substrate. Case studies are conducted to demonstrate the capability of this methodology in automatically retrieving layout information and assisting users to deploy pre-determined layout. The results show that the developed methodology enables rapid retrieval of position information from the physical system layout into the digital twin simulation and optimization and facilitates convenient deployment of an optimized layout determined in the digital twin into the physical system.     

增强现实游戏的场景重建和运动物体跟踪技术

针对增强现实游戏应用,介绍了一种基于自然特征的实时跟踪算法,它可以在一个未知但纹理丰富的环境中,不需要任何预先的学习,对相机和若干出现在相机视野里的刚体进行并行跟踪,获得相机和各个刚体的姿态和运动信息,可以在刚体移出相机视野以及相机剧烈运动导致跟踪丢失的情况下进行自动恢复。基于静态环境中并行相机跟踪和三维重建的算法框架进行扩展和改进,引入了用于交互的运动刚体。结果表明,利用现有的多核处理技术,在含有运动物体的环境中进行相机和运动物体的并行跟踪是可行的,为增强现实环境中同时支持显示和交互提出了廉价易行的解决方案。     

A virtual try-on system in augmented reality using RGB-D cameras for footwear personalization

This paper presents a system for design evaluation of footwear using commercial depth-sensing technologies. In a mixed reality environment, the system allows users to virtually try on 3D shoe models in a live video stream. A two-stage object tracking algorithm was developed to correctly align shoe models to moving feet during the try-on process. Color markers on the user's foot enabled markerless tracking. Tracking was driven by an iterative closest point (ICP) algorithm that superimposed the captured depth data and predefined reference foot models. Test data showed that the two-stage approach resulted in increased positional accuracy compared with tracking using only surface registration. Trimming the reference model using the instant view angle increased the computational efficiency of the ICP algorithm. The proposed virtual try-on function is an effective tool for realizing human-centered design. This study also demonstrated a new application of RGB-D cameras to product design.     

Robot programming using augmented reality: An interactive method for planning collision-free paths

Current robot programming approaches lack the intuitiveness required for quick and simple applications. As new robotic applications are being identified, there is a greater need to be able to programme robots safely and quickly. This paper discusses the use of an augmented reality (AR) environment for facilitating intuitive robot programming, and presents a novel methodology for planning collision-free paths for an n-d.o.f. (degree-of-freedom) manipulator in a 3D AR environment. The methodology is interactive because the human is involved in defining the free space or collision-free volume (CFV), and selecting the start and goal configurations. The methodology uses a heuristic beam search algorithm to generate the paths. A number of possible scenarios are discussed.     

Registration of technical drawings and calibrated images for industrial augmented reality

Despite tremendous progress in 3D modelling technology, most sites in traditional industries do not have a computer model of their facilities at their disposal. In these industries, 2D technical drawings are typically the most commonly used documents. In many cases, a database of fully calibrated and oriented photogrammetric images of parts of the plant is also available. These images are often used for metric measurement and 3D as-built modelling. For planning revamps and maintenance, it is necessary to use industrial drawings as well as images and 3D models represented in a common “world” coordinate system. This paper proposes a method for full integration of technical drawings, calibrated images and as-built 3D models. A new algorithm is developed in order to use only a few correspondences between points on a technical drawing and multiple images to estimate a metric planar transformation between the drawing and the world coordinate system. The paper describes the mathematical relationship between this transformation and the set of homographies needed for merging the technical drawing with all the calibrated images. The method is implemented and fully integrated into an industrial software we developed for 3D as-built reconstruction. We present examples of a real application, in which the method is successfully applied to create an augmented reality representation of a waste water plant. Accepted: 13 August 2001     

结构光视觉引导的轨迹跟踪系统的标定技术

为实现机械臂对工业现场三维特征点的跟踪,提出一种基于结构光视觉,六轴机械臂的整体标定方法。利用自制平面靶标求取单映射矩阵,根据张正友标定方法标定相机内参以及对应外参,利用LM算法优化参数,根据激光光条在靶标平面的参数方程和对应的外参标定出激光光平面在相机坐标系下的平面方程,通过求解[AX=XB]方程得到相机坐标系与机械臂末端坐标系的齐次变换矩阵[X],并对特定位姿下的工件偏移进行修正。实际测试表明：静态跟踪误差在±1.2 mm,满足工业现场应用要求。