Real‐time generation of augmented video sequences by background tracking

We present a real‐time system that outputs a multi‐layer augmented video from a normal video stream. The input sequence is captured in a known environment: a panoramic image (the reference background) is provided before shooting. We use background tracking to register each input frame with this reference, and background subtraction to segment the foreground objects. Our background tracking method is a coarse‐to‐fine integration of three state‐of‐the‐art algorithms that we make resistant to occlusions. In particular, we introduce an algebraic technique to properly adapt the Jurie and Dhome (JD) tracker (the most robust of them). We report experimental results on real and synthetic data which validate our approach. We generate our augmented video sequences by compositing layers: a natural or synthetic background and several natural or synthetic foregrounds. Copyright © 2006 John Wiley & Sons, Ltd.     

增强现实中基于自然特征的实时跟踪方法

在增强现实系统的复杂场景中,对目标的实时跟踪受到场景中诸多因素的制约,导致实时跟踪方法效率低且不准确,为此提出一种基于自然特征的实时跟踪方法。设计了一种螺旋分割模型,对捕获的图像进行螺旋分割,利用SURF算法在分割子块中提取特征点,并进行匹配。在对目标进行跟踪定位时,利用前一帧来预测当前帧目标出现的位置,以减少SURF算法的扫描区域,加速系统运算效率。实验中分别对场景光线强弱、视点和仿射变化以及目标被部分遮挡等不同情况进行测试,该方法均表现出较高的跟踪效率。     

The integration of optical and magnetic tracking for multi-user augmented reality

Multi-user augmented reality systems are especially dependent on precise registration. In this paper, we present a hybrid tracking system that combines optical and magnetic tracking. The magnetic tracking is used to give a robust estimate of position and orientation which then can be refined in realtime by optical tracking. The system is more precise than a magnetic tracker and both faster and more reliable than an optical tracker.     

结合图像识别的移动增强现实系统设计与应用

目的 在移动互联网时代下,移动增强现实应用得到越来越快的发展。然而户外场景中存在许多相似结构的建筑,且手机的存储和计算能力有限,因此应用多集中于室内小范围环境,对于室外大规模复杂场景的适应性较弱。对此,建立一套基于云端图像识别的移动增强现实系统。方法 为解决相似特征的误匹配问题,算法中将重力信息加入到SURF和BRISK特征描述中去,构建Gravity-SURF和Gravity-BRISK特征描述。云端系统对增强信息进行有效管理,采用基于Gravity-SURF特征的VLAD方法对大规模图像进行识别;在智能终端上的应用中呈现识别图像的增强信息,并利用识别图像的Gravity-BRISK特征和光流结合的方法对相机进行跟踪,采用Unity3D渲染引擎实时绘制3维模型。结果 在包含重力信息的4 000幅户外图像的数据库中进行实验。采用结合重力信息的特征描述算法,能够增强具有相似特征的描述符的区分性,并提高匹配正确率。图像识别算法的识别率能达到88%以上,识别时间在420 ms左右;光流跟踪的RMS误差小于1.2像素,帧率能达到23 帧/s。结论 本文针对室外大规模复杂场景建立的基于图像识别的移动增强现实系统,能方便对不同应用的增强现实数据进行管理。系统被应用到谷歌眼镜和新闻领域上,不局限于单一的应用领域。结果表明,识别算法和跟踪注册算法能够满足系统的精度和实时性要求。     

增强现实中基于视觉与磁力跟踪器的三维注册方法

讨论了如何实现了一种基于三个标志点的视觉式三维注册方法，对电磁式三维注册方法进行了必要的说明，给出了实验平台，同时对实验结果进行了必要的分析说明。实验表明，此方法比较有效。     

平板图案的跟踪及其在增强现实中的应用

介绍一种实时的基于单摄像机的平板图案跟踪算法。该算法可以在增强现实应用中用于摄像机定位和虚拟物体的放置。针对跟踪中由运动模糊和目标较小所引起的跟踪失败情况做出了讨论,提出结合使用LK跟踪算法和基于匈牙利算法的角点匹配来解决这些问题。还提出使用基于约束的非线性优化来降低噪声对摄像机方位计算的影响。实验结果表明,该算法能够在复杂情况下,可靠地跟踪目标图案,获取准确平滑的摄像机定标结果。     

Multi-sensor based real-time 6-DoF pose tracking for wearable augmented reality

Wearable augmented reality (WAR) combines a live view of a real scene with computer-generated graphic on resource-limited platforms. One of the crucial technologies for WAR is a real-time 6-DoF pose tracking, facilitating registration of virtual components within in a real scene. Generally, artificial markers are typically applied to provide pose tracking for WAR applications. However, these marker-based methods suffer from marker occlusions or large viewpoint changes. Thus, a multi-sensor based tracking approach is applied in this paper, and it can perform real-time 6-DoF pose tracking with real-time scale estimation for WAR on a consumer smartphone. By combining a wide-angle monocular camera and an inertial sensor, a more robust 6-DoF motion tracking is demonstrated with the mutual compensations of the heterogeneous sensors. Moreover, with the help of the depth sensor, the scale initialization of the monocular tracking is addressed, where the initial scale is propagated within the subsequent sensor-fusion process, alleviating the scale drift in traditional monocular tracking approaches. In addition, a sliding-window based Kalman filter framework is used to provide a low jitter pose tracking for WAR. Finally, experiments are carried out to demonstrate the feasibility and robustness of the proposed tracking method for WAR applications.     

增强现实游戏的场景重建和运动物体跟踪技术

针对增强现实游戏应用,介绍了一种基于自然特征的实时跟踪算法,它可以在一个未知但纹理丰富的环境中,不需要任何预先的学习,对相机和若干出现在相机视野里的刚体进行并行跟踪,获得相机和各个刚体的姿态和运动信息,可以在刚体移出相机视野以及相机剧烈运动导致跟踪丢失的情况下进行自动恢复。基于静态环境中并行相机跟踪和三维重建的算法框架进行扩展和改进,引入了用于交互的运动刚体。结果表明,利用现有的多核处理技术,在含有运动物体的环境中进行相机和运动物体的并行跟踪是可行的,为增强现实环境中同时支持显示和交互提出了廉价易行的解决方案。     

增强现实系统中的抛掷动作实时识别

在增强现实系统中,以抛掷作为主要交互手段,必须要具备高效的动作识别和响应能力。从抛掷动作的特征入手,着重分析了它的速度变化规律。依据此规律,设计了滑动平均结合模糊综合评判的算法,实现了对抛掷动作的准确判断,和虚拟物体的起抛速度的精确计算。实验结果表明,系统不受抛掷速度和种类的限制,有很好的实时性。     

An augmented reality system to guide radio‐frequency tumour ablation

Radio‐frequency ablation is a difficult operative task that requires a precise needle positioning in the centre of the pathology. This article presents an augmented reality system for hepatic therapy guidance that superimposes in real‐time 3D reconstructions (from CT acquisition) and a virtual model of the needle on external views of a patient. The superimposition of reconstructed models is performed with a 3D/2D registration based on radio‐opaque markers stuck on to the patient's skin. The characteristics of the problem (accuracy, robustness and time processing) led us to develop automatic procedures to extract and match the markers and to track the needle in real time. Experimental studies confirmed that our algorithms are robust and reliable. Preliminary experiments conducted on a human abdomen phantom showed that our system is highly accurate (needle positioning error within 3 mm) and enables the surgeon to reach a target in less than 1 minute on average. Our next step will be to perform an in vivo evaluation. Copyright © 2005 John Wiley & Sons, Ltd.     

面向移动增强现实的室外阴影实时检测技术

针对目前移动视点下视频阴影检测算法存在的误检测率高和边缘连续性差的问题,提出了一种基于边跟踪、边检测框架的实时阴影检测算法.首先对前后2帧重叠的阴影部分进行2次光流跟踪,并筛选掉前后向跟踪误差较大的点,通过Canny边缘置信保证跟踪边缘的准确性;然后通过基于光流的区域划分法得到待检测的新增区域;其次,针对纹理边缘误检测...     

Nonlinear control with integral sliding properties for circular aerial robot trajectory tracking: Real‐time validation

A nonlinear control algorithm for tracking dynamic trajectories using an aerial vehicle is developed in this work. The control structure is designed using a sliding mode methodology, which contains integral sliding properties. The stability analysis of the closed‐loop system is proved using the Lyapunov formalism, ensuring convergence in a desired finite time and robustness toward unknown and external perturbations from the first time instant, even for high frequency disturbances. In addition, a dynamic trajectory is constructed with the translational dynamics of an aerial robot for autonomous take‐off, surveillance missions, and landing. This trajectory respects the constraints imposed by the vehicle characteristics, allowing free initial trajectory conditions. Simulation results demonstrate the good performance of the controller in closed‐loop system when a quadrotor follows the designed trajectory. In addition, flight tests are developed to validate the trajectory and the controller behavior in real time.     

Real‐time generation of smoothed‐particle hydrodynamics‐based special effects in character animation

In the previous works, the real‐time fluid‐character animation could hardly be achieved because of the intensive processing demand on the character's movement and fluid simulation. This paper presents an effective approach to the real‐time generation of the fluid flow driven by the motion of a character in full 3D space, based on smoothed‐particle hydrodynamics method. The novel method of conducting and constraining the fluid particles by the geometric properties of the character motion trajectory is introduced. Furthermore, the optimized algorithms of particle searching and rendering are proposed, by taking advantage of the graphics processing unit parallelization. Consequently, both simulation and rendering of the 3D liquid effects with realistic character interactions can be implemented by our framework and performed in real‐time on a conventional PC. Copyright © 2013 John Wiley & Sons, Ltd.     

A plastic holographic waveguide combiner for light‐weight and highly‐transparent augmented reality glasses

There is a high demand for light‐weight, stylishly designed augmented reality (AR) glasses with natural see‐through capabilities for the wide‐spread distribution of novel wearable device to general consumers. We have successfully developed a unique production process of a holographic waveguide combiner that enables us to laminate holographic optical elements (HOEs) onto a plastic substrate with optical grade quality. The plastic substrate waveguide combiner has a number of advantages over conventional glass substrate combiners; the plastic substrate makes AR glasses lighter in weight and unbreakable. With the lamination process of HOEs, we can apply them to a various designs to satisfy general customers' wide range of preferences for the style. We also potentially made it possible for the holographic waveguide combiner to be produced in larger volumes at lower costs by using our novel roll‐to‐roll hologram recording and laminating process. In this paper, we present our approach of the plastic substrate HOE production process for AR glasses.     

Handling occlusions in video‐based augmented reality using depth information

Augmented Reality (AR) composes virtual objects with real scenes in a mixed environment where human–computer interaction has more semantic meanings. To seamlessly merge virtual objects with real scenes, correct occlusion handling is a significant challenge. We present an approach to separate occluded objects in multiple layers by utilizing depth, color, and neighborhood information. Scene depth is obtained by stereo cameras and two Gaussian local kernels are used to represent color, spatial smoothness. These three cues are intelligently fused in a probability framework, where the occlusion information can be safely estimated. We apply our method to handle occlusions in video‐based AR where virtual objects are simply overlapped on real scenes. Experiment results show the approach can correctly register virtual and real objects in different depth layers, and provide a spatial‐awareness interaction environment. Copyright © 2009 John Wiley & Sons, Ltd.     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

A survey of mobile and wireless technologies for augmented reality systems

Recent advances in hardware and software for mobile computing have enabled a new breed of mobile augmented reality (AR) systems and applications. A new breed of computing called ‘augmented ubiquitous computing’ has resulted from the convergence of wearable computing, wireless networking, and mobile AR interfaces. In this paper, we provide a survey of different mobile and wireless technologies and how they have impact AR. Our goal is to place them into different categories so that it becomes easier to understand the state of art and to help identify new directions of research. Copyright © 2008 John Wiley & Sons, Ltd.     

A multi‐plane optical see‐through head mounted display design for augmented reality applications

Augmented reality (AR) display technology greatly enhances users' perception of and interaction with the real world by superimposing a computer‐generated virtual scene on the real physical world. The main problem of the state‐of‐the‐art 3D AR head‐mounted displays (HMDs) is the accommodation‐vergence conflict because the 2D images displayed by flat panel devices are at a fixed distance from the eyes. In this paper, we present a design for an optical see‐through HMD utilizing multi‐plane display technology for AR applications. This approach manages to provide correct depth information and solves the accommodation‐vergence conflict problem. In our system, a projector projects slices of a 3D scene onto a stack of polymer‐stabilized liquid crystal scattering shutters in time sequence to reconstruct the 3D scene. The polymer‐stabilized liquid crystal shutters have sub‐millisecond switching time that enables sufficient number of shutters to achieve high depth resolution. A proof‐of‐concept two‐plane optical see‐through HMD prototype is demonstrated. Our design can be made lightweight, compact, with high resolution, and large depth range from near the eye to infinity and thus holds great potential for fatigue‐free AR HMDs.     

Real‐time multi‐spectral image fusion

This paper describes a novel real‐time multi‐spectral imaging capability for surveillance applications. The capability combines a new high‐performance multi‐spectral camera system with a distributed algorithm that computes a spectral‐screening principal component transform (PCT). The camera system uses a novel filter wheel design together with a high‐bandwidth CCD camera to allow image cubes to be delivered at 110 frames s with a spectral coverage between 400 and 1000 nm. The filters used in a particular application are selected to highlight a particular object based on its spectral signature. The distributed algorithm allows image streams from a dispersed collection of cameras to be disseminated, viewed, and interpreted by a distributed group of analysts in real‐time. It operates on networks of commercial‐off‐the‐shelf multiprocessors connected with high‐performance (e.g. gigabit) networking, taking advantage of multi‐threading where appropriate. The algorithm uses a concurrent formulation of the PCT to de‐correlate and compress a multi‐spectral image cube. Spectral screening is used to give features that occur infrequently (e.g. mechanized vehicles in a forest) equal importance to those that occur frequently (e.g. trees in the forest). A human‐centered color‐mapping scheme is used to maximize the impact of spectral contrast on the human visual system. To demonstrate the efficacy of the multi‐spectral system, plant‐life scenes with both real and artificial foliage are used. These scenes demonstrate the systems ability to distinguish elements of a scene that cannot be distinguished with the naked eye. The capability is evaluated in terms of visual performance, scalability, and real‐time throughput. Our previous work on predictive analytical modeling is extended to answer practical design questions such as ‘For a specified cost, what system can be constructed and what performance will it attain?’ Copyright © 2001 John Wiley & Sons, Ltd.