Harmonic rendering for visual coherence on mobile outdoor AR environment

Rapid developments in augmented reality (AR) and related technologies have led to increasing interest in immersive content. AR environments are created by combining virtual 3D models with a real-world video background. It is important to merge these two worlds seamlessly if users are to enjoy AR applications, but, all too often, the illumination and shading of virtual objects is not consider the real world lighting condition or does not match that of nearby real objects. In addition, visual artifacts produced when blending real and virtual objects further limit realism. In this paper, we propose a harmonic rendering technique that minimizes the visual discrepancy between the real and virtual environments to maintain visual coherence in outdoor AR. To do this, we introduce a method of estimating and approximating the Sun’s position and the sunlight direction to estimate the real sunlight intensity, as this is the most significant illumination source in outdoor AR and it provides a more realistic lighting environment for such content, reducing the mismatch between real and virtual objects.

     

基于AR的国家开放大学计算机专业体验式教学软件的设计开发

AR是一种新型多媒体计算机技术,融模式识别技术、机器学习技术、多媒体现实技术、3D建模技术于一体,创建一个具有智能性、交互性、沉浸感的虚实结合的系统。计算机专业体验式教学软件的设计对AR技术的应用与传播提供了有力支持,能够根据实际需求创设一个装配式开发环境,对当前所使用的软件开发环境进行扩展与延伸,借助多种类型建模组件将软件开发过程直观地展现出来,无需购置或者构建大型应用环境,将虚拟世界与真实世界无缝链接,从而提高国家开放大学学生的学习效率。     

基于实时流媒体技术的全景视频系统

将实时视频采集和虚拟现实技术相结合,探索出一种利用相对经济的采集设备,生成高信息量、强沉浸感的数字媒体的方案。文中介绍了软件系统的设计和实现,并详细说明了系统实现使用的关键技术和相应算法,解决了将流媒体技术与虚拟现实技术相结合的多个难点。全景视频系统系统基于C/S结构在Windows平台上实现,主要利用微软的DirectShow和D3D的开发包,为以后视频会议、数字媒体系统等提供了新的应用前景。     

Towards video-based immersive environments

Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.     

基于AR-VR混合技术的儿童智力开发系统研究

传统的儿童智力开发大多采用观看图书或是视频的方式,存在信息表达单一、缺乏生动性、交互性差等缺陷。在儿童智力开发理论基础上,将增强现实技术（AR）与虚拟现实技术（VR）结合,基于Unity3D平台设计,开发出具有游戏式风格的儿童智力开发系统。VR技术用于开发虚拟漫游模块,通过沉浸式交互体验,培养儿童对身边事物的认知及观察能力。AR技术用于开发识图模块,通过3D的学习方式,为儿童传递更丰富更具体的学习体验。实验结果表明,AR-VR混合技术能构建出高度真实的学习环境,具有多感官交互效果,增强儿童对现实环境的感知和学习能力,相较于传统的儿童智力开发,该系统能帮助儿童获得更好的学习体验和交互方式,激发儿童的学习兴趣。     

Interactive digital graffiti canvas system

This paper presents an interactive digital painting system that allows a user to draw graffiti on a virtual 3D canvas with a digital spray can. The system visualizes a stereoscopic representation of the canvas by tracking the user's head. It also emulates real‐time spray painting by tracking the spray can in the user's hand as well as sensing the button pressure of the spray device. After painting a 3D object, the user can interact with the object on the display and see it flying in the 3D environment through a tracked head‐mounted display. As demonstrated in the results of our evaluation, we verified that the system resembles real graffiti in regard to a natural and realistic graffiti experience. Copyright © 2015 John Wiley & Sons, Ltd.     

An augmented reality interface for visualizing and interacting with virtual content

In this paper, a novel AR interface is proposed that provides generic solutions to the tasks involved in augmenting simultaneously different types of virtual information and processing of tracking data for natural interaction. Participants within the system can experience a real-time mixture of 3D objects, static video, images, textual information and 3D sound with the real environment. The user-friendly AR interface can achieve maximum interaction using simple but effective forms of collaboration based on the combinations of human–computer interaction techniques. To prove the feasibility of the interface, the use of indoor AR techniques are employed to construct innovative applications and demonstrate examples from heritage to learning systems. Finally, an initial evaluation of the AR interface including some initial results is presented.     

A multi‐plane optical see‐through head mounted display design for augmented reality applications

Augmented reality (AR) display technology greatly enhances users' perception of and interaction with the real world by superimposing a computer‐generated virtual scene on the real physical world. The main problem of the state‐of‐the‐art 3D AR head‐mounted displays (HMDs) is the accommodation‐vergence conflict because the 2D images displayed by flat panel devices are at a fixed distance from the eyes. In this paper, we present a design for an optical see‐through HMD utilizing multi‐plane display technology for AR applications. This approach manages to provide correct depth information and solves the accommodation‐vergence conflict problem. In our system, a projector projects slices of a 3D scene onto a stack of polymer‐stabilized liquid crystal scattering shutters in time sequence to reconstruct the 3D scene. The polymer‐stabilized liquid crystal shutters have sub‐millisecond switching time that enables sufficient number of shutters to achieve high depth resolution. A proof‐of‐concept two‐plane optical see‐through HMD prototype is demonstrated. Our design can be made lightweight, compact, with high resolution, and large depth range from near the eye to infinity and thus holds great potential for fatigue‐free AR HMDs.     

增强现实技术在虚拟演播室系统中的应用

虚拟演播室是虚拟现实技术和视频合成技术相结合的产物,其场景是计算机生成的三维场景,由于人们对虚拟场景和复杂度的无限要求,使得场景的实时显示十分困难,使用基于图像的绘制技术构造虚拟空间能够较好地解决这个问题。在虚拟演播室中,演员需要与三维运动虚拟物体进行交互,运用增强现实技术,可以将三维虚拟物体与基于图象绘制的虚拟场景融合在一起。     

Generation of augmented video sequences combining behavioral animation and multi‐object tracking

In this paper we present a novel approach to generate augmented video sequences in real‐time, involving interactions between virtual and real agents in real scenarios. On the one hand, real agent motion is estimated by means of a multi‐object tracking algorithm, which determines real objects' position over the scenario for each time step. On the other hand, virtual agents are provided with behavior models considering their interaction with the environment and with other agents. The resulting framework allows to generate video sequences involving behavior‐based virtual agents that react to real agent behavior and has applications in education, simulation, and in the game and movie industries. We show the performance of the proposed approach in an indoor and outdoor scenario simulating human and vehicle agents. Copyright © 2009 John Wiley & Sons, Ltd.     

The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality Environments

Three-dimensional sound's effectiveness in virtual reality (VR) environments has been widely studied. However, due to the big differences between VR and augmented reality (AR) systems in registration, calibration, perceptual difference of immersiveness, navigation, and localization, it is important to develop new approaches to seamlessly register virtual 3-D sound in AR environments and conduct studies on 3-D sound's effectiveness in AR context. In this paper, we design two experimental AR environments to study the effectiveness of 3-D sound both quantitatively and qualitatively. Two different tracking methods are applied to retrieve the 3-D position of virtual sound sources in each experiment. We examine the impacts of 3-D sound on improving depth perception and shortening task completion time. We also investigate its impacts on immersive and realistic perception, different spatial objects identification, and subjective feeling of "human presence and collaboration". Our studies show that applying 3-D sound is an effective way to complement visual AR environments. It helps depth perception and task performance, and facilitates collaborations between users. Moreover, it enables a more realistic environment and more immersive feeling of being inside the AR environment by both visual and auditory means. In order to make full use of the intensity cues provided by 3-D sound, a process to scale the intensity difference of 3-D sound at different depths is designed to cater small AR environments. The user study results show that the scaled 3-D sound significantly increases the accuracy of depth judgments and shortens the searching task completion time. This method provides a necessary foundation for implementing 3-D sound in small AR environments. Our user study results also show that this process does not degrade the intuitiveness and realism of an augmented audio reality environment     

Vision‐tangible interactive display method for mixed and virtual reality: Toward the human‐centered editable reality

Building a human‐centered editable world can be fully realized in a virtual environment. Both mixed reality (MR) and virtual reality (VR) are feasible solutions to support the attribute of edition. Based on the current development of MR and VR, we present the vision‐tangible interactive display method and its implementation in both MR and VR. We address the issue of MR and VR together because they are similar regarding the proposed method. The editable mixed and virtual reality system is useful for studies, which exploit it as a platform. In this paper, we construct a virtual reality environment based on the Oculus Rift, and an MR system based on a binocular optical see‐through head‐mounted display. In the MR system about manipulating the Rubik's cube, and the VR system about deforming the virtual objects, the proposed vision‐tangible interactive display method is utilized to provide users with a more immersive environment. Experimental results indicate that the vision‐tangible interactive display method can improve the user experience and can be a promising way to make the virtual environment better.     

基于 OSG 的漫游系统的设计与实现

虚拟现实技术的提出,凭借真实的沉浸感、可交互性以及可构想性等这些优势,轻而易举地把现实世界通过计算机数字化中呈现出来。在城市建设、室内设计、工业设计、道路桥梁规划、旅游教学、房地产、古迹恢复、电力水利、地质灾害、虚拟校园等各个领域广泛应用虚拟现实技术,并对相应领域的发展产生了极大的帮助和推动作用。基于OSG的漫游系统正是基于此技术上的一项应用,该系统具备模型加载,点、线、面渲染,导航,漫游,雨、雪、雾特效,等功能。最终目标是希望实时呈现了一个逼真的、立体化的,能给人视觉、听觉、触觉感受的虚拟漫游系统。该漫游系统的实现有利于景区旅游资源充分开发利用,对广西师范大学建设规划具有重要的现实意义。同时,也是弘扬和传播广西师范大学校园文化的重要途径。     

Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

Designing a Highly Immersive Interactive Environment: The Virtual Mine

To achieve a full‐scale simulation of a pyrite mine, a highly immersive environment becomes necessary and this research has led to a complex system enabling users to walk through a virtual mine in real time, presenting all the behaviours present in such environment. Some of the problems encountered are the tunnels behaviours, including highly contrasted images due to the presence of the head light, narrow paths, elevators, sound reverberation and tunnels texture shades. The use of immersive virtual reality enables the generation of high‐quality simulations, because it is possible to control several feedback mechanisms such as the degree of luminance of produced imagery and spatial sound. In this research, a projection infrastructure and tracking system were specified and developed, aiming at producing the best results for this kind of simulation. To achieve our purposes, distributed algorithms were developed to run in a cluster solution that drives a four‐sided CAVE‐like environment. The complete production pipeline is presented, ranging from the developed authoring techniques, enabling fast production of new content for the simulation, to the tracking techniques produced for the improvement of the interaction.     

Real-time optical markerless tracking for augmented reality applications

Augmented reality (AR) technology consists in adding computer-generated information (2D/3D) to a real video sequence in such a manner that the real and virtual objects appear coexisting in the same world. To get a realistic illusion, the real and virtual objects must be properly aligned with respect to each other, which requires a robust real-time tracking strategy—one of the bottlenecks of AR applications. In this paper, we describe the limitations and advantages of different optical tracking technologies, and we present our customized implementation of both recursive tracking and tracking by detection approaches. The second approach requires the implementation of a classifier and we propose the use of a Random Forest classifier. We evaluated both approaches in the context of an AR application for design review. Some conclusions regarding the performance of each approach are given.     

Immersive Multiplayer Games With Tangible and Physical Interaction

In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game (for VR and AR environments, respectively) and a sword game (for the AR environment). Our immersive system provides an intuitive way for users to interact with the VR or AR world by physically moving around the real world and aiming freely with tangible objects. This encourages physical interaction between players as they compete or collaborate with other players. Evaluation of our system consists of users' subjective opinions and their objective performances. Our design principles and evaluation results can be applied to similar immersive game applications based on AR/VR.     

基于多视频的虚实融合系统

提出了一种基于多视频的虚实融合可视化系统的构建方法,旨在将真实世界中的图像和视频融合到虚拟场景中,用视频图像中的纹理和动态信息去丰富虚拟场景,提高虚拟环境的真实性,得到一种增强的虚拟环境.利用无人机采集图像来重建虚拟场景,并借助图像特征点的匹配来实现视频图像的注册.然后利用投影纹理映射技术,将图像投影到虚拟场景中.视频中的动态物体由于在虚拟环境中缺失对应的三维模型,直接投影,当视点发生变化时会产生畸变.首先检测和追踪这些物体,然后尝试使用多种显示方式来解决畸变问题.此外,系统还考虑有重叠区域的多视频之间的融合.实验结果表明,所构造的虚实融合环境是十分有益的.     

Video-based interactive storytelling using real-time video compositing techniques

Interactive storytelling systems usually adopt computer graphics to represent virtual story worlds, which facilitates the dynamic generation of visual content. However, the quality of the images and motion produced by these systems is still inferior compared to the high quality experience found in live-action films. Interactive rates in photorealistic rendering for the film industry will not be possible for decades to come. A promising alternative is the replacement of 3D virtual characters with video sequences with real actors. In this paper, we propose a new method for video-based interactive narratives that uses video compositing algorithms that run at truly interactive frame rates. The proposed method is consistent with plots that are generated by nondeterministic planning algorithms. Moreover, we propose a system of artificial intelligent agents that perform the same roles played by filmmaking professionals. A user evaluation of the proposed method is presented. We believe that future improvements of the techniques proposed in this paper represent an important contribution to the quest for new and more immersive forms of interactive cinema.

     

电子商务中的虚拟现实技术刍议

不断成熟的虚拟现实技术对电子商务发展起到一定的促进作用,其与电子商务相结合能够营造出前所未有的逼真购物环境,给顾客带来身临其境购物体验。文章对虚拟现实技术进行了简要介绍,给出了基于虚拟现实技术的电子商务系统的逻辑结构和场景制作流程等,虚拟现实技术应用于电子商务是其发展的必然趋势之一。