Optically based direct manipulation for augmented reality

Augmented reality (AR) constitutes a very powerful three-dimensional user interface for many “hands-on” application scenarios. To fully exploit the AR paradigm, the computer must not only augment the real world, but also accept feedback from it. In this paper, we present an optical approach for collecting such feedback by analyzing video sequences to track users and the objects they work with. Our system can be set up in any room after quickly placing a few known optical targets in the scene. We present two demonstration scenarios to illustrate the overall concept and potential of our approach and then discuss the research issues involved.     

An augmented reality interface for visualizing and interacting with virtual content

In this paper, a novel AR interface is proposed that provides generic solutions to the tasks involved in augmenting simultaneously different types of virtual information and processing of tracking data for natural interaction. Participants within the system can experience a real-time mixture of 3D objects, static video, images, textual information and 3D sound with the real environment. The user-friendly AR interface can achieve maximum interaction using simple but effective forms of collaboration based on the combinations of human–computer interaction techniques. To prove the feasibility of the interface, the use of indoor AR techniques are employed to construct innovative applications and demonstrate examples from heritage to learning systems. Finally, an initial evaluation of the AR interface including some initial results is presented.     

Real-time advanced spinal surgery via visible patient model and augmented reality system

This paper presents an advanced augmented reality system for spinal surgery assistance, and develops entry-point guidance prior to vertebroplasty spinal surgery. Based on image-based marker detection and tracking, the proposed camera-projector system superimposes pre-operative 3-D images onto patients. The patients’ preoperative 3-D image model is registered by projecting it onto the patient such that the synthetic 3-D model merges with the real patient image, enabling the surgeon to see through the patients’ anatomy. The proposed method is much simpler than heavy and computationally challenging navigation systems, and also reduces radiation exposure. The system is experimentally tested on a preoperative 3D model, dummy patient model and animal cadaver model. The feasibility and accuracy of the proposed system is verified on three patients undergoing spinal surgery in the operating theater. The results of these clinical trials are extremely promising, with surgeons reporting favorably on the reduced time of finding a suitable entry point and reduced radiation dose to patients.     

Spatial augmented reality as a method for a mobile robot to communicate intended movement

Our work evaluates a mobile robot’s ability to communicate intended movements to humans via projection of visual arrows and a simplified map. Humans utilize a variety of techniques to signal intended movement in a co-occupied space. We evaluated an augmented reality projection provided by the robot. The projection is on the floor and consists of arrows and a simplified map. Two pilots and one quasi-experiment were conducted to examine the effectiveness of visual projection of arrows by a robot for signaling intended movement. The pilot work demonstrates the effectiveness of utilizing arrows as a communication medium. The experiment examined the effectiveness of a simplified map and arrows for signaling the short-, mid-range, and long-term intended movement. Two pilot experiments confirm that arrows are an effective symbol for a robot to use to signal intent. A field experiment demonstrates that a robot can use a projected arrow and simplified map to signal its intended movement and people understand the projection for upcoming short-, medium-, and long-term movement. Augmented reality, such as projected arrows and simplified map, are an effective tool for robots to use when signaling their upcoming movement to humans. Telepresence robots in organizations, museum docents, information kiosks, hospital assistants, factories, and as members of search and rescue teams are typical applications where mobile robots reside and interact with people.     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

A haptic pedal for surgery assistance

The research and development of mechatronic aids for surgery is a persistent challenge in the field of robotic surgery. This paper presents a new haptic pedal conceived to assist surgeons in the operating room by transmitting real-time surgical information through the foot. An effective human–robot interaction system for medical practice must exchange appropriate information with the operator as quickly and accurately as possible. Moreover, information must flow through the appropriate sensory modalities for a natural and simple interaction. However, users of current robotic systems might experience cognitive overload and be increasingly overwhelmed by data streams from multiple modalities. A new haptic channel is thus explored to complement and improve existing systems. A preliminary set of experiments has been carried out to evaluate the performance of the proposed system in a virtual surgical drilling task. The results of the experiments show the effectiveness of the haptic pedal in providing surgical information through the foot.     

增强现实环境下的产品装配引导技术

以增强现实环境下引导产品装配为目标,建立了面向增强装配过程的统一 信息模型,管理文字、几何和产品装配特征等可视化引导信息。采用基于标志物的视觉跟踪 技术实现虚拟零件和视频中真实零件的注册定位,通过建立虚拟和真实装配场景的深度图处 理增强装配场景中虚实物体的遮挡关系。利用虚实零件的注册位置把装配引导信息叠加到装 配视频场景中。并开发了演示系统,分析和说明了增强现实环境下引导产品进行装配的过程。     

Lessons learned on the way to industrial augmented reality applications, a retrospective on ARVIKA

This paper presents and discusses the results of the project ARVIKA. After a short introduction and presentation of the project goals, the augmented reality system is described. It consists in a Plug-In for internet-browser in which all the software components have been integrated. The advantages and disadvantages of the adopted approaches, as well as the lessons learned from the project are presented in details. Finally, the open issues are discussed and future work to be achieved until a possible implementation of augmented reality in an industrial context is proposed.     

基于双目视觉的盆腔微创手术增强现实导航仿真系统的设计

基于内窥镜视觉导航的盆腔微创手术,往往因为病灶的复杂解剖位置以及内窥镜视野的局限性,对手术的顺利开展提出了重要的挑战。针对这个问题,设计并开发一套基于立体视觉的盆腔微创手术的增强现实（AR）导航仿真系统。首先,利用术前的CT影像重建骨盆3D模型以及盆腔的真实手术视频,生成带有纹理信息的骨盆3D模型,仿真一套具有真实轨迹的手术视频;然后,利用基于可视点颜色一致性的2D/3D配准技术,实现术前重建模型与手术视野的初始化配准。利用立体视觉跟踪算法,对手术过程中的内窥镜位置进行跟踪,根据内窥镜多自由度的变换矩阵来实现术前3D模型与手术视野的融合与增强现实导航。估计轨迹与真实轨迹的均方根误差为2.3933 mm,仿真实验表明,导航系统为视觉导航提供良好的增强现实显示效果。     

An interactive virtual guide for the AR based visit of archaeological sites

One of the most interesting research lines about avatars is the design and the implementation of a synthetic behaviour able to drive avatar's actions according to an adaptive interaction paradigm. This aspect, indeed, is of fundamental importance to many advanced applications involving avatars and humans. This study presents a novel framework exploiting augmented reality to visualize a synthetic 3D virtual guide inside an exhibit or a public gallery, to assist visitors wearing a Head Mounted Display during their visit and providing them with both visual and informative enhancements not available in a standard tour. The Human–avatar interaction is approached through a model based on timed automata to address the conversational issues and to improve the quality of interaction by means of an effective synchronization. A usability study conducted on an application of this research to the “avatar assisted tour” of a roman villa, confirms the efficacy of the approach.     

An augmented reality human–computer interface for object localization in a cognitive vision system

The European Cognitive Vision project VAMPIRE uses mobile AR-kits to interact with a visual active memory for teaching and retrieval purposes. This paper describes concept and technical realization of the used mobile AR-kits and discusses interactive learning and retrieval in office environments, and the active memory infrastructure. The focus is on 3D interaction for pointing in a scene coordinate system. This is achieved by 3D augmented pointing, which combines inside-out tracking for head pose recovery and 3D stereo human–computer interaction. Experimental evaluation shows that the accuracy of this 3D cursor is within a few centimeters, which is sufficient to point at an object in an office. Finally, an application of the cursor in VAMPIRE is presented, where in addition to the mobile system, at least one stationary active camera is used to obtain different views of an object. There are many potential applications, for example an improved view-based object recognition.     

基于增强现实与异型屏实时互动系统的研究*

提出并实现了一种基于增强现实的多人实时互动的平台和系统。首次提出了在展览展示领域融合应用AR技术与多异型屏技术,研究并实现了AR系统多人实时互动相关技术,设计了AR交互系统的渲染平台、交互平台和网络通信平台。提出利用人工免疫算法解决AR系统实时碰撞检测的问题。系统将在世博会进行展示应用,具有良好的技术与经济效益前景。     

基于虚拟现实技术的移动机器人视觉伺服控制系统设计

为了有效确保移动机器人视觉伺服控制效果,提高移动机器人视觉伺服控制精度,设计了基于虚拟现实技术的移动机器人视觉伺服控制系统。通过三维视觉传感器和立体显示器等虚拟环境的I/O设备、位姿传感器、视觉图像处理器以及伺服控制器元件,完成系统硬件设计。从运动学和动力学两个方面,搭建移动机器人数学模型,利用标定的视觉相机,生成移动机器人实时视觉图像,通过图像滤波、畸变校正等步骤,完成图像的预处理。利用视觉图像,构建移动机器人虚拟移动环境。在虚拟现实技术下,通过目标定位、路线生成、碰撞检测、路线调整等步骤,规划移动机器人行动路线,通过控制量的计算,实现视觉伺服控制功能。系统测试结果表明,所设计控制系统的位置控制误差较小,姿态角和移动速度控制误差仅为0.05°和0.12m/s,移动机器人碰撞次数较少,具有较好的移动机器人视觉伺服控制效果,能够有效提高移动机器人视觉伺服控制精度。     

Treating cockroach phobia using a serious game on a mobile phone and augmented reality exposure: A single case study

In vivo exposure has proved its efficacy in the treatment of specific phobias; however, not all patients benefit from it. Communication and information technologies such as Virtual Reality (VR) and Augmented Reality (AR) have improved exposure treatment adherence and acceptance. Serious games (SG) could also be used in order to facilitate exposure treatment. A line of research on SG is emerging which focuses on health issues. We have developed a SG for the treatment of cockroach phobia that uses a mobile phone as the application device. This work examines results of an N = 1 study about whether the use of this mobile game can facilitate treatment of this specific phobia preparing her for the AR exposure. A 25-year-old woman with cockroach phobia participated in the study. Results showed that the use of the mobile game reduced her level of fear and avoidance before a “one-session” AR exposure treatment was applied, following the guidelines by Öst. The participant found very helpful the use of the SG before the AR exposure session and she was willing to use it after the AR exposure session as a homework assignment. Although the results of this study are preliminary, SG appears to be a line of research of high interest in clinical psychology for the treatment of specific phobias.     

基于增强现实的虚拟城市规划关键技术研究

增强现实是虚拟现实技术的重要分支,已经被成功的应用到众多领域.探讨了增强现实应用于虚拟城市规划的可行性,并给出了一种新颖的基于增强现实技术的城市规划方法.它可以满足城市规划对虚实交互、人机交互的要求,充分体现出增强现实技术在城市规划应用中的优势.介绍了系统结构与工作流程,着重分析了所采用的基于标识的虚实注册方法,并给出了运行结果.     

Toc‐Tum mini‐games: An educational game accessible for deaf culture based on virtual reality

Human hearing is an important sense, which complements the other senses, and is essential for children to begin to acquire basic concepts of the world; however, in severe cases of disability, children become marginalized from these benefits. An example of this is with music, a deaf individual is to be able to perceive it. With this in mind, we developed a novel medical expert system based on virtual reality, called Toc‐Tum mini‐games, applying basic concepts of music in an accessible way to the deaf culture. The intelligent proposed system is validated in two different tests: (a) with the target audience to evaluate the interest in the game and (b) with professionals knowledgeable in the field of music or who have had contact with the deaf, evaluate the impact of the game teaching music to the deaf. The tests with the target audience showed that the methods of interaction allowed the children to understand most of the stages and that they were animated during the test days. Regarding the professionals, they were interested in the possibility of using the proposed system in question, even if some modifications in the design had to be made. The results showed that the Toc‐Tum mini‐games is an intelligent tool capable of introducing music in a playful and manageable way using a virtual environment.     

Effects of using a force feedback haptic augmented simulation on the attitudes of the gifted students towards studying chemical bonds in virtual reality environment

The aim of this study is to identify the effects of force feedback haptic applications developed in virtual reality environments (VREs), which is an important field of study in computer science and engineering, on gifted students’ attitudes towards chemistry education in learning process. A 3D 6 DOF (Degree of Freedom) haptik device (Phantom Omni?) has been used to develop the algorithm in this study. It can be used to transmit force and motion using a haptic device. Visual C++ was choosen as the software development environment. OpenGL? and Haptic Device Application Programming Interface have been used for rendering graphics. At the 3D image creation state Wrap 1200?, which is a kind of head-mounted display, has been chosen. The sample of this study consists of 52 students identified as gifted and are attending 6th and 7th grades at the Istanbul Science &; Art Center in Istanbul. The experimental group studied chemical bonds using an application developed by using a force feedback haptic device in VRE and the control group studied it by traditional teaching methods. The study reveals that there is a relation between using force feedback haptic applications which are developed in VREs and gifted students’ attitudes towards educational programs.     

基于3D虚拟现实技术的弹药检测虚拟训练系统设计

目前提出的弹药检测虚拟训练系统接收检测信号接收率较低,检测虚拟训练成功率较低。基于3D虚拟现实技术设计一种新的弹药检测虚拟训练系统,硬件由主控操作机、检测总线接口以及资源测试器组成,软件部分在3D虚拟现实技术下构建虚拟空间,在开放式软件结构构建的基础上进行检测指标划分,对用户界面的文件进行整改,按照弹药检测的指标数据提升整改的方向正确性,选用自检公式对检测的信息进行系统自检,依据检测到的问题的发生形式判断下一次问题发生的触发机制,由此避免下一次问题的产生,从而实现软件流程。实验结果表明,基于3D虚拟现实技术的弹药检测虚拟训练系统能有效提高信号接收率,增强虚拟训练接收成功率,具有较强的应用性。     

Multi-user mixed reality system ‘Gulliver’s World’: a case study on collaborative edutainment at the intersection of material and virtual worlds

This case study is about ‘Gulliver’s World’, a multi-user mixed reality environment that functions simultaneously as interactive edutainment platform and learning environment as well as flexible infrastructure for the expansion of mixed reality environments via innovations in technology and media art. As an exhibition project, the installation is characterized by a nonlinear exhibition concept that activates interaction between individual users and different modes of virtual reality as well as collaboration among the users themselves. At seven workstations, people of all age groups range along the Reality–Virtuality continuum while collaboratively creating 3D worlds. Results of these creation activities are interactive worlds at the nexus of theatre, digital film production and game environment. As a research project, ‘Gulliver’s World’ features multilevel infrastructure with exemplary content in which the latest insights and models to emerge from HCI research as well as concepts of mixed reality and virtual environments and their supporting technology are brought together and developed further.