Interactive 3D content insertion in images for multimedia applications

This article addresses the problem of creating interactive mixed reality applications where virtual objects interact in images of real world scenarios. This is relevant to create games and architectural or space planning applications that interact with visual elements in the images such as walls, floors and empty spaces. These scenarios are intended to be captured by the users with regular cameras or using previously taken photographs. Introducing virtual objects in photographs presents several challenges, such as pose estimation and the creation of a visually correct interaction between virtual objects and the boundaries of the scene. The two main research questions addressed in this article include, the study of the feasibility of creating interactive augmented reality (AR) applications where virtual objects interact in a real world scenario using the image detected high-level features and, also, verifying if untrained users are capable and motivated enough to perform AR initialization steps. The proposed system detects the scene automatically from an image with additional features obtained using basic annotations from the user. This operation is significantly simple to accommodate the needs of non-expert users. The system analyzes one or more photos captured by the user and detects high-level features such as vanishing points, floor and scene orientation. Using these features it will be possible to create mixed and augmented reality applications where the user interactively introduces virtual objects that blend with the picture in real time and respond to the physical environment. To validate the solution several system tests are described and compared using available external image datasets.     

Object interaction and task programming by demonstration in visuo-haptic augmented reality

A visuo-haptic augmented reality system is presented for object manipulation and task learning from human demonstration. The proposed system consists of a desktop augmented reality setup where users operate a haptic device for object interaction. Users of the haptic device are not co-located with the environment where real objects are present. A three degrees of freedom haptic device, providing force feedback, is adopted for object interaction by pushing, selection, translation and rotation. The system also supports physics-based animation of rigid bodies. Virtual objects are simulated in a physically plausible manner and seem to coexist with real objects in the augmented reality space. Algorithms for calibration, object recognition, registration and haptic rendering have been developed. Automatic model-based object recognition and registration are performed from 3D range data acquired by a moving laser scanner mounted on a robot arm. Several experiments have been performed to evaluate the augmented reality system in both single-user and collaborative tasks. Moreover, the potential of the system for programming robot manipulation tasks by demonstration is investigated. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations and that the learned task can be executed by a robot system.     

Modelling and design of an augmented reality differential drive mobility aid in an enabled environment

In this paper the authors discuss the modelling and design of an augmented reality platform for disabled wheeled mobility studies. This design consists of a virtual environment with two degrees of freedom motion platform and integrated ground contact force feedback. Differential drive mobility users continue to be in touch with reality on their own mobility aid while interacting with virtual objects. The major domain related to the differential drive mobility of the disabled members of society which coincides with the use of manual wheelchairs, electric wheelchairs and mobility scooters. In order to account for environmental and dynamic effects, the wheeled mobility user needs to map the intended trajectory into the virtual world. Motion and inertia force feedback produced on the augmented simulator give the users a haptic sensory stimulus input regarding spatial movement and ground contact forces. The main objective is to model and design an augmented reality platform with real world kinematic and dynamic properties to place a wheeled mobility user closer to real world encounters. The use of the designed augmented reality platform will be beneficial to disabled wheeled mobility users in need of occupational therapist training and evaluation.     

视触觉融合的增强现实三维注册方法

增强现实技术是近年来人机交互领域的研究热点。在增强现实环境下加入触觉感知,可使用户在真实场景中看到并感知到虚拟对象。为了实现增强现实环境下与虚拟对象之间更加自然的交互,提出一种视触觉融合的三维注册方法。基于图像视觉技术获得三维注册矩阵;借助空间转换关系求解出触觉空间与图像空间的转换关系;结合两者与摄像头空间的关系实现视触觉融合的增强现实交互场景。为验证该方法的有效性,设计了一个基于视触觉增强现实的组装机器人项目。用户可触摸并移动真实环境中的机器人零件,还能在触摸时感受到反馈力,使交互更具真实感。     

增强现实中基于轮廓深度恢复的虚实遮挡方法研究

针对增强现实系统中的虚实场景遮挡问题,提出一种基于立体视觉与前景轮廓深度重建的虚实遮挡恢复新方法.首先设计了一种基于色彩与亮度相结合的前景物体轮廓提取方法,以避免阴影对轮廓提取所造成的负面影响,系统随后利用立体视觉原理与插值技术恢复出前景轮廓以及轮廓内部的深度信息,为恢复虚拟与真实场景之间的正确遮挡关系提供依据.实验结...     

增强现实中虚拟物体制作的研究

增强现实是把计算机产生的虚拟物体合成到用户看到的真实世界中的一种技术。介绍了增强现实中虚拟物体所涉及的一些关键技术,包括虚拟物体的建模、摄像头标定、骨骼动画以及虚拟场景的优化,说明了如何将这些技术应用到实际系统中。     

Harmonic rendering for visual coherence on mobile outdoor AR environment

Rapid developments in augmented reality (AR) and related technologies have led to increasing interest in immersive content. AR environments are created by combining virtual 3D models with a real-world video background. It is important to merge these two worlds seamlessly if users are to enjoy AR applications, but, all too often, the illumination and shading of virtual objects is not consider the real world lighting condition or does not match that of nearby real objects. In addition, visual artifacts produced when blending real and virtual objects further limit realism. In this paper, we propose a harmonic rendering technique that minimizes the visual discrepancy between the real and virtual environments to maintain visual coherence in outdoor AR. To do this, we introduce a method of estimating and approximating the Sun’s position and the sunlight direction to estimate the real sunlight intensity, as this is the most significant illumination source in outdoor AR and it provides a more realistic lighting environment for such content, reducing the mismatch between real and virtual objects.

     

Two-handed tangible interaction techniques for composing augmented blocks

Modeling tools typically have their own interaction methods for combining virtual objects. For realistic composition in 3D space, many researchers from the fields of virtual and augmented reality have been trying to develop intuitive interactive techniques using novel interfaces. However, many modeling applications require a long learning time for novice users because of unmanageable interfaces. In this paper, we propose two-handed tangible augmented reality interaction techniques that provide an easy-to-learn and natural combination method using simple augmented blocks. We have designed a novel interface called the cubical user interface, which has two tangible cubes that are tracked by marker tracking. Using the interface, we suggest two types of interactions based on familiar metaphors from real object assembly. The first, the screw-driving method, recognizes the user??s rotation gestures and allows them to screw virtual objects together. The second, the block-assembly method, adds objects based on their direction and position relative to predefined structures. We evaluate the proposed methods in detail with a user experiment that compares the different methods.     

Tracking in unprepared environments for augmented reality systems

Many augmented reality applications require accurate tracking. Existing tracking techniques require prepared environments to ensure accurate results. This paper motivates the need to pursue augmented reality tracking techniques that work in unprepared environments, where users are not allowed to modify the real environment, such as in outdoor applications. Accurate tracking in such situations is difficult, requiring hybrid approaches. This paper summarizes two 3DOF results: a real-time system with a compass — inertial hybrid, and a non-real-time system fusing optical and inertial inputs. We then describe the preliminary results of 5- and 6-DOF tracking methods run in simulation. Future work and limitations are described.     

Gesture-based interactive augmented reality content authoring system using HMD

This paper proposes an augmented reality content authoring system that enables ordinary users who do not have programming capabilities to easily apply interactive features to virtual objects on a marker via gestures. The purpose of this system is to simplify augmented reality (AR) technology usage for ordinary users, especially parents and preschool children who are unfamiliar with AR technology. The system provides an immersive AR environment with a head-mounted display and recognizes users’ gestures via an RGB-D camera. Users can freely create the AR content that they will be using without any special programming ability simply by connecting virtual objects stored in a database to the system. Following recognition of the marker via the system’s RGB-D camera worn by the user, he/she can apply various interactive features to the marker-based AR content using simple gestures. Interactive features applied to AR content can enlarge, shrink, rotate, and transfer virtual objects with hand gestures. In addition to this gesture-interactive feature, the proposed system also allows for tangible interaction using markers. The AR content that the user edits is stored in a database, and is retrieved whenever the markers are recognized. The results of comparative experiments conducted indicate that the proposed system is easier to use and has a higher interaction satisfaction level than AR environments such as fixed-monitor and touch-based interaction on mobile screens.     

Multi‐view reconstruction of dynamic real‐world objects and their integration in augmented and virtual reality applications

We present a flexible system for high‐quality three‐dimensional reconstruction of dynamic real‐world objects based on a modular multi‐camera capture setup. The proposed algorithmic pipeline aims at the acquisition and digitization of natural and realistic representations of real people that can be easily integrated into augmented and virtual reality applications. In this context, we discuss the reduction of mesh complexity as one of the key challenges for visualizing reconstructed three‐dimensional content with augmented and virtual reality glasses and demonstrate different fields of application.     

Dynamic superimposition of synthetic objects on rigid and simple-deformable real objects

A current challenge in augmented reality applications is the accurate superimposition of synthetic objects on real objects within the environment. This challenge is heightened when the real objects are in motion and/or are non-rigid. In this article, we present a robust method for real-time, optical superimposition of synthetic objects on dynamic, rigid and simple-deformable real objects. Moreover, we illustrate this general method with the VRDA Tool, a medical education application enabling the visualization of internal human knee joint anatomy on a real human knee.     

Distributed design review using tangible augmented technical drawings

In this work we integrate augmented reality technology in a product development process using real technical drawings as a tangible interface for design review. We present an original collaborative framework for Augmented Design Review Over Network (ADRON). It provides the following features: augmented technical drawings, interactive FEM simulation, multimodal annotation and chat tools, web content integration and collaborative client/server architecture. Our framework is intended to use common hardware instead of expensive and complex virtual or augmented facilities. We designed the interface specifically for users with little or no augmented reality expertise proposing tangible interfaces for data review and visual editing for all the functions and configurations. Two case studies are presented and discussed: a real-time “touch and see” stress/strain simulation and a collaborative distributed design review session of an industrial component.     

Framework for context-aware smartphone applications

This paper presents research and development of a dedicated system architecture, designed to enable its users to interact with each other as well as to access information on points of interest that exist in their immediate environment. This is accomplished through managing personal preferences and contextual information in a distributed manner and in real time. The advantage of this system is that it uses mobile devices, heterogeneous sensors and a selection of user interface paradigms to produce a sociotechnical framework to enhance the perception of the environment and promote intuitive interactions. Representation of the real-world objects, their spatial relations and other captured features are visualised on scalable interfaces, ranging from 2D to 3D models and from photorealism to stylised clues and symbols. The conceptual design and implementation of our location and orientation based algorithm for mobile augmented reality is presented in detail. The framework is fit for use in unknown environments and therefore suitable for ubiquitous operation. The presented prototype is multifaceted and capable of supporting peer-to-peer exchange of information in a pervasive fashion, usable in various contexts. The modalities of these interactions are explored and laid out particularly in the context of entertainment and urban navigation.     

Exploring legibility of augmented reality X-ray

Virtual objects can be visualized inside real objects using augmented reality (AR). This visualization is called AR X-ray because it gives the impression of seeing through the real object. In standard AR, virtual information is overlaid on top of the real world. To position a virtual object inside an object, AR X-ray requires partially occluding the virtual object with visually important regions of the real object. In effect, the virtual object becomes less legible compared to when it is completely unoccluded. Legibility is an important consideration for various applications of AR X-ray. In this research, we explored legibility in two implementations of AR X-ray, namely, edge-based and saliency-based. In our first experiment, we explored on the tolerable amounts of occlusion to comfortably distinguish small virtual objects. In our second experiment, we compared edge-based and saliency-based AR X-ray methods when visualizing virtual objects inside various real objects. Moreover, we benchmarked the legibility of these two methods against alpha blending. From our experiments, we observed that users have varied preferences for proper amounts of occlusion cues for both methods. The partial occlusions generated by the edge-based and saliency-based methods need to be adjusted depending on the lighting condition and the texture complexity of the occluding object. In most cases, users identify objects faster with saliency-based AR X-ray than with edge-based AR X-ray. Insights from this research can be directly applied to the development of AR X-ray applications.     

语义驱动下水面场景中AR效果提升方法

增强现实应用场景不断拓展,但水面检测领域由于物理光学的反射等特性的影响制约了水面增强现实的应用。对此提出一种语义驱动下的基于ORB-SLAM2系统在水面场景下进行实时增强现实效果提升的方法。将视频帧传入深度学习语义分割网络ICNet中,把分割后得到的标签图和原视频帧一同输入SLAM系统的前端进行追踪及地图构建。在语义的指导下,水域内拟合平面并根据水面反射原理将3D模型放置在平面的同时,在关于平面镜面对称的位置生成其倒影。倒影的颜色依据提出的混色模型进行渲染。实验结果表明,在户外水面场景下增强现实的效果得到了提升,虚拟物体与真实物体视觉一致性也更为连贯。     

三维单目虚实物体球对面碰撞检测算法

为加强增强现实的沉浸感与真实性,实时地进行虚实物体间的碰撞检测至关重要.因此,提出一种基于增强现实和单目视觉的任意形状虚实物体碰撞检测估计算法.通过改进现有的单目二维虚实碰撞检测及响应算法,针对现有碰撞检测算法存在的计算复杂度高的问题,提出一种仅需计算实际物体4个特征点的三维碰撞检测算法;并通过对象分割、特征点提取、碰撞检测和碰撞响应等过程取得与真实世界物理特性一致的三维虚实碰撞响应估计效果.在增强现实有标记和无标记环境下分别进行实验结果表明,碰撞检测的计算量与二维算法相近,且具有景深效果,实现了基于单目视觉的增强现实三维虚实碰撞响应预测和处理.     

Rapid modeling of cones and cylinders from a single calibrated image using minimum 2D control points

Three-dimensional modeling from a single image is a very useful technique in applications, such as augment reality, architecture, urban digitalization, heritage preservation, etc. While the modeling of lines and planes has been well studied, a convenient way of modeling cones and cylinders, which are commonly encountered objects, is not available yet. Although fully automatic approaches exist, they cannot guarantee the precision demanded by a real application, and are not appropriate for modeling a complex environment, where users need full control of the modeling process. In addition, approaches relying on the detection of full object contours cannot be directly transformed to simple interactive modeling procedures. This paper presents novel approaches to modeling cone and cylinder objects based on a set of minimum two-dimensional control points. The approaches enable intuitive, fast and accurate modeling from a single image. Both synthetic and real data experiments demonstrate the advantages of the proposed approach: desired accuracy with minimal user interactions.     

卡通风格增强现实系统

增强现实技术将计算机生成的虚拟物体叠加到真实场景中.在传统的AR系统中,虚拟物体和真实场景视觉上存在较为明显的差异,达不到虚拟物体和真实场景无缝结合的要求.本文将增强现实技术与非真实感的绘制技术有机的结合起来,减小这种视觉差异,研究并实现了卡通风格的增强现实系统.     

水彩画风格实时增强现实技术

在传统的AR系统中,虚拟物体和真实场景在视觉上存在较为明显的差异,达不到虚拟物体和真实场景无缝结合的要求。将增强现实技术与NPR有机地结合起来,减小这种视觉差异,研究并实现了水彩画风格的增强现实系统。