Tangled reality

Leonardo da Vinci was a strong advocate for using sketches to stimulate the human imagination. Sketching is often considered to be integral to the process of design, providing an open workspace for ideas. For these same reasons, children use sketching as a simple way to express visual ideas. By merging the abstraction of human drawings and the freedom of virtual reality with the tangibility of physical tokens, Tangled Reality creates a rich mixed reality workspace. Tangled Reality allows users to build virtual environments based on simple colored sketches and traverse them using physical vehicles overlayed with virtual imagery. This setup allows the user to “build” and “experience” mixed reality simulations without ever touching a standard computer interface.     

Tangible authoring of 3D virtual scenes in dynamic augmented reality environment

This paper proposes tangible interfaces and interactions for authoring 3D virtual and immersive scenes easily and intuitively in tangible augmented reality (AR) environment. It provides tangible interfaces for manipulating virtual objects in a natural and intuitive manner and supports adaptive and accurate vision-based tracking in AR environments. In particular, RFID is used to directly integrate physical objects with virtual objects and to systematically support the tangible query of the relation between physical objects and virtual ones, which can provide more intuitive tangibility and a new way of virtual object manipulation. Moreover, the proposed approach offers an easy and intuitive switching mechanism between tangible environment and virtual environment. This paper also proposes a context-adaptive marker tracking method which can remove an inconsistent problem while embedding virtual objects into physical ones in tangible AR environments. The context-adaptive tracking method not only adjusts the locations of invisible markers by interpolating the locations of existing reference markers and those of previous ones, but also removes a jumping effect of movable virtual objects when their references are changed from one marker to another. Several case studies for generating tangible virtual scenes and comparison with previous work are given to show the effectiveness and novelty of the proposed approach.     

A research agenda for augmented and virtual reality in architecture,engineering and construction

This paper presents a study on the usage landscape of augmented reality (AR) and virtual reality (VR) in the architecture, engineering and construction sectors, and proposes a research agenda to address the existing gaps in required capabilities. A series of exploratory workshops and questionnaires were conducted with the participation of 54 experts from 36 organisations from industry and academia. Based on the data collected from the workshops, six AR and VR use-cases were defined: stakeholder engagement, design support, design review, construction support, operations and management support, and training. Three main research categories for a future research agenda have been proposed, i.e.: (i) engineering-grade devices, which encompasses research that enables robust devices that can be used in practice, e.g. the rough and complex conditions of construction sites; (ii) workflow and data management; to effectively manage data and processes required by AR and VR technologies; and (iii) new capabilities; which includes new research required that will add new features that are necessary for the specific construction industry demands. This study provides essential information for practitioners to inform adoption decisions. To researchers, it provides a research road map to inform their future research efforts. This is a foundational study that formalises and categorises the existing usage of AR and VR in the construction industry and provides a roadmap to guide future research efforts.     

Storing user experiences in mixed reality using hypermedia

This paper describes HyperMem, a system to store and replay user experiences in mixed environments. The experience is stored as a set of hypermedia nodes and links, with the information that was displayed along with the video of the real world that was navigated. It uses a generic hypermedia model, implemented as software components, to handle mixed reality environments. This model includes components for storing and replaying experiences and integrating them in the overall set of hypermedia graphs that can be accessed by a given user. The paper presents the goals of the system, the underlying hypermedia model, the application scenarios, and the architecture and tools for replaying and repurposing stored information.     

ARSC: Augmented reality student card

Augmented Reality (AR) is the technology of adding virtual objects to real scenes through enabling the addition of missing information in real life. As the lack of resources is a problem that can be solved through AR, this paper presents and explains the usage of AR technology we introduce Augmented Reality Student Card (ARSC) as an application of AR in the field of education. ARSC uses single static markers combined in one card for assigning different objects, while leaving the choice to the computer application for minimizing the tracking process. ARSC is designed to be a useful low cost solution for serving the education field. ARSC can represent any lesson in a 3D format that helps students to visualize different learning objects, interact with theories and deal with the information in a totally new, effective, and interactive way. ARSC can be used in offline, online and game applications with seven markers, four of them are used as a joystick game controller. One of the novelties in this paper is that experimental tests had been made for the ARTag marker set for sorting them according to their efficiency. The results of those tests were used in this research to choose the most efficient markers for ARSC, and can be used for further research. The experimental work in this paper also shows the constraints for marker creation for an AR application. As we need to work in both online and offline application, merging of toolkits and libraries has been made, as presented in this paper. ARSC was examined by a number of students of both genders with average age between 10 and 17 years and it found great acceptance among them.     

A method of creating a transparent space based on mixed reality

In this article, we propose a novel idea to create a transparent space. Accidents often happen when a person, a motorcycle, a car, etc., comes out suddenly from a blind corner. In such a place, if the shielding objects become transparent, hidden objects reveal themselves. This may contribute to preventing a lot of accidents. The proposed method creates a transparent space by the employment of a mixed-reality technique and a simple three-dimensional (3-D) recovery technique. The 3-D virtual space is created by a single camera image in real-time, and it is superposed onto a real space employing see-through vision. The observer can watch a dynamic scene of the invisible space through the object in real-time. In the experiment performed, an observer could watch persons walking around in a space hidden by an opaque partition. Future application of this technique may include its installation at various corners in buildings or at intersections on roads with bad visibility. This would surely contribute to reducing traffic accidents in future. This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006     

Mobile phone-based mixed reality: the Snap2Play game

The ubiquity of camera phones provides a convenient platform to develop immersive mixed-reality games. In this paper we introduce such a game which is loosely based on the popular card game “Memory”, where players are asked to match a pair of identical cards among a set of overturned cards by revealing only two cards at a time. In our game, the players are asked to match a “digital card”, which corresponds to a scene in a virtual world, to a “physical card”, which is an image of a scene in the real world. The objective is to convey a mixed-reality sensation. Cards are matched with a scene identification engine which consists of multiple classifiers trained on previously collected images. We present our comprehensive overall game design, as well as implementation details and results. We also describe how we constructed our scene identification engine and its performance. Finally, we present an analysis of player surveys to gauge the potential market acceptance.

Presence and interaction in mixed reality environments

In this paper, we present a simple and robust mixed reality (MR) framework that allows for real-time interaction with virtual humans in mixed reality environments under consistent illumination. We will look at three crucial parts of this system: interaction, animation and global illumination of virtual humans for an integrated and enhanced presence. The interaction system comprises of a dialogue module, which is interfaced with a speech recognition and synthesis system. Next to speech output, the dialogue system generates face and body motions, which are in turn managed by the virtual human animation layer. Our fast animation engine can handle various types of motions, such as normal key-frame animations, or motions that are generated on-the-fly by adapting previously recorded clips. Real-time idle motions are an example of the latter category. All these different motions are generated and blended on-line, resulting in a flexible and realistic animation. Our robust rendering method operates in accordance with the previous animation layer, based on an extended for virtual humans precomputed radiance transfer (PRT) illumination model, resulting in a realistic rendition of such interactive virtual characters in mixed reality environments. Finally, we present a scenario that illustrates the interplay and application of our methods, glued under a unique framework for presence and interaction in MR.     

An interactive 3D movement path manipulation method in an augmented reality environment

In this paper, we evaluate a path editing method using a tangible user interface to generate and manipulate the movement path of a 3D object in an Augmented Reality (AR) scene. To generate the movement path, each translation point of a real 3D manipulation prop is examined to determine which point should be used as a control point for the path. Interpolation using splines is then used to reconstruct the path with a smooth line. A dynamic score-based selection method is also used to effectively select small and dense control points of the path. In an experimental evaluation, our method took the same time and generated a similar amount of errors as a more traditional approach, however the number of control points needed was significantly reduced. For control manipulation, the task completion time was quicker and there was less hand movement needed. Our method can be applied to drawing or curve editing methods in AR educational, gaming, and simulation applications.     

Age invaders: social and physical inter-generational mixed reality family entertainment

Age invaders (AI), is a novel interactive intergeneration social-physical game which allows the elderly to play harmoniously together with children in the physical space, while parents can participate in the game play in real time remotely in the virtual world through the internet. Traditional digital games are designed for the young where normally the player sits in front of a computer or game console. Unlike standard computer games, age invaders brings the game play to a physical platform, and requires and encourages physical body movements rather than constraining the user in front of a computer for many hours. Age invaders is an interactive social-physical family digital game designed specially for a harmonious game play between the elderly and young. Adjusting game properties automatically compensates for potential elderly disadvantages, for example slower reaction time and slow movement.     

Design for emergence: experiments with a mixed reality urban playground game

In this paper we present our work in the design of ubiquitous social experiences, aiming to foster group participation and spontaneous playful behaviours in a city environment. We outline our approach of design for emergence: to provide just enough of a game context and challenge for people to be creative, to extend and enrich the experience of play through their interaction in the real world. CitiTag is our mixed reality testbed, a wireless location-based multiplayer game based on the concept of playground ‘tag’. We describe the design and implementation of CitiTag and discuss results from two user studies.     

An augmented reality interface for visualizing and interacting with virtual content

In this paper, a novel AR interface is proposed that provides generic solutions to the tasks involved in augmenting simultaneously different types of virtual information and processing of tracking data for natural interaction. Participants within the system can experience a real-time mixture of 3D objects, static video, images, textual information and 3D sound with the real environment. The user-friendly AR interface can achieve maximum interaction using simple but effective forms of collaboration based on the combinations of human–computer interaction techniques. To prove the feasibility of the interface, the use of indoor AR techniques are employed to construct innovative applications and demonstrate examples from heritage to learning systems. Finally, an initial evaluation of the AR interface including some initial results is presented.     

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.     

Automotive technicians' training as a community-of-practice: implications for the design of an augmented reality teaching aid

The paper presents an ergonomic analysis carried out in the early phases of an R&D project. The purpose was to investigate the functioning of today's Automotive Service Technicians (ASTs) training in order to inform the design of an Augmented Reality (AR) teaching aid. The first part of the paper presents a literature review of some major problems encountered by ASTs today. The benefits of AR as technological aid are also introduced. Then, the methodology and the results of two case studies are presented. The first study is based on interviews with trainers and trainees; the second one on observations in real training settings. The results support the assumption that today's ASTs' training could be regarded as a community-of-practice (CoP). Therefore, AR could be useful as a collaboration tool, offering a shared virtual representation of real vehicle's parts, which are normally invisible unless dismantled (e.g. the parts of a hydraulic automatic transmission). We conclude on the methods and the technologies to support the automotive CoP.     

Virtual Reality： A State-of-the-Art Survey

This paper presents a survey on virtual reality systems and provides an in-depth understanding toward the notion of immersion, according to the semantic meanings of the terms ``virtual' and ``reality'. The paper analyses the structure and functions of a virtual reality system and takes the three dimensional display as the immersive medium to identify the key issues for construction of virtual environments. The paper also reviews the development of virtual reality technology and introduces new image processing techniques into the design of virtual reality systems and virtual environments.     

Interactive mixed reality white cane simulation for the training of the blind and the visually impaired

This paper presents a mixed reality tool developed for the training of the visually impaired based on haptic and auditory feedback. The proposed approach focuses on the development of a highly interactive and extensible Haptic Mixed Reality training system that allows visually impaired to navigate into real size Virtual Reality environments. The system is based on the use of the CyberGrasp™ haptic device. An efficient collision detection algorithm based on superquadrics is also integrated into the system so as to allow real time collision detection in complex environments. A set of evaluation tests is designed in order to identify the importance of haptic, auditory and multimodal feedback and to compare the MR cane against the existing Virtual Reality cane simulation system.     

Interactive theatre via mixed reality and Ambient Intelligence

Our system of Mixed Reality and 3D Live with Ambient Intelligence (AmI) is indented to bring performance art to the people while offering to the performance artists a creative tool to extend the grammar of the traditional theatre. Actors and dancers at different places are captured by multiple cameras and their images are rendered in 3D form in such a way that they can play and dance together on the same place in real-time. Our Quanticum Man is an allegory of the time of the General Relativity and the matter of the Quantum Mechanics. The new type of interactive theatre enables social networking by supporting simultaneous participants in human-to-human social manner.     

Construction of a computer-simulated mixed reality environment for virtual factory layout planning

To survive the cut-throat competition in the manufacturing industry, many companies have introduced digital manufacturing technology. Digital manufacturing technology not only shortens the product development cycle times but also improves the precision of engineering simulation. However, building the virtual objects needed for a digital manufacturing environment requires skilled human resources; it is also costly and time-consuming. A high precision environment with the similar resources is also needed for a high precision simulation. In this paper, we propose a method of constructing a mixed reality-based digital manufacturing environment. The method integrates real objects, such as real images, with the virtual objects of a virtual manufacturing system. This type of integration minimizes the cost of implementing virtual objects and enhances the user's sense of reality. We studied several methods and derived a general framework for the system. Finally, we developed our idea into a virtual factory layout planning system. To assign the pose and position of real objects in virtual space, we applied a circle-based tracking method which uses a safety sign instead of the planar-square-shaped marker generally used for registration. Furthermore, we developed the framework to encapsulate simulation data from legacy data and process data for visualization based on mixed reality.