Mixed reality storytelling environments

The focus of this paper is the collaborative storytelling environments created as part of a three-year multi-disciplinary research project, KidStory. The project team, from the UK, Sweden and the USA, developed a series of story creation and telling tools and virtual environments for children aged 5–7 years. This paper concerns work with a Nottingham primary school to design and develop collaborative storytelling tools, including tangible interfaces devices and reactive spaces, with the aim of integrating these within the school context. The final set-up allowed children to dynamically produce story content, to create basic narrative structures and to retell their stories in a collaborative and adaptable physical space. Acknowledgements.This work has been conducted as part of the EU Esprit 29310 KidStory project. The authors acknowledge support from all partners (SICS/Maryland/KTH). Thanks also to the teachers and children at the primary school in Nottingham for their collaboration in this work. To download KidPad see http://www.kidpad.org.     

A method of creating a transparent space based on mixed reality

In this article, we propose a novel idea to create a transparent space. Accidents often happen when a person, a motorcycle, a car, etc., comes out suddenly from a blind corner. In such a place, if the shielding objects become transparent, hidden objects reveal themselves. This may contribute to preventing a lot of accidents. The proposed method creates a transparent space by the employment of a mixed-reality technique and a simple three-dimensional (3-D) recovery technique. The 3-D virtual space is created by a single camera image in real-time, and it is superposed onto a real space employing see-through vision. The observer can watch a dynamic scene of the invisible space through the object in real-time. In the experiment performed, an observer could watch persons walking around in a space hidden by an opaque partition. Future application of this technique may include its installation at various corners in buildings or at intersections on roads with bad visibility. This would surely contribute to reducing traffic accidents in future. This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006     

Storing user experiences in mixed reality using hypermedia

This paper describes HyperMem, a system to store and replay user experiences in mixed environments. The experience is stored as a set of hypermedia nodes and links, with the information that was displayed along with the video of the real world that was navigated. It uses a generic hypermedia model, implemented as software components, to handle mixed reality environments. This model includes components for storing and replaying experiences and integrating them in the overall set of hypermedia graphs that can be accessed by a given user. The paper presents the goals of the system, the underlying hypermedia model, the application scenarios, and the architecture and tools for replaying and repurposing stored information.     

Virtual Bounds: a teleoperated mixed reality

This paper introduces a mixed reality workspace that allows users to combine physical and computer-generated artifacts, and to control and simulate them within one fused world. All interactions are captured, monitored, modeled and represented with pseudo-real world physics. The objective of the presented research is to create a novel system in which the virtual and physical world would have a symbiotic relationship. In this type of system, virtual objects can impose forces on the physical world and physical world objects can impose forces on the virtual world. Virtual Bounds is an exploratory study allowing a physical probe to navigate a virtual world while observing constraints, forces, and interactions from both worlds. This scenario provides the user with the ability to create a virtual environment and to learn to operate real-life probes through its virtual terrain.     

Interactive theatre via mixed reality and Ambient Intelligence

Our system of Mixed Reality and 3D Live with Ambient Intelligence (AmI) is indented to bring performance art to the people while offering to the performance artists a creative tool to extend the grammar of the traditional theatre. Actors and dancers at different places are captured by multiple cameras and their images are rendered in 3D form in such a way that they can play and dance together on the same place in real-time. Our Quanticum Man is an allegory of the time of the General Relativity and the matter of the Quantum Mechanics. The new type of interactive theatre enables social networking by supporting simultaneous participants in human-to-human social manner.     

Age invaders: social and physical inter-generational mixed reality family entertainment

Age invaders (AI), is a novel interactive intergeneration social-physical game which allows the elderly to play harmoniously together with children in the physical space, while parents can participate in the game play in real time remotely in the virtual world through the internet. Traditional digital games are designed for the young where normally the player sits in front of a computer or game console. Unlike standard computer games, age invaders brings the game play to a physical platform, and requires and encourages physical body movements rather than constraining the user in front of a computer for many hours. Age invaders is an interactive social-physical family digital game designed specially for a harmonious game play between the elderly and young. Adjusting game properties automatically compensates for potential elderly disadvantages, for example slower reaction time and slow movement.     

Interactive mixed reality white cane simulation for the training of the blind and the visually impaired

This paper presents a mixed reality tool developed for the training of the visually impaired based on haptic and auditory feedback. The proposed approach focuses on the development of a highly interactive and extensible Haptic Mixed Reality training system that allows visually impaired to navigate into real size Virtual Reality environments. The system is based on the use of the CyberGrasp™ haptic device. An efficient collision detection algorithm based on superquadrics is also integrated into the system so as to allow real time collision detection in complex environments. A set of evaluation tests is designed in order to identify the importance of haptic, auditory and multimodal feedback and to compare the MR cane against the existing Virtual Reality cane simulation system.     

Conceptualising mixed spaces of interaction for designing continuous interaction

Recent progress in the overlay and registration of digital information on the users workspace in a spatially meaningful way has allowed mixed reality (MR) to become a more effective operational medium. However, research in software structures, design methods and design support tools for MR systems is still in its infancy. In this paper, we propose a conceptual classification of the design space to support the development of MR systems. The proposed design space (DeSMiR) is an abstract tool for systematically exploring several design alternatives at an early stage of interaction design, without being biassed towards a particular modality or technology. Once the abstract design possibilities have been identified and a concrete design decision has been taken (i.e. a specific modality has been selected), a concrete MR application can be considered in order to analyse the interaction techniques in terms of continuous interaction properties. We suggest that our design space can be applied to the design of several kinds of MR applications, especially those in which very little user focus distraction can be tolerated, and where smooth connections and interactions between real and virtual worlds is critical for the system development. An image-guided surgery system (IGS) is used as a case study.

Tangled reality

Leonardo da Vinci was a strong advocate for using sketches to stimulate the human imagination. Sketching is often considered to be integral to the process of design, providing an open workspace for ideas. For these same reasons, children use sketching as a simple way to express visual ideas. By merging the abstraction of human drawings and the freedom of virtual reality with the tangibility of physical tokens, Tangled Reality creates a rich mixed reality workspace. Tangled Reality allows users to build virtual environments based on simple colored sketches and traverse them using physical vehicles overlayed with virtual imagery. This setup allows the user to “build” and “experience” mixed reality simulations without ever touching a standard computer interface.     

Mobile phone-based mixed reality: the Snap2Play game

The ubiquity of camera phones provides a convenient platform to develop immersive mixed-reality games. In this paper we introduce such a game which is loosely based on the popular card game “Memory”, where players are asked to match a pair of identical cards among a set of overturned cards by revealing only two cards at a time. In our game, the players are asked to match a “digital card”, which corresponds to a scene in a virtual world, to a “physical card”, which is an image of a scene in the real world. The objective is to convey a mixed-reality sensation. Cards are matched with a scene identification engine which consists of multiple classifiers trained on previously collected images. We present our comprehensive overall game design, as well as implementation details and results. We also describe how we constructed our scene identification engine and its performance. Finally, we present an analysis of player surveys to gauge the potential market acceptance.

Afforded actions as a behavioral assessment of physical presence in virtual environments

A particular affordance was used as a potential candidate for behavioral assessment of physical presence in virtual environments. The subjects’ task was to walk through a virtual aperture of variable widths. In the case of presence, the subjects’ body orientation, while walking, was hypothesized to be adapted to the width of the aperture and to their own shoulder width. Results show that most subjects adapted their behavior to both their body architecture and the virtual width constraints. These subjects exhibited a behavioral transition from frontal walking to body rotation while walking through broad to narrow apertures. The same behavioral transition has already been documented in real environments (Warren and Whang in J Exp Psychol Human Percept Perform 13(3):371–383, 1987). This behavioral adjustment is thus assumed to be an objective indication of presence. Beyond these results, the present study suggests that every afforded action could be a potential tool for sensorimotor assessment of physical presence. Parts of this research have been first presented at the 11th Annual International Workshop of Presence, Padova, Italy, 16-18 October 2008.     

Absorption,dissociation, locus of control and presence in virtual reality

The present study aimed to explore the relationship between a number of psychological variables and a reported sense of presence in immersive virtual reality (IVR). It was hypothesised that participants’ scores on measures of absorption, dissociation, and external locus of control would be positively and significantly correlated with a measure of their sense of presence in IVR. A total of 64 people took part. Significant correlations were found between presence and dissociation (r = 0.403, p < 0.01), and presence and locus of control (r = 0.268, p < 0.05). However, the correlation between presence and absorption was not significant (r = −0.037, p = 0.386). The findings reported here suggest a complex interrelationship of psychological variables in relation to presence in IVR that warrants further research.     

Social cooperation and competition in the mixed reality space eXperience Induction Machine XIM

Although the architecture of mixed reality spaces is becoming increasingly more complex, our understanding of human behavior in such spaces is still limited. Despite the sophisticated methods deployed in ethology and behavioral biology to track and analyze the actions and movements of animals, we rarely find studies that focus on the understanding of human behavior using such instruments. Here, we address this issue by analyzing social behavior and physical actions of multiple humans who are engaging in a game. As a paradigm of social interaction, we constructed a mixed reality football game in which two teams of two players have to cooperate and compete in order to win. This paradigm was deployed in the, so-called, eXperience Induction Machine (XIM), a human accessible, fully instrumented space that supports full body interaction in mixed reality without the need for body-mounted sensors. Our results show that winning and losing strategies can be discerned by specific behavioral patterns and proxemics. This demonstrates that mixed reality systems such as XIM provide new paradigms for the investigation of human social behavior.     

Hybrid ecologies: understanding cooperative interaction in emerging physical-digital environments

We consider the emergence of hybrid ecologies, which marry mixed reality environments and ubiquitous computing environments together to bridge the physical-digital divide. Hybrid ecologies are new class of digital ecology that merge multiple environments, physical and digital, together. Collaboration in these emerging environments is characterized by ‘fragmented interaction’ in that it is mediated by interaction mechanisms that are differentially distributed. Unpacking the collaborative nature of fragmented interaction requires that we uncover the ordinary interactional competences that users exploit to make differentially distributed mechanisms of interaction work and the distributed practices that articulate ‘seamful’ representations and provide for awareness and coordination.

Design for emergence: experiments with a mixed reality urban playground game

In this paper we present our work in the design of ubiquitous social experiences, aiming to foster group participation and spontaneous playful behaviours in a city environment. We outline our approach of design for emergence: to provide just enough of a game context and challenge for people to be creative, to extend and enrich the experience of play through their interaction in the real world. CitiTag is our mixed reality testbed, a wireless location-based multiplayer game based on the concept of playground ‘tag’. We describe the design and implementation of CitiTag and discuss results from two user studies.     

Construction of a computer-simulated mixed reality environment for virtual factory layout planning

To survive the cut-throat competition in the manufacturing industry, many companies have introduced digital manufacturing technology. Digital manufacturing technology not only shortens the product development cycle times but also improves the precision of engineering simulation. However, building the virtual objects needed for a digital manufacturing environment requires skilled human resources; it is also costly and time-consuming. A high precision environment with the similar resources is also needed for a high precision simulation. In this paper, we propose a method of constructing a mixed reality-based digital manufacturing environment. The method integrates real objects, such as real images, with the virtual objects of a virtual manufacturing system. This type of integration minimizes the cost of implementing virtual objects and enhances the user's sense of reality. We studied several methods and derived a general framework for the system. Finally, we developed our idea into a virtual factory layout planning system. To assign the pose and position of real objects in virtual space, we applied a circle-based tracking method which uses a safety sign instead of the planar-square-shaped marker generally used for registration. Furthermore, we developed the framework to encapsulate simulation data from legacy data and process data for visualization based on mixed reality.     

Using virtual reality technology for aircraft visual inspection training: presence and comparison studies

The aircraft maintenance industry is a complex system consisting of several interrelated human and machine components. Recognizing this, the Federal Aviation Administration (FAA) has pursued human factors related research. In the maintenance arena the research has focused on the aircraft inspection process and the aircraft inspector. Training has been identified as the primary intervention strategy to improve the quality and reliability of aircraft inspection. If training is to be successful, it is critical that we provide aircraft inspectors with appropriate training tools and environment. In response to this need, the paper outlines the development of a virtual reality (VR) system for aircraft inspection training.

VR has generated much excitement but little formal proof that it is useful. However, since VR interfaces are difficult and expensive to build, the computer graphics community needs to be able to predict which applications will benefit from VR. To address this important issue, this research measured the degree of immersion and presence felt by subjects in a virtual environment simulator. Specifically, it conducted two controlled studies using the VR system developed for visual inspection task of an aft-cargo bay at the VR Lab of Clemson University. Beyond assembling the visual inspection virtual environment, a significant goal of this project was to explore subjective presence as it affects task performance. The results of this study indicated that the system scored high on the issues related to the degree of presence felt by the subjects. As a next logical step, this study, then, compared VR to an existing PC-based aircraft inspection simulator. The results showed that the VR system was better and preferred over the PC-based training tool.     

Virtual interpersonal touch: Haptic interaction and copresence in collaborative virtual environments

As digital communication becomes more commonplace and sensory rich, understanding the manner in which people interact with one another is crucial. In the current study, we examined the manners in which people touch digital representations of people, and compared those behaviors to the manner in which they touch digital representations of nonhuman objects. Results demonstrated that people used less force when touching people than other nonhuman objects, and that people touched the face with less force than the torso area. Finally, male digital representations were touched with more force than female representations by subjects of both genders. We discuss the implications of these data to the development of haptic communication systems as well as for a methodology of measuring the amount of copresence in virtual environments.