Towards video-based immersive environments

Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.     

Context-driven interaction in immersive virtual environments

There are many interaction tasks a user may wish to accomplish in an immersive virtual environment. A careful examination of these tasks reveals that they are often performed under different contexts. For each task and context, specialized interaction techniques can be developed. We present the context-driven interaction model: a design pattern that represents contextual information as a first-class, quantifiable component within a user interface and supports the development of context-sensitive applications by decoupling context recognition, context representation, and interaction technique development. As a primary contribution, this model provides an enumeration of important representations of contextual information gathered from across the literature and describes how these representations can effect the selection of an appropriate interaction technique. We also identify how several popular 3D interaction techniques adhere to this design pattern and describe how the pattern itself can lead to a more focused development of effective interfaces. We have constructed a formalized programming toolkit and runtime system that serves as a reference implementation of the context-driven model and a discussion is provided explaining how the toolkit can be used to implement a collection of representative 3D interaction interfaces.     

Using immersive virtual environments for certification

As virtual environments move from research laboratories to commercial use, there is a need to define the appropriate role of VEs in certifying the systems they simulate. This article describes the unique features of this developing technology, as well some key issues to consider when using VEs for certification     

Distance learning immersive environments: Sense of presence exploration

Distance learning technologies are becoming more important for the delivery of content as distance education seems rapidly growing adoption worldwide. These technologies allow collaboration between students and teachers in globally disparate environments. Immersive environments are one recent innovation with significant promise, offering a sense of presence—the sense of “being there”—which can strengthen and reinforce distance learning. The primary purpose of this study is to research the sense of presence using an immersive environment. While this is a work-in-progress, the authors provide preliminary results and conclusions.     

A study of locomotion paradigms for immersive medical simulation environments

Immersive virtual environments are increasingly used for medical training and rehearsal. Immersive environments can provide realistic context for team training, where success relies on practiced coordination between individual members. Using immersive virtual environments, medical teams can practice in situations that would otherwise be difficult or expensive to create. It has been shown that individuals perform poorly when the training environment differs significantly from practice 2005. Efforts have been made to close this gap using virtual environments. Interacting in a virtual space requires a robust locomotion paradigm. Locomotion paradigms are methods that allow an individual to move and navigate through virtual environments. Locomotion paradigms should be intuitive to the user, and not distract from the central task of medical training. In this paper, we describe and evaluate four locomotion paradigms, Look & Go, Push & Go, Point & Go, and Grab & Drag, using objective metrics to evaluate navigational efficiency. This study was performed with 98 volunteers predominantly of clinical backgrounds. With the comparison between the performances of game-playing and non-game-playing subjects, we have shown that game-playing experiences do not significantly affect the locomotion performances with the four proposed paradigms. The results of this study suggests the Grab & Drag as the best method among four locomotion paradigms in triage/trauma scenarios, where trainees need to find and help patients scattered in a large area.     

Studying the effectiveness of multi-user immersive environments for collaborative evaluation tasks

Massively Multiuser On-line Learning (MMOL) Platforms, often called “virtual learning worlds”, constitute a still unexplored context for communication-enhanced learning, where synchronous communication skills in an explicit social setting enhance the potential of effective collaboration. In this paper, we report on an experimental study of collaborative evaluation in an MMOL setting with 21 graduate students enrolled in university courses in technology-mediated teaching and learning. This study was carried out using a prototype of a 3D MMOL platform built around an interactive space called “MadriPolis”. This space was used to recreate an adequate scenario for a collaborative experience about Learning Object evaluation using the mainstream Learning Object Review Instrument (LORI), which is based on a Convergent Participation Model (CPM). The same experience was carried out using a conventional LCMS (Learning Content Management System) platform with the aim of contrasting the outcomes and interaction patterns in the two settings. This study makes use of Social Network Analysis (SNA) measures to describe the interactions between tutors and learners. By dwelling on the advantages of immersive environments, SNA indexes revealed that these interactions were rather dense and that student participation was rather broad-based in the case of the MMOL. The results suggest that MMOL platforms could be used in collaborative evaluation tasks as a means to enhance both tutor interaction patterns and the strength of the group's relationship.     

High-resolution multiprojector display walls

Meeting the demands of the current tera-scale visualization community requires the use of large- and small-scale tiled displays. The Lawrence Livermore National Laboratory (LLNL) efforts in this area have led to the creation of one of the largest interactive tiled displays built to date, and a number of new efforts for “personal” tiled displays. Visualization hardware and software now being built will display up to 15 times the number of pixels in a typical desktop display. We outline the system implemented at LLNL for the creation and support of large, tiled, multi-pipe graphics displays. The system builds on a simple, portable, parallel API for tiling OpenGL commands in parallel to multiple contexts on a set of X servers. This API has allowed us to retrofit existing visualization and analysis codes to support these displays with very little effort. While this work represents a first step toward the ultimate goal of scalable visualization in individual office spaces, the result has been the creation of useful new visualization workspaces and tools for our users     

Redirecting walking and driving for natural navigation in immersive virtual environments

Walking is the most natural form of locomotion for humans, and real walking interfaces have demonstrated their benefits for several navigation tasks. With recently proposed redirection techniques it becomes possible to overcome space limitations as imposed by tracking sensors or laboratory setups, and, theoretically, it is now possible to walk through arbitrarily large virtual environments. However, walking as sole locomotion technique has drawbacks, in particular, for long distances, such that even in the real world we tend to support walking with passive or active transportation for longer-distance travel. In this article we show that concepts from the field of redirected walking can be applied to movements with transportation devices. We conducted psychophysical experiments to determine perceptual detection thresholds for redirected driving, and set these in relation to results from redirected walking. We show that redirected walking-and-driving approaches can easily be realized in immersive virtual reality laboratories, e. g., with electric wheelchairs, and show that such systems can combine advantages of real walking in confined spaces with benefits of using vehicle-based self-motion for longer-distance travel.     

Camera Autocalibration and Horopter Curves

We describe a new algorithm for the obtainment of the affine and Euclidean calibration of a camera under general motion. The algorithm exploits the relationships of the horopter curves associated to each pair of cameras with the plane at infinity and the absolute conic. Using these properties we define cost functions whose minimization by means of general purpose techniques provides the required calibration. The experiments show the good convergence properties, computational efficiency and robust performance of the new techniques.     

A methodology for the evaluation of travel techniques for immersive virtual environments

We present a framework for the analysis and evaluation oftravel, or viewpoint motion control, techniques for use in immersive virtual environments (VEs). In previous work, we presented a taxonomy of travel techniques and a set of experiments mapping parts of the taxonomy to various performance metrics. Since these initial experiments, we have expanded the framework to allow evaluation of not only the effects of different travel techniques, but also the effects of many outside factors simultaneously. Combining this expanded framework with the measurement of multiple response variables epitomises the philosophy oftestbed evaluation. This experimental philosophy leads to a deeper understanding of the interaction and the technique(s) in question, as well as to broadly generalisable results. We also present an example experiment within this expanded framework, which evaluates the user's ability to gather information while travelling through a virtual environment. Results indicate that, of the variables tested, the complexity of the environment is by far the most important factor.     

Rich socio-cognitive agents for immersive training environments: case of NonKin Village

Demand is on the rise for scientifically based human-behavior models that can be quickly customized and inserted into immersive training environments to recreate a given society or culture. At the same time, there are no readily available science model-driven environments for this purpose (see survey in Sect. 2). In researching how to overcome this obstacle, we have created rich (complex) socio-cognitive agents that include a large number of social science models (cognitive, sociologic, economic, political, etc) needed to enhance the realism of immersive, artificial agent societies. We describe current efforts to apply model-driven development concepts and how to permit other models to be plugged in should a developer prefer them instead. The current, default library of behavioral models is a metamodel, or authoring language, capable of generating immersive social worlds. Section 3 explores the specific metamodels currently in this library (cognitive, socio-political, economic, conversational, etc.) and Sect. 4 illustrates them with an implementation that results in a virtual Afghan village as a platform-independent model. This is instantiated into a server that then works across a bridge to control the agents in an immersive, platform-specific 3D gameworld (client). Section 4 also provides examples of interacting in the resulting gameworld and some of the training a player receives. We end with lessons learned and next steps for improving both the process and the gameworld. The seeming paradox of this research is that as agent complexity increases, the easier it becomes for the agents to explain their world, their dilemmas, and their social networks to a player or trainee.     

An evaluation of immersive viewing on spatial knowledge acquisition in spherical panoramic environments

We report the results of an experiment conducted to examine the effects of immersive viewing on a common spatial knowledge acquisition task of spatial updating task in a spherical panoramic environment (SPE). A spherical panoramic environment, such as Google Street View, is an environment that is comprised of spherical images captured at regular intervals in a real world setting augmented with virtual navigational aids such as paths, dynamic maps, and textual annotations. Participants navigated the National Mall area of Washington, DC, in Google Street View in one of two viewing conditions; desktop monitor or a head-mounted display with a head orientation tracker. In an exploration phase, participants were first asked to navigate and observe landmarks on a pre-specified path. Then, in a testing phase, participants were asked to travel the same path and to rotate their view in order to look in the direction of the perceived landmarks at certain waypoints. The angular difference between participants’ gaze directions and the landmark directions was recorded. We found no significant difference between the immersive and desktop viewing conditions on participants’ accuracy of direction to landmarks as well as no difference in their sense of presence scores. However, based on responses to a post-experiment questionnaire, participants in both conditions tended to use a cognitive or procedural technique to inform direction to landmarks. Taken together, these findings suggest that in both conditions where participants experience travel based on teleportation between waypoints, the visual cues available in the SPE, such as street signs, buildings and trees, seem to have a stronger influence in determining the directions to landmarks than the egocentric cues such as first-person perspective and natural head-coupled motion experienced in the immersive viewing condition.     

Examining the effect of body ownership in immersive virtual and augmented reality environments

The traditional rubber hand illusion is a psychological experiment where participants are under the illusion that a rubber hand is part of their own body. This paper examines the use of real, virtual and augmented reality environments for identifying the elements that influence body ownership in healthy participants. Compared to the classical experiment where a plastic rubber hand was used, a realistic 3D representation was chosen to create the same illusion this time in both immersive virtual reality and augmented reality. Experiments were performed on 30 volunteers undergoing testing session composed of three stages. Participants were asked to complete two different questionnaires, one measuring their cognitive workload and another one regarding their experience with the rubber hand illusion. In addition, EEG signals of the individuals were recorded, resulting in 90 electroencephalogram datasets. Results indicate correlations between ownership statements with beta and gamma electroencephalogram bands in premotor cortex activity. Link between higher gamma production in ventral premotor area during the illusion was established in previous studies.     

User experimentation: an evaluation of velocity control techniques in immersive virtual environments

While many of the existing velocity control techniques are well designed, the techniques are often application-specific, making it difficult to compare their effectiveness. In this paper, we evaluate five known velocity control techniques using the same experimental settings. We compare the techniques based on the assumption that a good travel technique should be easy to learn and easy to use, should cause the user to have few collisions with the VE, should allow the user to complete tasks faster, and should promote better recollection of the environment afterwards. In our experiments, we ask twenty users to use each velocity control technique to navigate through virtual corridors while performing information-gathering tasks. In all cases, the users use pointing to indicate the direction of travel. We then measure the users’ ability to recollect the information they see in the VE, as well as how much time they spend in the VE and how often they collide with the virtual walls. After each test, we use questionnaires to evaluate the ease of learning and ease of use of the velocity control technique, and the users’ sense of presence in the environment. Each of the travel techniques is then evaluated based on the users’ performances in the VE and the results of their questionnaires.     

Autocalibration of a projector-camera system

This paper presents a method for calibrating a projector-camera system that consists of multiple projectors (or multiple poses of a single projector), a camera, and a planar screen. We consider the problem of estimating the homography between the screen and the image plane of the camera or the screen-camera homography, in the case where there is no prior knowledge regarding the screen surface that enables the direct computation of the homography. It is assumed that the pose of each projector is unknown while its internal geometry is known. Subsequently, it is shown that the screen-camera homography can be determined from only the images projected by the projectors and then obtained by the camera, up to a transformation with four degrees of freedom. This transformation corresponds to arbitrariness in choosing a two-dimensional coordinate system on the screen surface and when this coordinate system is chosen in some manner, the screen-camera homography as well as the unknown poses of the projectors can be uniquely determined. A noniterative algorithm is presented, which computes the homography from three or more images. Several experimental results on synthetic as well as real images are shown to demonstrate the effectiveness of the method.     

The effect of visual and interaction fidelity on spatial cognition in immersive virtual environments

Accuracy of memory performance per se is an imperfect reflection of the cognitive activity (awareness states) that underlies performance in memory tasks. The aim of this research is to investigate the effect of varied visual and interaction fidelity of immersive virtual environments on memory awareness states. A between groups experiment was carried out to explore the effect of rendering quality on location-based recognition memory for objects and associated states of awareness. The experimental space, consisting of two interconnected rooms, was rendered either flat-shaded or using radiosity rendering. The computer graphics simulations were displayed on a stereo head-tracked head mounted display. Participants completed a recognition memory task after exposure to the experimental space and reported one of four states of awareness following object recognition. These reflected the level of visual mental imagery involved during retrieval, the familiarity of the recollection, and also included guesses. Experimental results revealed variations in the distribution of participants' awareness states across conditions while memory performance failed to reveal any. Interestingly, results revealed a higher proportion of recollections associated with mental imagery in the flat-shaded condition. These findings comply with similar effects revealed in two earlier studies summarized here, which demonstrated that the less "naturalistic" interaction interface or interface of low interaction fidelity provoked a higher proportion of recognitions based on visual mental images.     

Camera-based calibration techniques for seamless multiprojector displays

Multiprojector, large-scale displays are used in scientific visualization, virtual reality, and other visually intensive applications. In recent years, a number of camera-based computer vision techniques have been proposed to register the geometry and color of tiled projection-based display. These automated techniques use cameras to "calibrate" display geometry and photometry, computing per-projector corrective warps and intensity corrections that are necessary to produce seamless imagery across projector mosaics. These techniques replace the traditional labor-intensive manual alignment and maintenance steps, making such displays cost-effective, flexible, and accessible. In this paper, we present a survey of different camera-based geometric and photometric registration techniques reported in the literature to date. We discuss several techniques that have been proposed and demonstrated, each addressing particular display configurations and modes of operation. We overview each of these approaches and discuss their advantages and disadvantages. We examine techniques that address registration on both planar (video walls) and arbitrary display surfaces and photometric correction for different kinds of display surfaces. We conclude with a discussion of the remaining challenges and research opportunities for multiprojector displays     

Towards user centred design (UCD) in architecture based on immersive virtual environments

This paper describes a generic concept of how to combine the experience of user centred design (UCD) in the field of Human Computer Interaction (HCI) with the traditional approach of participatory design (PD) in an architectural design process. Even if some basic requirements of this generic method are not available yet, this paper will also describe an approach, which enables planners even now to involve end users by using virtual environments (VE) as immersive and spatial prototype. It will be described and illustrated by the way of example using the building project Centre of Virtual Engineering of the Fraunhofer Institute for Industrial Engineering (IAO) in Stuttgart. It demonstrates that the transfer of the UCD approach to architectural planning combined with the provision of an adequate prototype can make a significant contribution towards an increase in quality and performance in building and construction projects.     

Designing presence for real locomotion in immersive virtual environments: an affordance-based experiential approach

This paper describes a framework for designing systems for real locomotion in virtual environments (VEs) in order to achieve an intense sense of presence. The main outcome of the present research is a list of design features that the virtual reality technology should have in order to achieve such a goal. To identify these features, an approach based on the combination of two design strategies was followed. The first was based on the theory of affordances and was utilized to design a generic VE in which the affordances of the corresponding real environment could be evoked. The second was the experiential design applied to VEs and was utilized to create an experience of locomotion corresponding to that achievable in a real environment. These design strategies were chosen because of their potential to enhance the sense of presence. The proposed list of features can be utilized as an instrument that allows VE designers to evaluate the maturity of their systems and to pinpoint directions for future developments. A survey analysis was performed using the proposed framework, which involved three case studies to determine how many features of the proposed framework were present and their status. The result of such analysis represented a measure of the completeness of the systems design, of the affordances provided to the user, and a prediction of the sense of presence.