Modeling immersive media experiences by sensing impact on subjects

As immersive technologies target to provide higher quality of multimedia experiences, it is important to understand the quality of experience (QoE) perceived by users from various multimedia rendering schemes, in order to design and optimize human-centric immersive multimedia systems. In this study, various QoE-related aspects, such as depth perception, sensation of reality, content preference, and perceived quality are investigated and compared for presentation of 2D and 3D contents. Since the advantages of implicit over explicit QoE assessment have become essential, the way these QoE-related aspects influence brain and periphery is also investigated. In particular, two classification schemes using electroencephalography (EEG) and peripheral signals (electrocardiography and respiration) are carried out, to explore if it is possible to automatically recognize the QoE-related aspects under investigation. In addition, a decision-fusion scheme is applied to EEG and peripheral features, to explore the advantage of integrating information from the two modalities. The results reveal that the highest monomodal average informedness is achieved in the high beta EEG band (0.14 % ± 0.09, p < 0.01), when recognizing sensation of reality. The highest and significantly non-random multimodal average informedness is achieved when high beta EEG band is fused with peripheral features (0.17 % ± 0.1, p < 0.01), for the case of sensation of reality. Finally, a temporal analysis is conducted to explore how the EEG correlates for the case of sensation of reality change over time. The results reveal that the right cortex is more involved when sensation of reality is low, and the left when sensation of reality is high, indicating that approach and withdrawal-related processes occur during sensation of reality.     

The immersive visualization probe for exploring n-dimensional spaces

A new tool uses two types of dimensional navigation to display continuous 4D subsets of n-dimensional data. Thanks to the tool's embedded coordinate systems, researchers can better understand a model's underlying physical or mathematical process. Here, we describe a new method for visualizing data structure or models defined in higher-dimensional spaces. This technique is suitable for applications in which a scalar function, defined mathematically or procedurally, depends on n variables or parameters. The function's values essentially describe a set of points in n-dimensional space. To visualize these sets, we fix all but three of the parameters and then sample the resulting 4D set (the three parameters and the function's resulting value) on several discrete grids located on planes in the 3D space. We compose the image by using color to represent the fourth dimension (the function's value) at discrete locations on these grids. Interactive control over the way the parameters are fixed results in a highly dynamic system that researchers can easily use to explore the n-dimensional space's structure.     

Human factors in the design of an immersive display

In 1988 Fakespace began building a telepresence camera system for the Virtual Environment Workstation (View) project at NASA Ames Research Center. The complete system combined a teleoperated camera platform and 3D viewing system. Fakespace has installed descendents of this system for day-to-day use in environments ranging from research laboratories to office cubicles. By observing how people use image-generation and viewing technologies, we have evaluated the human factors involved in building and deploying effective immersive visualization systems. This article describes the application of these factors in the design of the Fakespace BOOM (Binocular Omni-Orientation Monitor). The Fakespace BOOM represents a class of immersive display devices known as counterbalanced displays. The choices that led to the design of this device apply to the design of many types of immersive displays     

Towards 6DoF live video streaming system for immersive media

Based on the three rotational degrees (video in three dimensions, on the X, Y and Z axes) of freedom provided by VR, the viewer is free to control the viewing point and has six degrees of freedom (6DoF). When watching a sports game, the audience is no longer limited by the position of the camera, and can freely choose the viewing angle and position just like watching in the real world, which can greatly improve the immersion of viewing. However, the major barrier that prevents 6DoF video live from being industrialized lies in the extremely high computational complexity, of which multi-view depth estimation and Depth Image Based Rendering (DIBR) is difficult to realize. And existing devices do not have hardware interfaces that support multi-views coding technology. Therefore, we need new technologies for depth estimation and virtual view synthesis, and we need to use existing hardware coding/decoding interfaces to reduce power consumption. In this paper, we provide a 6DoF live video system, which includes multi-view depth estimation technique based on unsupervised learning, virtual viewpoint real-time rendering technology and 6DoF video coding. Experimental results demonstrate that our proposed acceleration method can speed up the original depth estimation algorithm by more than 34x, and can speed up the original DIBR algorithm by more than 168x. With our 6DoF video coding method, experimental results show that the bit rate achieves an average of 70%, 64%, 33%, 60% and 66% bitrate saving for AVC, HEVC, AV1, AVS3, VVC codec standard respectively.

     

Collaborative immersive authoring tool for real-time creation of multisensory VR experiences

Multimedia Tools and Applications - With the appearance of innovative virtual reality (VR) technologies, the need to create immersive content arose. Although there are already some non-immersive...     

Ambient media for peripheral information display

In our everyday lives, we use surrounding living and working environments to arrange physical information artefacts such as books, pictures and calendars. However, when it comes to consumption of computer-based information, this is almost entirely based on attending to screens that separate us from our surroundings. In this paper, we explore the augmentative use of non-computer artefacts in our surroundings for peripheral display of digital information. We discuss system integration of such ambient media, the mapping of information to ambient media, and issues of flexibility and user control. Ambient media have been in everyday use in our work environment, and we report observations from which we conclude their utility as extensions to our digital information spaces.     

Image modification for immersive projection display based on pseudo‐projection models

This paper describes a practical method that enables actual images to be converted so that they can be projected onto an immersive projection display (IPD) screen. IPD screens are particularly unique in that their angle of view is extremely wide; therefore, the images projected onto them need to be taken on a special format. In practice, however, it is generally very difficult to shoot images that completely satisfy the specifications of the targeting IPD environment due to cost, technical problems or other reasons. To overcome these problems, we developed a method to modify the images by abandoning geometrical consistency. We were able to utilize this method by assuming that the given image was shot according to a special projection model. Because this model differed from the actual projection model with which the image was taken, we termed it the pseudo‐projection model. Since our method uses simple geometry, and can easily be expressed by a parametric function, the degree of modification or the time sequence for modification can readily be adjusted according to the features of each type of content. Copyright © 2003 John Wiley & Sons, Ltd.     

Labyrinth 3D. Cultural archetypes for exploring media archives

Since the advent of the World Wide Web, online media archives have changed their audience from a restricted number of professionals and amateurs to the general public. This shift is not without consequences: if, on the one side, it represents an important opportunity for archives to engage in a dialogue with a larger audience, on the other side, it advocates novel forms of access that go beyond the highly specialized models underlying traditional access tools. In this paper, we propose to use 3-D graphics for designing novel tools of exploratory search in cultural heritage archives. Our approach has been deployed as an online virtual environment where the user can navigate the meaning relations over the items in the archive. Targeted at cultural heritage, the application, called Labyrinth 3D, relies on the use of cultural archetypes to inform the conceptualization of the archive and the creation of the virtual environment, with the goal of engaging the user in the exploration of the archive through the creation of personal paths.     

Staggered striping: A flexible technique to display continuous media

During the past decade, information technology has evolved to where it is economically viable to store and retrieve continuous media data types, e.g., audio and video objects. Systems that support this data type are expected to play a major role in many applications including library information systems, entertainment technology, and educational applications. The objects of this new data type should be retrieved at a pre-specified bandwidth. If an object is retrieved at a rate lower than its pre-specified bandwidth then its display will suffer from frequent disruptions and delays termed hiccups.This paper describes staggered striping as a novel technique to support a hiccup-free retrieval of continuous media data types. Its design is based on a multidisk architecture. It is a flexible technique that can support media types whose bandwidth requirements are either lower or higher than the bandwidth of a single disk. Its design allows a system to scale to thousands of disk drives as its incurred overhead is fixed and does not increase as a function of additional disk drives.This research was supported in part by grants from AT&T/NCR/Teradata, Hewlett-Packard, IBM grant SJ92488, NSF grants IRI-9110522, IRI-9203389, and CDA-9216321 and DARPA contract DABT63-91-C-0001. An earlier version of this paper appeared in Proceedings of SIGMOD '94.     

Cognitive load in immersive media settings: the role of spatial presence and cybersickness

Virtual Reality - Faced with the ongoing diversification and commercial success of highly immersive media technologies (e.g., VR headsets), both content producers and scientific scholars have...     

A framework for fidelity evaluation of immersive virtual reality systems

Virtual Reality - Developments in visual and tracking systems have expanded virtual reality (VR) applications and led to VR becoming a powerful tool for decision making, planning, and conducting...     

VirSchool: The effect of background music and immersive display systems on memory for facts learned in an educational virtual environment

Game technology has been widely used for educational applications, however, despite the common use of background music in games, its effect on learning has been largely unexplored. This paper discusses how music played in the background of a computer-animated history lesson affected participants’ memory for facts. A virtual history lesson was presented to participants with different background stimuli (music or no-music) to test the effect of music on memory. To test the role of immersion on memory and its possible relationship to the music, two different display systems (3-monitor display system or immersive Reality Center) were used in the study. Overall, participants remembered a significantly higher number of facts using the 3-monitor display system, particularly if no background music was played in the second half of the history lesson. Conversely, for participants using the Reality Center, significantly higher recall of facts was found when participants listened to music in the second half of the history lesson. Cognitive load/overload and (un-)familiarity with the technology are offered as explanations.     

A cooperative robotic platform for adaptive and immersive artistic installations

In this paper we shall review the major approaches done in the past concerning the development of immersive artistic installations. This will permit one to identify the basic features to be attained by this kind of installations and based on them a new cooperative and autonomous robotic platform will be proposed. The major novelty of the approach presented in this paper relies on its short- and long-term adaptation capabilities. The technical details of its basic building blocks and the results of the preliminary field tests will be provided, showing that it constitutes a very efficient platform for the construction of actual adaptive and immersive artistic installations.     

A study of locomotion paradigms for immersive medical simulation environments

Immersive virtual environments are increasingly used for medical training and rehearsal. Immersive environments can provide realistic context for team training, where success relies on practiced coordination between individual members. Using immersive virtual environments, medical teams can practice in situations that would otherwise be difficult or expensive to create. It has been shown that individuals perform poorly when the training environment differs significantly from practice 2005. Efforts have been made to close this gap using virtual environments. Interacting in a virtual space requires a robust locomotion paradigm. Locomotion paradigms are methods that allow an individual to move and navigate through virtual environments. Locomotion paradigms should be intuitive to the user, and not distract from the central task of medical training. In this paper, we describe and evaluate four locomotion paradigms, Look & Go, Push & Go, Point & Go, and Grab & Drag, using objective metrics to evaluate navigational efficiency. This study was performed with 98 volunteers predominantly of clinical backgrounds. With the comparison between the performances of game-playing and non-game-playing subjects, we have shown that game-playing experiences do not significantly affect the locomotion performances with the four proposed paradigms. The results of this study suggests the Grab & Drag as the best method among four locomotion paradigms in triage/trauma scenarios, where trainees need to find and help patients scattered in a large area.     

A new indexing method for complex similarity queries in immersive multimedia systems

This paper proposes a novel indexing method for complex similarity queries in high-dimensional image and video systems. In order to provide the indexing method with the flexibility in dealing with multiple features and multiple query objects, we treat every dimension independently. The efficiency of our method is realized by a specialized bitmap indexing that represents all objects in a database as a set of bitmaps. The percentage of data accessed in our indexing method is inversely proportional to the overall dimensionality, and thus the performance deterioration with the increasing dimensionality does not occur. To demonstrate the efficacy of our method we conducted extensive experiments and compared the performance with the VA-file-based index and the linear scan by using real image and video datasets, and obtained a remarkable speed-up over them.     

A methodology for the evaluation of travel techniques for immersive virtual environments

We present a framework for the analysis and evaluation oftravel, or viewpoint motion control, techniques for use in immersive virtual environments (VEs). In previous work, we presented a taxonomy of travel techniques and a set of experiments mapping parts of the taxonomy to various performance metrics. Since these initial experiments, we have expanded the framework to allow evaluation of not only the effects of different travel techniques, but also the effects of many outside factors simultaneously. Combining this expanded framework with the measurement of multiple response variables epitomises the philosophy oftestbed evaluation. This experimental philosophy leads to a deeper understanding of the interaction and the technique(s) in question, as well as to broadly generalisable results. We also present an example experiment within this expanded framework, which evaluates the user's ability to gather information while travelling through a virtual environment. Results indicate that, of the variables tested, the complexity of the environment is by far the most important factor.     

A Spanish dataset for reproducible benchmarked offline handwriting recognition

Language Resources and Evaluation - In this paper, a public dataset for Offline Handwriting Recognition, along with an appropriate evaluation method to provide benchmark indicators at sentence...     

A simulator for both manual and powered wheelchairs in immersive virtual reality CAVE

Virtual Reality - A large number of people in the world need to use a wheelchair because of different disabilities. Driving a wheelchair requires complex physical and cognitive abilities which need...