Bayesian tracking fusion framework with online classifier ensemble for immersive visual applications

During the last decade, the development of the immersive virtual reality (VR) has achieved a great progress in different application areas. For more advanced large-scale immersive VR environments or systems, one of the most challenge is to accurately track the position of the user’s body part such as head when he/she is immersived in the environment to feel the changes among the synthetic stereoscopic image sequences. Unfortunately, accurate tracking is not easy in the virtual reality scenarios due to the variety types of existing intrinsic and extrinsic changes when tracking is on-the-fly. Especially for the single tracker, a long time accurate tracking is usually not possible because of the model adaption problem in different environments. Recent trend of research in tracking is to incorporate multiple trackers into a compositive learning framework and utilize the advantages of different trackers for more effective tracking. Therefore, in this paper, we propose a novel Bayesian tracking fusion framework with online classifier ensemble strategy. The proposed tracking formulates a fusion framework for online learning of multiple trackers by modeling a cumulative loss minimization process. With an optimal pair-wise sampling scheme for the SVM classifier, the proposed fusion framework can achieve more accurate tracking performance when compared with the other state-of-art trackers. In addition, the experiments on the standard benchmark database also verify that the proposed tracking is able to handle the challenges in many immersive VR applications and environments.     

Designing a Highly Immersive Interactive Environment: The Virtual Mine

To achieve a full‐scale simulation of a pyrite mine, a highly immersive environment becomes necessary and this research has led to a complex system enabling users to walk through a virtual mine in real time, presenting all the behaviours present in such environment. Some of the problems encountered are the tunnels behaviours, including highly contrasted images due to the presence of the head light, narrow paths, elevators, sound reverberation and tunnels texture shades. The use of immersive virtual reality enables the generation of high‐quality simulations, because it is possible to control several feedback mechanisms such as the degree of luminance of produced imagery and spatial sound. In this research, a projection infrastructure and tracking system were specified and developed, aiming at producing the best results for this kind of simulation. To achieve our purposes, distributed algorithms were developed to run in a cluster solution that drives a four‐sided CAVE‐like environment. The complete production pipeline is presented, ranging from the developed authoring techniques, enabling fast production of new content for the simulation, to the tracking techniques produced for the improvement of the interaction.     

Global hand pose estimation by multiple camera ellipse tracking

Immersive virtual environments with life-like interaction capabilities have very demanding requirements including high-precision motion capture and high-processing speed. These issues raise many challenges for computer vision-based motion estimation algorithms. In this study, we consider the problem of hand tracking using multiple cameras and estimating its 3D global pose (i.e., position and orientation of the palm). Our interest is in developing an accurate and robust algorithm to be employed in an immersive virtual training environment, called “Virtual GloveboX” (VGX) (Twombly et al. in J Syst Cybern Inf 2:30–34, 2005), which is currently under development at NASA Ames. In this context, we present a marker-based, hand tracking and 3D global pose estimation algorithm that operates in a controlled, multi-camera, environment built to track the user’s hand inside VGX. The key idea of the proposed algorithm is tracking the 3D position and orientation of an elliptical marker placed on the dorsal part of the hand using model-based tracking approaches and active camera selection. It should be noted that, the use of markers is well justified in the context of our application since VGX naturally allows for the use of gloves without disrupting the fidelity of the interaction. Our experimental results and comparisons illustrate that the proposed approach is more accurate and robust than related approaches. A byproduct of our multi-camera ellipse tracking algorithm is that, with only minor modifications, the same algorithm can be used to automatically re-calibrate (i.e., fine-tune) the extrinsic parameters of a multi-camera system leading to more accurate pose estimates.     

一种基于全景视频的6自由度野外虚拟环境快速构建方法

野外环境的虚拟化是一个具有重要意义的研究领域,关于如何更高效、更便捷地构建野外虚拟环境,以及如何在构建的虚拟环境中获得更高的交互自由度与更好的沉浸式体验,是值得思考的重要问题。因此,提出了一种基于全景视频的6自由度野外虚拟环境快速构建方法,通过对从中提取的全景影像进行深度估计、摄影建模等处理,得到划分为不同远近层次的重建场景;并根据用户的漫游位置实现了近远景的实时动态匹配融合,最终获得了支持一定范围内6自由度漫游的沉浸式野外虚拟环境;通过图像质量的客观指标进行了量化分析验证,并实现了从构建到使用完整流程的野外虚拟环境原型系统作为应用。结果表明,该方法有着高效地使用方式与优良的应用效果,在虚拟化野外地质实习教学等方面有着良好的前景。     

沉浸式所触即所得网络可视分析方法

沉浸式网络可视化在空间沉浸、用户参与、多维感知等方面具有天然的优势。受用户与日常物体交互方式所启发,基于所触即所得(WYTIWYG)的理念提出一种沉浸式网络可视分析方法来挖掘网络特征和关联模式。首先提出手势舒适度评估模型来指导手势动作设计,并引入窗口状态模型来优化手势识别稳定性。此外,将网络分析交互需求与手势动作语义绑定,定义沉浸式网络手势交互范式。与真实世界中抓取交互类似,用户可利用自然交互手势在沉浸式环境下执行移动、高亮、布局维度变换、边绑定等操作。最后,案例研究验证了方法的有效性。     

Evaluation of direct manipulation using finger tracking for complex tasks in an immersive cube

A solution for interaction using finger tracking in a cubic immersive virtual reality system (or immersive cube) is presented. Rather than using a traditional wand device, users can manipulate objects with fingers of both hands in a close-to-natural manner for moderately complex, general purpose tasks. Our solution couples finger tracking with a real-time physics engine, combined with a heuristic approach for hand manipulation, which is robust to tracker noise and simulation instabilities. A first study has been performed to evaluate our interface, with tasks involving complex manipulations, such as balancing objects while walking in the cube. The user’s finger-tracked manipulation was compared to manipulation with a 6 degree-of-freedom wand (or flystick), as well as with carrying out the same task in the real world. Users were also asked to perform a free task, allowing us to observe their perceived level of presence in the scene. Our results show that our approach provides a feasible interface for immersive cube environments and is perceived by users as being closer to the real experience compared to the wand. However, the wand outperforms direct manipulation in terms of speed and precision. We conclude with a discussion of the results and implications for further research.     

Measuring flow experience in an immersive virtual environment for collaborative learning

In contexts other than immersive virtual environments, theoretical and empirical work has identified flow experience as a major factor in learning and human–computer interaction. Flow is defined as a ‘holistic sensation that people feel when they act with total involvement’. We applied the concept of flow to modeling the experience of collaborative learning in an immersive virtual environment. The aims were, first, to psychometrically evaluate a measurement model of flow and, second, to test a structural model of flow. Pairs of small teams engaged in collaborative problem‐solving tasks while communicating by way of an immersive virtual environment. Flow was measured after each session, using Guo and Poole's inventory for measuring flow in human–computer interaction. In relation to the first aim, partial‐least‐squares analysis demonstrated strong evidence for the measurement model. In relation to the second aim, the structural model was supported: the effect of learning‐task characteristics on flow experience was mediated by its precursors, with extraneous variables held constant. It is reasoned that the experiment and resultant analysis of this work contributes to the development of measurement models and structural models of flow in immersive virtual environments.     

支持力反馈的沉浸式物理学习环境的构建

随着虚拟现实(VR)技术的发展,沉浸式学习环境在教育教学领域应用前景日趋广阔,如物理实验仿真教学.然而,现有虚拟学习环境大多只能提供视觉与听觉的交互,不支持力触觉交互,存在弊端.项目组将力反馈技术应用于虚拟学习环境,描述一个支持力反馈的沉浸式物理学习环境的总体框架和开发流程.使用Touch力反馈设备,借助Unity3D...     

Intelligent virtual environments for virtual reality art

The development of virtual reality (VR) art installations is faced with considerable difficulties, especially when one wishes to explore complex notions related to user interaction. We describe the development of a VR platform, which supports the development of such installations, from an art+science perspective. The system is based on a CAVE™-like immersive display using a game engine to support visualisation and interaction, which has been adapted for stereoscopic visualisation and real-time tracking. In addition, some architectural elements of game engines, such as their reliance on event-based systems have been used to support the principled definition of alternative laws of Physics. We illustrate this research through the development of a fully implemented artistic brief that explores the notion of causality in a virtual environment. After describing the hardware architecture supporting immersive visualisation we show how causality can be redefined using artificial intelligence technologies inspired from action representation in planning and how this symbolic definition of behaviour can support new forms of user experience in VR.     

State of the Art of Molecular Visualization in Immersive Virtual Environments

Visualization plays a crucial role in molecular and structural biology. It has been successfully applied to a variety of tasks, including structural analysis and interactive drug design. While some of the challenges in this area can be overcome with more advanced visualization and interaction techniques, others are challenging primarily due to the limitations of the hardware devices used to interact with the visualized content. Consequently, visualization researchers are increasingly trying to take advantage of new technologies to facilitate the work of domain scientists. Some typical problems associated with classic 2D interfaces, such as regular desktop computers, are a lack of natural spatial understanding and interaction, and a limited field of view. These problems could be solved by immersive virtual environments and corresponding hardware, such as virtual reality head-mounted displays. Thus, researchers are investigating the potential of immersive virtual environments in the field of molecular visualization. There is already a body of work ranging from educational approaches to protein visualization to applications for collaborative drug design. This review focuses on molecular visualization in immersive virtual environments as a whole, aiming to cover this area comprehensively. We divide the existing papers into different groups based on their application areas, and types of tasks performed. Furthermore, we also include a list of available software tools. We conclude the report with a discussion of potential future research on molecular visualization in immersive environments.     

Applications drive VR interface selection

Physical interfaces to virtual environments fall into two broad categories: devices for navigation and physical interaction (such as deskbound and flying “mice”, gestural devices and navigation controls built into viewing devices); and viewing devices (such as head-mounted displays, counterbalanced displays, shutter glasses and a new class of immersive “desktop” viewers). The visual image is critical to creating an effective virtual experience. Beginning with immersive displays, this survey describes the range of available devices and discusses how application considerations affect the selection of components when configuring a virtual reality (VR) system     

Immersive Multiplayer Games With Tangible and Physical Interaction

In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game (for VR and AR environments, respectively) and a sword game (for the AR environment). Our immersive system provides an intuitive way for users to interact with the VR or AR world by physically moving around the real world and aiming freely with tangible objects. This encourages physical interaction between players as they compete or collaborate with other players. Evaluation of our system consists of users' subjective opinions and their objective performances. Our design principles and evaluation results can be applied to similar immersive game applications based on AR/VR.     

ICOME: An Immersive Collaborative 3D Object Modelling Environment

In most existing immersive virtual environments, 3D geometry is imported from external packages. Within ICOME (an immersive Collaborative 3D Object Modelling Environment) we focus on the immersive construction of 3D geometrical objects within the environment itself. Moreover, the framework allows multiple people to simultaneously undertake 3D modelling tasks in a collaborative way. This article describes the overall architecture, which conceptually follows a client/server approach. The various types of clients, which are implemented, are described in detail. Some illustrative 3D object modelling examples are given. Extensions to the system with regard to 3D audio are also mentioned.This paper is a republished version of Virtual Reality 1999; 4: 129–138 with corrections     

Walk-up VR: virtual reality beyond projection screens

Today, VR research and development efforts often focus on the continual innovation of interaction styles and metaphors for virtual environments (VEs). New tools and interaction devices aim to increase the immersive experience rather than support seamless integration with real work scenarios. Even though users may soon perceive odours within VEs, real, task-oriented interaction within these environments will continue to lag behind. Combining these efforts can result in new user interfaces that reduce the cumbersome barriers prevalent in VEs today, finally unleashing the latent impact of this technology in everyday life. Implementing this vision requires an interdisciplinary and applied approach to integrate VR into the workplace. Mixed-reality display capabilities, useful multimodal interaction and perceptual, intuitive interfaces are major components of such an application-oriented and human-centered approach, for which we coined the term walk-up VR. The paper discusses augmented and virtual reality as contributing technologies     

Towards video-based immersive environments

Video provides a comprehensive visual record of environment activity over time. Thus, video data is an attractive source of information for the creation of virtual worlds which require some real-world fidelity. This paper describes the use of multiple streams of video data for the creation of immersive virtual environments. We outline our multiple perspective interactive video (MPI-Video) architecture which provides the infrastructure for the processing and analysis of multiple streams of video data. Our MPI-Video system performs automated analysis of the raw video and constructs a model of the environment and object activity within this environment. This model provides a comprehensive representation of the world monitored by the cameras which, in turn, can be used in the construction of a virtual world. In addition, using the information produced and maintained by the MPI-Video system, our immersive video system generates virtual video sequences. These are sequences of the dynamic environment from an arbitrary view point generated using the real camera data. Such sequences allow a user to navigate through the environment and provide a sense of immersion in the scene. We discuss results from our MPI-Video prototype, outline algorithms for the construction of virtual views and provide examples of a variety of such immersive video sequences.     

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.     

Trade-off between resolution and interactivity in spatial taskperformance

Virtual reality displays usually lag far behind classical computer graphics displays in static image quality parameters, such as resolution. Both the popular press and scientific papers often stress that resolution will have to increase greatly before users can experience virtual environments as “the real thing”. Nevertheless, it is already possible to do some useful work in VR environments. The point we experimentally demonstrate here is that resolution is much less important for interactive tasks that employ immersive VR, where users can explore the environment by moving their heads and bodies, than it is in classical computer graphics applications, where users can only explore by gazing at a single picture. In the context of unmanned aerial vehicles, frame rate (read: passive camera movement) is more important than resolution for target detection, recognition, designation, and tracking. In the experiments reported here, we investigated the relative importance of various image parameters like spatial resolution (number of pixels per video frame), intensity resolution (number of gray levels per pixel), and temporal resolution (number of frame updates per second). Most experimental data concerning these resolutions come from classical psychophysics. However, experimental conditions in classical psychophysics feature stationary observers looking at short-term, pointlike flashes on stationary displays, and are thus far more representative of human interaction with pictures and photographs than with highly interactive systems like those employed in virtual reality     

VR-CAD integration: Multimodal immersive interaction and advanced haptic paradigms for implicit edition of CAD models

This paper presents an approach for the integration of Virtual Reality (VR) and Computer-Aided Design (CAD). Our general goal is to develop a VR-CAD framework making possible intuitive and direct 3D edition on CAD objects within Virtual Environments (VE). Such a framework can be applied to collaborative part design activities and to immersive project reviews. The cornerstone of our approach is a model that manages implicit editing of CAD objects. This model uses a naming technique of B-Rep components and a set of logical rules to provide straight access to the operators of Construction History Graphs (CHG). Another set of logical rules and the replay capacities of CHG make it possible to modify in real-time the parameters of these operators according to the user’s 3D interactions. A demonstrator of our model has been developed on the OpenCASCADE geometric kernel, but we explain how it can be applied to more standard CAD systems such as CATIA. We combined our VR-CAD framework with multimodal immersive interaction (using 6 DoF tracking, speech and gesture recognition systems) to gain direct and intuitive deformation of the objects’ shapes within a VE, thus avoiding explicit interactions with the CHG within a classical WIMP interface. In addition, we present several haptic paradigms specially conceptualized and evaluated to provide an accurate perception of B-Rep components and to help the user during his/her 3D interactions. Finally, we conclude on some issues for future researches in the field of VR-CAD integration.     

Multimodal identity tracking in a smart room

The automatic detection, tracking, and identification of multiple people in intelligent environments are important building blocks on which smart interaction systems can be designed. Those could be, e.g., gesture recognizers, head pose estimators or far-field speech recognizers and dialog systems. In this paper, we present a system which is capable of tracking multiple people in a smart room environment while inferring their identities in a completely automatic and unobtrusive way. It relies on a set of fixed and active cameras to track the users and get close-ups of their faces for identification, and on several microphone arrays to determine active speakers and steer the attention of the system. Information coming asynchronously from several sources, such as position updates from audio or visual trackers and identification events from identification modules, is fused at higher level to gradually refine the room’s situation model. The system has been trained on a small set of users and showed good performance at acquiring and keeping their identities in a smart room environment. An erratum to this article can be found at     

Immersive structural biology: a new approach to hybrid modeling of macromolecular assemblies

Advanced biophysical imaging techniques, such as cryo-electron microscopy or tomography, enable 3D volumetric reconstructions of large macromolecular complexes in a near-native environment. However, pure volumetric data is insufficient for a detailed understanding of the underlying protein–protein interactions. This obstacle can be overcome by assembling an atomic model of the whole protein complex from known atomic structures, which are available from either X-ray crystallography or homology modeling. Due to many factors such as noise, conformational variability, experimental artifacts, and inexact model structures, existing automatic docking procedures are known to report false positives for a significant number of cases. The present paper focuses on a new technique to combine an offline exhaustive search algorithm with interactive visualization, collision detection, and haptic rendering. The resulting software system is highly immersive and allows the user to efficiently solve even difficult multi-resolution docking problems. Stereoscopic viewing, combined with head tracking and force feedback, generates an ideal virtual environment for true interaction with and solution of hybrid biomolecular modeling problems.