支持力反馈的沉浸式物理学习环境的构建

随着虚拟现实(VR)技术的发展,沉浸式学习环境在教育教学领域应用前景日趋广阔,如物理实验 仿真教学。然而,现有虚拟学习环境大多只能提供视觉与听觉的交互,不支持力触觉交互,存在弊端。项目组将 力反馈技术应用于虚拟学习环境,描述一个支持力反馈的沉浸式物理学习环境的总体框架和开发流程。使用 Touch 力反馈设备,借助 Unity3D 软件和 OpenHaptics 力觉函数库,设计实现典型案例：重力和摩擦力实验。通过实验 得出：支持力反馈的沉浸式物理学习环境,具有更好的沉浸感和自然交互性,可加深学生对抽象物理概念和规律 的理解,提高学习兴趣和自主性。     

Online unsupervised feature learning for visual tracking

We propose a method for visual tracking-by-detection based on online feature learning. Our learning framework performs feature encoding with respect to an over-complete dictionary, followed by spatial pyramid pooling. We then learn a linear classifier based on the resulting feature encoding. Unlike previous work, we learn the dictionary online and update it to help capture the appearance of the tracked target as well as the background. In more detail, given a test image window, we extract local image patches from it and each local patch is encoded with respect to the dictionary. The encoded features are then pooled over a spatial pyramid to form an aggregated feature vector. Finally, a simple linear classifier is trained on these features.Our experiments show that the proposed powerful—albeit simple—tracker, outperforms all the state-of-the-art tracking methods that we have tested. Moreover, we evaluate the performance of different dictionary learning and feature encoding methods in the proposed tracking framework, and analyze the impact of each component in the tracking scenario. In particular, we show that a small dictionary, learned and updated online is as effective and more efficient than a huge dictionary learned offline. We further demonstrate the flexibility of feature learning by showing how it can be used within a structured learning tracking framework. The outcome is one of the best trackers reported to date, which facilitates the advantages of both feature learning and structured output prediction. We also implement a multi-object tracker, which achieves state-of-the-art performance.     

The Role of 3-D Sound in Human Reaction and Performance in Augmented Reality Environments

Three-dimensional sound's effectiveness in virtual reality (VR) environments has been widely studied. However, due to the big differences between VR and augmented reality (AR) systems in registration, calibration, perceptual difference of immersiveness, navigation, and localization, it is important to develop new approaches to seamlessly register virtual 3-D sound in AR environments and conduct studies on 3-D sound's effectiveness in AR context. In this paper, we design two experimental AR environments to study the effectiveness of 3-D sound both quantitatively and qualitatively. Two different tracking methods are applied to retrieve the 3-D position of virtual sound sources in each experiment. We examine the impacts of 3-D sound on improving depth perception and shortening task completion time. We also investigate its impacts on immersive and realistic perception, different spatial objects identification, and subjective feeling of "human presence and collaboration". Our studies show that applying 3-D sound is an effective way to complement visual AR environments. It helps depth perception and task performance, and facilitates collaborations between users. Moreover, it enables a more realistic environment and more immersive feeling of being inside the AR environment by both visual and auditory means. In order to make full use of the intensity cues provided by 3-D sound, a process to scale the intensity difference of 3-D sound at different depths is designed to cater small AR environments. The user study results show that the scaled 3-D sound significantly increases the accuracy of depth judgments and shortens the searching task completion time. This method provides a necessary foundation for implementing 3-D sound in small AR environments. Our user study results also show that this process does not degrade the intuitiveness and realism of an augmented audio reality environment     

Immersive Multiplayer Games With Tangible and Physical Interaction

In this paper, we present a new immersive multiplayer game system developed for two different environments, namely, virtual reality (VR) and augmented reality (AR). To evaluate our system, we developed three game applications-a first-person-shooter game (for VR and AR environments, respectively) and a sword game (for the AR environment). Our immersive system provides an intuitive way for users to interact with the VR or AR world by physically moving around the real world and aiming freely with tangible objects. This encourages physical interaction between players as they compete or collaborate with other players. Evaluation of our system consists of users' subjective opinions and their objective performances. Our design principles and evaluation results can be applied to similar immersive game applications based on AR/VR.     

Multi-modal virtual environments for education with haptic and olfactory feedback

It has been suggested that immersive virtual reality (VR) technology allows knowledge-building experiences and in this way provides an alternative educational process. Important key features of constructivist educational computer-based environments for science teaching and learning, include interaction, size, transduction and reification. Indeed, multi-sensory VR technology suits very well the needs of sciences that require a higher level of visualization and interaction. Haptics that refers to physical interactions with virtual environments (VEs) may be coupled with other sensory modalities such as vision and audition but are hardly ever associated with other feedback channels, such as olfactory feedback. A survey of theory and existing VEs including haptic or olfactory feedback, especially in the field of education is provided. Our multi-modal human-scale VE VIREPSE (virtual reality platform for simulation and experimentation) that provides haptic interaction using a string-based interface called SPIDAR (space interface device for artificial reality), olfactory and auditory feedbacks is described. An application that allows students experiencing the abstract concept of the Bohr atomic model and the quantization of the energy levels has been developed. Different configurations that support interaction, size and reification through the use of immersive and multi-modal (visual, haptic, auditory and olfactory) feedback are proposed for further evaluation. Haptic interaction is achieved using different techniques ranging from desktop pseudo-haptic feedback to human-scale haptic interaction. Olfactory information is provided using different fan-based olfactory displays (ODs). Significance of developing such multi-modal VEs for education is discussed.     

An experimental investigation of menu selection for immersive virtual environments: fixed versus handheld menus

Virtual Reality - With the development of consumer-grade virtual reality (VR) systems, the interface and interaction design for immersive virtual environments have become a critical issue for VR...     

Tangible authoring of 3D virtual scenes in dynamic augmented reality environment

This paper proposes tangible interfaces and interactions for authoring 3D virtual and immersive scenes easily and intuitively in tangible augmented reality (AR) environment. It provides tangible interfaces for manipulating virtual objects in a natural and intuitive manner and supports adaptive and accurate vision-based tracking in AR environments. In particular, RFID is used to directly integrate physical objects with virtual objects and to systematically support the tangible query of the relation between physical objects and virtual ones, which can provide more intuitive tangibility and a new way of virtual object manipulation. Moreover, the proposed approach offers an easy and intuitive switching mechanism between tangible environment and virtual environment. This paper also proposes a context-adaptive marker tracking method which can remove an inconsistent problem while embedding virtual objects into physical ones in tangible AR environments. The context-adaptive tracking method not only adjusts the locations of invisible markers by interpolating the locations of existing reference markers and those of previous ones, but also removes a jumping effect of movable virtual objects when their references are changed from one marker to another. Several case studies for generating tangible virtual scenes and comparison with previous work are given to show the effectiveness and novelty of the proposed approach.     

Distributed, multi-sensor tracking of multiple participants within immersive environments using a 2-cohort camera setup

Vision-based human motion tracking has gained significant interest in recent years, as the need for more intuitive human–computer interaction paradigms are sought after. Immersive environments, such as large-scale displays and virtual reality systems are particularly suitable for such interaction mechanisms, as they provide a controlled basis for experimentation and evaluation. However, despite the growing interest, human-motion tracking within immersive environments is still in its infancy. The lack of visible light and the presence of multiple people within such environments pose several challenges to the problem. In this research a robust framework for real-time, multi-camera human-motion tracking within an immersive environment is presented. The static nature of the environment is exploited to utilise a novel 2-cohort camera layout. This layout proves to be highly useful for tracking, providing several benefits over existing techniques, which treat all cameras as the same.     

The virtual playground: an educational virtual reality environment for evaluating interactivity and conceptual learning

The research presented in this paper aims at investigating user interaction in immersive virtual learning environments, focusing on the role and the effect of interactivity on conceptual learning. The goal has been to examine if the learning of young users improves through interacting in (i.e. exploring, reacting to, and acting upon) an immersive virtual environment (VE) compared to non-interactive or non-immersive environments. Empirical work was carried out with more than 55 primary school students between the ages of 8 and 12, in different between-group experiments: an exploratory study, a pilot study, and a large-scale experiment. The latter was conducted in a virtual environment designed to simulate a playground. In this “Virtual Playground,” each participant was asked to complete a set of tasks designed to address arithmetical “fractions” problems. Three different conditions, two experimental virtual reality (VR) conditions and a non-VR condition, that varied the levels of activity and interactivity, were designed to evaluate how children accomplish the various tasks. Pre-tests, post-tests, interviews, video, audio, and log files were collected for each participant, and analysed both quantitatively and qualitatively. This paper presents a selection of case studies extracted from the qualitative analysis, which illustrate the variety of approaches taken by children in the VEs in response to visual cues and system feedback. Results suggest that the fully interactive VE aided children in problem solving but did not provide a strong evidence of conceptual change as expected; rather, it was the passive VR environment, where activity was guided by a virtual robot, that seemed to support student reflection and recall, leading to indications of conceptual change.     

基于Web的虚拟展厅的设计与实现

当前的VR标准，可以描述一个被动的VR系统，即系统应用VR技术来实现一些预先设计的3D内容的可视化。然而，现实中的一个挑战在于构造主动VR应用，这种应用允许动态合成虚拟场景。为了能构造主动VR应用，该文结合目前工作描述了网络环境下虚拟展览厅系统，提出虚拟世界的数据库建模方法。文中阐述了该系统的用途和功能，给出了系统的结构模型等。     

A parallel and robust object tracking approach synthesizing adaptive Bayesian learning and improved incremental subspace learning

This paper presents a novel tracking algorithm which integrates two complementary trackers. Firstly, an improved Bayesian tracker(B-tracker) with adaptive learning rate is presented. The classification score of B-tracker reflects tracking reliability, and a low score usually results from large appearance change. Therefore, if the score is low, we decrease the learning rate to update the classifier fast so that B-tracker can adapt to the variation and vice versa. In this way, B-tracker is more suitable than its traditional version to solve appearance change problem. Secondly, we present an improved incremental subspace learning method tracker(Stracker). We propose to calculate projected coordinates using maximum posterior probability, which results in a more accurate reconstruction error than traditional subspace learning tracker. Instead of updating at every time, we present a stopstrategy to deal with occlusion problem. Finally, we present an integrated framework(BAST), in which the pair of trackers run in parallel and return two candidate target states separately. For each candidate state, we define a tracking reliability metrics to measure whether the candidate state is reliable or not, and the reliable candidate state will be chosen as the target state at the end of each frame. Experimental results on challenging sequences show that the proposed approach is very robust and effective in comparison to the state-of-the-art trackers.     

基于特征融合与分类器在线学习的目标跟踪算法

为了解决目标在复杂环境下表观变化引起的跟踪漂移问题,提出一种基于多特征融合与分类器在线学习的目标跟踪算法.该算法利用不同表观特征训练子分类器,通过构建损失函数求得各子分类器可信度,进而加权融合子预测结果,得到当前帧最佳目标状态估计;同时,依据最近-最远边界原则和协同训练理论粗更新训练样本集,并通过精选择准则得到更具代表性的训练样本集,实现子分类器自适应更新.实验结果表明,所提出的算法在多种典型测试场景中都能取得较鲁棒的跟踪效果.     

Exploring the design space of eyes-free target acquisition in virtual environments

Virtual Reality - Supporting smooth target acquisition is an important objective in immersive virtual reality (VR) environments. However, users were obliged to search for objects relying on the...     

Inclusion of immersive virtual learning environments and visual control systems to support the learning of students with Asperger syndrome

This paper presents the use of immersive virtual reality systems in the educational intervention with Asperger students. The starting points of this study are features of these students' cognitive style that requires an explicit teaching style supported by visual aids and highly structured environments. The proposed immersive virtual reality system, not only to assess the student's behavior and progress, but also is able to adapt itself to the student's specific needs. Additionally, the immersive reality system is equipped with sensors that can determine certain behaviors of the students. This paper determines the possible inclusion of immersive virtual reality as a support tool and learning strategy in these particular students' intervention. With this objective two task protocols have been defined with which the behavior and interaction situations performed by participant students are recorded. The conclusions from this study talks in favor of the inclusion of these virtual immersive environments as a support tool in the educational intervention of Asperger syndrome students as their social competences and executive functions have improved.     

Applications drive VR interface selection

Physical interfaces to virtual environments fall into two broad categories: devices for navigation and physical interaction (such as deskbound and flying “mice”, gestural devices and navigation controls built into viewing devices); and viewing devices (such as head-mounted displays, counterbalanced displays, shutter glasses and a new class of immersive “desktop” viewers). The visual image is critical to creating an effective virtual experience. Beginning with immersive displays, this survey describes the range of available devices and discusses how application considerations affect the selection of components when configuring a virtual reality (VR) system     

Web-based learning in the computer-aided design curriculum

Abstract This study attempts to apply the principle of constructivism and virtual reality (VR) technologies to computer-aided design (CAD) curriculum by integrating network, CAD and VR into a web-based learning environment. Through VR technologies, it is expected that the traditional two-dimensional (2D) computer graphics course can be expanded into a three-dimensional (3D) real-time simulation one. VR technologies provide a novel method to enhance user visualisation of complex three-dimensional graphics and environments. Experience and environmental interaction allow users more readily to perceive the dimensional interrelations of graphics which are typically portrayed through static multiview or pictorial representations. A web-based learning system ( WebDeGrator ) has been developed to simulate a computer graphics learning system for learning. Future developments of the proposed web-based learning framework are also discussed.     

Platform is not destiny: Embodied learning effects comparing 2D desktop to 3D virtual reality STEM experiences

Experiences in immersive 3D virtual reality (VR) are more presence-inducing, and so it may be tempting to claim that content will be learned better in VR. This randomized control trial study on natural selection challenges that assumption. This study answers the question of whether learning STEM in an immersive 3D VR environment is always superior to learning via a 2D monitor (PC). This is a 2 × 2 × 3 design. The first factor is platform immersivity (low = 2D PC, or high = 3D VR), the second factor is level of embodiment (lower = watching playback video, or higher = using mouse/controller to agentically manipulate content), and the third factor is test time (pretest, posttest, and follow-up). There was a significant main effect for embodiment, the high embodied and agentic groups learned the most. There was not a main effect for platform, because the participants in the low embodied VR group performed significantly worse than the three other groups. Although, the one high embodied, VR group learned and retained the most knowledge. A path-analysis revealed that the effect of platform was significantly mediated by presence, agency, and engagement. The smaller learning gain in the low embodied VR condition suggests that participants come to the immersive VR experience with expectations about agency and control of the virtual content, and when those expectations were not met, the disconnect was deleterious for learning. More agentic and interactive control of manipulable virtual content is encouraged. Design is critical, and platform is not destiny.     

Seeing size and feeling weight: the size-weight illusion in natural and virtual reality

OBJECTIVE: We experimentally tested the degree that the size-weight illusion depends on perceptual conditions allowing the observer to assume that both the visual and the kinesthetic stimuli of a weight seen and lifted emanate from the same object. We expected that the degree of the illusion depended on the "realism" provided by different kinds of virtual reality (VR) used when the weights are seen in virtual reality and at the same time lifted in natural reality. BACKGROUND: Welch and Warren (1980) reported that an intermodal influence can be expected only if perceptual information of different modalities is compellingly related to only one object. METHOD: Objects of different sizes and weights were presented to 50 participants in natural reality or in four virtual realities: two immersive head-mounted display VRs (with or without head tracking) and two nonimmersive desktop VRs (with or without screening from input of the natural environment using a visor). The objects' heaviness was scaled using the magnitude estimation method. RESULTS: Data show that the degree of the illusion is largest in immersive and lowest in nonimmersive virtual realities. CONCLUSION: The higher the degree of the illusion is, the more compelling the situation is perceived and the more the observed data are in correspondence with the data predicted for the illusion in natural reality. This shows that the kind of mediating technology used strongly influences the presence experienced. APPLICATION: The size-weight illusion's sensitivity to conditions that affect the sense of presence makes it a promising objective presence measure.     

Audio–Visual Active Speaker Tracking in Cluttered Indoors Environments $^{ast}$

We propose a system for detecting the active speaker in cluttered and reverberant environments where more than one person speaks and moves. Rather than using only audio information, the system utilizes audiovisual information from multiple acoustic and video sensors that feed separate audio and video tracking modules. The audio module operates using a particle filter (PF) and an information-theoretic framework to provide accurate acoustic source location under reverberant conditions. The video subsystem combines in 3-D a number of 2-D trackers based on a variation of Stauffer's adaptive background algorithm with spatiotemporal adaptation of the learning parameters and a Kalman tracker in a feedback configuration. Extensive experiments show that gains are to be expected when fusion of the separate modalities is performed to detect the active speaker.      

Visual tracking via weakly supervised learning from multiple imperfect oracles

Notwithstanding many years of progress, visual tracking is still a difficult but important problem. Since most top-performing tracking methods have their strengths and weaknesses and are suited for handling only a certain type of variation, one of the next challenges is to integrate all these methods and address the problem of long-term persistent tracking in ever-changing environments. Towards this goal, we consider visual tracking in a novel weakly supervised learning scenario where (possibly noisy) labels but no ground truth are provided by multiple imperfect oracles (i.e., different trackers). These trackers naturally have intrinsic diversity due to their different design strategies, and we propose a probabilistic method to simultaneously infer the most likely object position by considering the outputs of all trackers, and estimate the accuracy of each tracker. An online evaluation strategy of trackers and a heuristic training data selection scheme are adopted to make the inference more effective and efficient. Consequently, the proposed method can avoid the pitfalls of purely single tracking methods and get reliably labeled samples to incrementally update each tracker (if it is an appearance-adaptive tracker) to capture the appearance changes. Extensive experiments on challenging video sequences demonstrate the robustness and effectiveness of the proposed method.