Multi-modal virtual environments for education with haptic and olfactory feedback

It has been suggested that immersive virtual reality (VR) technology allows knowledge-building experiences and in this way provides an alternative educational process. Important key features of constructivist educational computer-based environments for science teaching and learning, include interaction, size, transduction and reification. Indeed, multi-sensory VR technology suits very well the needs of sciences that require a higher level of visualization and interaction. Haptics that refers to physical interactions with virtual environments (VEs) may be coupled with other sensory modalities such as vision and audition but are hardly ever associated with other feedback channels, such as olfactory feedback. A survey of theory and existing VEs including haptic or olfactory feedback, especially in the field of education is provided. Our multi-modal human-scale VE VIREPSE (virtual reality platform for simulation and experimentation) that provides haptic interaction using a string-based interface called SPIDAR (space interface device for artificial reality), olfactory and auditory feedbacks is described. An application that allows students experiencing the abstract concept of the Bohr atomic model and the quantization of the energy levels has been developed. Different configurations that support interaction, size and reification through the use of immersive and multi-modal (visual, haptic, auditory and olfactory) feedback are proposed for further evaluation. Haptic interaction is achieved using different techniques ranging from desktop pseudo-haptic feedback to human-scale haptic interaction. Olfactory information is provided using different fan-based olfactory displays (ODs). Significance of developing such multi-modal VEs for education is discussed.     

Whole-hand kinesthetic feedback and haptic perception in dextrous virtual manipulation

One of the key requirements for a Virtual Reality system is the multimodal, real-time interaction between the human operator and a computer simulated and animated environment. This paper investigates problems related particularly to the haptic interaction between the human operator and a virtual environment. The work presented here focuses on two issues: 1) the synthesis of whole-hand kinesthetic feedback, based on the application of forces (torques) on individual phalanges (joints) of the human hand, and 2) the experimental evaluation of this haptic feedback system, in terms of human haptic perception of virtual physical properties (such as the weight of a virtual manipulated object), using psychophysical methods. The proposed kinesthetic feedback methodology is based on the solution of a generalized force distribution problem for the human hand during virtual manipulation tasks. The solution is computationally efficient and has been experimentally implemented using an exoskeleton force-feedback glove. A series of experiments is reported concerning the perception of weight of manipulated virtual objects and the obtained results demonstrate the feasibility of the concept. Issues related to the use of sensory substitution techniques for the application of haptic feedback on the human hand are also discussed.     

Helping older pedestrians navigate in the city: comparisons of visual,auditory and haptic guidance instructions in a virtual environment

Preserving older pedestrians’ navigation skills in urban environments is a challenge for maintaining their quality of life. However, existing aids do not take into account older people’s perceptual and cognitive declines nor their user experience, and they call upon sensory modalities that are already used during walking. The present study was aimed at comparing different guidance instructions using visual, auditory, and haptic feedback in order to identify the most efficient and best accepted one(s). Sixteen middle-age (non-retired) adults, 21 younger-old (young-retired) adults, and 21 older-old (old-retired) adults performed a navigation task in a virtual environment. The task was performed with visual feedback (directional arrows superimposed on the visual scenes), auditory feedback (sounds in the left/right ear), haptic feedback (vibrotactile information delivered by a wristband), combinations of different types of sensory feedback, or a paper map. The results showed that older people benefited from the sensory guidance instructions, as compared to the paper map. Visual and auditory feedbacks were associated with better performance and user experience than haptic feedback or the paper map, and the benefits were the greatest among the older-old participants, even though the paper-map familiarity was appreciated. Several recommendations for designing pedestrian navigation aids are proposed.     

虚拟体空间中的触觉雕刻

目前，在虚拟环境中大多数的信息获取是通过视觉、听觉等非接触感觉获得的。然而缺乏触觉反馈的信息减少了很大一部分的信息源。在看和听之外，能够触摸、感觉和操纵物体，在很大程度上提高了虚拟环境的真实性。该文研究了触觉绘制的基本模型，提出了采用虚平面作为中介实现体数据的实时触觉绘制。并在此基础上探讨了体的局部变形及结合触觉反馈模型，实现了具有触觉反馈的虚拟雕刻交互系统。该系统可应用于融化、燃烧、印记、构造和着色实时交互操作。     

Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

Multimodal selection techniques for dense and occluded 3D virtual environments

Object selection is a primary interaction technique which must be supported by any interactive three-dimensional virtual reality application. Although numerous techniques exist, few have been designed to support the selection of objects in dense target environments, or the selection of objects which are occluded from the user's viewpoint. There is, thus, a limited understanding on how these important factors will affect selection performance. In this paper, we present a set of design guidelines and strategies to aid the development of selection techniques which can compensate for environment density and target visibility. Based on these guidelines, we present new forms of the ray casting and bubble cursor selection techniques, which are augmented with visual, audio, and haptic feedback, to support selection within dense and occluded 3D target environments. We perform a series of experiments to evaluate these new techniques, varying both the environment density and target visibility. The results provide an initial understanding of how these factors affect selection performance. Furthermore, the results showed that our new techniques adequately allowed users to select targets which were not visible from their initial viewpoint. The audio and haptic feedback did not provide significant improvements, and our analysis indicated that our introduced visual feedback played the most critical role in aiding the selection task.     

The Effects of Haptic Feedback and Visual Distraction on Pointing Task Performance

Previous research has not fully examined the effect of additional sensory feedback, particularly delivered through the haptic modality, in pointing task performance with visual distractions. This study examined the effect of haptic feedback and visual distraction on pointing task performance in a 3D virtual environment. Results indicate a strong positive effect of haptic feedback on performance in terms of task time and root mean square error of motion. Level of similarity between distractor objects and the target object significantly reduced performance, and subjective ratings indicated a sense of increased task difficulty as similarity increased. Participants produced the best performance in trials where distractor objects had a different color but the same shape as the target object and constant haptic assistive feedback was provided. Overall, this study provides insight toward the effect of object features and similarity and the effect of haptic feedback on pointing task performance.     

A virtual 3D interactive painting method for Chinese calligraphy and painting based on real-time force feedback technology

A novel 3D interactive painting method for Chinese calligraphy and painting based on force feedback technology is proposed. The relationship between the force exerted on the brush and the resulting brush deformation is analyzed and a spring-mass model is used to build a model of the 3D Chinese brush. The 2D brush footprint between the brush and the plane of the paper or object is calculated according to the deformation of the 3D brush when force is exerted on the 3D brush. Then the 3D brush footprint is obtained by projecting the 2D brush footprint onto the surface of the 3D object in real time, and a complete 3D brushstroke is obtained by superimposing 3D brush footprints along the painting direction. The proposed method has been successfully applied in a virtual 3D interactive drawing system based on force feedback technology. In this system, users can paint 3D brushstrokes in real time with a Phantom Desktop haptic device, which can effectively serve as a virtual reality interface to the simulated painting environment for users.     

视触觉融合的增强现实三维注册方法

增强现实技术是近年来人机交互领域的研究热点。在增强现实环境下加入触觉感知,可使用户在真实场景中看到并感知到虚拟对象。为了实现增强现实环境下与虚拟对象之间更加自然的交互,提出一种视触觉融合的三维注册方法。基于图像视觉技术获得三维注册矩阵;借助空间转换关系求解出触觉空间与图像空间的转换关系;结合两者与摄像头空间的关系实现视触觉融合的增强现实交互场景。为验证该方法的有效性,设计了一个基于视触觉增强现实的组装机器人项目。用户可触摸并移动真实环境中的机器人零件,还能在触摸时感受到反馈力,使交互更具真实感。     

The contribution of virtual reality to research on sensory feedback in remote control

Here we consider research on the kinds of sensory information most effective as feedback during remote control of machines, and the role of virtual reality and telepresence in that research. We argue that full automation is a distant goal and that remote control deserves continued attention and improvement. Visual feedback to controllers has developed in various ways but autostereoscopic displays have yet to be proven. Haptic force feedback, in both real and virtual settings, has been demonstrated to offer much to the remote control environment and has led to a greater understanding of the kinesthetic and cutaneous components of haptics, and their role in multimodal processes, such as sensory capture and integration. We suggest that many displays using primarily visual feedback would benefit from the addition of haptic information but that much is yet to be learned about optimizing such displays.     

Beyond the visuals: tactile augmentation and sensory enhancement in an arthroscopy simulator

This paper considers tactile augmentation, the addition of a physical object within a virtual environment (VE) to provide haptic feedback. The resulting mixed reality environment is limited in terms of the ease with which changes can be made to the haptic properties of objects within it. Therefore sensory enhancements or illusions that make use of visual cues to alter the perceived hardness of a physical object allowing variation in haptic properties are considered. Experimental work demonstrates that a single physical surface can be made to ‘feel’ both softer and harder than it is in reality by the accompanying visual information presented. The strong impact visual cues have on the overall perception of object hardness, indicates haptic accuracy may not be essential for a realistic virtual experience. The experimental results are related specifically to the development of a VE for surgical training; however, the conclusions drawn are broadly applicable to the simulation of touch and the understanding of haptic perception within VEs.     

Study on interaction-induced symptoms with respect to virtual grasping and manipulation

Owing to the popularity of various hand tracking interfaces, there have been numerous applications developed to provide intuitive hand interaction with the virtual world. As users start with great anticipation, they end up with dissatisfaction due to difficulties of manipulation or physical tiredness coming very short. Although the task itself is rather trivial in a real life situation, it requires much effort in the virtual environment. We address this awkwardness as ‘VR interaction-induced fatigue symptom’ and hypothesize its causes based on our observations. We argue that the source of the fatigue comes from the restricted sensory information of the VR interfaces, and that users try to accommodate the missing sensory feedback by excessive motion leading to wrong posture or bad timing. We demonstrate our hypothesis by conducting experiments of two types of virtual interaction scenarios: object transport and 3D selection. Furthermore, by analyzing the behaviors of users' action collected from our experiment, we derive essential factors to be considered in designing VR applications, and propose a conceptual interaction model for orchestrating virtual grasping.     

A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment

This study presents a 3D virtual reality (VR) keyboard system with realistic haptic feedback. The system uses two five-fingered data gloves to track finger positions and postures, uses micro-speakers to create simulated vibrations, and uses a head-mounted display (HMD) for 3D display. When users press a virtual key in the VR environment, the system can provide realistic simulated key click haptic feedback to users. The results of this study show that the advantages of the haptic VR keyboard are that users can use it when wearing HMDs (users do not need to remove HMDs to use the VR keyboard), the haptic VR keyboard can pop-up display at any location in the VR environments (users do not need to go to a specific location to use an actual physical keyboard), and the haptic VR keyboard can be used to provide realistic key click haptic feedback (which other studies have shown enhances user performance). The results also show that the haptic VR keyboard system can be used to create complex vibrations that simulate measured vibrations from a real keyboard and enhance keyboard interaction in a fully immersive VR environment.     

A sequenced multimodal learning approach to support students' development of conceptual learning

Virtual learning environments can now be enriched not only with visual and auditory information, but also with tactile and kinesthetic feedback. However, the way to successfully integrate haptic feedback on a multimodal learning environment is still unclear. This study aims to provide guidelines on how visuohaptic simulations can be implemented effectively, thus the research question is: Under what conditions do visual and tactile information support students' development of conceptual learning of force‐related concepts? Participants comprised 170 undergraduate students of a Midwestern University enrolled in a physics for elementary education class. Four experiments were conducted using four different configurations of multimodal learning environments: Visual feedback only, haptic force feedback only, visual and haptic force feedback at the same time, and sequenced modality of haptic feedback first and visual feedback second. Our results suggest that haptic force feedback has the potential to enrich learning when compared with visual only environments. Also, haptic and visual modalities interact better when sequenced one after another rather than presented simultaneously. Finally, exposure to virtual learning environments enhanced by haptic forced feedback was a positive experience, but the ease of use and ease of interpretation was not so evident.     

Sensation-preserving haptic rendering

Most human-computer interactive systems focus primarily on the graphical rendering of visual information and, to a lesser extent, on the display of auditory information. Haptic interfaces have the potential to increase the quality of human-computer interaction by accommodating the sense of touch. They provide an attractive augmentation to visual display and enhance the level of understanding of complex data sets. A haptic rendering system generates contact or restoring forces to prevent penetration into the virtual objects and create a sense of touch. The system computes contact forces by first detecting if a collision or penetration has occurred. Then, the system determines the (projected) contact points on the model surface. Finally, it computes restoring forces based on the amount of penetration. Researchers have recently investigated the problem of rendering the contact forces and torques between 3D virtual objects. This problem is known as six-degrees-of-freedom (6-DOF) haptic rendering, as the computed output includes both 3-DOF forces and 3-DOF torques. This article presents an overview of our work in this area. We suggest different approximation methods based on the principle of preserving the dominant perceptual factors in haptic exploration.     

Networked multiplayer cooperative interaction using decoupled motion control method in a shared virtual environment with haptic,visual and movement feedback

This paper focuses on multiplayer cooperative interaction in a shared haptic environment based on a local area network. Decoupled motion control, which allows one user to manipulate a haptic interface to control only one‐dimensional movement of an avatar, is presented as a new type haptic‐based cooperation among multiple users. Users respectively move an avatar along one coordinate axis so that the motion of the avatar is the synthesis of movements along all axes. It is different from previous haptic cooperation where all users can apply forces on an avatar along any direction to move it, the motion of which completely depends on the resultant force. A novel concept of movement feedback is put forward where one user can sense other users’ hand motions through his or her own haptic interface. The concept can also be explained wherein one person who is required to move a virtual object along only one axis can also feel the motions of the virtual object along other axes. Movement feedback, which is a feeling of motion, differs from force feedback, such as gravity, collision force and resistance. A spring‐damper force model is proposed for the computation of motion feedback to implement movement transmission among users through haptic devices. Experimental results validate that movement feedback is beneficial for performance enhancement of such kind of haptic‐based cooperation, and the effect of movement feedback in performance improvement is also evaluated by all subjects.Copyright © 2012 John Wiley & Sons, Ltd.     

Influences of haptic communication on a shared manual task

With the advent of new haptic feedback devices, researchers are giving serious consideration to the incorporation of haptic communication in collaborative virtual environments. For instance, haptic interactions based tools can be used for medical and related education whereby students can train in minimal invasive surgery using virtual reality before approaching human subjects. To design virtual environments that support haptic communication, a deeper understanding of humans′ haptic interactions is required. In this paper, human′s haptic collaboration is investigated. A collaborative virtual environment was designed to support performing a shared manual task. To evaluate this system, 60 medical students participated to an experimental study. Participants were asked to perform in dyads a needle insertion task after a training period. Results show that compared to conventional training methods, a visual-haptic training improves user′s collaborative performance. In addition, we found that haptic interaction influences the partners′ verbal communication when sharing haptic information. This indicates that the haptic communication training changes the nature of the users′ mental representations. Finally, we found that haptic interactions increased the sense of copresence in the virtual environment: haptic communication facilitates users′ collaboration in a shared manual task within a shared virtual environment. Design implications for including haptic communication in virtual environments are outlined.     

An investigation on the impact of auditory and haptic feedback on rhythmic walking interactions

This paper presents a system capable of rhythmic walking interaction by auditory and haptic display. Likewise, it summarizes the results of the research on the influence of the audio and haptic stimulation on rhythmic walking interaction. The system detects user’s footsteps and either provides interactive real-time feedback or suggests a pace using a synthetic walking sound or vibration. This pace is either a constant tempo or adapts to the walker. Auditory and haptic feedback signals are either ecological physically-based synthetic walking signals or simple sinusoidal beeps. In the experiment, the different auditory and haptic feedback and interaction modes are studied with respect to their effect on the walking tempo. The results show that participants synchronise equally well with the tempo with either audio or haptic cues, but indicate the audio–haptic conditions as the easiest to synchronise with. Moreover, results indicate that multimodal audio–haptic feedback provide the most natural feeling. These results have implications on the design of interactive entertainment or therapeutical applications.     

Virtual DesignWorks—designing 3D CAD models via haptic interaction

Haptic devices (force feedback interfaces) show great promise for use in the design of CAD models. However, current haptic systems are used primarily to verify rather than to interact with CAD systems to design a model.The paper discusses a haptic-based CAD system—Virtual DesignWorks. It is the world's first haptic application for the design of CAD models based on component technology (COM+). With haptic devices, the system allows engineers, in virtual space, to directly touch a native B-Rep CAD model, and deform it by pushing, pulling and dragging its surfaces in a natural three-dimensional environment. Virtual DesignWorks adopts the novel COM-based haptic model, which demonstrates significant advantages compared to the traditional haptic models. Force feedback gives designers the greatest flexibility for the design of complex surfaces, and haptics has the potential to become a critical interface for design applications.