Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

On the development of a haptic system for rapid product development

This paper presents a system development that extends haptic modeling to a number of key aspects in product development. Since haptic modeling has been developed based on physical laws, it is anticipated that a natural link between the virtual world and practical applications can be established based on haptic interaction. In the proposed system, a haptic device is used as the central mechanism for reverse engineering, shape modeling, real time mechanical property analysis, machining tool path planning and coordinate measuring machine (CMM) tolerance inspection path planning. With all these features in a single haptic system, it is possible to construct a three dimensional part by either haptic shape modeling or reverse engineering, then performing real-time mechanical property analysis in which the stiffness of a part can be felt and intuitively evaluated by the user, or generating collision free cutter tool path and CMM tolerance inspection path. Due to the force feed back in all of the above activities, the product development process is more intuitive, efficient and user-friendly. A prototype system has been developed to demonstrate the proposed capabilities.     

A large haptic device for aircraft engine maintainability

The virtual reality for maintainability (Revima) VR system supports maintainability simulation in aeronautics. Within this project we have developed and integrated a haptic device, the large haptic interface for aeronautic maintainability (LHIfAM). We use this device to track hand movements and provide force feedback within the large geometric models that describe aircraft engines. The user movements are the same as those that occur when testing physical mock-ups. An integrated haptic device and VR system for testing aircraft engines reduces development costs and avoids the necessity of physical mock-ups formaintainability.     

A reverse engineering method based on haptic volume removing

This paper presents a new reverse engineering methodology that is based on haptic volume removing. When a physical object is to be digitized, it is first buried in a piece of virtual clay that is generated with the help of a fixture. Now digitizing the physical object is by simply chipping away the virtual clay with a position tracker that is attached to a haptic device PHANToM^®. While chipping away the clay, the user can see on the computer monitor what is emerging and at the same time feel the chipping force from the haptic device. By so doing, reverse engineering is seamlessly integrated into haptic volume sculpting that is now widely used for conceptual design. Furthermore, the proposed method has eliminated the need to merge point clouds that are digitized from different views using current digitizers. The virtual clay volume is represented by a spatial run-length encoding scheme. A prototype system has been developed to demonstrate the feasibility of the proposed new method through a case study. The strengths and weaknesses of the presented method are analyzed and the applicability is discussed.     

Haptic Direct-Drive Robot Control Scheme in Virtual Reality

This paper explores the use of a 2-D (Direct-Drive Arm) manipulator for mechanism design applications based on virtual reality (VR). This article reviews the system include a user interface, a simulator, and a robot control scheme. The user interface is a combination of a virtual clay environment and human arm dynamics via robot effector handler. The model of the VR system is built based on a haptic interface device behavior that enables the operator to feel the actual force feedback from the virtual environment just as s/he would from the real environment. A primary stabilizing controller is used to develop a haptic interface device where realistic simulations of the dynamic interaction forces between a human operator and the simulated virtual object/mechanism are required. The stability and performance of the system are studied and analyzed based on the Nyquist stability criterion. Experiments on cutting virtual clay are used to validate the theoretical developments. It was shown that the experimental and theoretical results are in good agreement and that the designed controller is robust to constrained/unconstrained environment.     

桌面式虚拟现实维修训练系统的研究与应用

计算机虚拟现实技术提供一个具有直观感和沉浸感的人机界面,可作为一种用于维修训练的有效工具,但目前的相关研究主要针对沉浸式和增强型虚拟现实,开发成本高且应用性不强。该文提出一种面向对象的桌面式虚拟现实维修训练系统原型,利用面向对象的方法,将基于最优拆卸序列的维修模式运用到系统中。通过某型机械装置的虚拟拆卸,验证了该系统的有效性。     

An information-theoretic treatment of passive haptic media

Haptic rendering has been long considered as the process of estimating the force that stems from the interaction of a user and an object. Even if this approach follows the principles of natural haptic interaction, it places severe limitations in processing haptic media. This paper presents an information theoretic framework that aims to provide a new view of haptic rendering that can accommodate for open-loop synthetic haptic media, where interaction-based rendering is a special case. As a result, using the proposed information-theoretic approach, the haptic signal can be precomputed as a force field, stored and then filtered by taking into account device and perceptual capabilities of the receiver in order to lower the required bandwidth of the resulting stream, thus opening new possibilities for the representation and processing of haptic media.     

Interaction model between elastic objects for haptic feedback considering collisions of soft tissue

The simulation of organ–organ interaction is indispensable for practical and advanced medical VR simulator such as open surgery and indirect palpation. This paper describes a method to represent real-time interaction between elastic objects for accurate force feedback in medical VR simulation. The proposed model defines boundary deformation of colliding elements based on temporary surface forces calculated by temporary deformation. The model produces accurate deformation and force feedback considering collisions of objects as well as prevents unrealistic overlap of objects. A prototype simulator of rectal palpation is constructed on general desktop PC with a haptic device, PHANToM. The system allows users to feel different stiffness of a rear elastic object located behind another elastic object. The results of experiments confirmed the method expresses organ–organ interaction in real-time and produces realistic and perceivable force feedback.     

A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment

This study presents a 3D virtual reality (VR) keyboard system with realistic haptic feedback. The system uses two five-fingered data gloves to track finger positions and postures, uses micro-speakers to create simulated vibrations, and uses a head-mounted display (HMD) for 3D display. When users press a virtual key in the VR environment, the system can provide realistic simulated key click haptic feedback to users. The results of this study show that the advantages of the haptic VR keyboard are that users can use it when wearing HMDs (users do not need to remove HMDs to use the VR keyboard), the haptic VR keyboard can pop-up display at any location in the VR environments (users do not need to go to a specific location to use an actual physical keyboard), and the haptic VR keyboard can be used to provide realistic key click haptic feedback (which other studies have shown enhances user performance). The results also show that the haptic VR keyboard system can be used to create complex vibrations that simulate measured vibrations from a real keyboard and enhance keyboard interaction in a fully immersive VR environment.     

Robot-enabled tangible virtual assembly with coordinated midair object placement

The assembly in Virtual Reality (VR) enables users to fit virtual parts into existing 3D models immersively. However, users cannot physically feel the haptic feedback when connecting the parts with the virtual model. This work presents a robot-enabled tangible interface that dynamically moves a physical structure with a robotic arm to provide physical feedback for holding a handheld proxy in VR. This enables the system to provide force feedback during virtual assembly. The cooperation between the physical support and the handheld proxy produces realistic physical force feedback, providing a tangible experience for various virtual parts in virtual assembly scenarios. We developed a prototype system that allowed the operator to place a virtual part onto other models in VR by placing the proxy onto the matched structure attached to a robotic arm. We conducted a user evaluation to explore user performance and system usability in a virtual assembly task. The results indicated that the robot-enabled tangible support increased the task completion time but significantly improved the system usability and sense of presence with a more realistic haptic experience.     

A brush device with visual and haptic feedback for virtual painting of 3D virtual objects

We have previously developed a mixed reality (MR) painting system with which a user could take a physical object in the real world and apply virtual paint to it. However, this system could not provide the sensation of painting on virtual objects in MR space. Therefore, we subsequently proposed and developed mechanisms that simulated the effect of touch and movement when a brush device was used to paint on a virtual canvas. In this paper, we use visual and haptic feedback to provide the sensation of painting on virtual three-dimensional objects using a new brush device called the MAI Painting Brush++. We evaluate and confirm its effectiveness through several user studies.     

Neural dynamics-based Poisson propagation for deformable modelling

This paper presents a new methodology from the standpoint of energy propagation for real-time and nonlinear modelling of deformable objects. It formulates the deformation process of a soft object as a process of energy propagation, in which the mechanical load applied to the object to cause deformation is viewed as the equivalent potential energy based on the law of conservation of energy and is further propagated among masses of the object based on the nonlinear Poisson propagation. Poisson propagation of mechanical load in conjunction with non-rigid mechanics of motion is developed to govern the dynamics of soft object deformation. Further, these two governing processes are modelled with cellular neural networks to achieve real-time computational performance. A prototype simulation system with a haptic device is implemented for real-time simulation of deformable objects with haptic feedback. Simulations, experiments as well as comparisons demonstrate that the proposed methodology exhibits nonlinear force–displacement relationship, capable of modelling large-range deformation. It can also accommodate homogeneous, anisotropic and heterogeneous materials by simply changing the constitutive coefficient value of mass points.

     

Construction of a supermicro sense of force feedback and vision for micro-objects: development of a haptic device

This research aims to develop a combined sense system that uses both the force feedback and visual feedback to establish the shape of microscopic features of a microsample. It is thought that the efficiency of minute procedures would be improved if the operator could obtain a sense of force while using a manipulator. We used a cantilever to touch a minute object and obtain a reaction force from the degree of bending. We made a haptic device which gives a sense of that force to the operator, who can feel the force when a user touches a sample with a cantilever. In addition, when the haptic device is used in simulations, the user can feel a force just as if the user had touched a sample.     

A desktop VR prototype for industrial training applications

The recent advances in computer graphics has spurred interest from both academics and industries in virtual reality (VR) enabled training applications. This paper presents a desktop VR prototype for industrial training applications. It is designed and implemented as a general shell by providing the data interface to import both the virtual environment models and specific domain knowledge. The geometric models of the virtual environment are constructed using feature-based modelling and assembly function by external CAD tools, and then transferred into the prototype through a conversion module. A hierarchical structure is proposed to partition and organise these imported virtual environment models. Based on this structure, a visibility culling approach is developed for fast rendering and user interaction. The case study has demonstrated the functionality of the proposed prototype system by applying it to a maintenance training application for a refinery bump system, which, in general, has a large number of polygons and a certain depth complexity. Significant speedup in both context rendering and response to user manipulations has been achieved to provide the user with a fast system response within the desktop virtual environment. Compared with the immersive VR system, the proposed system has offered an affordable and portable training media for industrial applications.The work was done in Nanyang Technological University.     

Object interaction and task programming by demonstration in visuo-haptic augmented reality

A visuo-haptic augmented reality system is presented for object manipulation and task learning from human demonstration. The proposed system consists of a desktop augmented reality setup where users operate a haptic device for object interaction. Users of the haptic device are not co-located with the environment where real objects are present. A three degrees of freedom haptic device, providing force feedback, is adopted for object interaction by pushing, selection, translation and rotation. The system also supports physics-based animation of rigid bodies. Virtual objects are simulated in a physically plausible manner and seem to coexist with real objects in the augmented reality space. Algorithms for calibration, object recognition, registration and haptic rendering have been developed. Automatic model-based object recognition and registration are performed from 3D range data acquired by a moving laser scanner mounted on a robot arm. Several experiments have been performed to evaluate the augmented reality system in both single-user and collaborative tasks. Moreover, the potential of the system for programming robot manipulation tasks by demonstration is investigated. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations and that the learned task can be executed by a robot system.     

Haptic palpation for medical simulation in virtual environments

Palpation is a physical examination technique where objects, e.g., organs or body parts, are touched with fingers to determine their size, shape, consistency and location. Many medical procedures utilize palpation as a supplementary interaction technique and it can be therefore considered as an essential basic method. However, palpation is mostly neglected in medical training simulators, with the exception of very specialized simulators that solely focus on palpation, e.g., for manual cancer detection. In this article we propose a novel approach to enable haptic palpation interaction for virtual reality-based medical simulators. The main contribution is an extensive user study conducted with a large group of medical experts. To provide a plausible simulation framework for this user study, we contribute a novel and detailed interaction algorithm for palpation with tissue dragging, which utilizes a multi-object force algorithm to support multiple layers of anatomy and a pulse force algorithm for simulation of an arterial pulse. Furthermore, we propose a modification for an off–the–shelf haptic device by adding a lightweight palpation pad to support a more realistic finger grip configuration for palpation tasks. The user study itself has been conducted on a medical training simulator prototype with a specific procedure from regional anesthesia, which strongly depends on palpation. The prototype utilizes a co-rotational finite-element approach for soft tissue simulation and provides bimanual interaction by combining the aforementioned techniques with needle insertion for the other hand. The results of the user study suggest reasonable face validity of the simulator prototype and in particular validate medical plausibility of the proposed palpation interaction algorithm.     

Generating haptic texture using solid noise

Texture enhances haptic interaction by providing unique, distinguishable, and versatile surfaces. In computer haptics, texture can render environments more realistic and provide useful information. In this paper, an algorithm is proposed for virtual texture simulation by using solid noise, where only a few parameters need to be altered to generate a range of realistic and diverse textures by reproducing different frequencies similar to that of real vibrational signals in a virtual environment. The proposed method can capture the textural effect in a haptic simulation while retaining a simple overall geometry and stable update rate. This method also allows the user to change the texture at runtime and can be easily incorporated into any existing code and used in any traditional haptic device without affecting overall haptic-rendering performance. Moreover, the solid noise texture is independent of object geometry and can be applied to any shape without additional computations. We conducted a human-subject study to evaluate the recognition accuracy for each generated haptic texture as well as its realism and correspondence to real texture. The results indicated the high performance of the method and its ability to generate haptic textures with a very high recognition rate that were highly realistic.     

Bridging the HASM: An OWL ontology for modeling the information pathways in haptic interfaces software

Haptics technology has received enormous attention to enhance human computer interaction. The last decade has witnessed a rapid progress in haptic application software development due to the fact that the underlying technology has become mature and has opened up novel research areas. In an attempt to organize the path between cause and effect we envision a need for a standard for haptic application software modeling. In order for the software to better enhance the tactile information sensation, flow and perception and also make interaction between humans and haptics more efficient and natural, we need a formal representation of the haptics domain. This article proposes the use of HASM, a haptic applications software modeling ontology to formally model the haptics domain in order to be used during the specifications and design phases of developing software applications for haptic interfaces. The presented ontology captures the existing knowledge in haptics domain, using OWL, and defines the pathways that the haptic information follows between the human and the machine haptic system, using SWRL rules. The haptic ontology that has been developed will be used as a basis to design effective user interfaces and assist the development of software modeling for haptic devices. A case study is demonstrating how this haptic ontology can be used to design a software model that analyzes the perception of a haptic property of an object by interacting with a haptic device.     

Intelligent Control for Haptic Displays

Usually, a mouse is used for input activities only, whereas output from the computer is sent via the monitor and one or two loudspeakers. But why not use the mouse for output, too? For instance, if it would be possible to predict the next interaction object the user wants to click on, a mouse with a mechanical brake could stop the cursor movement at the desired position. This kind of aid is especially attractive for small targets like resize handles of windows or small buttons. In this paper, we present an approach for the integration of haptic feedback in everyday graphical user interfaces. We use a specialized mouse which, is able to apply simple haptic information, to the user's hand and index finger. A multi-agent system has been designed which ‘observes’ the ‘user in order to predict the next interaction object and launch haptic feedback, thus supporting positioning actions with the mouse. Although primarily designed in order to provide intelligent’ haptic feedback, the system can be combined with other output modalities as well, due to its modular and flexible architecture.     

Developing speech input for virtual reality applications: A reality based interaction approach

An input device should be natural and convenient for a user to transmit information to a computer, and should be designed from an understanding of the task to be performed and the interrelationship between the task and the device from the perspective of the user. In order to investigate the potential of speech input as a reality based interaction device, this paper presents the findings of a study that investigated speech input in a VR application. Two independent user trials were combined within the same experimental design to evaluate the commands that users employed when they used free speech in which they were not restricted to a specific vocabulary. The study also investigated when participants were told they were either talking to a machine (e.g. a speech recognition system) or instructing another person to complete a VR based task. Previous research has illustrated that when users are limited to a specific vocabulary, this can alter the interaction style employed. The findings from this research illustrate that the interaction style users employ are very different when they are told they are talking to a machine or another person. Using this knowledge, recommendations can be drawn for the development of speech input vocabularies for future VR applications.