Haptics in virtual reality and multimedia

Imagine that you're chatting with a good friend who tells you about her new cashmere pullover, and that you're able to touch it and appreciate the soft sensation of the cashmere wool - remotely over the Web in your virtual chatroom! In this article, the authors present their work on haptic sensing of virtual textiles. This is work done in the context of the Haptex research project that aims at integrating the human sense of touch into multimodal user interfaces. Being able to support haptic perception in the user interface will be a great step toward next-generation immersive multimedia experiences.     

Multimodal Interaction on Automultiscopic Content with Mobile Surface Haptics

In this work, we present interactive automultiscopic content with mobile surface haptics for multimodal interaction. Our system consists of a 40‐view automultiscopic display and a tablet supporting surface haptics in an immersive room. Animated graphics are projected onto the walls of the room. The 40‐view automultiscopic display is placed at the center of the front wall. The haptic tablet is installed at the mobile station to enable the user to interact with the tablet. The 40‐view real‐time rendering and multiplexing technology is applied by establishing virtual cameras in the convergence layout. Surface haptics rendering is synchronized with three‐dimensional (3D) objects on the display for real‐time haptic interaction. We conduct an experiment to evaluate user experiences of the proposed system. The results demonstrate that the system's multimodal interaction provides positive user experiences of immersion, control, user interface intuitiveness, and 3D effects.     

Phantom-based multimodal interactions for medical education and training: the Munich knee joint simulator

Simulation environments based on virtual reality technologies can support medical education and training. In this paper, the novel approach of an "interactive phantom" is presented that allows a realistic display of haptic contact information typically generated when touching and moving human organs or segments. The key idea of the haptic interface is to attach passive phantom objects to a mechanical actuator. The phantoms look and feel as real anatomical objects. Additional visualization of internal anatomical and physiological information and sound generated during the interaction with the phantom yield a multimodal approach that can increase performance, didactic value, and immersion into the virtual environment. Compared to classical approaches, this multimodal display is convenient to use, provides realistic tactile properties, and can be partly adjusted to different, e.g., pathological properties. The interactive phantom is exemplified by a virtual human knee joint that can support orthopedic education, especially for the training of clinical knee joint evaluation. It is suggested that the technical principle can be transferred to many other fields of medical education and training such as obstetrics and dentistry.     

Testing usability of multimodal applications with visually impaired children

Testing multimodal applications with visually impaired children requires specific testing methods and procedures. In this article, we used a SensAble Phantom device to produce haptic feedback together with stereo sound and visual feedback. We also experimented with low-cost haptic devices, such as force-feedback game controllers and tactile-feedback devices. Here we discuss how to conduct usability testing for multimodal applications with visually impaired children.     

Haptic holography: a primitive computational plastic

We describe our work on haptic holography, a combination of computational modeling and multimodal spatial display, which allows a person to see, feel, and interact with three-dimensional freestanding holographic images of material surfaces. In this paper, we combine various holographic displays with a force-feedback device to render multimodal images with programmatically prescribed material properties and behavior. After a brief overview of related work which situates visual display within the manual workspace, we describe our holo-haptic approach and survey three implementations, Touch, Lathe, and Poke, each named for the primitive functional affordance it offers. In Touch, static holographic images of simple geometric scenes are reconstructed in front of the hologram plane, and coregistered with a force model of the same geometry. These images can be visually inspected and haptically explored using a handheld interface. In Lathe, a holo-haptic image can be reshaped by haptic interaction in a dynamic but constrained manner. Finally in Poke, using a new technique for updating interference-modeled holographic fringe patterns, we render a holo-haptic image that permits more flexible interactive reshaping of its reconstructed surface. We situate this work within the context of related research and describe the strengths, shortcomings, and implications of our approach.     

Toward multimodal human-computer interface

Recent advances in various signal processing technologies, coupled with an explosion in the available computing power, have given rise to a number of novel human-computer interaction (HCI) modalities: speech, vision-based gesture recognition, eye tracking, electroencephalograph, etc. Successful embodiment of these modalities into an interface has the potential of easing the HCI bottleneck that has become noticeable with the advances in computing and communication. It has also become increasingly evident that the difficulties encountered in the analysis and interpretation of individual sensing modalities may be overcome by integrating them into a multimodal human-computer interface. We examine several promising directions toward achieving multimodal HCI. We consider some of the emerging novel input modalities for HCI and the fundamental issues in integrating them at various levels, from early signal level to intermediate feature level to late decision level. We discuss the different computational approaches that may be applied at the different levels of modality integration. We also briefly review several demonstrated multimodal HCI systems and applications. Despite all the recent developments, it is clear that further research is needed for interpreting and fitting multiple sensing modalities in the context of HCI. This research can benefit from many disparate fields of study that increase our understanding of the different human communication modalities and their potential role in HCI     

Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality

Virtual reality and augmented reality (VR/AR) are evolving. The market demands and innovation efforts call for a shift in the key VR/AR technologies and engaging people virtually. Tele-haptics with multimodal and bilateral interactions are emerging as the future of the VR/AR industry. By transmitting and receiving haptic sensations wirelessly, tele-haptics allow human-to-human interactions beyond the traditional VR/AR interactions. The core technologies for tele-haptics include multimodal tactile sensing and feedback based on highly advanced sensors and actuators. Recent developments of haptic innovations based on active materials show that active materials can significantly contribute to addressing the needs and challenges for the current and future VR/AR technologies. Thus, this paper intends to review the current status and opportunities of active material-based haptic technology with a focus on VR/AR applications. It first provides an overview of the current VR/AR applications of active materials for haptic sensing and actuation. It then highlights the state-of-the-art haptic interfaces that are relevant to the materials with an aim to provide perspectives on the role of active materials and their potential integration in haptic devices. This paper concludes with the perspective and outlook of immersive multimodal tele-haptic interaction technologies.     

Adaptive impedance control of a haptic interface

Teleoperation enables an operator to manipulate remote objects. One of the main goals in teleoperation research is to provide the operator with the feeling of the telepresent object and of being present at the remote site. In order for this to happen, a master robot must be designed as a bilateral control system that can transmit position commands to a slave robot and reflect the interaction force. A newly proposed adaptive impedance algorithm is applied to the force control of a haptic interface that has been developed as a master robot. With the movement of the haptic interface for position command generation, the impedance between an operator and the haptic interface varies dynamically. When the impedance parameters and the dynamics of the haptic interface are known precisely, many model based control theories and methods can be used to control the interface accurately. However, due to the parameters’ variations and the uncertainty in the dynamic model, it is difficult to control the interface precisely. Therefore, this paper proposes a new adaptive impedance control algorithm and experimentally verifies the effectiveness of the algorithm for control of the haptic interface.     

Perceptual Issues in Haptic Digital Watermarking

The growing interest in haptic applications suggests that haptic digital media will soon become widely available, and the need will arise to protect digital haptic data from misuse. In this article, we present our study and findings on psychophysical experiments regarding human abilities to perceive a digital watermark, or hidden signal, through a haptic interface.     

User-centered modeling and evaluation of multimodal interfaces

Historically, the development of computer interfaces has been a technology-driven phenomenon. However, new multimodal interfaces are composed of recognition-based technologies that must interpret human speech, gesture, gaze, movement patterns, and other complex natural behaviors, which involve highly automatized skills that are not under full conscious control. As a result, it now is widely acknowledged that multimodal interface design requires modeling of the modality-centered behavior and integration patterns upon which multimodal systems aim to build. This paper summarizes research on the cognitive science foundations of multimodal interaction, and on the essential role that user-centered modeling has played in prototyping, guiding, and evaluating the design of next-generation multimodal interfaces. In particular, it discusses the properties of different modalities and the information content they carry, the unique features of multimodal language and its processability, as well as when users are likely to interact multimodally and how their multimodal input is integrated and synchronized. It also reviews research on typical performance and linguistic efficiencies associated with multimodal interaction, and on the user-centered reasons why multimodal interaction minimizes errors and expedites error handling. In addition, this paper describes the important role that selective methodologies and evaluation metrics have played in shaping next-generation multimodal systems, and it concludes by highlighting future directions for designing a new class of adaptive multimodal-multisensor interfaces.     

The effect of force saturation on the haptic perception of detail

This paper presents a quantitative study of the effects of maximum capable force magnitude of a haptic interface on the haptic perception of detail. Specifically, the haptic perception of detail is characterized by identification, detection, and discrimination of round and square cross-section ridges, in addition to corner detection tests. Test results indicate that performance, measured as a percent correct score in the perception experiments, improves in a nonlinear fashion as the maximum allowable level of force in the simulation increases. Further, all test subjects appeared to reach a limit in their perception capabilities at maximum-force output levels of 3-4 N, while the hardware was capable of 10 N of maximum continuous force output. These results indicate that haptic interface hardware may be able to convey sufficient perceptual information to the user with relatively low levels of force feedback. The data is compiled to aid those who wish to design a stylus-type haptic interface to meet certain requirements for the display of physical detail within a haptic simulation.     

Development of micromanipulator and haptic interface for networkedmicromanipulation

Telemicromanipulation systems with haptic feedback, which are connected through a network, are proposed. It is based on scaled bilateral teleoperation systems between different structures. These systems are composed of an original 6 degree of freedom (DOF) parallel link manipulator to carry out micromanipulation and a 6-DOF haptic interface with force feedback. A parallel mechanism is adopted as a slave micromanipulator because of its good features of accuracy and stiffness. The system modeling and control of the parallel manipulator system are conducted. Parallel manipulator feasibility as a micromanipulator, positioning accuracy and device control characteristics are investigated. A haptic master interface is developed for micromanipulation systems. System modeling and a model reference adaptive controller are applied to compensate friction force, which spoils free motion performance and force response isotropy of the haptic interface. These systems aim to make the micromanipulation more productive constructing a better human interface through the microenvironment force and scale expansion     

Programmable and Ultrasensitive Haptic Interfaces Enabling Closed-Loop Human–Machine Interactions

Intuitive, efficient, and unconstrained interactions require human–machine interfaces (HMIs) to accurately recognize users' manipulation intents. Susceptibility to interference and conditional usage mode of HMIs will lead to poor experiences that limit their great interaction potential. Herein, a programmable and ultrasensitive haptic interface enabling closed-loop human–machine interactions is reported. A cross-scale architecture design strategy is proposed to fabricate the haptic interface, which optimizes the hierarchical contact process. The synergistic optimization of the cross-scale architecture between carbon nanotubes and the multiscale sensing structure realizes a haptic interface with ultrahigh sensitivity and a wide detection range of 15.1 kPa⁻¹ and 180 kPa, which are improved by more than 900% over the performance of the common interface. The rapid response time of <5 ms and the limit of detection of 8 Pa of the haptic interface far surpass the somatosensory perception of human skin, which enables the haptic interface to accurately recognize interactive intents. A wireless pressure-data interactive glove (wireless PDI glove) is designed and realizes a round-the-clock operation, noise immunity, and efficient interactive control, which perfectly compensate for the flaws of typical vision and voice recognition modes.     

Haptic Interaction Stability With Respect to Grasp Force

This paper addresses the contact instability of admittance control of a haptic interface. A high level of rigidity of the grasp of a subject operating the haptic interface will result in unstable behavior of the haptic interaction. Experiments with a system dedicated to measuring grasp force were performed to explore the conditions when grasp force has reached the critical grasp force that destabilizes the haptic interface. The critical grasp force was quantified for various values of virtual environment parameters. The experimental results are compared to simulation results obtained with a model of haptic interaction. To improve stability, two methods were applied: one with virtual coupling, the other with a compensator filter. A model was used to define the structure of the compensator filter and to determine the parameters of the virtual coupling and the compensator filter. Experimental and simulation results confirmed an improvement of stability. Both methods allow higher grasp forces of the human operator, and experiments show that the compensator filter allows higher grasp forces than the virtual coupling.     

Haptic display with an interface device capable of continuous-time impedance display within a sampling period

Stability of a haptic interface is an important issue in virtual reality because an operator directly touches haptic interface devices. Stability is influenced by the sampling period and the discrete-time property of the control system. For decreasing the sampling system influence, this paper proposes a haptic device with an analog circuit which is placed between the computer and the haptic device. The circuit functions as springs and dampers. The control system can specify stiffness, damping coefficients, and their equilibrium. Since the impedance generated by the electric spring and damper can work continuously within the sampling period, it is effective for making the system more stable. We also develop a control method for displaying a static virtual object with the proposed device. Further, we discuss the effects of the proposed approach analytically, using passivity analysis for a 1-degree-of-freedom display system. Finally, some experimental results in a two-dimensional virtual environment are presented.     

Design of a haptic arm exoskeleton for training and rehabilitation

A high-quality haptic interface is typically characterized by low apparent inertia and damping, high structural stiffness, minimal backlash, and absence of mechanical singularities in the workspace. In addition to these specifications, exoskeleton haptic interface design involves consideration of space and weight limitations, workspace requirements, and the kinematic constraints placed on the device by the human arm. These constraints impose conflicting design requirements on the engineer attempting to design an arm exoskeleton. In this paper, the authors present a detailed review of the requirements and constraints that are involved in the design of a high-quality haptic arm exoskeleton. In this context, the design of a five-degree-of-freedom haptic arm exoskeleton for training and rehabilitation in virtual environments is presented. The device is capable of providing kinesthetic feedback to the joints of the lower arm and wrist of the operator, and will be used in future work for robot-assisted rehabilitation and training. Motivation for such applications is based on findings that show robot-assisted physical therapy aids in the rehabilitation process following neurological injuries. As a training tool, the device provides a means to implement flexible, repeatable, and safe training methodologies.     

Design of a force control grasper through stiffness modulation for endosurgery: theory and experiments

In endosurgery, tissue manipulation is performed by long graspers which have a poor force-reflection property, i.e. they do not reflect the grasping force to the hand of the surgeon. In this paper, a novel design of an electro-mechanical system is considered which can enhance the force–reflecting capability of endosurgical graspers. The type of synthesis of such a haptic interface leads to the application of a tunable spring. The design of a tunable spring is optimized based on the haptic and surgical requirements. Moreover, by using suitable control laws, it is shown that the primary requirements of the design application can be met. Simulation and experimental results are presented using a prototype model of such a haptic interface to demonstrate the practicality of such a design concept.     

Haptic Interface of the KAIST-Ewha Colonoscopy Simulator II

This paper presents an improved haptic interface for the Korea Advanced Institute of Science and Technology Ewha Colonoscopy Simulator II. The haptic interface enables the distal portion of the colonoscope to be freely bent while guaranteeing sufficient workspace and reflective forces for colonoscopy simulation. Its force–torque sensor measures the profiles of the user. Manipulation of the colonoscope tip is monitored by four deflection sensors and triggers computations to render accurate graphic images corresponding to the rotation of the angle knob. Tack sensors are attached to the valve-actuation buttons of the colonoscope to simulate air injection or suction as well as the corresponding deformation of the colon. A survey study for face validation was conducted, and the result shows that the developed haptic interface provides realistic haptic feedback for colonoscopy simulations.      

A General-Purpose 7 DOF Haptic Device: Applications Toward Robot-Assisted Surgery

A 7 DOF haptic device has been designed and developed with applications towards robot-assisted minimally invasive surgery. The device consists of four degrees of force feedback (X, Y, Z, and grasping) capability and seven degrees of position feedback capability. It has a closed kinematic chain with two halves (user interface and spatial mechanism) that connect together via a universal joint. The user interface contains four degrees of position feedback, namely, the roll, pitch, yaw, and linear motion of the hand and forearm. In addition, a grasping mechanism with two thimbles mounted at the end of the user interface provides force feedback to the fingers of the user. The spatial mechanism provides force feedback to the user interface through a universal joint located at the grasping mechanism. This paper presents the design and development of this haptic device. In addition, a kinematic and workspace analysis of the device has been completed to compute the position of the slave robot and end-effector tool. Friction estimation has been presented to enable a higher transparency of the haptic device. Finally, a simulation of needle insertion into soft tissue was developed to test the device.