Haptic Perception,Mechanics, and Material Technologies for Virtual Reality

Emerging virtual and augmented reality technologies can transform human activities in myriad domains, lending tangible, embodied form to digital data, services, and information. Haptic technologies will play a critical role in enabling human to touch and interact with the contents of these virtual environments. The immense variety of skilled manual tasks that humans perform in real environments are only possible through the coordination of touch sensation, perception, and movement that together comprise the haptic modality. Consequently, many research groups are vigorously investigating haptic technologies for virtual reality. A longstanding research goal in this area has been to create haptic interfaces that allow their users to touch and feel plausibly realistic virtual objects. In this progress report, the perspective on this unresolved research challenge is shared, guided by the observation that no technologies can even approximately match the capabilities of the human sense of touch. Factors that have it challenging to engineer haptic technologies for virtual reality, including the extraordinary spatial and temporal tactile acuity of the skin, and the complex interplay between continuum mechanics, haptic perception, and interaction are identified. The perspective on how these challenges may be overcome through convergent research on haptic perception, mechanics, electronics, and material technologies is presented.     

Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality

Virtual reality and augmented reality (VR/AR) are evolving. The market demands and innovation efforts call for a shift in the key VR/AR technologies and engaging people virtually. Tele-haptics with multimodal and bilateral interactions are emerging as the future of the VR/AR industry. By transmitting and receiving haptic sensations wirelessly, tele-haptics allow human-to-human interactions beyond the traditional VR/AR interactions. The core technologies for tele-haptics include multimodal tactile sensing and feedback based on highly advanced sensors and actuators. Recent developments of haptic innovations based on active materials show that active materials can significantly contribute to addressing the needs and challenges for the current and future VR/AR technologies. Thus, this paper intends to review the current status and opportunities of active material-based haptic technology with a focus on VR/AR applications. It first provides an overview of the current VR/AR applications of active materials for haptic sensing and actuation. It then highlights the state-of-the-art haptic interfaces that are relevant to the materials with an aim to provide perspectives on the role of active materials and their potential integration in haptic devices. This paper concludes with the perspective and outlook of immersive multimodal tele-haptic interaction technologies.     

Thermo-Haptic Materials and Devices for Wearable Virtual and Augmented Reality

The current generation of virtual and augmented realities (VR/AR) has substantially advanced in the past decade because of the rapid development of converging technologies in various engineering and scientific fields. However, the current VR/AR technologies rely mainly on visual and auditory senses to physically replicate the virtual environment, although tactile senses play a significant role in the daily life since a myriad of tactile information is received through physical touch. Of the tactile senses, thermal senses are of great importance to be reproduced in the VR/AR field, since heat is transferred constantly and further interact with the surrounding environment. To date, there has been a huge amount of research studies on functional materials, thermo-haptic devices, and wearable electronics that have all converged to form the fundamental groundwork for the development of wearable thermal VR/AR devices. In this progress report, a review on various physical mechanisms and research is provided that can potentially be applied in the next generation of thermal VR/AR technologies and discuss the essential challenges that need to be addressed.     

Stretchable Skin‐Like Cooling/Heating Device for Reconstruction of Artificial Thermal Sensation in Virtual Reality

Along with visual and tactile sensations, thermal sensation by temperature feeling on the skin can provide rich physical information on the environment and objects. With a simple touch of objects, relative temperature can be sensed and even objects can be differentiated with different thermal properties without any visual cue. Thus, artificially reproducing accurate/controllable thermal sensation haptic signals on human epidermis will certainly be a major research area to reconstruct a more realistic virtual reality (VR) environment. In this study, for the first time, a skin‐like, highly soft and stretchable and bi‐functional (both cold and hot sensation) thermo‐haptic device is reported for wearable VR applications with a single device structure (not separate heater and cooler). The skin‐like thermo‐haptic (STH) device can actively cool down and heat up deformable skin surfaces with instantaneous and accurate adjustment of temperature based upon a feedback control algorithm to mimic desirable thermal sensation with 230% stretchability. As a proof‐of‐concept, the STH device is integrated with a finger‐motion tracking glove to provide artificial thermal sensation information to the skin in various situations such as touching cold beer bottles and hot coffee cups in virtual space. This new type of STH device can offer potential implications for next‐generation haptic devices to provide unique thermal information for a more realistic virtual‐world field and medical thermal treatment.     

Enabling 4-DoF hand guidance using a portable haptic device exerting tangential force on the user's finger pads

Body-grounded kinesthetic haptic devices can provide cues for movement in multiple degrees of freedom by exerting forces directly on users, as in dexterous robot teleoperation tasks. However, these haptic devices have limited workspaces, can destabilize a teleoperation control loop, and can be expensive. Portable haptic devices can approximate the sensations of a kinesthetic device by exploiting diverse human sense of touch principles without these shortcomings. Our goal is to analyze the feasibility of hand guidance (HG) using tangential force stimuli. Here we reveal and quantify users’ interpretation of simultaneous tactile stimulation (STS) applied to multiple finger pads of the same hand. We completed an extensive experiment on different users to reveal a maximum number of understandable cues which can be used as movement commands for HG. As expected, many tactile stimuli tested were meaningless for users, but a few could be clearly interpreted — we call these “intuitive movement cues”. For the experiment, we designed a device that can be held in the palm and exerts tactile stimuli to the user's finger pads on the thumb and index fingers, or the thumb and middle fingers. We performed two studies in which we identified the extent of salience of different movement cues. In particular, commands to redirect the hand position and orientation in four axes: moving forward/backward, wrist twisting right/left (rotate clockwise/counter-clockwise), moving right/left, and wrist tilting up/down (rotate upwards/downwards). The results revealed that this approach provided 7 intuitive directional movement cues for relative HG in 3D space. The proposed HG principle is promising for applications such as robotic surgery training, laparoscopic training, and needle insertion training, during which surgical trainees must learn dexterous hand movements involving motion paths. There are many applications for 3D movement guidance outside the medical domain that could benefit from this haptics technology, including training for precise manipulation and assembly tasks, augmented teleoperation, and communication during shared control in collaborative human-machine systems.     

Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

Haptic information in Internet-based teleoperation

Many tasks can be done easily by humans turn out to be very difficult to accomplish with a teleoperated robot. The main reason for this is the lack of tactile sensing, which cannot be replaced by visual feedback alone. Once haptic devices are developed, their potential in many fields is obvious. Especially, in teleoperation systems, where haptic feedback can increase the efficiency and even render some tasks feasible. This paper studies Internet-based teleoperation systems that include haptic feedback, concentrating on the control of such systems and their performance. The potential of this technology and its advantages are explored. In addition, key issues, such as stability, synchronization, and transparency are analyzed and studied. Specifically, an event-based planning and control of Internet-based teleoperation systems is presented with experimental results of several implemented system scenarios in micro- and macro-scales     

Phantom-based multimodal interactions for medical education and training: the Munich knee joint simulator

Simulation environments based on virtual reality technologies can support medical education and training. In this paper, the novel approach of an "interactive phantom" is presented that allows a realistic display of haptic contact information typically generated when touching and moving human organs or segments. The key idea of the haptic interface is to attach passive phantom objects to a mechanical actuator. The phantoms look and feel as real anatomical objects. Additional visualization of internal anatomical and physiological information and sound generated during the interaction with the phantom yield a multimodal approach that can increase performance, didactic value, and immersion into the virtual environment. Compared to classical approaches, this multimodal display is convenient to use, provides realistic tactile properties, and can be partly adjusted to different, e.g., pathological properties. The interactive phantom is exemplified by a virtual human knee joint that can support orthopedic education, especially for the training of clinical knee joint evaluation. It is suggested that the technical principle can be transferred to many other fields of medical education and training such as obstetrics and dentistry.     

Liquid Crystal Soft Actuators and Robots toward Mixed Reality

Liquid crystals (LCs) are soft but smart materials that can adjust its chemical or physical properties in response to various external stimuli. Using these materials to construct soft actuators and robots, referred as LC actuators and robots, is expected to replace current machinery part, obtaining lighter and smaller equipment with adjustable and complex functions. Especially, combining these LC actuator and robots with existing virtual reality and augmented reality technologies will produce a new world of mixed reality (MR) with the visual, auditory, and somatosensory interaction. In this review, the recent work on responsive LC actuators and robots is introduced, emphasizing on their potentials in haptic use. By discussing their programmable control via suitable stimuli, the LC actuators and robots are summarized for mechanical outputs, environmental mimic, and fine-tuning of surface texture and roughness. It is anticipated that the continuous development on LC actuators and robots will accelerate the MR technology toward practical application.     

Ionic Tactile Sensors for Emerging Human‐Interactive Technologies: A Review of Recent Progress

Ionic tactile sensors (ITS) represent a new class of deformable sensory platforms that mimic not only the tactile functions and topological structures but also the mechanotransduction mechanism across the biological ion channels in human skin, which can demonstrate a more advanced biological interface for targeting emerging human‐interactive technologies compared to conventional e‐skin devices. Recently, flexible and even stretchable ITS have been developed using novel structural designs and strategies in materials and devices. These skin‐like tactile sensors can effectively sense pressure, strain, shear, torsion, and other external stimuli with high sensitivity, high reliability, and rapid response beyond those of human perception. In this review, the recent developments of the ITS based on the novel concepts, structural designs, and strategies in materials innovation are entirely highlighted. In particular, biomimetic approaches have led to the development of the ITS that extend beyond the tactile sensory capabilities of human skin such as sensitivity, pressure detection range, and multimodality. Furthermore, the recent progress in self‐powered and self‐healable ITS, which should be strongly required to allow human‐interactive artificial sensory platforms is reviewed. The applications of ITS in human‐interactive technologies including artificial skin, wearable medical devices, and user‐interactive interfaces are highlighted. Last, perspectives on the current challenges and the future directions of this field are presented.     

A Realization of Haptic Training System by Multilateral Control

In recent years, the realization of a haptic system has been desired strongly in the fields of medical treatment and expert's skill acquisition. An integrated system design of the analysis of the interaction between a robot and the environment required for stable contact operation, an observation method of the reaction force from the environment, and the architecture of a bilateral control system are absolutely essential for acquisition and reproduction of a vivid tactile sensation. In this paper, a haptic training system is realized based on multilateral control. The law of action and reaction is attained by three robots. Bilateral control is extended, and multilateral control is introduced. Multilateral control is designed similarly as bilateral control based on modal decomposition; force is controlled in the common mode, and position is controlled in the differential mode. The scaling factors of position and force are set independently. Therefore, it is possible to change a trainer's assistant force according to a trainee's skill level. The proposed training system based on multilateral control will be a fundamental technology for the evolution of haptic devices     

A touch of the future: contact-expressive devices

Contact-expressive devices could improve our lives-from practical improvements that let us interact with devices when other modalities are impossible, inappropriate, or inadequate to devices that have important consequences for physical well-being. We can also imagine using contact expressions in devices that are pleasurable, playful, artistic, or otherwise important to a human's experience. This paper discusses the technical development of sensor-motor systems, tactile interfaces, and technologies that can embody and understand affect. Contact-expressive devices are technologies that understand and use touch in meaningful ways-that can distinguish between a press and a caress. Also discussed are the following issues: technical requirements and the nature of contact expressions. Although a fairly large body of research exists regarding nonverbal communication, the majority of this work is on visual expression (facial expressions, physical appearance, direction of gaze, physical posture, and other visible forms of body language) and paralinguistics (such as vocal inflection, pitch, volume, and speech rate). Work needs to begin on studies of human contact with the goal of leveraging those insights for the design of contact-expressive devices-and work needs to begin on the implementation of technologies that have rich contact-expressive range and appeal.     

4-inch transparent plates based on thin-film AlN actuators for haptic applications

Numerous applications require tactile interfaces today. In particular, many customers’ applications such as automotive, Smartphone, tablet PC or touch pad can be concerned by high performances, low voltage haptic interfaces which allow the user to interact with its environment by the sense of touch. This technology is already used but with limitations such as high power consumption and limited feedback effect because today a simple vibration is commonly obtained. We chose to work on the squeeze-film effect. It consists in changing the friction between the finger and a plate resonator. It provides high granularity level of haptic sensation. This paper deals with the design, realization and characterization of high performances actuators in order to promote the squeeze-film effect on a 4-inch transparent plate (diagonal of the plate). Using Finite Element Method (FEM) models, we select the best design, able to generate the highest plate displacement amplitude as possible. We built demonstrators using a generic technology based on thin-film Aluminum Nitride (AlN) actuators on glass substrate. Electromechanical characterizations prove that it is possible to obtain the focused substrate vibration amplitude using only 35 V in amplitude. The integration of the thin-film actuator plate in a haptic demonstrator is now ongoing.     

Skin-Integrated Vibrohaptic Interfaces for Virtual and Augmented Reality

Haptic technology involves the use of electrical or mechanical means to stimulate afferent nerves or mechanoreceptors in the skin as the basis for creating sensations of physical touch that can qualitatively expand virtual or augmented reality experiences beyond those supported by visual and auditory cues alone. An emerging direction in this field involves the development of platforms that provide spatiotemporal patterns of sensation to the skin across not only the fingertips, but to any and all regions of the body, using thin, skin-like technologies that impose negligible physical burden on the user. This review highlights the biological basis for skin interfaces of this type and the latest advances in haptics in the context of this ambitious goal, including electrotactile and vibrotactile devices that support perceptions of touch in form factors that have potential as skin-integrated interfaces. The content includes a discussion of schemes for integrating these stimulators into programmable arrays, with an emphasis on scalable materials and designs that have the potential to support soft interfaces across large areas of the skin. A concluding section summarizes the potential consequences of successful research efforts in this area, along with key multidisciplinary challenges and associated research opportunities in materials science and engineering.     

Force controlled and teleoperated endoscopic grasper for minimallyinvasive surgery-experimental performance evaluation

Minimally invasive surgery generates new user interfaces which create visual and haptic distortion when compared to traditional surgery. In order to regain the tactile and kinesthetic information that is lost, a computerized force feedback endoscopic surgical grasper (FREG) was developed with computer control and a haptic user interface. The system uses standard unmodified grasper shafts and tips. The FREG can control grasping forces either by surgeon teleoperation control, or under software control. The FREG performance was evaluated using an automated palpation function (programmed series of compressions) in which the grasper measures mechanical properties of the grasped materials. The material parameters obtained from measurements showed the ability of the FREG to discriminate between different types of normal soft tissues (small bowel, lung, spleen, liver, colon, and stomach) and different kinds of artificial soft tissue replication materials (latex/silicone) for simulation purposes. In addition, subjective tests of ranking stiffness of silicone materials using the FREG teleoperation mode showed significant improvement in the performance compared to the standard endoscopic grasper. Moreover, the FREG performance was closer to the performance of the human hand than the standard endoscopic grasper. The FREG as a tool incorporating the force feedback teleoperation technology may provide the basis for application in telesurgery, clinical endoscopic surgery, surgical training, and research.     

Force display using a hybrid haptic device composed of motors and brakes

In the virtual environment, force feedback to the human operator makes virtual experiences more realistic. However, the force feedback using active actuators such as motors can make the system active and sometimes unstable. To ensure the safe operation and enhance the haptic feeling, system stability should be guaranteed. Both active actuators such as motors and passive ones such as brakes are commonly used for haptic devices. Motors can generate a torque in any direction, but they can make the system active and thus, sometimes unstable during operation. On the other hand, brakes can generate a torque only against their rotation, but they dissipate energy during operation and this dissipation makes the system intrinsically stable. Consequently, motors and brakes are complementary to each other. In this research, a two degree-of-freedom (DOF) haptic device equipped with motors and brakes is designed, in which each DOF is actuated by a pair of motor and brake. Simultaneous operation of motors and brakes is analyzed. Models for some environments, virtual wall contact and frictional effect, are proposed. The results for the hybrid haptic system are compared with those for the active haptic system and the passivity based control system. The experimental results show that the hybrid haptic device is more suited to some applications than the other haptic systems.     

Review Article Tactile sensing for mechatronics—a state of the art survey

In this paper we examine the state of the art in tactile sensing for mechatronics. We define a tactile sensor as a device or system that can measure a given property of an object or contact event through physical contact between the sensor and the object. We consider any property that can be measured through contact, including the shape of an object, texture, temperature, hardness, moisture content, etc.A comprehensive search of the literature revealed that there was a significant increase in publications on tactile sensing from 1991 onwards. Considerable effort in the 1980s was spent investigating transduction techniques and developing new sensors, whilst emphasis in more recent research has focused on experiments using tactile sensors to perform a variety of tasks.This paper reports on progress in tactile sensing in the following areas: cutaneous sensors, sensing fingers, soft materials, industrial robot grippers, multifingered hands, probes and whiskers, analysis of sensing devices, haptic perception, processing sensory data and new application areas.We conclude that the predominant choice of transduction method is piezoelectric, with arrays using resistive or capacitive sensing. We found that increased emphasis on understanding tactile sensing and perception issues has opened up potential for new application areas. The predicted growth in applications in industrial automation has not eventuated. New applications for tactile sensing including surgery, rehabilitation and service robotics, and food processing automation show considerable potential and are now receiving significant levels of research attention.     

Mechanoreceptor-Inspired Dynamic Mechanical Stimuli Perception based on Switchable Ionic Polarization

Diverse touch experiences offer a path toward greater human–machine interaction, which is essential for the development of haptic technology. Recent advances in triboelectricity-based touch sensors provide great advantages in terms of cost, simplicity of design, and use of a broader range of materials. Since performance solely relies on the level of contact electrification between materials, triboelectricity-based touch sensors cannot effectively be used to measure the extent of deformation of materials under a given mechanical force. Here, an ion-doped gelatin hydrogel (IGH)-based touch sensor is reported to identify not only contact with an object but also deformation under a certain level of force. Switchable ionic polarization of the gelatin hydrogel is found to be instrumental in allowing for different sensing mechanisms when it is contacted and deformed. The results show that ionic polarization relies on conductivity of the hydrogels. Quantitative studies using voltage sweeps demonstrate that higher ion mobility and shorter Debye length serve to improve the performance of the mechanical stimuli-perceptible sensor. It is successfully demonstrated that this sensor offers dynamic deformation-responsive signals that can be used to control the motion of a miniature car. This study broadens the potential applications for ionic hydrogel-based sensors in a human–machine communication system.     

Metrology Challenges for Emerging Research Devices and Materials

As silicon complimentary metal-oxide-semiconductor (CMOS) technology approaches its limits, new device structures and computational paradigms will be required to replace and augment standard CMOS devices for ULSI circuits. These possible emerging technologies span the realm from transistors made from silicon nanowires to heteroepitaxial layers for spin transistors to devices made from nanoscale molecules. One theme that pervades these seemingly disparate emerging technologies is that the electronic properties of these nanodevices are extremely susceptible to small perturbations in structural and material properties such as dimension, structure, roughness, and defects. The extreme sensitivity of the electronic properties of these devices to their nanoscale physical properties defines a significant need for precise accurate metrology. This paper will describe some of the most critical metrology required to characterize materials and devices in the research and exploratory stage and how these requirements would potentially change if these research devices were to start into a technology development effort