A Liquid Metal Based Multimodal Sensor and Haptic Feedback Device for Thermal and Tactile Sensation Generation in Virtual Reality

Virtual reality (VR) has been widely used for training, gaming, and entertainment, and the value of VR is continually increasing as a contact-free technology. For an immersive VR experience, measuring finger movements and providing appropriate feedback to the hand are as important as visual information, given the necessity of the hands for activities in daily life. Thus, a hand-worn VR device with motion sensors and haptic feedback is desirable. In this paper, a multimodal sensing and feedback glove is developed with soft, stretchable, lightweight, and compact sensor and heater sheets manufactured by direct ink writing (DIW) of liquid metal, eutectic gallium-indium (eGaIn). In the sensor sheet, ten sensors and three vibrators are embedded to measure finger movements and provide vibro-haptic feedback. The other heater sheet provides thermo-haptic sensation in accurate and rapid manner via model-based feedback control even under stretched conditions. The multimodal sensing and feedback glove allows users to feel the contact status and discriminate materials with different temperature. Performance of the proposed multimodal glove is verified under VR environments including touching and pushing two blocks of different materials and grabbing a heated metal ball submerged in hot water.     

Haptic Perception,Mechanics, and Material Technologies for Virtual Reality

Emerging virtual and augmented reality technologies can transform human activities in myriad domains, lending tangible, embodied form to digital data, services, and information. Haptic technologies will play a critical role in enabling human to touch and interact with the contents of these virtual environments. The immense variety of skilled manual tasks that humans perform in real environments are only possible through the coordination of touch sensation, perception, and movement that together comprise the haptic modality. Consequently, many research groups are vigorously investigating haptic technologies for virtual reality. A longstanding research goal in this area has been to create haptic interfaces that allow their users to touch and feel plausibly realistic virtual objects. In this progress report, the perspective on this unresolved research challenge is shared, guided by the observation that no technologies can even approximately match the capabilities of the human sense of touch. Factors that have it challenging to engineer haptic technologies for virtual reality, including the extraordinary spatial and temporal tactile acuity of the skin, and the complex interplay between continuum mechanics, haptic perception, and interaction are identified. The perspective on how these challenges may be overcome through convergent research on haptic perception, mechanics, electronics, and material technologies is presented.     

Stereo‐Vision‐Based Human‐Computer Interaction with Tactile Stimulation

If a virtual object in a virtual environment represented by a stereo vision system could be touched by a user with some tactile feeling on his/her fingertip, the sense of reality would be heightened. To create a visual impression as if the user were directly pointing to a desired point on a virtual object with his/her own finger, we need to align virtual space coordinates and physical space coordinates. Also, if there is no tactile feeling when the user touches a virtual object, the virtual object would seem to be a ghost. Therefore, a haptic interface device is required to give some tactile sensation to the user. We have constructed such a human‐computer interaction system in the form of a simple virtual reality game using a stereo vision system, a vibro‐tactile device module, and two position/orientation sensors.     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

Nanowire-Based Soft Wearable Human–Machine Interfaces for Future Virtual and Augmented Reality Applications

A virtual world has now become a reality as augmented reality (AR) and virtual reality (VR) technology become commercially available. Similar to how humans interact with the physical world, AR and VR systems rely on human–machine interface (HMI) sensors to interact with the virtual world. Currently, this is achieved via state of-the-art wearable visual and auditory tools that are rigid, bulky, and burdensome, thereby causing discomfort during practical application. To this end, a skin sensory interface has the potential to serve as the next-generation AR/VR technology because skin-like wearable sensors have advantages in that they can be ultrathin, ultra-soft, conformal, and imperceptible, which provides the ultimate comfort and immersive experience for users. In this progress report, nanowire-based soft wearable HMI sensors including acoustic, strain, pressure sensors, and physiological sensors are reviewed that may be adopted as skin sensory inputs in future AR/VR systems. Further, nanowire-based soft contact lenses, haptic force, and thermal and vibration actuators are covered as potential means of feedback for future AR/VR systems. Considering the possible effects of the virtual world on human health, skin-like wearable artery pulses, glucose, and lactate sensors are also described, which may enable imperceptible health monitoring during future AR/VR practices.     

Pseudo-haptics for rigid tool/soft surface interaction feedback in virtual environments

This paper proposes a novel pseudo-haptic soft surface stiffness simulation technique achieved by displaying the deformation of the soft surface and maneuvering an indenter avatar over a virtual soft surface by means of a touch-sensitive tablet. The visual feedback of the surface deformation and the alterations to the indenter avatar behavior produced by the proposed technique create the illusion of interaction with a hard inclusion embedded in the virtual soft surface. The proposed pseudo-haptics technique is validated with a series of experiments conducted by employing a tablet computer with an S-pen input and a tablet computer with a bare finger input. Tablet computers provide unique opportunities for presenting the pseudo-haptic (indenter avatar speed), haptic (contact reaction force from the device surface) and visual cues (surface information) at the same active point of interaction which facilitates information fusion. Hence, here, we evaluate the performance of tablet computers in identification of hard inclusions within virtual soft objects and compare it with the performance of a touchpad input device. A direct hand-soft surface interaction is used for benchmarking of this study. We found that compared with using a touchpad, both the sensitivity and the positive predictive value of the hard inclusion detection can be significantly improved by 33.3% and 13.9%, respectively, by employing tablet computers. Using tablet computers could produce results comparable to the direct hand-soft surface interaction in detecting hard inclusions in a soft object. The experimental results presented here confirm the potential of the proposed technique for conveying haptic information in rigid tool/soft surface interaction in virtual environments.     

Phantom-based multimodal interactions for medical education and training: the Munich knee joint simulator

Simulation environments based on virtual reality technologies can support medical education and training. In this paper, the novel approach of an "interactive phantom" is presented that allows a realistic display of haptic contact information typically generated when touching and moving human organs or segments. The key idea of the haptic interface is to attach passive phantom objects to a mechanical actuator. The phantoms look and feel as real anatomical objects. Additional visualization of internal anatomical and physiological information and sound generated during the interaction with the phantom yield a multimodal approach that can increase performance, didactic value, and immersion into the virtual environment. Compared to classical approaches, this multimodal display is convenient to use, provides realistic tactile properties, and can be partly adjusted to different, e.g., pathological properties. The interactive phantom is exemplified by a virtual human knee joint that can support orthopedic education, especially for the training of clinical knee joint evaluation. It is suggested that the technical principle can be transferred to many other fields of medical education and training such as obstetrics and dentistry.     

An Adaptive and Wearable Thermal Camouflage Device

Thermal cloaking and camouflage have attracted increasing attention with the progress of infrared surveillance technologies. Previous studies have been mainly focused on emissivity manipulation or using sophisticated thermal metamaterials. However, emissivity control is only applicable for objects that are warmer than the environment and lower emissivity is usually accompanied with high reflectance of the surrounding thermal signals if they have nonuniform temperature. Metamaterial‐based thermal camouflage holds great promise but their applications on human subjects are yet to be realized. Direct temperature control represents a more desirable strategy to realize dynamically adjustable camouflage within a wide ambient temperature range, but a wearable, portable, and adjustable thermo‐regulation system that is suitable for human subjects has not been developed. This work demonstrates a wearable and adaptive infrared camouflage device responding to the background temperature change based on the thermoelectric cooling and heating effect. The flexible thermoelectric device can realize the infrared camouflage effect to effectively shield the metabolic heat from skin within a wide range of background temperature: 7 °C below and 15 °C above the ambient temperature, showing promise for a broad range of potential applications, such as security, counter‐surveillance, and adaptive heat shielding and thermal control.     

Emerging Thermal Technology Enabled Augmented Reality

In the past decade, remarkable progress has been made in the domain of augmented reality/virtual reality (AR/VR). The need for realistic and immersive augmentation has propelled the development of haptics interfaces-enabled AR/VR. The haptics interfaces facilitate direct interaction and manipulation with both real and virtual objects, thus augmenting the perception and experiences of the users. The level of augmentation can be significantly improved by thermal stimulation or sensing, which facilitates a higher degree of object identification and discrimination. This review discusses the thermal technology-enabled augmented reality and summarizes the recent progress in the development of different thermal technology such as thermal haptics including thermo-resistive heater and Peltier devices, thermal sensors including resistive, pyroelectric, and thermoelectric sensors, which can be utilized to improve the realism of augmentation. The fundamental mechanism, design strategies, and the rational guidelines for the adoption of these technologies in AR/VR is explicitly discussed. The conclusion provides an outlook on the existing challenges and outlines the future roadmap for the realization of next-generation thermo-haptics enabled augmented reality.     

Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

Augmented System for Immersive 3D Expansion and Interaction

In the field of augmented reality technologies, commercial optical see‐through‐type wearable displays have difficulty providing immersive visual experiences, because users perceive different depths between virtual views on display surfaces and see‐through views to the real world. Many cases of augmented reality applications have adopted eyeglasses‐type displays (EGDs) for visualizing simple 2D information, or video see‐through‐type displays for minimizing virtual‐ and real‐scene mismatch errors. In this paper, we introduce an innovative optical see‐through‐type wearable display hardware, called an EGD. In contrast to common head‐mounted displays, which are intended for a wide field of view, our EGD provides more comfortable visual feedback at close range. Users of an EGD device can accurately manipulate close‐range virtual objects and expand their view to distant real environments. To verify the feasibility of the EGD technology, subject‐based experiments and analysis are performed. The analysis results and EGD‐related application examples show that EGD is useful for visually expanding immersive 3D augmented environments consisting of multiple displays.     

Reversible Water Transportation Diode: Temperature‐Adaptive Smart Janus Textile for Moisture/Thermal Management

“Air‐conditioning” textiles with thermal‐ or moisture‐managing functions are of high interest for not only improving human comfort but also reducing energy consumption. However, making the textile sensitive to the surrounding environment and exhibit adaptive thermal/moisture management still remains a great challenge. Herein, a double‐sided synergetic Janus textile is developed, featuring reversible diode‐like water transportation and adjustable thermal convection upon temperature change. The incorporated responsive polymer networks with inverse transitions on the opposite sides provide synergistic surface energy gradients and capillary gradients that generate drying and cooling effects (with 50% faster water evaporation and 1.2–2.3 °C cooler than with cotton fabric) in hot weather while offering thermal preservation (120 s longer needed to be cooled down and maximumly 3.3 °C warmer than with cotton fabric) in a cold environment. This method could provide ideas for the development of more adaptive textiles and clothing to address maximum personal comfort in demanding situations.     

Force display using a hybrid haptic device composed of motors and brakes

In the virtual environment, force feedback to the human operator makes virtual experiences more realistic. However, the force feedback using active actuators such as motors can make the system active and sometimes unstable. To ensure the safe operation and enhance the haptic feeling, system stability should be guaranteed. Both active actuators such as motors and passive ones such as brakes are commonly used for haptic devices. Motors can generate a torque in any direction, but they can make the system active and thus, sometimes unstable during operation. On the other hand, brakes can generate a torque only against their rotation, but they dissipate energy during operation and this dissipation makes the system intrinsically stable. Consequently, motors and brakes are complementary to each other. In this research, a two degree-of-freedom (DOF) haptic device equipped with motors and brakes is designed, in which each DOF is actuated by a pair of motor and brake. Simultaneous operation of motors and brakes is analyzed. Models for some environments, virtual wall contact and frictional effect, are proposed. The results for the hybrid haptic system are compared with those for the active haptic system and the passivity based control system. The experimental results show that the hybrid haptic device is more suited to some applications than the other haptic systems.     

Thermo-Haptic Materials and Devices for Wearable Virtual and Augmented Reality

The current generation of virtual and augmented realities (VR/AR) has substantially advanced in the past decade because of the rapid development of converging technologies in various engineering and scientific fields. However, the current VR/AR technologies rely mainly on visual and auditory senses to physically replicate the virtual environment, although tactile senses play a significant role in the daily life since a myriad of tactile information is received through physical touch. Of the tactile senses, thermal senses are of great importance to be reproduced in the VR/AR field, since heat is transferred constantly and further interact with the surrounding environment. To date, there has been a huge amount of research studies on functional materials, thermo-haptic devices, and wearable electronics that have all converged to form the fundamental groundwork for the development of wearable thermal VR/AR devices. In this progress report, a review on various physical mechanisms and research is provided that can potentially be applied in the next generation of thermal VR/AR technologies and discuss the essential challenges that need to be addressed.     

Bracelet-Shaped Thermal Display for Representing Numerical Data

A thermal display, a type of haptic display, is effective for providing intuitive information about temperature. In many thermal display studies, users have assumed sitting positions when using these devices. However, their use in a large-scale virtual-reality system requires users to be in a standing position, as they generally observe three-dimensional (3D) objects while standing or walking around. Thus, we developed thermal displays that are suitable for large-scale virtual-reality systems. From another standpoint, in scientific visualization, response time is very important for observing physical phenomena, especially for dynamic numerical simulation. One way to optimize this parameter is to provide two types of thermal information: the rate of thermal change, and the actual temperature. To this end, we propose a bracelet-shaped thermal display with three Peltier elements that can provide both types of information. Finally, we present an example of visualizing and haptizing the result of a molecular dynamics simulation.     

面向VR应用的花卉植物物理渲染技术研究与实现

花卉植物高真实感的仿真交互是目前虚拟植物可视化研究的一个重要方向。随着虚拟现实技术的普及,越来越多的应用采用了VR头戴设备的呈现方式。VR系统需要高度真实的沉浸感画面,通用的植物建模和图形引擎渲染功能已不能满足该需求。该文通过分析光照原理并融合基于物理的渲染技术,提出基于双向散射分布函数BSDF的花卉植物高度真实感的物理渲染算法,利用ShaderLab,对几种盆栽花卉植物在光照下进行仿真,并对融合算法做优化处理。针对VR头盔设备HTC Vive的成像效果,对图像进行扭曲变形优化,使画面更符合人眼双目立体视觉成像效果,增强系统沉浸感。最后基于该方法设计并实现了一个头盔式VR花卉植物仿真模拟系统,获得了逼真的场景漫游体验效果。     

On the Design of Miniature Haptic Devices for Upper Extremity Prosthetics

We have developed three different versions of a multifunction haptic device that can display touch, pressure, vibration, shear force, and temperature to the skin of an upper extremity amputee, especially the one who has undergone targeted nerve reinnervation (TR) surgery. In TR patients, sensation from the reinnervated skin is projected to the missing hand. This paper addresses the design of the mechanical display, the portion responsible for contact, pressure, vibration, and shear force. A variety of different overall design approaches satisfying the design specifications and the performance requirements are considered. The designs of the fully prototyped haptic devices are compared through open-loop frequency response, closed-loop force response, and tapping response in constrained motion. We emphasize the tradeoffs between key design factors, including force capability, workspace, size, bandwidth, weight, and mechanism complexity.      

Teleoperated touch feedback from the surfaces at the nanoscale: modeling and experiments

In this paper, a teleoperated nanoscale touching system is proposed, and continuum nanoscale contact mechanics models are introduced. The tele-nanorobotic system consists of a piezoresistive nanoprobe with a sharp tip as the nanorobot and force-topology sensor, a custom-made 1-degree-of-freedom haptic device for force-feedback, three-dimensional (3D) virtual reality (VR) graphics display of the nano world for visual feedback, and a force-reflecting servo type scaled teleoperation controller. Using this system, one-dimensional and 3D touching experiments and VR simulations are realized. Scaling of nano-forces is one of the major issues of the scaled teleoperation system since nanometer scale forces are dominated by surface forces instead of inertial forces as in the macro world. As the force scaling approach, a heuristic rule is introduced where nano-forces are linearly scaled with an experimentally determined scaling parameter. Simulation results and preliminary experiments of touching silicon and InAs quantum dot nanostructures show that adhesion forces at the nanoscale can be felt repeatedly at the operator's hand, and the proposed system enables the nanoscale surface topography and contact/noncontact nano-force feedback.     

Recent Advances in Wearable Sensors and Integrated Functional Devices for Virtual and Augmented Reality Applications

The advancement in virtual reality/augmented reality (VR/AR) has been achieved by breakthroughs in the realistic perception of virtual elements. Although VR/AR technology is advancing fast, enhanced sensor functions, long-term wearability, and seamless integration with other electronic components are still required for more natural interactions with the virtual world. Here, this report reviews the recent advances in multifunctional wearable sensors and integrated functional devices for VR/AR applications. Specified device designs, packaging strategies, and interactive physiological sensors are summarized based on their methodological approaches for sensory inputs and virtual feedback. In addition, limitations of the existing systems, key challenges, and future directions are discussed. It is envisioned that this progress report's outcomes will expand the insights on wearable functional sensors and device interfaces toward next-generation VR/AR technologies.     

Untethered Feel-Through Haptics Using 18-µm Thick Dielectric Elastomer Actuators

Head-mounted displays for virtual reality (VR) and augmented reality (AR) allow users to see highly realistic virtual worlds. The wearable haptics that enable feeling and touching these virtual objects are typically bulky, tethered, and provide only low fidelity feedback. A particularly challenging type of wearable human-machine interface is feel-through haptics: ultra-thin wearables so soft as to be mechanically imperceptible when turned off, yet generating sufficient force when actuated to make virtual objects feel tangible, or to change the perceived texture of a physical object. Here, 18 µm thick soft dielectric elastomer actuators (DEA), directly applied on the skin, reports rich vibrotactile feedback generation from 1 Hz to 500 Hz. Users correctly identifies different frequency and sequence patterns with success rates from 73 to 97% for devices applied on their fingertips. An untethered version weighing only 1.3 grams allowed blindfolded users to correctly identify letters by “seeing” them through their fingers. The silicone-based DEA membrane is mechanically transparent, enabling wearable haptics for the many applications where hand dexterity is critical. The feel-through DEA can be placed in array format anywhere on the body.