Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality

Virtual reality and augmented reality (VR/AR) are evolving. The market demands and innovation efforts call for a shift in the key VR/AR technologies and engaging people virtually. Tele-haptics with multimodal and bilateral interactions are emerging as the future of the VR/AR industry. By transmitting and receiving haptic sensations wirelessly, tele-haptics allow human-to-human interactions beyond the traditional VR/AR interactions. The core technologies for tele-haptics include multimodal tactile sensing and feedback based on highly advanced sensors and actuators. Recent developments of haptic innovations based on active materials show that active materials can significantly contribute to addressing the needs and challenges for the current and future VR/AR technologies. Thus, this paper intends to review the current status and opportunities of active material-based haptic technology with a focus on VR/AR applications. It first provides an overview of the current VR/AR applications of active materials for haptic sensing and actuation. It then highlights the state-of-the-art haptic interfaces that are relevant to the materials with an aim to provide perspectives on the role of active materials and their potential integration in haptic devices. This paper concludes with the perspective and outlook of immersive multimodal tele-haptic interaction technologies.     

A control-system architecture for robots used to simulate dynamicforce and moment interaction between humans and virtual objects

Haptic or kinesthetic feedback is essential in many important virtual reality and telepresence applications. Previous research focuses on simulating static forces such as those encountered when interacting with a stiff object such as a wall. Past studies usually employ custom-made devices that are not readily available to other researchers. Consequently, many of the results found in the haptic feedback literature cannot be replicated independently. With experimental results, the paper demonstrates that “off the shelf,” general purpose robotics equipment can be incorporated into an effective haptic/kinesthetic feedback system. Such a system can accommodate a wide variety of virtual reality applications including training and telerobotics. An admittance control scheme is utilized, which enables the simulation of dynamic force and moment interaction as well as contact with stiff objects. The paper shows that the mechanical deficiencies (e.g., friction, inertia, and backlash) often associated with general purpose manipulators can be overcome with a suitable control system architecture     

Haptic holography: a primitive computational plastic

We describe our work on haptic holography, a combination of computational modeling and multimodal spatial display, which allows a person to see, feel, and interact with three-dimensional freestanding holographic images of material surfaces. In this paper, we combine various holographic displays with a force-feedback device to render multimodal images with programmatically prescribed material properties and behavior. After a brief overview of related work which situates visual display within the manual workspace, we describe our holo-haptic approach and survey three implementations, Touch, Lathe, and Poke, each named for the primitive functional affordance it offers. In Touch, static holographic images of simple geometric scenes are reconstructed in front of the hologram plane, and coregistered with a force model of the same geometry. These images can be visually inspected and haptically explored using a handheld interface. In Lathe, a holo-haptic image can be reshaped by haptic interaction in a dynamic but constrained manner. Finally in Poke, using a new technique for updating interference-modeled holographic fringe patterns, we render a holo-haptic image that permits more flexible interactive reshaping of its reconstructed surface. We situate this work within the context of related research and describe the strengths, shortcomings, and implications of our approach.     

Touch and Go — Designing Haptic Feedback for a Hand-Held Mobile Device

Increasingly, our mobile devices are acquiring the ability to be aware of their surroundings. These devices are also acquiring the ability to sense what is happening to them — how they are being held and moved. The coincidence of connectedness, awareness and richly multimodal input and output capabilities brings into the hand a device capable of supporting an entirely new class of haptic or touch-based interactions, where gestures can be captured and reactions to these gestures conveyed as haptic feedback directly into the hand. Thus, one can literally shake the hand of a friend, toss a file off one's PDA, or be led by the hand to a desired location in a strange city. In this paper I will propose that, for the mobile user negotiating these multiple frames of reference for their actions, a better understanding of the senses of touch, of the body's motion and its sense of its own motion, may be the key to providing a meaningful bridge between these interleaved and interdependent spaces.     

Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

Force display using a hybrid haptic device composed of motors and brakes

In the virtual environment, force feedback to the human operator makes virtual experiences more realistic. However, the force feedback using active actuators such as motors can make the system active and sometimes unstable. To ensure the safe operation and enhance the haptic feeling, system stability should be guaranteed. Both active actuators such as motors and passive ones such as brakes are commonly used for haptic devices. Motors can generate a torque in any direction, but they can make the system active and thus, sometimes unstable during operation. On the other hand, brakes can generate a torque only against their rotation, but they dissipate energy during operation and this dissipation makes the system intrinsically stable. Consequently, motors and brakes are complementary to each other. In this research, a two degree-of-freedom (DOF) haptic device equipped with motors and brakes is designed, in which each DOF is actuated by a pair of motor and brake. Simultaneous operation of motors and brakes is analyzed. Models for some environments, virtual wall contact and frictional effect, are proposed. The results for the hybrid haptic system are compared with those for the active haptic system and the passivity based control system. The experimental results show that the hybrid haptic device is more suited to some applications than the other haptic systems.     

Air Permeable Vibrotactile Actuators for Wearable Wireless Haptics

Vibrotactile actuators can evoke mechanical stimulations on human skins to induce haptic feedbacks for various human machine interaction applications. However, efforts toward their practical usages encounter several engineering challenges, including wearable comfortability and output abilities. Here, air permeable actuators are developed and embedded in common fabrics for vibrotactile actuation, achieving excellent air permeability of 108 L m⁻² s⁻¹, low preload requirement of 10 mN, high output sensitivity of 0.2 mN/V, and good mechanical durability by surviving 11 million testing cycles. As demonstration examples, a wireless haptic feedback glove is shown to distinguish 32 different English characters and symbols with an overall accuracy of 97.8%, and large size actuators (10 × 10 cm²) are also proved for providing haptic feedback for parts of human body. As such, the proposed system opens a new class of wearable vibrotactile actuators for potential applications in wide fields of metaverse, teleoperation, smart textiles, and robotics.     

A Liquid Metal Based Multimodal Sensor and Haptic Feedback Device for Thermal and Tactile Sensation Generation in Virtual Reality

Virtual reality (VR) has been widely used for training, gaming, and entertainment, and the value of VR is continually increasing as a contact-free technology. For an immersive VR experience, measuring finger movements and providing appropriate feedback to the hand are as important as visual information, given the necessity of the hands for activities in daily life. Thus, a hand-worn VR device with motion sensors and haptic feedback is desirable. In this paper, a multimodal sensing and feedback glove is developed with soft, stretchable, lightweight, and compact sensor and heater sheets manufactured by direct ink writing (DIW) of liquid metal, eutectic gallium-indium (eGaIn). In the sensor sheet, ten sensors and three vibrators are embedded to measure finger movements and provide vibro-haptic feedback. The other heater sheet provides thermo-haptic sensation in accurate and rapid manner via model-based feedback control even under stretched conditions. The multimodal sensing and feedback glove allows users to feel the contact status and discriminate materials with different temperature. Performance of the proposed multimodal glove is verified under VR environments including touching and pushing two blocks of different materials and grabbing a heated metal ball submerged in hot water.     

Electronic sensory systems for the visually impaired

In this article, I have summarized the required features in electronic sensory systems for the visually impaired (ESSVI) development and described how we have addressed these features at the Department of Electrical Electronic and System Engineering (DIEES). The "Committee on Vision" document on electronic travel aids explored the needs of the visually impaired (VI) and the features of devices that are needed to support them in their daily activities. High-performance devices can improve the life quality of visually impaired people. Many aids have already been developed that extend their "range of observation". Design engineers who understand the individual needs of the VI can build effective aids.     

Haptic information in Internet-based teleoperation

Many tasks can be done easily by humans turn out to be very difficult to accomplish with a teleoperated robot. The main reason for this is the lack of tactile sensing, which cannot be replaced by visual feedback alone. Once haptic devices are developed, their potential in many fields is obvious. Especially, in teleoperation systems, where haptic feedback can increase the efficiency and even render some tasks feasible. This paper studies Internet-based teleoperation systems that include haptic feedback, concentrating on the control of such systems and their performance. The potential of this technology and its advantages are explored. In addition, key issues, such as stability, synchronization, and transparency are analyzed and studied. Specifically, an event-based planning and control of Internet-based teleoperation systems is presented with experimental results of several implemented system scenarios in micro- and macro-scales     

A high bandwidth interface for haptic human computer interaction

Current force feedback, haptic interface devices are generally limited to the display of low frequency, high amplitude spatial data. A typical device consists of a low impedance framework of one or more degrees-of-freedom (dof), allowing a user to explore a pre-defined workspace via an end effector such as a handle, thimble, probe or stylus. The movement of the device is then constrained using high gain positional feedback, thus reducing the apparent dof of the device and conveying the illusion of hard contact to the user. Such devices are, however, limited to a narrow bandwidth of frequencies, typically below 30 Hz, and are not well suited to the display of surface properties, such as object texture. This paper details a device to augment an existing force feedback haptic display with a vibrotactile display, thus providing a means of conveying low amplitude, high frequency spatial information of object surface properties.     

Stretchable Skin‐Like Cooling/Heating Device for Reconstruction of Artificial Thermal Sensation in Virtual Reality

Along with visual and tactile sensations, thermal sensation by temperature feeling on the skin can provide rich physical information on the environment and objects. With a simple touch of objects, relative temperature can be sensed and even objects can be differentiated with different thermal properties without any visual cue. Thus, artificially reproducing accurate/controllable thermal sensation haptic signals on human epidermis will certainly be a major research area to reconstruct a more realistic virtual reality (VR) environment. In this study, for the first time, a skin‐like, highly soft and stretchable and bi‐functional (both cold and hot sensation) thermo‐haptic device is reported for wearable VR applications with a single device structure (not separate heater and cooler). The skin‐like thermo‐haptic (STH) device can actively cool down and heat up deformable skin surfaces with instantaneous and accurate adjustment of temperature based upon a feedback control algorithm to mimic desirable thermal sensation with 230% stretchability. As a proof‐of‐concept, the STH device is integrated with a finger‐motion tracking glove to provide artificial thermal sensation information to the skin in various situations such as touching cold beer bottles and hot coffee cups in virtual space. This new type of STH device can offer potential implications for next‐generation haptic devices to provide unique thermal information for a more realistic virtual‐world field and medical thermal treatment.     

A spherical permanent magnet actuator for a high-fidelity force-feedback joystick

The paper describes the basic features and performance characteristics of a novel spherical permanent magnet actuator. It is capable of two degrees-of-freedom, and has potential for use in applications such as haptic feedback and position servo devices. Due to the simplicity of its mechanical construction and rotor position sensing system, and the absence of transmission components, the actuator is particularly suitable for high-fidelity force-feedback joysticks, for which a high bandwidth, low cost and small size are major requirements. In order to assess its potential, comprehensive measurements have been carried out, which confirm that the spherical actuator offers a considerably higher bandwidth than current commercially available force-feedback joysticks.     

A haptic-enabled multimodal interface for the planning of hip arthroplasty

Multimodal environments help fuse a diverse range of sensory modalities, which is particularly important when integrating the complex data involved in surgical preoperative planning. The authors apply a multimodal interface for preoperative planning of hip arthroplasty with a user interface that integrates immersive stereo displays and haptic modalities. This article overviews this multimodal application framework and discusses the benefits of incorporating the haptic modality in this area.     

Virtual environments for medical training: graphical and hapticsimulation of laparoscopic common bile duct exploration

We develop a computer-based training system to simulate laparoscopic procedures in virtual environments for medical training. The major hardware components of our system include a computer monitor to display visual interactions between 3D virtual models of organs and instruments together with a pair of force feedback devices interfaced with laparoscopic instruments to simulate haptic interactions. We simulate a surgical procedure that involves inserting a catheter into the cystic duct using a pair of laparoscopic forceps. This procedure is performed during laparoscopic cholecystectomy to search for gallstones in the common bile duct. Using the proposed system, the user can be trained to grasp and insert a flexible and freely moving catheter into the deformable cystic duct in virtual environments. The associated deformations are displayed on the computer screen and the reaction forces are fed back to the user through the force feedback devices. A hybrid modeling approach was developed to simulate the real-time visual and haptic interactions that take place between the forceps and the catheter, as well as the duct; and between the catheter and the duct     

A General-Purpose 7 DOF Haptic Device: Applications Toward Robot-Assisted Surgery

A 7 DOF haptic device has been designed and developed with applications towards robot-assisted minimally invasive surgery. The device consists of four degrees of force feedback (X, Y, Z, and grasping) capability and seven degrees of position feedback capability. It has a closed kinematic chain with two halves (user interface and spatial mechanism) that connect together via a universal joint. The user interface contains four degrees of position feedback, namely, the roll, pitch, yaw, and linear motion of the hand and forearm. In addition, a grasping mechanism with two thimbles mounted at the end of the user interface provides force feedback to the fingers of the user. The spatial mechanism provides force feedback to the user interface through a universal joint located at the grasping mechanism. This paper presents the design and development of this haptic device. In addition, a kinematic and workspace analysis of the device has been completed to compute the position of the slave robot and end-effector tool. Friction estimation has been presented to enable a higher transparency of the haptic device. Finally, a simulation of needle insertion into soft tissue was developed to test the device.     

Multimodal Interaction on Automultiscopic Content with Mobile Surface Haptics

In this work, we present interactive automultiscopic content with mobile surface haptics for multimodal interaction. Our system consists of a 40‐view automultiscopic display and a tablet supporting surface haptics in an immersive room. Animated graphics are projected onto the walls of the room. The 40‐view automultiscopic display is placed at the center of the front wall. The haptic tablet is installed at the mobile station to enable the user to interact with the tablet. The 40‐view real‐time rendering and multiplexing technology is applied by establishing virtual cameras in the convergence layout. Surface haptics rendering is synchronized with three‐dimensional (3D) objects on the display for real‐time haptic interaction. We conduct an experiment to evaluate user experiences of the proposed system. The results demonstrate that the system's multimodal interaction provides positive user experiences of immersion, control, user interface intuitiveness, and 3D effects.     

4-inch transparent plates based on thin-film AlN actuators for haptic applications

Numerous applications require tactile interfaces today. In particular, many customers’ applications such as automotive, Smartphone, tablet PC or touch pad can be concerned by high performances, low voltage haptic interfaces which allow the user to interact with its environment by the sense of touch. This technology is already used but with limitations such as high power consumption and limited feedback effect because today a simple vibration is commonly obtained. We chose to work on the squeeze-film effect. It consists in changing the friction between the finger and a plate resonator. It provides high granularity level of haptic sensation. This paper deals with the design, realization and characterization of high performances actuators in order to promote the squeeze-film effect on a 4-inch transparent plate (diagonal of the plate). Using Finite Element Method (FEM) models, we select the best design, able to generate the highest plate displacement amplitude as possible. We built demonstrators using a generic technology based on thin-film Aluminum Nitride (AlN) actuators on glass substrate. Electromechanical characterizations prove that it is possible to obtain the focused substrate vibration amplitude using only 35 V in amplitude. The integration of the thin-film actuator plate in a haptic demonstrator is now ongoing.     

Enhancing human-computer interaction in medical segmentation

Medical image acquisition devices are becoming increasingly multidimensional, and predictions assume 5000 images per patient study within the next decade. Therefore, new paradigms for computerized medical image analysis and visualization are of fundamental importance to make possibly full use of the information buried in the enormous flood of image data. The aim of the presented project is the optimal cooperation between computer-based image analysis algorithms and human operators using new closed-loop segmentation systems for improved information flow. This paper describes an enhanced, visuo-haptic interaction tool we have developed for medical segmentation. Evaluation studies with the system, which confirm the value of adding haptic feedback, are also presented.     

Controlling a robotic fish with a smart phone

Recently, mobile devices such as Apple’s iDevices have acquired the ability to host a variety of functions beyond merely initiating and receiving telephone calls. These devices have great potential for educational applications, especially when integrated with disparate technologies, such as environmental sensors, microscopes, and robotics. In this work, we present the development of an iDevice application and communication hardware for interacting with a miniature robotic fish. The application includes a user interface for controlling the robot’s motion, a detailed tutorial featuring an animated fish, and a link to information about scientific research using the robot. This platform has been field-tested with children in a broad age range and refined based on their feedback. The application has been found to be significantly easier to use than a traditional joystick controller by a survey of middle school children.