Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

Skin Electronics: Next-Generation Device Platform for Virtual and Augmented Reality

Virtual reality (VR) and augmented reality (AR) are overcoming the physical limits of real-life using advances in devices and software. In particular, the recent restrictions in transportation from the coronavirus disease 2019 (COVID-19) pandemic are making people more interested in these virtual experiences. However, to minimize the differences between artificial and natural perception, more human-interactive and human-like devices are necessary. The skin is the largest organ of the human body and interacts with the environment as the site of interfacing and sensing. Recent progress in skin electronics has enabled the use of the skin as the mounting object of functional devices and the signal pathway bridging humans and computers, with opening its potential in future VR and AR applications. In this review, the current skin electronics are summarized as one of the most promising device solutions for future VR/AR devices, especially focusing on the recent materials and structures. After defining and explaining VR/AR systems and the components, the advantages of skin electronics for VR/AR applications are emphasized. Next, the detailed functionalities of skin electronic devices, including the input, output, energy devices, and integrated systems, are reviewed for future VR/AR applications.     

Recent Advances in Wearable Sensors and Integrated Functional Devices for Virtual and Augmented Reality Applications

The advancement in virtual reality/augmented reality (VR/AR) has been achieved by breakthroughs in the realistic perception of virtual elements. Although VR/AR technology is advancing fast, enhanced sensor functions, long-term wearability, and seamless integration with other electronic components are still required for more natural interactions with the virtual world. Here, this report reviews the recent advances in multifunctional wearable sensors and integrated functional devices for VR/AR applications. Specified device designs, packaging strategies, and interactive physiological sensors are summarized based on their methodological approaches for sensory inputs and virtual feedback. In addition, limitations of the existing systems, key challenges, and future directions are discussed. It is envisioned that this progress report's outcomes will expand the insights on wearable functional sensors and device interfaces toward next-generation VR/AR technologies.     

A Liquid Metal Based Multimodal Sensor and Haptic Feedback Device for Thermal and Tactile Sensation Generation in Virtual Reality

Virtual reality (VR) has been widely used for training, gaming, and entertainment, and the value of VR is continually increasing as a contact-free technology. For an immersive VR experience, measuring finger movements and providing appropriate feedback to the hand are as important as visual information, given the necessity of the hands for activities in daily life. Thus, a hand-worn VR device with motion sensors and haptic feedback is desirable. In this paper, a multimodal sensing and feedback glove is developed with soft, stretchable, lightweight, and compact sensor and heater sheets manufactured by direct ink writing (DIW) of liquid metal, eutectic gallium-indium (eGaIn). In the sensor sheet, ten sensors and three vibrators are embedded to measure finger movements and provide vibro-haptic feedback. The other heater sheet provides thermo-haptic sensation in accurate and rapid manner via model-based feedback control even under stretched conditions. The multimodal sensing and feedback glove allows users to feel the contact status and discriminate materials with different temperature. Performance of the proposed multimodal glove is verified under VR environments including touching and pushing two blocks of different materials and grabbing a heated metal ball submerged in hot water.     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

虚拟现实技术在医学领域的研究现状与进展

虚拟现实(VR)与其所衍生的增强现实(AR)和混合现实(MR)能把携带三维信息的虚拟场景与真实世界相互叠加,极大地提升用户感官世界的直观性、精准性、实时性。该技术的推广与应用,将会给医学领域带来变革式发展。本文剖析VR/AR/MR的概念并简述其发展历程,分别对虚拟现实和增强现实在医学领域的应用进行阐述,并通过微软产品HoloLens的特点分析基于混合现实的解决方案在医学领域的优势。最后对VR/AR/MR目前在医学领域所存在的不足进行归纳,并对未来的发展趋势进行展望。     

Nanowire-Based Soft Wearable Human–Machine Interfaces for Future Virtual and Augmented Reality Applications

A virtual world has now become a reality as augmented reality (AR) and virtual reality (VR) technology become commercially available. Similar to how humans interact with the physical world, AR and VR systems rely on human–machine interface (HMI) sensors to interact with the virtual world. Currently, this is achieved via state of-the-art wearable visual and auditory tools that are rigid, bulky, and burdensome, thereby causing discomfort during practical application. To this end, a skin sensory interface has the potential to serve as the next-generation AR/VR technology because skin-like wearable sensors have advantages in that they can be ultrathin, ultra-soft, conformal, and imperceptible, which provides the ultimate comfort and immersive experience for users. In this progress report, nanowire-based soft wearable HMI sensors including acoustic, strain, pressure sensors, and physiological sensors are reviewed that may be adopted as skin sensory inputs in future AR/VR systems. Further, nanowire-based soft contact lenses, haptic force, and thermal and vibration actuators are covered as potential means of feedback for future AR/VR systems. Considering the possible effects of the virtual world on human health, skin-like wearable artery pulses, glucose, and lactate sensors are also described, which may enable imperceptible health monitoring during future AR/VR practices.     

Emerging Thermal Technology Enabled Augmented Reality

In the past decade, remarkable progress has been made in the domain of augmented reality/virtual reality (AR/VR). The need for realistic and immersive augmentation has propelled the development of haptics interfaces-enabled AR/VR. The haptics interfaces facilitate direct interaction and manipulation with both real and virtual objects, thus augmenting the perception and experiences of the users. The level of augmentation can be significantly improved by thermal stimulation or sensing, which facilitates a higher degree of object identification and discrimination. This review discusses the thermal technology-enabled augmented reality and summarizes the recent progress in the development of different thermal technology such as thermal haptics including thermo-resistive heater and Peltier devices, thermal sensors including resistive, pyroelectric, and thermoelectric sensors, which can be utilized to improve the realism of augmentation. The fundamental mechanism, design strategies, and the rational guidelines for the adoption of these technologies in AR/VR is explicitly discussed. The conclusion provides an outlook on the existing challenges and outlines the future roadmap for the realization of next-generation thermo-haptics enabled augmented reality.     

Advanced graphics behind medical virtual reality: evolution ofalgorithms, hardware, and software interfaces

Applications of virtual reality (VR) and augmented reality (AR) in medicine require real-time visualization and modeling of large three-dimensional data sets. Consequently, these applications require powerful computation, extensive high-bandwidth memory, and fast communication links. In the past, the manufacturers of medical imaging equipment produced their own special-purpose proprietary hardware for image processing and solid graphics. Due to the developments in computer hardware in general and in graphics accelerators in particular, there is a trend toward replacing the proprietary hardware off-the-shelf (OTS) equipment. Computer graphics itself has advanced in its quest for realism. Generic algorithms such as shading, texture mapping, and volume rendering have been developed to meet the resultant ever increasing requirements. Advances in both the OTS CPU and graphics hardware have enabled real-time implementations of these algorithms, thereby facilitating many of the medical VR/AR applications used today. The development of graphics libraries such as OpenGL has also been an important factor. These libraries provide an underlying portable software platform that optimizes the utilization of the available graphics hardware. OpenGL has become a standard graphics application programming interface, particularly for graphics-intensive applications, and more and more OTS systems provide hardware implementations of OpenGL commands. The review paper follows the evolution of these technologies and examines their crucial role in enabling the appearance of the current VR/AR applications in medicine and provides a look at current trends and future possibilities     

Thermo-Haptic Materials and Devices for Wearable Virtual and Augmented Reality

The current generation of virtual and augmented realities (VR/AR) has substantially advanced in the past decade because of the rapid development of converging technologies in various engineering and scientific fields. However, the current VR/AR technologies rely mainly on visual and auditory senses to physically replicate the virtual environment, although tactile senses play a significant role in the daily life since a myriad of tactile information is received through physical touch. Of the tactile senses, thermal senses are of great importance to be reproduced in the VR/AR field, since heat is transferred constantly and further interact with the surrounding environment. To date, there has been a huge amount of research studies on functional materials, thermo-haptic devices, and wearable electronics that have all converged to form the fundamental groundwork for the development of wearable thermal VR/AR devices. In this progress report, a review on various physical mechanisms and research is provided that can potentially be applied in the next generation of thermal VR/AR technologies and discuss the essential challenges that need to be addressed.     

Textile Electronics for VR/AR Applications

Textile electronics addresses fibers or fiber assemblies with electronic functions to generate, transmit, modulate, and detect electrons. Interactive textile electronic devices may provide suitable platforms for virtual reality (VR)/augmented reality (AR) applications because of their excellent performance and unique immersive features such as lightweight, handiness, flexibility, comfort, and low strain even under high deformations. This paper presents a systematic review of the literature on the state-of-the-art of interactive devices, fabrication technologies, system integration, promising applications, and challenges involved in textile-based VR/AR systems.     

Covalent Surface Modification Effects on Single‐Walled Carbon Nanotubes for Targeted Sensing and Optical Imaging

Optical nanoscale technologies often implement covalent or noncovalent strategies for the modification of nanoparticles, whereby both functionalizations are leveraged for multimodal applications but can affect the intrinsic fluorescence of nanoparticles. Specifically, single‐walled carbon nanotubes (SWCNTs) can enable real‐time imaging and cellular delivery; however, the introduction of covalent SWCNT sidewall functionalizations often attenuates SWCNT fluorescence. Recent advances in SWCNT covalent functionalization chemistries preserve the SWCNT's pristine graphitic lattice and intrinsic fluorescence, and here, such covalently functionalized SWCNTs maintain intrinsic fluorescence‐based molecular recognition of neurotransmitter and protein analytes. The covalently modified SWCNT nanosensor preserves its fluorescence response towards its analyte for certain nanosensors, presumably dependent on the intermolecular interactions between SWCNTs or the steric hindrance introduced by the covalent functionalization that hinders noncovalent interactions with the SWCNT surface. These SWCNT nanosensors are further functionalized via their covalent handles with a targeting ligand, biotin, to self‐assemble on passivated microscopy slides, and these dual‐functionalized SWCNT materials are explored for future use in multiplexed sensing and imaging applications.     

Flex Ethernet Technology and Application in 5G Mobile Transport Network

With the advent of 5G era,the rise of cloud services,virtual reality/virtual reality(AR/VR),vehicle networking and other technologies has put forward new requirements for the bandwidth and delay of the bearer network.Traditional Ethernet technology cannot meet the new requirements very well.Flex Ethernet(FlexE)technology has emerged as the times require.This paper introduces the background,standardization process,functional principle,application mode and technical advantages of FlexE technology,and finally analyses its application prospects and shortcomings in 5G mobile transport network.     

Flexible Hybrid Sensor Systems with Feedback Functions

Skin-like wearable sensors are regarded as key technologies toward home-based healthcare, human–machine interfaces, robotics, prostheses, and enhanced augmented/virtual reality (AR/VR). Inspired by human somatosensory functions, artificial sensory feedback systems play vital roles in shaping interactions with complex environments and timely decision-making. This study presents an overview of recent advances in feedback-driven, closed-loop skin-inspired flexible sensor systems that make use of emerging functional nanomaterials and elaborate structures. Drawing on feedback solutions, four categories of sensor systems are highlighted, which include prosthesis- and AR/VR-based human–machine interfaces, smartphone-based approaches for point-of-care detection, and smart wearable displays for direct signal visualizations. Furthermore, the progress of machine learning on the reliable recognition of massive quantities of signals generated by flexible sensor networks is briefly discussed. The state-of-the-art hybrid sensor techniques, along with other emerging strategies, will enable total sensory feedback loop systems to be developed for next-generation electronic skins.     

A student's perspective: getting real with virtual reality

As with many up and coming technologies, the road to wide acceptance is not always smooth. Virtual reality (VR) is one such technology whose bumpy ascent is well underway. VR is not all that new but the steady price/performance gains for computational power have helped it gain a broader base of applications and even a little popularity along the way. The author's current research is one example of how VR can be used as an analysis as well as a visualization tool. He is developing interactive scenarios where the user can pose "what if?" questions in a VR environment and the VR environment will respond with an array of possibilities. The user will then be able to experience and better assess the impacts of each alternative. The interactive aspects of VR make it an ideal platform for developing scenarios across a broad range of applications.     

Printable Semiconductors for Backplane TFTs of Flexible OLED Displays

Studies on printable semiconductors and technologies have increased rapidly over recent decades, pioneering novel applications in many fields, such as energy, sensing, logic circuits, and information displays. The newest display technologies are already turning to metal oxide semiconductors, i.e., indium gallium zinc oxide, for the improvements needed to drive active matrix organic light‐emitting diodes. Convenience and portability will be realized with flexible and wearable displays in the future. This report summarizes recent progress on the development of solution‐processed thin film transistors, especially those deposited at low temperatures for next‐generation flexible smart displays. The first part provides an overview on the history and current status of displays. Then, recent advances in state‐of‐the‐art solution‐processed transistors based on different semiconductors are presented, including metal oxides, organic materials, perovskites, and carbon nanotubes. Finally, conclusions are drawn and the remaining challenges and future perspectives are discussed.     

Conducting a virtual orchestra

The virtual orchestra is a multimodal system architecture based on complex 3D sound environment processing and a compact user interface. The system architecture supports high-performance sound content for VR (virtual reality) applications, and the user controls the performance with a magnetically tracked handheld device. The objective of this work is to give music amateurs the opportunity to conduct a group of musicians and produce a new kind of multimedia experience. To explore this idea, the authors developed a semi-immersive virtual reality system, based on a large projection screen, designed to create an entertaining multimedia experience for nonexpert users. They decided to implement a music performance simulator, a VR system where several virtual characters play different musical instruments and adapt their performance according to the user's instructions.     

IPTV架构下VR业务关键技术与解决方案

虚拟现实技术作为一种全新的技术，在视频方面的革新带给用户全新的交互体验。介绍了国内外VR/AR产业的发展情况，分析了IPTV架构下VR业务的需求；对编解码、投影、业务平台、机顶盒和网络传输等关键技术进行了研究解析；然后从VR的两种不同使用场景出发，提出了IPTV架构下VR业务的解决方案及业务流程，为IPTV VR的发展提供了一种可供参考的解决方案。     

Stretchable Skin‐Like Cooling/Heating Device for Reconstruction of Artificial Thermal Sensation in Virtual Reality

Along with visual and tactile sensations, thermal sensation by temperature feeling on the skin can provide rich physical information on the environment and objects. With a simple touch of objects, relative temperature can be sensed and even objects can be differentiated with different thermal properties without any visual cue. Thus, artificially reproducing accurate/controllable thermal sensation haptic signals on human epidermis will certainly be a major research area to reconstruct a more realistic virtual reality (VR) environment. In this study, for the first time, a skin‐like, highly soft and stretchable and bi‐functional (both cold and hot sensation) thermo‐haptic device is reported for wearable VR applications with a single device structure (not separate heater and cooler). The skin‐like thermo‐haptic (STH) device can actively cool down and heat up deformable skin surfaces with instantaneous and accurate adjustment of temperature based upon a feedback control algorithm to mimic desirable thermal sensation with 230% stretchability. As a proof‐of‐concept, the STH device is integrated with a finger‐motion tracking glove to provide artificial thermal sensation information to the skin in various situations such as touching cold beer bottles and hot coffee cups in virtual space. This new type of STH device can offer potential implications for next‐generation haptic devices to provide unique thermal information for a more realistic virtual‐world field and medical thermal treatment.     

Phantom-based multimodal interactions for medical education and training: the Munich knee joint simulator

Simulation environments based on virtual reality technologies can support medical education and training. In this paper, the novel approach of an "interactive phantom" is presented that allows a realistic display of haptic contact information typically generated when touching and moving human organs or segments. The key idea of the haptic interface is to attach passive phantom objects to a mechanical actuator. The phantoms look and feel as real anatomical objects. Additional visualization of internal anatomical and physiological information and sound generated during the interaction with the phantom yield a multimodal approach that can increase performance, didactic value, and immersion into the virtual environment. Compared to classical approaches, this multimodal display is convenient to use, provides realistic tactile properties, and can be partly adjusted to different, e.g., pathological properties. The interactive phantom is exemplified by a virtual human knee joint that can support orthopedic education, especially for the training of clinical knee joint evaluation. It is suggested that the technical principle can be transferred to many other fields of medical education and training such as obstetrics and dentistry.