Nanowire-Based Soft Wearable Human–Machine Interfaces for Future Virtual and Augmented Reality Applications

A virtual world has now become a reality as augmented reality (AR) and virtual reality (VR) technology become commercially available. Similar to how humans interact with the physical world, AR and VR systems rely on human–machine interface (HMI) sensors to interact with the virtual world. Currently, this is achieved via state of-the-art wearable visual and auditory tools that are rigid, bulky, and burdensome, thereby causing discomfort during practical application. To this end, a skin sensory interface has the potential to serve as the next-generation AR/VR technology because skin-like wearable sensors have advantages in that they can be ultrathin, ultra-soft, conformal, and imperceptible, which provides the ultimate comfort and immersive experience for users. In this progress report, nanowire-based soft wearable HMI sensors including acoustic, strain, pressure sensors, and physiological sensors are reviewed that may be adopted as skin sensory inputs in future AR/VR systems. Further, nanowire-based soft contact lenses, haptic force, and thermal and vibration actuators are covered as potential means of feedback for future AR/VR systems. Considering the possible effects of the virtual world on human health, skin-like wearable artery pulses, glucose, and lactate sensors are also described, which may enable imperceptible health monitoring during future AR/VR practices.     

Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality

Virtual reality and augmented reality (VR/AR) are evolving. The market demands and innovation efforts call for a shift in the key VR/AR technologies and engaging people virtually. Tele-haptics with multimodal and bilateral interactions are emerging as the future of the VR/AR industry. By transmitting and receiving haptic sensations wirelessly, tele-haptics allow human-to-human interactions beyond the traditional VR/AR interactions. The core technologies for tele-haptics include multimodal tactile sensing and feedback based on highly advanced sensors and actuators. Recent developments of haptic innovations based on active materials show that active materials can significantly contribute to addressing the needs and challenges for the current and future VR/AR technologies. Thus, this paper intends to review the current status and opportunities of active material-based haptic technology with a focus on VR/AR applications. It first provides an overview of the current VR/AR applications of active materials for haptic sensing and actuation. It then highlights the state-of-the-art haptic interfaces that are relevant to the materials with an aim to provide perspectives on the role of active materials and their potential integration in haptic devices. This paper concludes with the perspective and outlook of immersive multimodal tele-haptic interaction technologies.     

Recent Advances in Wearable Sensors and Integrated Functional Devices for Virtual and Augmented Reality Applications

The advancement in virtual reality/augmented reality (VR/AR) has been achieved by breakthroughs in the realistic perception of virtual elements. Although VR/AR technology is advancing fast, enhanced sensor functions, long-term wearability, and seamless integration with other electronic components are still required for more natural interactions with the virtual world. Here, this report reviews the recent advances in multifunctional wearable sensors and integrated functional devices for VR/AR applications. Specified device designs, packaging strategies, and interactive physiological sensors are summarized based on their methodological approaches for sensory inputs and virtual feedback. In addition, limitations of the existing systems, key challenges, and future directions are discussed. It is envisioned that this progress report's outcomes will expand the insights on wearable functional sensors and device interfaces toward next-generation VR/AR technologies.     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

Skin Electronics: Next-Generation Device Platform for Virtual and Augmented Reality

Virtual reality (VR) and augmented reality (AR) are overcoming the physical limits of real-life using advances in devices and software. In particular, the recent restrictions in transportation from the coronavirus disease 2019 (COVID-19) pandemic are making people more interested in these virtual experiences. However, to minimize the differences between artificial and natural perception, more human-interactive and human-like devices are necessary. The skin is the largest organ of the human body and interacts with the environment as the site of interfacing and sensing. Recent progress in skin electronics has enabled the use of the skin as the mounting object of functional devices and the signal pathway bridging humans and computers, with opening its potential in future VR and AR applications. In this review, the current skin electronics are summarized as one of the most promising device solutions for future VR/AR devices, especially focusing on the recent materials and structures. After defining and explaining VR/AR systems and the components, the advantages of skin electronics for VR/AR applications are emphasized. Next, the detailed functionalities of skin electronic devices, including the input, output, energy devices, and integrated systems, are reviewed for future VR/AR applications.     

Textile Electronics for VR/AR Applications

Textile electronics addresses fibers or fiber assemblies with electronic functions to generate, transmit, modulate, and detect electrons. Interactive textile electronic devices may provide suitable platforms for virtual reality (VR)/augmented reality (AR) applications because of their excellent performance and unique immersive features such as lightweight, handiness, flexibility, comfort, and low strain even under high deformations. This paper presents a systematic review of the literature on the state-of-the-art of interactive devices, fabrication technologies, system integration, promising applications, and challenges involved in textile-based VR/AR systems.     

Stretchable Skin‐Like Cooling/Heating Device for Reconstruction of Artificial Thermal Sensation in Virtual Reality

Along with visual and tactile sensations, thermal sensation by temperature feeling on the skin can provide rich physical information on the environment and objects. With a simple touch of objects, relative temperature can be sensed and even objects can be differentiated with different thermal properties without any visual cue. Thus, artificially reproducing accurate/controllable thermal sensation haptic signals on human epidermis will certainly be a major research area to reconstruct a more realistic virtual reality (VR) environment. In this study, for the first time, a skin‐like, highly soft and stretchable and bi‐functional (both cold and hot sensation) thermo‐haptic device is reported for wearable VR applications with a single device structure (not separate heater and cooler). The skin‐like thermo‐haptic (STH) device can actively cool down and heat up deformable skin surfaces with instantaneous and accurate adjustment of temperature based upon a feedback control algorithm to mimic desirable thermal sensation with 230% stretchability. As a proof‐of‐concept, the STH device is integrated with a finger‐motion tracking glove to provide artificial thermal sensation information to the skin in various situations such as touching cold beer bottles and hot coffee cups in virtual space. This new type of STH device can offer potential implications for next‐generation haptic devices to provide unique thermal information for a more realistic virtual‐world field and medical thermal treatment.     

虚拟现实技术在医学领域的研究现状与进展

虚拟现实(VR)与其所衍生的增强现实(AR)和混合现实(MR)能把携带三维信息的虚拟场景与真实世界相互叠加,极大地提升用户感官世界的直观性、精准性、实时性。该技术的推广与应用,将会给医学领域带来变革式发展。本文剖析VR/AR/MR的概念并简述其发展历程,分别对虚拟现实和增强现实在医学领域的应用进行阐述,并通过微软产品HoloLens的特点分析基于混合现实的解决方案在医学领域的优势。最后对VR/AR/MR目前在医学领域所存在的不足进行归纳,并对未来的发展趋势进行展望。     

Thermo-Haptic Materials and Devices for Wearable Virtual and Augmented Reality

The current generation of virtual and augmented realities (VR/AR) has substantially advanced in the past decade because of the rapid development of converging technologies in various engineering and scientific fields. However, the current VR/AR technologies rely mainly on visual and auditory senses to physically replicate the virtual environment, although tactile senses play a significant role in the daily life since a myriad of tactile information is received through physical touch. Of the tactile senses, thermal senses are of great importance to be reproduced in the VR/AR field, since heat is transferred constantly and further interact with the surrounding environment. To date, there has been a huge amount of research studies on functional materials, thermo-haptic devices, and wearable electronics that have all converged to form the fundamental groundwork for the development of wearable thermal VR/AR devices. In this progress report, a review on various physical mechanisms and research is provided that can potentially be applied in the next generation of thermal VR/AR technologies and discuss the essential challenges that need to be addressed.     

Liquid Crystal Soft Actuators and Robots toward Mixed Reality

Liquid crystals (LCs) are soft but smart materials that can adjust its chemical or physical properties in response to various external stimuli. Using these materials to construct soft actuators and robots, referred as LC actuators and robots, is expected to replace current machinery part, obtaining lighter and smaller equipment with adjustable and complex functions. Especially, combining these LC actuator and robots with existing virtual reality and augmented reality technologies will produce a new world of mixed reality (MR) with the visual, auditory, and somatosensory interaction. In this review, the recent work on responsive LC actuators and robots is introduced, emphasizing on their potentials in haptic use. By discussing their programmable control via suitable stimuli, the LC actuators and robots are summarized for mechanical outputs, environmental mimic, and fine-tuning of surface texture and roughness. It is anticipated that the continuous development on LC actuators and robots will accelerate the MR technology toward practical application.     

Emerging Thermal Technology Enabled Augmented Reality

In the past decade, remarkable progress has been made in the domain of augmented reality/virtual reality (AR/VR). The need for realistic and immersive augmentation has propelled the development of haptics interfaces-enabled AR/VR. The haptics interfaces facilitate direct interaction and manipulation with both real and virtual objects, thus augmenting the perception and experiences of the users. The level of augmentation can be significantly improved by thermal stimulation or sensing, which facilitates a higher degree of object identification and discrimination. This review discusses the thermal technology-enabled augmented reality and summarizes the recent progress in the development of different thermal technology such as thermal haptics including thermo-resistive heater and Peltier devices, thermal sensors including resistive, pyroelectric, and thermoelectric sensors, which can be utilized to improve the realism of augmentation. The fundamental mechanism, design strategies, and the rational guidelines for the adoption of these technologies in AR/VR is explicitly discussed. The conclusion provides an outlook on the existing challenges and outlines the future roadmap for the realization of next-generation thermo-haptics enabled augmented reality.     

Untethered Feel-Through Haptics Using 18-µm Thick Dielectric Elastomer Actuators

Head-mounted displays for virtual reality (VR) and augmented reality (AR) allow users to see highly realistic virtual worlds. The wearable haptics that enable feeling and touching these virtual objects are typically bulky, tethered, and provide only low fidelity feedback. A particularly challenging type of wearable human-machine interface is feel-through haptics: ultra-thin wearables so soft as to be mechanically imperceptible when turned off, yet generating sufficient force when actuated to make virtual objects feel tangible, or to change the perceived texture of a physical object. Here, 18 µm thick soft dielectric elastomer actuators (DEA), directly applied on the skin, reports rich vibrotactile feedback generation from 1 Hz to 500 Hz. Users correctly identifies different frequency and sequence patterns with success rates from 73 to 97% for devices applied on their fingertips. An untethered version weighing only 1.3 grams allowed blindfolded users to correctly identify letters by “seeing” them through their fingers. The silicone-based DEA membrane is mechanically transparent, enabling wearable haptics for the many applications where hand dexterity is critical. The feel-through DEA can be placed in array format anywhere on the body.     

A Liquid Metal Based Multimodal Sensor and Haptic Feedback Device for Thermal and Tactile Sensation Generation in Virtual Reality

Virtual reality (VR) has been widely used for training, gaming, and entertainment, and the value of VR is continually increasing as a contact-free technology. For an immersive VR experience, measuring finger movements and providing appropriate feedback to the hand are as important as visual information, given the necessity of the hands for activities in daily life. Thus, a hand-worn VR device with motion sensors and haptic feedback is desirable. In this paper, a multimodal sensing and feedback glove is developed with soft, stretchable, lightweight, and compact sensor and heater sheets manufactured by direct ink writing (DIW) of liquid metal, eutectic gallium-indium (eGaIn). In the sensor sheet, ten sensors and three vibrators are embedded to measure finger movements and provide vibro-haptic feedback. The other heater sheet provides thermo-haptic sensation in accurate and rapid manner via model-based feedback control even under stretched conditions. The multimodal sensing and feedback glove allows users to feel the contact status and discriminate materials with different temperature. Performance of the proposed multimodal glove is verified under VR environments including touching and pushing two blocks of different materials and grabbing a heated metal ball submerged in hot water.     

基于虚拟现实的产品三维展示技术研究

文章在对VR虚拟体验展厅、VR数字体验馆进行实例分析的基础上,对虚拟现实下产品的三维展示技术进行分析,包括动态环境建模技术、实时三维图形生成技术、立体显示和传感器技术以及系统集成技术。     

Immersive interactive technologies and virtual shopping experiences: Differences in consumer perceptions between augmented reality (AR) and virtual reality (VR)

Based on the concepts of the Reality-Virtuality (RV) continuum and the stimulus-organism-response (SOR) framework, this study investigated differences between AR and VR in their effects on vividness/interactivity, a sense of presence, users’ sensory brand app experience, attitude, and behavioral intention. For AR, participants downloaded an app to mobile phones. For VR, participants downloaded a VR app to smartphones attached to a Google Cardboard VR headset. Vividness and interactivity directly (or indirectly) impacted a sense of presence, sensory brand app experience, attitude towards technology, and behavioral intention. However, AR and VR differed in how vividness and interactivity influenced sensory brand app experience and attitude towards technology. Overall, a sense of presence was a significant mediator for 1) the relationship between vividness and sensory brand app experience and 2) the relationship between interactivity and attitude towards technology for VR but no such relationship was observed with AR. The relationships among sensory brand app experience, attitude, and behavioral intention were significant in both AR and VR settings. Employing real AR/VR shopping situations, the study provides for practitioners deep understandings of consumers’ actual perceptions of applications of AR/VR and offers practical insights as to the efficiency of AR and VR in improving consumer virtual shopping experiences.     

Haptic Perception,Mechanics, and Material Technologies for Virtual Reality

Emerging virtual and augmented reality technologies can transform human activities in myriad domains, lending tangible, embodied form to digital data, services, and information. Haptic technologies will play a critical role in enabling human to touch and interact with the contents of these virtual environments. The immense variety of skilled manual tasks that humans perform in real environments are only possible through the coordination of touch sensation, perception, and movement that together comprise the haptic modality. Consequently, many research groups are vigorously investigating haptic technologies for virtual reality. A longstanding research goal in this area has been to create haptic interfaces that allow their users to touch and feel plausibly realistic virtual objects. In this progress report, the perspective on this unresolved research challenge is shared, guided by the observation that no technologies can even approximately match the capabilities of the human sense of touch. Factors that have it challenging to engineer haptic technologies for virtual reality, including the extraordinary spatial and temporal tactile acuity of the skin, and the complex interplay between continuum mechanics, haptic perception, and interaction are identified. The perspective on how these challenges may be overcome through convergent research on haptic perception, mechanics, electronics, and material technologies is presented.     

虚拟现实视频的制作和应用场景分析

目前,我国科技水平已经上升到了前所未有的高度,同时也推动了现代信息技术的飞速发展,许多先进的信息传输技术和图像处理技术都得到了质的飞跃。在大数据时代背景下,虚拟现实技术得到了全面推广,并在各个领域都得到了极大的突破,随着虚拟现实视频的引入,为观赏者带来了前所未有的视觉盛宴,让用户体验可交互式视频体验。我国当前的虚拟现实视频包括全景3D视频、非全景3D视频、局部全景3D视频、VR全景视频以及全景3D交互视频等,按照交互体验来看,又分为强交互视频与若交互视频。虚拟现实视频是基于虚拟现实技术为基础,通过不同场景视频的制作,在影视、直播、综艺等多个领域中都有着广泛的应用。主要针对虚拟显示视频的制作和应用场景进行探究。     

基于三维重建的虚拟现实类鸟飞行系统

随着虚拟现实(virtual reality, VR)技术的飞速发展,各种应用层出不穷。然而目前大多应用仅限于定点的静态全景展示及游览,无法利用VR化不可能为可能的优越性。本文提出了一种融合VR技术与三维重建算法的类鸟飞行交互模拟系统。该系统搭建了结构稳定的硬件控制平台用以改变实际姿态,同时通过软件仿真了虚拟环境中鸟类的飞行,结合硬件驱动及软件模拟实现了深层次的飞行模拟。现有的虚拟环境大多通过人工建模、激光扫描仪或无人机(unmanned aerial vehicle, UAV)航拍构造,其中建模存在费时费力的缺点,激光扫描仪则无法适应大规模重建,而UAV成本高昂且需专业培训。本文引入了基于图像分簇的PMVS (patch based multi-view stereopesis)算法,只需输入特定场景的图片即可利用计算机自动恢复出三维结构,不仅快速而且可对任意场景重构。使用户在足不出户的情况下即可在世界各地体验飞行,为旅游事业及VR产业提供了一种新的可能。     

Flexible Hybrid Sensor Systems with Feedback Functions

Skin-like wearable sensors are regarded as key technologies toward home-based healthcare, human–machine interfaces, robotics, prostheses, and enhanced augmented/virtual reality (AR/VR). Inspired by human somatosensory functions, artificial sensory feedback systems play vital roles in shaping interactions with complex environments and timely decision-making. This study presents an overview of recent advances in feedback-driven, closed-loop skin-inspired flexible sensor systems that make use of emerging functional nanomaterials and elaborate structures. Drawing on feedback solutions, four categories of sensor systems are highlighted, which include prosthesis- and AR/VR-based human–machine interfaces, smartphone-based approaches for point-of-care detection, and smart wearable displays for direct signal visualizations. Furthermore, the progress of machine learning on the reliable recognition of massive quantities of signals generated by flexible sensor networks is briefly discussed. The state-of-the-art hybrid sensor techniques, along with other emerging strategies, will enable total sensory feedback loop systems to be developed for next-generation electronic skins.     

Wireless User Perspectives in Europe: HandSmart Mediaphone Interface

HandSmart is one example of wearable device that can be used as a user interface for advanced mediaphones and it is based on MARISIL a Mobile Augmented Reality Interface Sign Interpretation Language. This paper will describe the interface and some of the applications of these new types of personal devices. The user-centered development methodology is discussed in brief at the end of the paper. Evolutions in technology have provided a variety of new opportunities for exploring and discovering virtual 3D worlds. Head-mounted displays and data gloves enable us to interact and immerse much better into the artificial generated 3D environment. Such devices have been advertised in the entertainment media and are recognized by the public as the symbols of virtual reality (VR). Augmented Reality that has the attribute of being more related to real world than VR by overlaying virtual sounds, feelings or visions onto our senses within the real world, can therefore extend our natural experiences. The authors believe that the new generation of mediaphones can embed these new techniques.