Haptic User Interfaces for the Visually Impaired: Implications for Haptically Enhanced Science Learning Systems

The overall quality of haptic user interfaces designed to support visually impaired students' science learning through sensorial feedback was systematically studied to investigate task performance and user behavior. Fourteen 6th- to 11th-grade students with visual impairments recruited from a state-funded blind school were asked to perform three main tasks (i.e., menu selection, structure exploration, and force recognition) using haptic user interfaces and a haptic device. This study used several dependent measures that are categorized into three types of variables: (a) task performance including success rate, workload, and task completion time; (b) user behavior defined as cursor movements proportionately represented from the user's cursor positional data; and (c) user preference. Results showed that interface type has significant effects on task performance, user behavior, and user preference, with varying degree of impact to participants with severe visual impairments performing the tasks. The results of this study as well as a set of refined design guidelines and principles should provide insights to the future research of haptic user interfaces that can be used when developing haptically enhanced science learning systems for the visually impaired.     

Robot-enabled tangible virtual assembly with coordinated midair object placement

The assembly in Virtual Reality (VR) enables users to fit virtual parts into existing 3D models immersively. However, users cannot physically feel the haptic feedback when connecting the parts with the virtual model. This work presents a robot-enabled tangible interface that dynamically moves a physical structure with a robotic arm to provide physical feedback for holding a handheld proxy in VR. This enables the system to provide force feedback during virtual assembly. The cooperation between the physical support and the handheld proxy produces realistic physical force feedback, providing a tangible experience for various virtual parts in virtual assembly scenarios. We developed a prototype system that allowed the operator to place a virtual part onto other models in VR by placing the proxy onto the matched structure attached to a robotic arm. We conducted a user evaluation to explore user performance and system usability in a virtual assembly task. The results indicated that the robot-enabled tangible support increased the task completion time but significantly improved the system usability and sense of presence with a more realistic haptic experience.     

Haptic gas pedal feedback

Active driver support systems either automate a control task or present warnings to drivers when their safety is seriously degraded. In a novel approach, utilising neither automation nor discrete warnings, a haptic gas pedal (accelerator) interface was developed that continuously presents car-following support information, keeping the driver in the loop. This interface was tested in a fixed-base driving simulator. Twenty-one drivers between the ages of 24 and 30 years participated in a driving experiment to investigate the effects of haptic gas pedal feedback on car-following behaviour. Results of the experiment indicate that when haptic feedback was presented to the drivers, some improvement in car-following performance was achieved, while control activity decreased. Further research is needed to investigate the effectiveness of the system in more varied driving conditions. Haptics is an under-used modality in the application of human support interfaces, which usually draw on vision or hearing. This study demonstrates how haptics can be used to create an effective driver support interface.     

Haptic gas pedal feedback

Active driver support systems either automate a control task or present warnings to drivers when their safety is seriously degraded. In a novel approach, utilising neither automation nor discrete warnings, a haptic gas pedal (accelerator) interface was developed that continuously presents car-following support information, keeping the driver in the loop. This interface was tested in a fixed-base driving simulator. Twenty-one drivers between the ages of 24 and 30 years participated in a driving experiment to investigate the effects of haptic gas pedal feedback on car-following behaviour. Results of the experiment indicate that when haptic feedback was presented to the drivers, some improvement in car-following performance was achieved, while control activity decreased. Further research is needed to investigate the effectiveness of the system in more varied driving conditions. Haptics is an under-used modality in the application of human support interfaces, which usually draw on vision or hearing. This study demonstrates how haptics can be used to create an effective driver support interface.     

The affect of contact force sensations on user performance in virtual assembly tasks

The development of a realistic virtual assembly environment is challenging because of the complexity of the physical processes and the limitation of available VR technology. Many research activities in this domain primarily focused on particular aspects of the assembly task such as the feasibility of assembly operations in terms of interference between the manipulated parts. The virtual assembly environment reported in this research is focused on mechanical part assembly. The approach presented addresses the problem of part-to-part contacts during the mating phase of assembly tasks. The system described calculates contact force sensations by making their intensity dependent on the depth of penetration. However the penetration is not visible to the user who sees a separate model, which does not intersect the mating part model. The two 3D models of the part, the off-screen rendered model and the on-screen rendered model are connected by a spring-dumper arrangement. The force calculated is felt by the operator through the haptic interface when parts come in contact during the mating phase of the assembly task. An evaluation study investigating the effect of contact force sensation on user performance during part-to-part interface was conducted. The results showed statistically significant effect of contact force sensation on user performance in terms of task completion time. The subjective evaluation based on feedback from users confirmed that contact force sensation is a useful cue for the operator to find the relative positions of components in the final assembly state.     

Realistic haptic rendering of interacting deformable objects in virtual environments

A new computer haptics algorithm to be used in general interactive manipulations of deformable virtual objects is presented. In multimodal interactive simulations, haptic feedback computation often comes from contact forces. Subsequently, the fidelity of haptic rendering depends significantly on contact space modeling. Contact and friction laws between deformable models are often simplified in up to date methods. They do not allow a "realistic" rendering of the subtleties of contact space physical phenomena (such as slip and stick effects due to friction or mechanical coupling between contacts). In this paper, we use Signorini's contact law and Coulomb's friction law as a computer haptics basis. Real-time performance is made possible thanks to a linearization of the behavior in the contact space, formulated as the so-called Delassus operator, and iteratively solved by a Gauss-Seidel type algorithm. Dynamic deformation uses corotational global formulation to obtain the Delassus operator in which the mass and stiffness ratio are dissociated from the simulation time step. This last point is crucial to keep stable haptic feedback. This global approach has been packaged, implemented, and tested. Stable and realistic 6D haptic feedback is demonstrated through a clipping task experiment.     

Virtual DesignWorks—designing 3D CAD models via haptic interaction

Haptic devices (force feedback interfaces) show great promise for use in the design of CAD models. However, current haptic systems are used primarily to verify rather than to interact with CAD systems to design a model.The paper discusses a haptic-based CAD system—Virtual DesignWorks. It is the world's first haptic application for the design of CAD models based on component technology (COM+). With haptic devices, the system allows engineers, in virtual space, to directly touch a native B-Rep CAD model, and deform it by pushing, pulling and dragging its surfaces in a natural three-dimensional environment. Virtual DesignWorks adopts the novel COM-based haptic model, which demonstrates significant advantages compared to the traditional haptic models. Force feedback gives designers the greatest flexibility for the design of complex surfaces, and haptics has the potential to become a critical interface for design applications.     

虚拟装配中基于导纳控制的力觉渲染技术

传统的力触觉渲染多采用阻抗控制,不能很好地满足虚拟装配的应用要求,相比之下导纳控制模式更适用这一领域.为此提出一种基于导纳控制的双线程力觉渲染构架,并给出相应的力觉渲染算法.首先建立用于导纳控制的动力学模型,并讨论了碰撞和约束这2个状态下的力觉渲染;为了使用力觉交互接口进行虚拟装配中的小间隙装配,提出物理约束与几何约束结合的力觉渲染方法;最后针对物理计算和力反馈循环2个线程刷新频率不匹配的问题,利用二次拉格朗日多项式进行数值插值,实现了力觉交互接口的平稳输出.通过力反馈设备与自主开发的虚拟装配原型系统VAPP的连接与应用,验证了所提出的算法满足虚拟装配系统中力觉交互的应用要求.     

Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations

We survey the haptics literature and identify conditions under which haptic interaction displays can enhance human perception and performance. Integrating haptic interactions in multimodal systems requires understanding user's sensory, perceptual, and cognitive abilities and limitations. Haptic design guidelines can aid developers of multimodal interactive systems. Haptic interaction relates to all aspects of touch and body movement and the application of these senses to computer interaction. This involves not only sensation and perception, but also motor and cognitive aspects of active movement (that is, self-initiated movement) for which detailed motor plans are created, stored in memory, and compared to receptor feedback from the muscles, joints, and skin.     

Visual-haptic feedback interaction in automotive touchscreens

Touchscreen interfaces offer benefits in terms of flexibility and ease of interaction and as such their use has increased rapidly in a range of devices, from mobile phones to in-car technology. However, traditional touchscreens impose an inevitable visual workload demand that has implications for safety, especially in automotive use. Recent developments in touchscreen technology have enabled feedback to be provided via the haptic channel. A study was conducted to investigate the effects of visual and haptic touchscreen feedback on visual workload, task performance and subjective response using a medium-fidelity driving simulator. Thirty-six experienced drivers performed touchscreen ‘search and select’ tasks while engaged in a motorway driving task. The study utilised a 3 × 2 within-subjects design, with three levels of visual feedback: ‘immediate’, ‘delayed’, ‘none’; and two levels of haptic feedback: ‘visual only’, ‘visual + haptic’. Results showed that visual workload was increased when visual feedback was delayed or absent; however, introducing haptic feedback counteracted this effect, with no increases observed in glance time and count. Task completion time was also reduced when haptic feedback was enabled, while driving performance showed no effect due to feedback type. Subjective responses indicated that haptic feedback improved the user experience and reduced perceived task difficulty.     

Effects of haptic feedback,stereoscopy, and image resolution on performance and presence in remote navigation

Traditionally, the main goal of teleoperation has been to successfully achieve a given task as if performing the task in local space. An emerging and related requirement is to also match the subjective sensation or the user experience of the remote environment, while maintaining reasonable task performance. This concept is often called “presence” or “(experiential) telepresence,” which is informally defined as “the sense of being in a mediated environment.” In this paper, haptic feedback is considered as an important element for providing improved presence and reasonable task performance in remote navigation. An approach for using haptic information to “experientially” teleoperate a mobile robot is described. Haptic feedback is computed from the range information obtained by a sonar array attached to the robot, and delivered to a user's hand via a haptic probe. This haptic feedback is provided to the user, in addition to stereoscopic images from a forward-facing stereo camera mounted on the mobile robot. The experiment with a user population in a real-world environment showed that haptic feedback significantly improved both task performance and user-felt presence. When considering user-felt presence, no interaction among haptic feedback, image resolution, and stereoscopy was observed, that is, haptic feedback was effective, regardless of the fidelity of visual elements. Stereoscopic images also significantly improved both task performance and user-felt presence, but high-resolution images only significantly improved user-felt presence. When considering task performance, however, it was found that there was an interaction between haptic feedback and stereoscopy, that is, stereoscopic images were only effective when no force feedback was applied. According to the multiple regression analysis, haptic feedback was a higher contributing factor to the improvement in performance and presence than image resolution and stereoscopy.     

The use of force feedback and auditory cues for performance of an assembly task in an immersive virtual environment

Using an immersive virtual environment, this study investigated whether the inclusion of force feedback or auditory cues improved manipulation performance and subjective reports of usability for an assembly task. Twenty-four volunteers (12 males and 12 females) were required to assemble and then disassemble five interconnecting virtual parts with either auditory, force, or no feedback cues provided. Performance for the assembly task was measured using completion time and number of collisions between parts, while the users preferences across conditions were evaluated using subjective reports of usability. The results indicated that the addition of force feedback slowed completion time and led to more collisions between parts for males. In contrast, females exhibited no change in the mean completion time for the assembly task but did show an increase in collision counts. Despite these negative performance findings when adding force feedback, users did report perceived increases in realism, helpfulness and utility towards the assembly task when force feedback was provided. Unlike force feedback, the results showed that auditory feedback, indicating that parts had collided during the assembly task, had no negative performance effects on the objective measures while still increasing perceived realism and overall user satisfaction. When auditory cues and force feedback were presented together, performance times, number of collisions, and usability were not improved compared to conditions containing just auditory cues or force feedback alone. Based on these results, and given the task and display devices used in the present study, the less costly option of excluding auditory and force feedback cues would produce the best performance when measured by the number of collisions and completion time. However, if increased ratings of usability for an assembly task are desired while maintaining objective performance levels and reduced cost, then the inclusion of auditory feedback cues is best.     

High-fidelity bilateral teleoperation systems and the effect of multimodal haptics.

In master-slave teleoperation applications that deal with a delicate and sensitive environment, it is important to provide haptic feedback of slave/environment interactions to the user's hand as it improves task performance and teleoperation transparency (fidelity), which is the extent of telepresence of the remote environment available to the user through the master-slave system. For haptic teleoperation, in addition to a haptics-capable master interface, often one or more force sensors are also used, which warrant new bilateral control architectures while increasing the cost and the complexity of the teleoperation system. In this paper, we investigate the added benefits of using force sensors that measure hand/master and slave/environment interactions and of utilizing local feedback loops on the teleoperation transparency. We compare the two-channel and the four-channel bilateral control systems in terms of stability and transparency, and study the stability and performance robustness of the four-channel method against nonidealities that arise during bilateral control implementation, which include master-slave communication latency and changes in the environment dynamics. The next issue addressed in the paper deals with the case where the master interface is not haptics capable, but the slave is equipped with a force sensor. In the context of robotics-assisted soft-tissue surgical applications, we explore through human factors experiments whether slave/environment force measurements can be of any help with regard to improving task performance. The last problem we study is whether slave/environment force information, with and without haptic capability in the master interface, can help improve outcomes under degraded visual conditions.     

Upper-body haptic system for snake robot teleoperation in pipelines

Snake robots have shown a great potential for operations in confined workplaces that are less accessible or dangerous to human workers, such as the in-pipe inspection. However, the snake robot teleoperation remains a nontrivial task due to the unique locomotion mechanism (e.g., helical motion) and the constraints of the workplaces including the low visibility and indistinguishable features. Most snake robot feedback systems are based on the live camera view only. It is hard for the human operator to develop a correct spatial understanding of the remote workplace, leading to problems such as disorientation and motion sickness in snake robot teleoperation. This study designs and evaluates an innovative haptic assistant system for snake robot teleoperation in the in-pipe inspection. An upper-body haptic suit with 40 vibrators on both the front and back sides of the human operator was developed to generate haptic feedback corresponding to the bottom and up sides of the snake robot, transferring the egocentric sensation of the snake robot to the human operator. A human-subject experiment (n = 31) was performed to evaluate the efficacy of the developed system. The results indicate that the proposed haptic assistant system outperformed other feedback systems in terms of both task performance and subjective workload and motion sickness evaluations. It inspires new control and feedback designs for the future snake robot in industrial operations.     

Effects of feedback type and modality on motor skill learning and retention

ABSTRACT

Two types of feedback, including knowledge of results (KR) and knowledge of performance (KP) are typically delivered as part of motor training using different sensory modalities. Twenty-four right-handed individuals performed a computer-based psychomotor training task using a contemporary haptic interface. Results revealed that KP was superior to KR for overall task performance retention, whereas KR was good for training specific aspects of performance. A combination of KP and KR through the same modality did not produce an additive positive learning effect for task performance. Auditory and haptic feedback improved retention over visual, but only with the use of multiple information cues. Design guidelines are provided for training systems for motor skill development and/or rehabilitation.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of 'drag-and-drops' while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of ‘drag-and-drops’ while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

Haptic rendering of dynamic volumetric data

With current methods for volume haptics in scientific visualization, features in time-varying data can freely move straight through the haptic probe without generating any haptic feedback the algorithms are simply not designed to handle variation with time but consider only the instantaneous configuration when the haptic feedback is calculated. This article introduces haptic rendering of dynamic volumetric data to provide a means for haptic exploration of dynamic behaviour in volumetric data. We show how haptic feedback can be produced that is consistent with volumetric data moving within the virtual environment and with data that, in itself, evolves over time. Haptic interaction with time-varying data is demonstrated by allowing palpation of a CT sequence of a beating human heart.     

Slingshot 3D: A synchronous haptic-audio-video game

Recent advances in multimedia and human computer interaction technologies have paved the way for rich contents across multiple media such as haptics, audio, and video. This paper introduces a multimodal game named the Slingshot 3D Game: an interactive and synchronous haptic-audio-video shooter game over the Internet network. The game incorporates two types of haptic feedback: tactile feedback using a haptic jacket and kinesthetic feedback using the Novint Falcon haptic interface. Furthermore, the game utilizes a depth camera to track the player's (upper) body movements and detect collisions between the player's body and the shot projectile. To promote availability and cost, the game uses the Internet network as the communication medium between the players, by utilizing the Admux communication framework [10]. The game design and implementation are detailed in this paper. Both the player performance analysis and the user satisfaction analysis have shown that the incorporation of synchronous haptic-video multimedia has enhanced the perception of player presence and the overall quality of performance.