An investigation on the impact of auditory and haptic feedback on rhythmic walking interactions

This paper presents a system capable of rhythmic walking interaction by auditory and haptic display. Likewise, it summarizes the results of the research on the influence of the audio and haptic stimulation on rhythmic walking interaction. The system detects user’s footsteps and either provides interactive real-time feedback or suggests a pace using a synthetic walking sound or vibration. This pace is either a constant tempo or adapts to the walker. Auditory and haptic feedback signals are either ecological physically-based synthetic walking signals or simple sinusoidal beeps. In the experiment, the different auditory and haptic feedback and interaction modes are studied with respect to their effect on the walking tempo. The results show that participants synchronise equally well with the tempo with either audio or haptic cues, but indicate the audio–haptic conditions as the easiest to synchronise with. Moreover, results indicate that multimodal audio–haptic feedback provide the most natural feeling. These results have implications on the design of interactive entertainment or therapeutical applications.     

A sequenced multimodal learning approach to support students' development of conceptual learning

Virtual learning environments can now be enriched not only with visual and auditory information, but also with tactile and kinesthetic feedback. However, the way to successfully integrate haptic feedback on a multimodal learning environment is still unclear. This study aims to provide guidelines on how visuohaptic simulations can be implemented effectively, thus the research question is: Under what conditions do visual and tactile information support students' development of conceptual learning of force‐related concepts? Participants comprised 170 undergraduate students of a Midwestern University enrolled in a physics for elementary education class. Four experiments were conducted using four different configurations of multimodal learning environments: Visual feedback only, haptic force feedback only, visual and haptic force feedback at the same time, and sequenced modality of haptic feedback first and visual feedback second. Our results suggest that haptic force feedback has the potential to enrich learning when compared with visual only environments. Also, haptic and visual modalities interact better when sequenced one after another rather than presented simultaneously. Finally, exposure to virtual learning environments enhanced by haptic forced feedback was a positive experience, but the ease of use and ease of interpretation was not so evident.     

Assisting Visually Impaired People to Acquire Targets on a Large Wall-Mounted Display

Large displays have become ubiquitous in our everyday lives, but these displays are designed for sighted people.This paper addresses the need for visually impaired people to access targets on large wall-mounted displays. We developed an assistive interface which exploits mid-air gesture input and haptic feedback, and examined its potential for pointing and steering tasks in human computer interaction（HCI）. In two experiments, blind and blindfolded users performed target acquisition tasks using mid-air gestures and two different kinds of feedback（i.e., haptic feedback and audio feedback）. Our results show that participants perform faster in Fitts＇ law pointing tasks using the haptic feedback interface rather than the audio feedback interface. Furthermore, a regression analysis between movement time（MT） and the index of difficulty（ID）demonstrates that the Fitts＇ law model and the steering law model are both effective for the evaluation of assistive interfaces for the blind. Our work and findings will serve as an initial step to assist visually impaired people to easily access required information on large public displays using haptic interfaces.     

Haptic Seat Interfaces for Driver Information and Warning Systems

The implementation of haptic interfaces in vehicles has important safety and flexibility implications for lessening visual and auditory overload during driving. The present study aims to design and evaluate haptic interfaces with vehicle seats. Three experiments were conducted by testing a haptic seat in a simulator with a total of 20 participants. The first experiment measured reaction time, subjective satisfaction, and subject workloads of the haptic, visual, and auditory displays for the four signals primarily used by vehicle navigation systems. The second experiment measured reaction time, subjective satisfaction, and subjective workloads of the haptic, auditory, and multimodal (haptic + auditory) displays for the ringing signal used by in-vehicle Bluetooth hands-free systems. The third experiment measured drivers' subjective awareness, urgency, usefulness, and disturbance levels at various vibration intensities and positions for a haptic warning signal used by a driver drowsiness warning system. The results indicated that haptic seat interfaces performed better than visual and auditory interfaces, but the unfamiliarity of the haptic interface caused a lower subjective satisfaction for some criteria. Generally, participants showed high subjective satisfaction levels and low subjective workloads toward haptic seat interfaces. This study provided guidance for implementing haptic seat interfaces and identified the possible benefits of their use. It is expected that haptic seats implemented in vehicles will improve safety and the interaction between driver and vehicle.     

Multimodal selection techniques for dense and occluded 3D virtual environments

Object selection is a primary interaction technique which must be supported by any interactive three-dimensional virtual reality application. Although numerous techniques exist, few have been designed to support the selection of objects in dense target environments, or the selection of objects which are occluded from the user's viewpoint. There is, thus, a limited understanding on how these important factors will affect selection performance. In this paper, we present a set of design guidelines and strategies to aid the development of selection techniques which can compensate for environment density and target visibility. Based on these guidelines, we present new forms of the ray casting and bubble cursor selection techniques, which are augmented with visual, audio, and haptic feedback, to support selection within dense and occluded 3D target environments. We perform a series of experiments to evaluate these new techniques, varying both the environment density and target visibility. The results provide an initial understanding of how these factors affect selection performance. Furthermore, the results showed that our new techniques adequately allowed users to select targets which were not visible from their initial viewpoint. The audio and haptic feedback did not provide significant improvements, and our analysis indicated that our introduced visual feedback played the most critical role in aiding the selection task.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of 'drag-and-drops' while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of ‘drag-and-drops’ while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

A novel multimodal interface for improving visually impaired people’s web accessibility

This paper introduces a novel interface designed to help blind and visually impaired people to explore and navigate on the Web. In contrast to traditionally used assistive tools, such as screen readers and magnifiers, the new interface employs a combination of both audio and haptic features to provide spatial and navigational information to users. The haptic features are presented via a low-cost force feedback mouse allowing blind people to interact with the Web, in a similar fashion to their sighted counterparts. The audio provides navigational and textual information through the use of non-speech sounds and synthesised speech. Interacting with the multimodal interface offers a novel experience to target users, especially to those with total blindness. A series of experiments have been conducted to ascertain the usability of the interface and compare its performance to that of a traditional screen reader. Results have shown the advantages that the new multimodal interface offers blind and visually impaired people. This includes the enhanced perception of the spatial layout of Web pages, and navigation towards elements on a page. Certain issues regarding the design of the haptic and audio features raised in the evaluation are discussed and presented in terms of recommendations for future work.     

Helping older pedestrians navigate in the city: comparisons of visual,auditory and haptic guidance instructions in a virtual environment

Preserving older pedestrians’ navigation skills in urban environments is a challenge for maintaining their quality of life. However, existing aids do not take into account older people’s perceptual and cognitive declines nor their user experience, and they call upon sensory modalities that are already used during walking. The present study was aimed at comparing different guidance instructions using visual, auditory, and haptic feedback in order to identify the most efficient and best accepted one(s). Sixteen middle-age (non-retired) adults, 21 younger-old (young-retired) adults, and 21 older-old (old-retired) adults performed a navigation task in a virtual environment. The task was performed with visual feedback (directional arrows superimposed on the visual scenes), auditory feedback (sounds in the left/right ear), haptic feedback (vibrotactile information delivered by a wristband), combinations of different types of sensory feedback, or a paper map. The results showed that older people benefited from the sensory guidance instructions, as compared to the paper map. Visual and auditory feedbacks were associated with better performance and user experience than haptic feedback or the paper map, and the benefits were the greatest among the older-old participants, even though the paper-map familiarity was appreciated. Several recommendations for designing pedestrian navigation aids are proposed.     

Facilitating peripheral interaction: design and evaluation of peripheral interaction for a gesture-based lighting control with multimodal feedback

Most interactions with today’s interfaces require a person’s full and focused attention. To alleviate the potential clutter of focal information, we investigated how interactions could be designed to take place in the background or periphery of attention. This paper explores whether gestural, multimodal interaction styles of an interactive light system allow for this. A study compared the performance of interactions with the light system in two conditions: the central condition in which participants interacted only with the light system, and the peripheral condition in which they interacted with the system while performing a high-attentional task simultaneously. Our study furthermore compared different feedback styles (visual, auditory, haptic, and a combination). Results indicated that especially for the combination feedback style, the interaction could take place without participants’ full visual attention, and performance did not significantly decrease in the peripheral condition. This seems to indicate that these interactions at least partly took place in their periphery of attention and that the multimodal feedback style aided this process.     

DiGVis: A system for comprehension and creation of directed graphs for the visually challenged

One of the major barriers for the social inclusion of blind persons is the limited access to graphics-based learning resources which are highly vision oriented. This paper presents a cost-effective tool which facilitates comprehension and creation of virtual directed graphs, such as flowcharts, using alternate modalities of audio and touch. It provides a physically accessible virtual spatial workspace and multimodal interface to non-visually represent directed graphs in interactive manner. The concept of spatial query is used to aid exploration and mental visualization through audio and tactile feedback. A unique aspect of the tool, named DiGVis, offers compatible representations of directed graphs for the sighted and non-sighted persons. A study with 28 visually challenged subjects indicates that the tool facilitates comprehension of layout and directional connectivity of elements in a virtual diagram. Further, in a pilot study, blind persons could independently comprehend a virtual flowchart layout and its logical steps. They were also able to create the flowchart data without sighted assistance using DiGVis. A comparison with sighted subjects using DiGVis for similar task demonstrates the effectiveness of the technique for inclusive education.     

Workspace analysis for haptic feedback manipulator in virtual cockpit system

To obtain natural space experience of haptic interaction for users in virtual cockpit systems (VCS), a haptic feedback system and a workspace analysis framework for haptic feedback manipulator (HFM) are presented in this paper. Firstly, improving the classical three-dimensional workspace obtained by the Monte Carlo method, a novel workspace representation method, oriented workspace, is presented, which can indicate both the position and the orientation of the end-effector. Then, aimed at the characters of HFMs, the oriented workspace is divided into the effective workspace and the prohibited area by extracting the control panel area. At last, the effective workspace volume and the control panel area are calculated by the double-directed extremum method, with the accuracy improved by repeatedly adding and extracting boundary points. By simulation, the area in which interactions between the manipulator and users hand performed is determined and accordingly the effective workspace along with its boundary and volume are obtained in a relative high precision, which lay a basis for haptic interaction in VCS.     

The Influence of workspace awareness on group intellective decision effectiveness

Workspace awareness is an understanding of members’ interactions within a shared workspace, and has been a fundamental concern to researchers examining how groups using group support systems can be more effective as they design and develop physical artifacts. We provide theoretical and empirical evidence that workspace awareness can be a significant factor influencing the effectiveness of groups employing group support systems for the development of intangible artifacts, such as decision quality and consensus. The workspace awareness literature typically does not examine causal relationships within awareness. We develop a theoretical model that divides workspace awareness into three elements (presence, behavior, and insight awareness), which are important for group decision tasks, and provide empirical evidence that: (1) understanding the reasons behind group member behaviors (insight awareness) is key to increasing decision quality and consensus; (2) greater insight awareness can be obtained when an individual is better able to track and characterize others’ behaviors (behavior awareness); and (3) behavior awareness depends on an individual's ability to identify and distinguish among the different individuals within the group (presence awareness). Empirical support is derived from a laboratory experiment.     

Multimodal Interface Technologies for UAV Ground Control Stations

This paper examines different technologies that can be applied in the design and development of a ground control station for Unmanned Aerial Vehicles (UAVs) equipped with multimodal interfaces. Multimodal technologies employ multiple sensory channels/modalities for information transmission as well as for system control. Examples of these technologies could be haptic feedback, head tracking, auditory information (3D audio), voice control, tactile displays, etc. The applicability and benefits of those technologies is analyzed for a task consisting in the acknowledgement of alerts in an UAV ground control station composed by three screens and managed by a single operator. For this purpose, several experiments were conducted with a group of individuals using different combinations of modal conditions (visual, aural and tactile).     

Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

Haptic feedback and students’ learning about levers: Unraveling the effect of simulated touch

While there has been extensive experimental research on haptics, less has been conducted on cross-modal interactions between visual and haptic perception and even less still on cross-modal applications in instructional settings. This study looks at a simulation on the principles of levers using both visual and haptic feedback: one group received visual and haptic feedback while the other just visual feedback. Using the triangulation of learning scores, eye tracking data, and video analysis of interaction with the levers, the efficacy of haptic feedback to improve learning was explored. The results indicate that while the total fixation time on the levers and numeric readout was greater for the visual and haptic group, very similar patterns of visual attention were seen between groups. Perhaps surprisingly, the visual only group scored higher on an embedded assessment. Explanations for these results are synthesized from theories of cross-modal perception and cognitive architecture.     

Visual-haptic feedback interaction in automotive touchscreens

Touchscreen interfaces offer benefits in terms of flexibility and ease of interaction and as such their use has increased rapidly in a range of devices, from mobile phones to in-car technology. However, traditional touchscreens impose an inevitable visual workload demand that has implications for safety, especially in automotive use. Recent developments in touchscreen technology have enabled feedback to be provided via the haptic channel. A study was conducted to investigate the effects of visual and haptic touchscreen feedback on visual workload, task performance and subjective response using a medium-fidelity driving simulator. Thirty-six experienced drivers performed touchscreen ‘search and select’ tasks while engaged in a motorway driving task. The study utilised a 3 × 2 within-subjects design, with three levels of visual feedback: ‘immediate’, ‘delayed’, ‘none’; and two levels of haptic feedback: ‘visual only’, ‘visual + haptic’. Results showed that visual workload was increased when visual feedback was delayed or absent; however, introducing haptic feedback counteracted this effect, with no increases observed in glance time and count. Task completion time was also reduced when haptic feedback was enabled, while driving performance showed no effect due to feedback type. Subjective responses indicated that haptic feedback improved the user experience and reduced perceived task difficulty.     

Foundation for improved interaction by individuals with visual impairments through multimodal feedback

Through an investigation of how the performance of people who have normal visual capabilities is affected by unimodal, bimodal, and trimodal feedback, this research establishes a foundation for presenting effective feedback to enhance the performance of individuals who have visual impairments. Interfaces that employ multiple feedback modalities, such as auditory, haptic, and visual, can enhance user performance for individuals with barriers limiting one or more channels of perception, such as a visual impairment. Results obtained demonstrate the effects of different feedback combinations on mental workload, accuracy, and performance time. Future, similar studies focused on participants with visual impairments will be grounded in this work. Published online: 6 November 2002     

Effects of auditory,haptic and visual feedback on performing gestures by gaze or by hand

Modern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality (auditory, haptic or visual) and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements (up, down, left and right). In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences.     

Haptic-feedback support for cognitive mapping of unknown spaces by people who are blind

Mental mapping of spaces is essential for the development of efficient orientation and mobility skills. Most of the information required for this mental mapping is gathered through the visual channel. People who are blind lack this information, and in consequence, they are required to use compensatory sensorial channels and alternative exploration methods. In this study, people who are blind use a virtual environment (VE) that provides haptic and audio feedback to explore an unknown space. The cognitive mapping of the space based on the VE and the subject's ability to apply this map to accomplish tasks in the real space are examined. Results show clearly that a robust and comprehensive map is constructed, contributing to successful performance in real space tasks.