Verbal interface design: Do verbal directional cues automatically orient visual spatial attention?

The last few years have seen a rapid growth of interest in the use of verbal information displays in many applied interface settings. However, to date, it is unclear what effect the presentation of verbal cues, such as the words ‘left’ or ‘right’, has on the spatial distribution of an interface operator’s attention. In the present study, we addressed this issue by investigating whether centrally-presented spatially-nonpredictive verbal directional cues elicit an automatic shift of visual spatial attention in the direction indicated by the cue. Participants performed a digit discrimination task for targets presented on either the left or right. Prior to target presentation, the directional word cues ‘left’ or ‘right’ were presented auditorily or visually from the centre of the display at cue-target stimulus onset asynchronies (SOAs) of 200, 400, or 600 ms. Visual discrimination performance was assessed both under conditions where the target digits were unmasked (auditory and visual cuing), and when the targets were masked (auditory cuing only). The results showed that unmasked visual target discrimination performance was facilitated on the cued (relative to the uncued) side at the shortest SOA following visual cuing, but was unaffected by auditorily-presented directional cues. Interestingly, our results also indicated improved visual sensitivity on the auditorily-cued side in the masked target condition. These findings are discussed in relation to previous laboratory-based and applied symbolic cuing studies that have investigated the consequences of the presentation of arrow, gaze direction, and/or head orientation directional cues on the spatial distribution of attention.     

A comparison of tactile, visual, and auditory warnings for rear-end collision prevention in simulated driving

OBJECTIVE: This study examined the effectiveness of rear-end collision warnings presented in different sensory modalities as a function of warning timing in a driving simulator. BACKGROUND: The proliferation of in-vehicle information and entertainment systems threatens driver attention and may increase the risk of rear-end collisions. Collision warning systems have been shown to improve inattentive and/or distracted driver response time (RT) in rear-end collision situations. However, most previous rear-end collision warning research has not directly compared auditory, visual, and tactile warnings. METHOD: Sixteen participants in a fixed-base driving simulator experienced four warning conditions: no warning, visual, auditory, and tactile. The warnings activated when the time-to-collision (TTC) reached a critical threshold of 3.0 or 5.0 s. Driver RT was captured from a warning below critical threshold to brake initiation. RESULTS: Drivers with a tactile warning had the shortest mean RT. Drivers with a tactile warning had significantly shorter RT than drivers without a warning and had a significant advantage over drivers with visual warnings. CONCLUSION: Tactile warnings show promise as effective rear-end collision warnings. APPLICATION: The results of this study can be applied to the future design and evaluation of automotive warnings designed to reduce rear-end collisions.     

Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment

The aim of the present study is to investigate interactions between vision and audition during a target acquisition task performed in a virtual environment. We measured the time taken to locate a visual target (acquisition time) signalled by auditory and/or visual cues in conditions of variable visual load. Visual load was increased by introducing a secondary visual task. The auditory cue was constructed using virtual three-dimensional (3D) sound techniques. The visual cue was constructed in the form of a 3D updating arrow. The results suggested that both auditory and visual cues reduced acquisition time as compared to an uncued condition. Whereas the visual cue elicited faster acquisition time than the auditory cue, the combination of the two cues produced the fastest acquisition time. The introduction of secondary visual task differentially affected acquisition time depending on cue modality. In conditions of high visual load, acquiring a target signalled by the auditory cue led to slower and more error-prone performance than acquiring a target signalled by either the visual cue alone or by both the visual and auditory cues.     

Comparing effects of 2-D and 3-D visual cues during aurally aided target acquisition

The aim of the present study was to investigate interactions between vision and audition during a visual target acquisition task performed in a virtual environment. In two experiments, participants were required to perform an acquisition task guided by auditory and/or visual cues. In both experiments the auditory cues were constructed using virtual 3-D sound techniques based on nonindividualized head-related transfer functions. In Experiment 1 the visual cue was constructed in the form of a continuously updated 2-D arrow. In Experiment 2 the visual cue was a nonstereoscopic, perspective-based 3-D arrow. The results suggested that virtual spatial auditory cues reduced acquisition time but were not as effective as the virtual visual cues. Experiencing the 3-D perspective-based arrow rather than the 2-D arrow produced a faster acquisition time not only in the visually aided conditions but also when the auditory cues were presented in isolation. Suggested novel applications include providing 3-D nonstereoscopic, perspective-based visual information on radar displays, which may lead to a better integration with spatial virtual auditory information.     

The differential effect of vibrotactile and auditory cues on visual spatial attention

Previous research has shown that the presentation of spatially predictive auditory and vibrotactile warning signals can facilitate driver responses to driving events seen through the windscreen or rearview mirror. The present study investigated whether this facilitation reflects the priming of the appropriate response (i.e. braking vs. accelerating) or an attentional cuing effect (i.e. a perceptual benefit that facilitates subsequent behavioural responding). In the experiments reported here, participants had to discriminate the colour of a number plate (red vs. blue) following the presentation of either spatially predictive vibrotactile (experiment 1) or auditory (experiment 2) warning signals that indicated the likely location (front or back) of the visual target, while simultaneously performing a highly attention-demanding rapid serial visual presentation task. Numberplate discrimination performance was facilitated following the presentation of valid auditory cues, but not following the presentation of equally informative vibrotactile cues. The use of an orthogonal spatial cuing design enabled with us to rule out of a potential response priming account of these data. The results suggest that whilst directional congruency between a warning signal and a target event may be sufficient to facilitate performance due to the priming of the appropriate response, attentional facilitation effects may also require the co-location of the cue and target within the same functional region of space.     

The differential effect of vibrotactile and auditory cues on visual spatial attention

Previous research has shown that the presentation of spatially predictive auditory and vibrotactile warning signals can facilitate driver responses to driving events seen through the windscreen or rearview mirror. The present study investigated whether this facilitation reflects the priming of the appropriate response (i.e. braking vs. accelerating) or an attentional cuing effect (i.e. a perceptual benefit that facilitates subsequent behavioural responding). In the experiments reported here, participants had to discriminate the colour of a number plate (red vs. blue) following the presentation of either spatially predictive vibrotactile (experiment 1) or auditory (experiment 2) warning signals that indicated the likely location (front or back) of the visual target, while simultaneously performing a highly attention-demanding rapid serial visual presentation task. Numberplate discrimination performance was facilitated following the presentation of valid auditory cues, but not following the presentation of equally informative vibrotactile cues. The use of an orthogonal spatial cuing design enabled with us to rule out of a potential response priming account of these data. The results suggest that whilst directional congruency between a warning signal and a target event may be sufficient to facilitate performance due to the priming of the appropriate response, attentional facilitation effects may also require the co-location of the cue and target within the same functional region of space.     

Using visual and auditory cues to locate out-of-view objects in head-mounted augmented reality

When looking for an object in a complex visual scene, Augmented Reality (AR) can assist search with visual cues persistently pointing in the target’s direction. The effectiveness of these visual cues can be reduced if they are placed at a different visual depth plane to the target they are indicating. To overcome this visual-depth problem, we test the effectiveness of adding simultaneous spatialized auditory cues that are fixed at the target’s location. In an experiment we manipulated which cue(s) were available (visual-only vs. visual + auditory), and which disparity plane relative to the target the visual cue was displayed on. Results show that participants were slower at finding targets when the visual cue was placed on a different disparity plane to the target. However, this slowdown in search performance could be substantially reduced with auditory cueing. These results demonstrate the importance of AR cross-modal cueing under conditions of visual uncertainty and show that designers should consider augmenting visual cues with auditory ones.     

Toward developing an approach for alerting drivers to the direction of a crash threat

OBJECTIVE: This study explored the potential for auditory and haptic spatial cuing approaches to alert drivers to the direction of a crash threat. BACKGROUND: For an automobile equipped with multiple crash avoidance systems, effective cuing of the crash threat direction may help the driver avoid the crash. Because the driver may not be looking in the direction of a visual crash alert, nonvisual crash alerts were explored as an additional means of directing attention to a potential crash situation. METHODS: In this in-traffic study, 32 drivers were asked to verbally report alert direction in the absence of any crash threats. Driver localization accuracy and response time were examined as a function of eight alert locations surrounding the vehicle and four directional alert approaches (auditory, haptic, haptic and auditory, and haptic and nondirectional auditory). The auditory directional alert approach used four speakers and broadband alert sounds, and the haptic directional alert approach used vibrations generated at various locations on the bottom of the driver's seat. RESULTS: Overall, relative to the auditory alert approach, the three approaches that included the haptic seat alert component reduced correct localization response times by 257 ms and increased percentage correct localization from 32% to 84%. CONCLUSION: These results suggest that seat vibration alerts are a promising candidate for alerting drivers to the direction of a crash threat. APPLICATION: These findings should facilitate developing a multimodality integrated crash alert approach for vehicles equipped with multiple crash avoidance systems.     

Effects of displacement magnitude and direction of auditory cues on auditory spatial facilitation of visual search

OBJECTIVE: The objective of this study is to examine the effects of cue error on auditory spatial facilitation (ASF) of visual search. BACKGROUND: ASF is the reduction in time needed to locate and identify a visual target when an auditory cue is presented at the location of the target. Although ASF has been shown to occur when the auditory cue coincides with the target location, it is important to determine whether facilitatory effects are also evident when the cue is displaced. METHOD: Participants performed a visual search task in the presence of an auditory cue that was presented at the center of the screen (uninformative), at the location of the target (accurate), or displaced up to 12 degrees from the target horizontally or vertically. RESULTS: Generally, displaced auditory cues reduced search times as compared with a condition in which the cue was uninformative. When the displacement was always along a single spatial dimension, the cue was as effective as a coincident cue if it was within the local visual area. However, when the dimension along which the cue was displaced varied randomly, the cue did not necessarily reduce search time and hurt performance when the visual search task was difficult. CONCLUSION: Designers of virtual audio displays should be aware that auditory cue accuracy will be affected by the difficulty of the visual task and the operators' knowledge of cue precision and reliability. APPLICATION: Findings from this study can be applied to the design of multimodal interfaces and augmented or virtual environments.     

Cross-modal links among vision, audition, and touch in complex environments

OBJECTIVES: This study sought to determine whether performance effects of cross-modal spatial links that were observed in earlier laboratory studies scale to more complex environments and need to be considered in multimodal interface design. It also revisits the unresolved issue of cross-modal cuing asymmetries. BACKGROUND: Previous laboratory studies employing simple cues, tasks, and/or targets have demonstrated that the efficiency of processing visual, auditory, and tactile stimuli is affected by the modality, lateralization, and timing of surrounding cues. Very few studies have investigated these cross-modal constraints in the context of more complex environments to determine whether they scale and how complexity affects the nature of cross-modal cuing asymmetries. METHOD: Amicroworld simulation of battlefield operations with a complex task set and meaningful visual, auditory, and tactile stimuli was used to investigate cuing effects for all cross-modal pairings. RESULTS: Significant asymmetric performance effects of cross-modal spatial links were observed. Auditory cues shortened response latencies for collocated visual targets but visual cues did not do the same for collocated auditory targets. Responses to contralateral (rather than ipsilateral) targets were faster for tactually cued auditory targets and each visual-tactile cue-target combination, suggesting an inhibition-of-return effect. CONCLUSIONS: The spatial relationships between multimodal cues and targets significantly affect target response times in complex environments. The performance effects of cross-modal links and the observed cross-modal cuing asymmetries need to be examined in more detail and considered in future interface design. APPLICATION: The findings from this study have implications for the design of multimodal and adaptive interfaces and for supporting attention management in complex, data-rich domains.     

Effects of in-vehicle warning information displays with or without spatial compatibility on driving behaviors and response performance

A driving simulator study was conducted to evaluate the effects of five in-vehicle warning information displays upon drivers' emergent response and decision performance. These displays include visual display, auditory displays with and without spatial compatibility, hybrid displays in both visual and auditory format with and without spatial compatibility. Thirty volunteer drivers were recruited to perform various tasks that involved driving, stimulus-response, divided attention and stress rating. Results show that for displays of single-modality, drivers benefited more when coping with visual display of warning information than auditory display with or without spatial compatibility. However, auditory display with spatial compatibility significantly improved drivers' performance in reacting to the divided attention task and making accurate S-R task decision. Drivers' best performance results were obtained for hybrid display with spatial compatibility. Hybrid displays enabled drivers to respond the fastest and achieve the best accuracy in both S-R and divided attention tasks.     

Haptic gas pedal feedback

Active driver support systems either automate a control task or present warnings to drivers when their safety is seriously degraded. In a novel approach, utilising neither automation nor discrete warnings, a haptic gas pedal (accelerator) interface was developed that continuously presents car-following support information, keeping the driver in the loop. This interface was tested in a fixed-base driving simulator. Twenty-one drivers between the ages of 24 and 30 years participated in a driving experiment to investigate the effects of haptic gas pedal feedback on car-following behaviour. Results of the experiment indicate that when haptic feedback was presented to the drivers, some improvement in car-following performance was achieved, while control activity decreased. Further research is needed to investigate the effectiveness of the system in more varied driving conditions. Haptics is an under-used modality in the application of human support interfaces, which usually draw on vision or hearing. This study demonstrates how haptics can be used to create an effective driver support interface.     

Haptic gas pedal feedback

Active driver support systems either automate a control task or present warnings to drivers when their safety is seriously degraded. In a novel approach, utilising neither automation nor discrete warnings, a haptic gas pedal (accelerator) interface was developed that continuously presents car-following support information, keeping the driver in the loop. This interface was tested in a fixed-base driving simulator. Twenty-one drivers between the ages of 24 and 30 years participated in a driving experiment to investigate the effects of haptic gas pedal feedback on car-following behaviour. Results of the experiment indicate that when haptic feedback was presented to the drivers, some improvement in car-following performance was achieved, while control activity decreased. Further research is needed to investigate the effectiveness of the system in more varied driving conditions. Haptics is an under-used modality in the application of human support interfaces, which usually draw on vision or hearing. This study demonstrates how haptics can be used to create an effective driver support interface.     

Interruption management: A comparison of auditory and tactile cues for both alerting and orienting

Tactile and auditory cues have been suggested as methods of interruption management for busy visual environments. The current experiment examined attentional mechanisms by which cues might improve performance. The findings indicate that when interruptive tasks are presented in a spatially diverse task environment, the orienting function of tactile cues is a critical component, which directs attention to the location of the interruption, resulting in superior interruptive task performance. Non-directional tactile cues did not degrade primary task performance, but also did not improve performance on the secondary task. Similar results were found for auditory cues. The results support Posner and Peterson's [1990. The attention system of the human brain. Annual Review of Neuroscience 13, 25–42] theory of independent functional networks of attention, and have practical applications for systems design in work environments that consist of multiple, visual tasks and time-sensitive information.     

Auditory feedback in haptic collaborative interfaces

The combined effect of haptic and auditory feedback in shared interfaces on the cooperation between visually impaired and sighted persons is under-investigated. A central challenge for cooperating group members lies in obtaining a common understanding of the elements of the workspace and maintaining awareness of the other members' actions, as well as one's own, during the group work process. The aim of the experimental study presented here was to investigate if adding audio cues in a haptic and visual interface makes collaboration between a sighted and a blindfolded person more efficient. Results showed that task performance was significantly faster in the audio, haptic and visual feedback condition compared to the haptic and visual feedback condition. One special focus was also to study how participants utilize the auditory and haptic force feedback in order to obtain a common understanding of the workspace and to maintain an awareness of the group members' actions. Results from a qualitative analysis showed that the auditory and haptic feedback was used in a number of important ways to support the group members' action awareness and in the participants' grounding process.     

Looks are (almost) everything: where drivers look to get information

OBJECTIVE: To describe the impact of Rockwell's early eye movements research. BACKGROUND: The advent of a new technology enabling measurements of eye movements in natural environments launched the seminal research of a Human Factors pioneer, Tom Rockwell, into how drivers process visual information. METHOD: In two seminal Human Factors articles -"Mapping Eye-Movement Pattern to the Visual Scene in Driving: An Exploratory Study" (Mourant & Rockwell, 1970) and "Strategies of Visual Search by Novice and Experienced Drivers" (Mourant & Rockwell, 1972)- Rockwell and his student, Ron Mourant, examined drivers' eye movements in naturalistic driving environments. RESULTS: The analyses of the visual fixations revealed systematic relationships between the sources of information the drivers needed to drive safely and the spatial distributions of their visual fixations. In addition, they showed that as drivers gain skill and experience, their pattern of fixations changes in a systematic manner. CONCLUSIONS: The research demonstrated that fixations and saccadic eye movements provide important insights into drivers' visual search behavior, information needs, and information acquisition processes. APPLICATION: This research has been a cornerstone for a myriad of driving-related studies, by Rockwell and other researchers. Building on Rockwell's pioneering work, these studies used eye-tracking systems to describe cognitive aspects of skill acquisition, and the effects of fatigue and other impairments on the process of attention and information gathering. A novel and potentially revolutionary application of this research is to use eye movement recordings for vehicle control and activation of in-vehicle safety systems.     

Accessible haptic user interface design approach for users with visual impairments

With the number of people with visual impairments (e.g., low vision and blind) continuing to increase, vision loss has become one of the most challenging disabilities. Today, haptic technology, using an alternative sense to vision, is deemed an important component for effectively accessing information systems. The most appropriately designed assistive technology is critical for those with visual impairments to adopt assistive technology and to access information, which will facilitate their tasks in personal and professional life. However, most of the existing design approaches are inapplicable and inappropriate to such design contexts as users with visual impairments interacting with non-graphical user interfaces (i.e., haptic technology). To resolve such design challenges, the present study modified a participatory design approach (i.e., PICTIVE, Plastic Interface for Collaborative Technology Initiatives Video Exploration) to be applicable to haptic technologies, by considering the brain plasticity theory. The sense of touch is integrated into the design activity of PICTIVE. Participants with visual impairments were able to effectively engage in designing non-visual interfaces (e.g., haptic interfaces) through non-visual communication methods (e.g., touch modality).     

Mild body cooling impairs attention via distraction from skin cooling

Many contemporary workers are routinely exposed to mild cold stress, which may compromise mental function and lead to accidents. A study investigated the effect of mild body cooling of 1.0 degree C rectal temperature (Tre) on vigilance (i.e. sustained attention) and the orienting of spatial attention (i.e. spatially selective processing of visual information). Vigilance and spatial attention tests were administered to 14 healthy males and six females at four stages (pre-immersion, deltaTre = 0, -0.5 and - 1.0 degree C ) of a gradual, head-out immersion cooling session (18-25 deltaC water), and in four time-matched stages of a contrast session, in which participants sat in an empty tub and no cooling took place. In the spatial attention test, target discrimination times were similar for all stages of the contrast session, but increased significantly in the cooling phase upon immersion (deltaTre = 0 degrees C), with no further increases at deltaTre = -0.5 and - 1.0 degree C. Despite global response slowing, cooling did not affect the normal pattern of spatial orienting. In the vigilance test, the variability of detection time was adversely affected in the cooling but not the contrast trials: variability increased at immersion but did not increase further with additional cooling. These findings suggest that attentional impairments are more closely linked to the distracting effects of cold skin temperature than decreases in body core temperature.     

Foundation for improved interaction by individuals with visual impairments through multimodal feedback

Through an investigation of how the performance of people who have normal visual capabilities is affected by unimodal, bimodal, and trimodal feedback, this research establishes a foundation for presenting effective feedback to enhance the performance of individuals who have visual impairments. Interfaces that employ multiple feedback modalities, such as auditory, haptic, and visual, can enhance user performance for individuals with barriers limiting one or more channels of perception, such as a visual impairment. Results obtained demonstrate the effects of different feedback combinations on mental workload, accuracy, and performance time. Future, similar studies focused on participants with visual impairments will be grounded in this work. Published online: 6 November 2002     

Head up versus head down: the costs of imprecision, unreliability, and visual clutter on cue effectiveness for display signaling

We conducted 2 experiments to investigate the clutter-scan trade-off between the cost of increasing clutter by overlaying complex information onto the forward field of view using a helmet-mounted display (HMD) and the cost of scanning when presenting this information on a handheld display. In the first experiment, this trade-off was examined in terms of the spatial accuracy of target cuing data in a relatively sparse display; in the second, the spatial accuracy of the cue was varied more radically in an information-rich display. Participants were asked to detect and identify targets hidden in the far domain while performing a monitoring task in the near domain using either an HMD or a handheld display. The results revealed that on a sparse display, the reduced scanning from the HMD presentation of cuing out-weighed the costs of clutter for cued targets, regardless of cue precision, but no benefit was found for uncued targets. When the HMD displayed task-irrelevant information, however, target detection was hindered by the extraneous clutter in the forward field of view relative to the handheld display condition, and this cost of clutter increased as the amount of data that needed to be inspected increased. Potential applications of this research include the development of design considerations for head-up displays for aviation and military applications.