Improving target detection in visual search through the augmenting multi-sensory cues

The present experiment tested 60 individuals on a multiple screen, visual target detection task. Using a within-participant design, individuals received no-cue augmentation, an augmenting tactile cue alone, an augmenting auditory cue alone or both of the latter augmentations in combination. Results showed significant and substantive improvements in performance such that successful search speed was facilitated by more than 43%, errors of omission were reduced by 86% and errors of commission were reduced by more than 77% in the combinatorial cueing condition compared with the non-cued control. These outcomes were not a trade of performance efficiency for associated mental effort because recorded levels of cognitive workload were also reduced by more than 30% in the multi-cued circumstance compared with the control condition. When the tactile modality was incorporated it led to the highest gain in performance speed, when the auditory modality was incorporated, it led to the best levels of performance accuracy. The combined condition rendered the best of each from of performance increment. Reasons for this outcome pattern are discussed alongside their manifest practical benefits.

Practitioner Summary: This experiment tested 60 individuals on a multiple screen, visual target detection task. Individuals received no-cue augmentation, tactile cue alone, an augmenting auditory cue alone or both of the latter augmentations in combination. Results showed significant and substantive improvements in the combinatorial cueing condition compared with the non-cued control.     

Interruption management: A comparison of auditory and tactile cues for both alerting and orienting

Tactile and auditory cues have been suggested as methods of interruption management for busy visual environments. The current experiment examined attentional mechanisms by which cues might improve performance. The findings indicate that when interruptive tasks are presented in a spatially diverse task environment, the orienting function of tactile cues is a critical component, which directs attention to the location of the interruption, resulting in superior interruptive task performance. Non-directional tactile cues did not degrade primary task performance, but also did not improve performance on the secondary task. Similar results were found for auditory cues. The results support Posner and Peterson's [1990. The attention system of the human brain. Annual Review of Neuroscience 13, 25–42] theory of independent functional networks of attention, and have practical applications for systems design in work environments that consist of multiple, visual tasks and time-sensitive information.     

Effect of physical workload and modality of information presentation on pattern recognition and navigation task performance by high-fit young males

Abstract

Many occupations require both physical exertion and cognitive task performance. Knowledge of any interaction between physical demands and modalities of cognitive task information presentation can provide a basis for optimising performance. This study examined the effect of physical exertion and modality of information presentation on pattern recognition and navigation-related information processing. Results indicated males of equivalent high fitness, between the ages of 18 and 34, rely more on visual cues vs auditory or haptic for pattern recognition when exertion level is high. We found that navigation response time was shorter under low and medium exertion levels as compared to high intensity. Navigation accuracy was lower under high level exertion compared to medium and low levels. In general, findings indicated that use of the haptic modality for cognitive task cueing decreased accuracy in pattern recognition responses.

Practitioner Summary: An examination was conducted on the effect of physical exertion and information presentation modality in pattern recognition and navigation. In occupations requiring information presentation to workers, who are simultaneously performing a physical task, the visual modality appears most effective under high level exertion while haptic cueing degrades performance.     

Using visual and auditory cues to locate out-of-view objects in head-mounted augmented reality

When looking for an object in a complex visual scene, Augmented Reality (AR) can assist search with visual cues persistently pointing in the target’s direction. The effectiveness of these visual cues can be reduced if they are placed at a different visual depth plane to the target they are indicating. To overcome this visual-depth problem, we test the effectiveness of adding simultaneous spatialized auditory cues that are fixed at the target’s location. In an experiment we manipulated which cue(s) were available (visual-only vs. visual + auditory), and which disparity plane relative to the target the visual cue was displayed on. Results show that participants were slower at finding targets when the visual cue was placed on a different disparity plane to the target. However, this slowdown in search performance could be substantially reduced with auditory cueing. These results demonstrate the importance of AR cross-modal cueing under conditions of visual uncertainty and show that designers should consider augmenting visual cues with auditory ones.     

Interruption management: the use of attention-directing tactile cues

Previous research has suggested that providing informative cues about interrupting stimuli aids management of multiple tasks. However, auditory and visual cues can be ineffective in certain situations. The objective of the present study was to explore whether attention-directing tactile cues aid or interfere with performance. A two-group posttest-only randomized experiment was conducted. Sixty-one participants completed a 30-min performance session consisting of aircraft-monitoring and gauge-reading computer tasks. Tactile signals were administered to a treatment group to indicate the arrival and location of interrupting tasks. Control participants had to remember to visually check for the interrupting tasks. Participants in the treatment group responded to more interrupting tasks and responded faster than did control participants. Groups did not differ on error rates for the interrupting tasks, performance of the primary task, or subjective workload perceptions. In the context of the tasks used in the present research, tactile cues allowed participants to effectively direct attention where needed without disrupting ongoing information processing. Tactile cues should be explored in a variety of other visual, interrupt-laden environments. Potential applications exist for aviation, user-interface design, vigilance tasks, and team environments.     

Effects of displacement magnitude and direction of auditory cues on auditory spatial facilitation of visual search

OBJECTIVE: The objective of this study is to examine the effects of cue error on auditory spatial facilitation (ASF) of visual search. BACKGROUND: ASF is the reduction in time needed to locate and identify a visual target when an auditory cue is presented at the location of the target. Although ASF has been shown to occur when the auditory cue coincides with the target location, it is important to determine whether facilitatory effects are also evident when the cue is displaced. METHOD: Participants performed a visual search task in the presence of an auditory cue that was presented at the center of the screen (uninformative), at the location of the target (accurate), or displaced up to 12 degrees from the target horizontally or vertically. RESULTS: Generally, displaced auditory cues reduced search times as compared with a condition in which the cue was uninformative. When the displacement was always along a single spatial dimension, the cue was as effective as a coincident cue if it was within the local visual area. However, when the dimension along which the cue was displaced varied randomly, the cue did not necessarily reduce search time and hurt performance when the visual search task was difficult. CONCLUSION: Designers of virtual audio displays should be aware that auditory cue accuracy will be affected by the difficulty of the visual task and the operators' knowledge of cue precision and reliability. APPLICATION: Findings from this study can be applied to the design of multimodal interfaces and augmented or virtual environments.     

Multisensory teamwork: using a tactile or an auditory display to exchange gaze information improves performance in joint visual search

In joint tasks, adjusting to the actions of others is critical for success. For joint visual search tasks, research has shown that when search partners visually receive information about each other’s gaze, they use this information to adjust to each other’s actions, resulting in faster search performance. The present study used a visual, a tactile and an auditory display, respectively, to provide search partners with information about each other’s gaze. Results showed that search partners performed faster when the gaze information was received via a tactile or auditory display in comparison to receiving it via a visual display or receiving no gaze information. Findings demonstrate the effectiveness of tactile and auditory displays for receiving task-relevant information in joint tasks and are applicable to circumstances in which little or no visual information is available or the visual modality is already taxed with a demanding task such as air-traffic control. Practitioner Summary: The present study demonstrates that tactile and auditory displays are effective for receiving information about actions of others in joint tasks. Findings are either applicable to circumstances in which little or no visual information is available or when the visual modality is already taxed with a demanding task.     

Effects of Auditory Cues for Linear and Differential Magnification Methods on Visual Inspection Performance

The unfavorable consequence of biased attention to peripheral objects was reported with the use of a newly proposed differential magnification method by the authors. Given the favorable influence of auditory cueing on attention orientation, this study investigated the effectiveness of auditory cues in directing participants’ attention from the peripheral to foveal areas of the differential magnifying lens for improving inspection performance. Thirty‐nine Chinese undergraduates performed a simulated inspection task using a particular magnification method (between‐participant factor). Three within‐participant factors of auditory cue interval, target difficulty, and magnification power were tested. Objective performance measures of speed and accuracy and subjective evaluation using NASA task load index were collected and analyzed. The results showed that positive effects of auditory cues were found for the differential magnification methods but not for the traditional linear magnification method. Significant effects of target difficulty and magnification power were also found. It was concluded that the aid of auditory cues, which did not share the same attentional resource with the visual inspection task, proved to be an appropriate reminder for the differential magnification methods to improve participants’ visual inspection performance. Attention should also be paid to selecting an appropriate auditory cue interval for search tasks of different target difficulties. The findings of this study have implications for the design of video magnifiers in the context of multimodal human–machine interfaces.     

Tactile interruption management: tactile cues as task-switching reminders

Tactile cuing has been suggested as a method of interruption management for busy visual environments. This study examined the effectiveness of tactile cues as an interruption management strategy in a multi-tasking environment. Sixty-four participants completed a continuous aircraft monitoring task with periodic interruptions of a discrete gauge memory task. Participants were randomly assigned to two groups; one group had to remember to monitor for interruptions while the other group received tactile cues indicating an interruption’s arrival and location. As expected, the cued participants evidenced superior performance on both tasks. The results are consistent with the notion that tactile cues transform the resource-intensive, time-based task of remembering to check for interruptions into a simpler, event-based task, where cues assume a portion of the workload, permitting the application of valuable resources to other task demands. This study is discussed in the context of multiple resource theory and has practical implications for systems design in environments consisting of multiple, visual tasks and time-sensitive information.     

Human interaction with robotic systems: performance and workload evaluations

Abstract

We first tested the effect of differing tactile informational forms (i.e. directional cues vs. static cues vs. dynamic cues) on objective performance and perceived workload in a collaborative human–robot task. A second experiment evaluated the influence of task load and informational message type (i.e. single words vs. grouped phrases) on that same collaborative task. In both experiments, the relationship of personal characteristics (attentional control and spatial ability) to performance and workload was also measured. In addition to objective performance and self-report of cognitive load, we evaluated different physiological responses in each experiment. Results showed a performance–workload association for directional cues, message type and task load. EEG measures however, proved generally insensitive to such task load manipulations. Where significant EEG effects were observed, right hemisphere amplitude differences predominated, although unexpectedly these latter relationships were negative. Although EEG measures were partially associated with performance, they appear to possess limited utility as measures of workload in association with tactile displays.

Practitioner Summary: As practitioners look to take advantage of innovative tactile displays in complex operational realms like human–robotic interaction, associated performance effects are mediated by cognitive workload. Despite some patterns of association, reliable reflections of operator state can be difficult to discern and employ as the number, complexity and sophistication of these respective measures themselves increase.     

Can a short nap and bright light function as implicit learning and visual search enhancers?

The present study examined effects of a short nap (20 min) and/or bright light (2000 lux) on visual search and implicit learning in a contextual cueing task. Fifteen participants performed a contextual cueing task twice a day (1200–1330 h and 1430–1600 h) and scored subjective sleepiness before and after a short afternoon nap or a break period. Participants served a total of four experimental conditions (control, short nap, bright light and short nap with bright light). During the second task, bright light treatment (BLT) was applied in the two of the four conditions. Participants performed both tasks in a dimly lit environment except during the light treatment. Results showed that a short nap reduced subjective sleepiness and improved visual search time, but it did not affect implicit learning. Bright light reduced subjective sleepiness. A short nap in the afternoon could be a countermeasure against sleepiness and an enhancer for visual search.

Practitioner Summary: The study examined effects of a short afternoon nap (20 min) and/or bright light (2000 lux) on visual search and implicit learning. A short nap is a powerful countermeasure against sleepiness compared to bright light exposure in the afternoon.     

Semantic congruency of auditory warnings

Abstract

The aim of this study was to explore operator experience and performance for semantically congruent and incongruent auditory icons and abstract alarm sounds. It was expected that performance advantages for congruent sounds would be present initially but would reduce over time for both alarm types. Twenty-four participants (12M/12F) were placed into auditory icon or abstract alarm groupings. For each group both congruent and incongruent alarms were used to represent different driving task scenarios. Once sounded, participants were required to respond to each alarm by selecting a corresponding driving scenario. User performance for all sound types improved over time, however even with experience a decrement in speed of response remained for the incongruent iconic sounds and in accuracy of performance for the abstract warning sounds when compared to the congruent auditory icons. Semantic congruency was found to be of more importance for auditory icons than for abstract sounds.

Practitioner Summary: Alarms are used in many operating systems as emergency, alerting, or continuous monitoring signals for instance. This study found that the type and representativeness of an auditory warning will influence operator performance over time. Semantically congruent iconic sounds produced performance advantages over both incongruent iconic sounds and abstract warnings.     

Effects of increasing visual load on aurally and visually guided target acquisition in a virtual environment

The aim of the present study is to investigate interactions between vision and audition during a target acquisition task performed in a virtual environment. We measured the time taken to locate a visual target (acquisition time) signalled by auditory and/or visual cues in conditions of variable visual load. Visual load was increased by introducing a secondary visual task. The auditory cue was constructed using virtual three-dimensional (3D) sound techniques. The visual cue was constructed in the form of a 3D updating arrow. The results suggested that both auditory and visual cues reduced acquisition time as compared to an uncued condition. Whereas the visual cue elicited faster acquisition time than the auditory cue, the combination of the two cues produced the fastest acquisition time. The introduction of secondary visual task differentially affected acquisition time depending on cue modality. In conditions of high visual load, acquiring a target signalled by the auditory cue led to slower and more error-prone performance than acquiring a target signalled by either the visual cue alone or by both the visual and auditory cues.     

Perceptual training for visual search

People are better at visual search than the best fully automated methods. Despite this, visual search remains a difficult perceptual task. The goal of this investigation was to experimentally test the ways in which visual search performance could be improved through two categories of training interventions: perceptual training and conceptual training. To determine the effects of each training on a later performance task, the two types of trainings were manipulated using a between-subjects design (conceptual vs. perceptual × training present vs. training absent). Perceptual training led to speed and accuracy improvements in visual search. Issues with the design and administration of the conceptual training limited conclusions on its effectiveness but provided useful lessons for conceptual training design. The results suggest that when the visual search task involves detecting heterogeneous or otherwise unpredictable stimuli, perceptual training can improve visual search performance. Similarly, careful consideration of the performance task and training design is required to evaluate the effectiveness of conceptual training.

Practitioner Summary: Visual search is a difficult, yet critical, task in industries such as baggage screening and radiology. This study investigated the effectiveness of perceptual training for visual search. The results suggest that when visual search involves detecting heterogeneous or otherwise unpredictable stimuli, perceptual training may improve the speed and accuracy of visual search.     

An empirical investigation into the design of auditory cues to enhance computer program comprehension

Decades of research have led to notable improvements in the representations used to aid human comprehension of computer programs. Much of this research has focused on visual representations, which leaves open the question of how best to design auditory representations of computer programs. While this question has particular relevance for visually impaired programmers, sighted programmers might also benefit from enhanced auditory representations of their programs. In order to investigate this question empirically, first, we introduce artifact encoding, a novel approach to rigorously measuring the comprehensibility of auditory representations of computer programs. Using this approach as a foundation, we present an experimental study that compared the comprehensibility of two alternative auditory program representations: one with lexical scoping cues that convey the nesting level of program statements, and another without such scoping cues. The results of our first experiment validate both artifact encoding and the scoping cues we used. To see whether auditory cues validated through our paradigm can aid program comprehension in a realistic task scenario, we experimentally compared programmers' ability to debug programs using three alternative environments: (1) an auditory execution environment with our empirically derived auditory cues; (2) an auditory execution environment with the current state-of-the-art auditory cues generated by a screen reader running on top of Microsoft Visual Studio; and (3) a visual version of the execution environment. The results of our second experiment showed that our comprehensible auditory cues are significantly better than the state-of-the-art, affording human performance approaching the effectiveness of visual representations within the statistical margin of error. This research contributes a novel methodology and foundational empirical data that can guide the design of effective auditory representations of computer programs.     

Learning from instructional animations: How does prior knowledge mediate the effect of visual cues?

The purpose of this study was to investigate the effects of cueing and prior knowledge on learning and mental effort of students studying an animation with narration. This study employed a 2 (no cueing vs. visual cueing) × 2 (low vs. high prior knowledge) between‐subjects factorial design. The results revealed a significant interaction effect between prior knowledge and cueing on learning. Low prior knowledge learners had higher scores after studying an instructional animation with visual cues, compared to those who studied the same instructional animation without visual cues. Conversely, when cues were not provided, high prior knowledge learners outperformed those high prior knowledge learners who studied with the cued version of an instructional animation. These results indicated that the effects of cueing in an instructional animation change depending on the learners' level of prior knowledge. Specifically, low prior knowledge learners benefited more when visual cues were provided, whereas cues did not facilitate learning for high prior knowledge learners.     

Tracking and discrete dual task performance with different spatial stimulus–response mappings

The effect of spatial compatibility on dual-task performance for various display–control configurations was studied using a tracking task and a discrete four-choice response task. A total of 36 participants took part in this study, and they were asked to perform the primary tracking task while at the same time to respond to an occasional signal. Different levels of compatibility between the stimuli and responses of the discrete response task were found to lead to different degrees of influence on the tracking task. However, degradation of performance was observed for both tasks, which was probably due to resource competition for the visual and spatial resources required for simultaneous task operation and required for bimanual responses. No right–left prevalence effect for the spatial compatibility task was observed in this study, implying that the use of unimanual two-finger responses may not provide the right conditions for a significant effect in the horizontal right–left dimension.

Practitioner Summary: The effect of spatial compatibility in multiple display–control configurations was examined in a dual-task paradigm. The analyses of keen competition for visual and spatial resources in processing the dual tasks under different degrees of stimulus–response compatibility provide useful ergonomics design implications and recommendations for visual interfaces requiring frequent visual scanning.     

Evaluation of automobile navigation systems with multisensory information channels

Although automobile navigation systems have conventionally utilised visual and auditory modalities to deliver information, recent advancement in tactile technology has introduced the possibility of integrating tactile actuators into navigation systems. To empirically test the effect of tactile navigational cues, we conducted simulated driving experiments in which participants (N?=?96) were exposed to four sets of information modalities (visual, visual?+?auditory, visual?+?tactile, and visual?+?auditory?+?tactile). The results indicate that multisensory systems including tactile cues allow participants to respond faster to unexpected road events and impart greater satisfaction with the overall driving experience. The implications and limitations of these findings are discussed.     

A comparison of tactile, visual, and auditory warnings for rear-end collision prevention in simulated driving

OBJECTIVE: This study examined the effectiveness of rear-end collision warnings presented in different sensory modalities as a function of warning timing in a driving simulator. BACKGROUND: The proliferation of in-vehicle information and entertainment systems threatens driver attention and may increase the risk of rear-end collisions. Collision warning systems have been shown to improve inattentive and/or distracted driver response time (RT) in rear-end collision situations. However, most previous rear-end collision warning research has not directly compared auditory, visual, and tactile warnings. METHOD: Sixteen participants in a fixed-base driving simulator experienced four warning conditions: no warning, visual, auditory, and tactile. The warnings activated when the time-to-collision (TTC) reached a critical threshold of 3.0 or 5.0 s. Driver RT was captured from a warning below critical threshold to brake initiation. RESULTS: Drivers with a tactile warning had the shortest mean RT. Drivers with a tactile warning had significantly shorter RT than drivers without a warning and had a significant advantage over drivers with visual warnings. CONCLUSION: Tactile warnings show promise as effective rear-end collision warnings. APPLICATION: The results of this study can be applied to the future design and evaluation of automotive warnings designed to reduce rear-end collisions.     

Good vibrations: tactile feedback in support of attention allocation and human-automation coordination in event-driven domains

Observed breakdowns in human-machine communication can be explained, in part, by the nature of current automation feedback, which relies heavily on focal visual attention. Such feedback is not well suited for capturing attention in case of unexpected changes and events or for supporting the parallel processing of large amounts of data in complex domains. As suggested by multiple-resource theory, one possible solution to this problem is to distribute information across various sensory modalities. A simulator study was conducted to compare the effectiveness of visual, tactile, and redundant visual and tactile cues for indicating unexpected changes in the status of an automated cockpit system. Both tactile conditions resulted in higher detection rates for, and faster response times to, uncommanded mode transitions. Tactile feedback did not interfere with, nor was its effectiveness affected by, the performance of concurrent visual tasks. The observed improvement in task-sharing performance indicates that the introduction of tactile feedback is a promising avenue toward better supporting human-machine communication in event-driven, information-rich domains.