Effectiveness of auditory and tactile crossmodal cues in a dual-task visual and auditory scenario

Abstract

In this study, we examined how spatially informative auditory and tactile cues affected participants’ performance on a visual search task while they simultaneously performed a secondary auditory task. Visual search task performance was assessed via reaction time and accuracy. Tactile and auditory cues provided the approximate location of the visual target within the search display. The inclusion of tactile and auditory cues improved performance in comparison to the no-cue baseline conditions. In comparison to the no-cue conditions, both tactile and auditory cues resulted in faster response times in the visual search only (single task) and visual–auditory (dual-task) conditions. However, the effectiveness of auditory and tactile cueing for visual task accuracy was shown to be dependent on task-type condition. Crossmodal cueing remains a viable strategy for improving task performance without increasing attentional load within a singular sensory modality.

Practitioner Summary: Crossmodal cueing with dual-task performance has not been widely explored, yet has practical applications. We examined the effects of auditory and tactile crossmodal cues on visual search performance, with and without a secondary auditory task. Tactile cues aided visual search accuracy when also engaged in a secondary auditory task, whereas auditory cues did not.     

The role of cue utilisation in reducing the workload in a train control task

Abstract

Skilled performance has been characterised, in part, by the capacity to accurately identify and respond to patterns as cues in the environment. The outcome is a reduction in cognitive load and a greater residual capacity to undertake concurrent tasks. The present study was designed to examine the relationship between cue utilisation and temporal pattern recognition in the context of a simulated, rail control task. Sixty-one university students undertook an assessment of cue utilisation and engaged in a rail control simulation. The appearance and movement of trains followed a consistent but implicit (undisclosed) pattern. Throughout the second half of the rail task, a secondary task was included. The results indicated that participants with relatively higher cue utilisation were more likely to identify the implicit pattern of rail movements, were more accurate and responded more rapidly under increased workload conditions. The results suggest that a propensity to identify patterns as cues may provide an opportunity to reduce cognitive demands, thereby facilitating performance in a novel task. Implications for selection and system design are discussed.

Practitioner Summary: This study was designed to explain differences in the way in which people learn, particularly when tasks involve recurring patterns. Using simulated rail control, the results indicated that participants who display behaviour that is indicative of the utilisation of cues also recognise patterns in the movement of simulated trains. This enables them to manage trains more effectively, even while undertaking other tasks.     

Using cerebral hemovelocity to measure workload during a spatialised auditory vigilance task in novice and experienced observers

This experiment was designed to assess cognitive load using transcranial Doppler sonography during the performance of a 40-min communication vigilance task in which messages were presented in different spatial locations or across a single monaural radio channel. In addition, some observers received 14 hours of practice to determine whether the neurophysiological measure was sensitive to a potential attenuation of workload. Critical messages were detected more frequently in the spatialised audio presentation mode condition, but there were no performance differences between experienced and novice observers. Neurophysiological data show that activation was greater in the novice condition than in the experienced condition, suggesting that novice observers expended greater effort. Furthermore, the neurophysiological measure showed more activation in the monaural radio condition than in the spatialised audio condition. The results support a resource account of vigilance and suggest that cerebral blood flow velocity can be used to diagnose the degree of attentional resource utilisation during vigilance tasks.

Practitioner Summary: Due to high workload experienced during vigilance tasks, displays and methods are sought which enhance performance. This study shows that spatialising auditory communications in a monitoring task enhances performance and attenuates mental workload. Also, experience mitigates excessive workload, and cerebral hemovelocity can be used to diagnose attentional resource utilisation.     

Medium-term effects of a two-desk sit/stand workstation on cognitive performance and workload for healthy people performing sedentary work: a secondary analysis of a randomised controlled trial

Implementing sit/stand workstations in sedentary work environments is a common way to reduce sedentary time, but their medium-term effect on cognitive performance is unclear. To address this circumstance, eighteen office workers participated in a two-arm, randomised controlled cross-over trial (ClinicalTrials.gov Identifier: NCT02825303), either working at a traditional (sit) or an interventional (sit/stand) workplace for 23 weeks. Cognitive performance (working speed, reaction time, concentration performance, accuracy), workload and relevant covariates (salivary cortisol level, heart rate, physical activity, sitting time) were measured pre- and post-intervention under laboratory conditions. MANOVA and RMANOVA results did not show differences in performance parameters and workload, respectively, between sit/stand and traditional workplace users. Differences in text editing accuracy and cortisol levels for sit/stand workstation users indicate potential connectivity to cognitive parameters which should be further examined with large-scale studies. Practitioner summary: Medium-term effects of working at sit/stand workstations on cognitive performance and workload are unexplored. This randomised controlled trial suggests that cognitive performance and workload are unaffected for sit/stand workstation users after 23?weeks of use. However, accuracy appeared to improve and physiological stress appeared to be altered.

Abbreviations: BMI: body mass index; IPAQ: International physical activity questionnaire; MET: metabolic equivalent of task; MANOVA: multivariate ANOVA; NASA TLX: NASA task load index; RMANOVA: repeated measures ANOVA     

Cervicobrachial muscle response to cognitive load in a dual-task scenario

People working in an office environment often have to deal with significant cognitive workload due to the coordination of multiple, simultaneous tasks. The objective of this research was to examine the impact of cognitive load in office-type tasks on physical-stress response, using a dual-task paradigm involving a primary cognitive task and secondary typing task. The central hypothesis of this research was that altering the demands of the cognitive task would lead to a difference in physical stress-level and performance. Cognitive load was manipulated by presenting participants with three different types of cognitive tasks described in Rasmussen's () taxonomy, including skill-, rule-, and knowledge-based tasks. Dependent variables examined in the study included: (1) electromyographic activity of the upper trapezius (pars descendens) and cervical erector spinae muscles, (2) performance in a secondary typing task, and (3) subjective measures of stress and cognitive workload. The results of this study revealed that the primary task causing the highest level of perceived workload also produced 61% higher muscle activity in the right trapezius, and 6 and 11% higher activity in the left and right cervical erector spinae, respectively, in comparison to muscle activity associated with the cognitive task causing the lowest perceived workload. With respect to performance, a 23% decrease was observed in typing productivity when the rule-based task was completed simultaneously vs. typing in the absence of any additional cognitive task (the baseline condition). This information may be used to better organize work activities in office environments to increase performance and reduce stress.     

Cervicobrachial muscle response to cognitive load in a dual-task scenario

People working in an office environment often have to deal with significant cognitive workload due to the coordination of multiple, simultaneous tasks. The objective of this research was to examine the impact of cognitive load in office-type tasks on physical-stress response, using a dual-task paradigm involving a primary cognitive task and secondary typing task. The central hypothesis of this research was that altering the demands of the cognitive task would lead to a difference in physical stress-level and performance. Cognitive load was manipulated by presenting participants with three different types of cognitive tasks described in taxonomy, including skill-, rule-, and knowledge-based tasks. Dependent variables examined in the study included: (1) electromyographic activity of the upper trapezius (pars descendens) and cervical erector spinae muscles, (2) performance in a secondary typing task, and (3) subjective measures of stress and cognitive workload. The results of this study revealed that the primary task causing the highest level of perceived workload also produced 61% higher muscle activity in the right trapezius, and 6 and 11% higher activity in the left and right cervical erector spinae, respectively, in comparison to muscle activity associated with the cognitive task causing the lowest perceived workload. With respect to performance, a 23% decrease was observed in typing productivity when the rule-based task was completed simultaneously vs. typing in the absence of any additional cognitive task (the baseline condition). This information may be used to better organize work activities in office environments to increase performance and reduce stress.     

Tactile interruption management: tactile cues as task-switching reminders

Tactile cuing has been suggested as a method of interruption management for busy visual environments. This study examined the effectiveness of tactile cues as an interruption management strategy in a multi-tasking environment. Sixty-four participants completed a continuous aircraft monitoring task with periodic interruptions of a discrete gauge memory task. Participants were randomly assigned to two groups; one group had to remember to monitor for interruptions while the other group received tactile cues indicating an interruption’s arrival and location. As expected, the cued participants evidenced superior performance on both tasks. The results are consistent with the notion that tactile cues transform the resource-intensive, time-based task of remembering to check for interruptions into a simpler, event-based task, where cues assume a portion of the workload, permitting the application of valuable resources to other task demands. This study is discussed in the context of multiple resource theory and has practical implications for systems design in environments consisting of multiple, visual tasks and time-sensitive information.     

Development and validation of a behavioural video coding scheme for detecting mental workload in manual assembly

Abstract

Manual assembly in the future Industry 4.0 workplace will put high demands on operators’ cognitive processing. The development of mental workload (MWL) measures therefore looms large. Physiological gauges such as electroencephalography (EEG) show promising possibilities, but still lack sufficient reliability when applied in the field. This study presents an alternative measure with a substantial ecological validity. First, we developed a behavioural video coding scheme identifying 11 assembly behaviours potentially revealing MWL being too high. Subsequently, we explored its validity by analysing videos of 24 participants performing a high and a low complexity assembly. Results showed that five of the behaviours identified, such as freezing and the amount of part rotations, significantly differed in occurrence and/or duration between the two conditions. The study hereby proposes a novel and naturalistic method that could help practitioners to map and redesign critical assembly phases, and researchers to enrich validation of MWL-measures through measurement triangulation.

Practitioner summary: Current physiological mental workload (MWL) measures still lack sufficient reliability when applied in the field. Therefore, we identified several observable assembly behaviours that could reveal MWL being too high. The results propose a method to map MWL by observing specific assembly behaviours such as freezing and rotating parts.

Abbreviations: MWL: mental workload; EEG: electroencephalography; fNIRS: functional near infrared spectroscopy; AOI: area of interest; SMI: SensoMotoric Instruments, ETG: Eye-Tracking Glasses; FPS: frames per second; BORIS: Behavioral Observation Research Interactive Software; IRR: inter-rater reliability; SWAT: Subjective Workload Assessment Technique; NASA-TLX: National Aeronautics and Space Administration Task Load Index; EL: emotional load; DSSQ: Dundee Stress State Questionnaire; PHL: physical load; SBO: Strategisch Basis Onderzoek     

Improving target detection in visual search through the augmenting multi-sensory cues

The present experiment tested 60 individuals on a multiple screen, visual target detection task. Using a within-participant design, individuals received no-cue augmentation, an augmenting tactile cue alone, an augmenting auditory cue alone or both of the latter augmentations in combination. Results showed significant and substantive improvements in performance such that successful search speed was facilitated by more than 43%, errors of omission were reduced by 86% and errors of commission were reduced by more than 77% in the combinatorial cueing condition compared with the non-cued control. These outcomes were not a trade of performance efficiency for associated mental effort because recorded levels of cognitive workload were also reduced by more than 30% in the multi-cued circumstance compared with the control condition. When the tactile modality was incorporated it led to the highest gain in performance speed, when the auditory modality was incorporated, it led to the best levels of performance accuracy. The combined condition rendered the best of each from of performance increment. Reasons for this outcome pattern are discussed alongside their manifest practical benefits.

Practitioner Summary: This experiment tested 60 individuals on a multiple screen, visual target detection task. Individuals received no-cue augmentation, tactile cue alone, an augmenting auditory cue alone or both of the latter augmentations in combination. Results showed significant and substantive improvements in the combinatorial cueing condition compared with the non-cued control.     

The influence of leads on cognitive load and learning in a hypertext environment

The purpose of this study was to determine the effect of leads (or hypertext node previews) on cognitive load and learning. Leads provided a brief summary of information in the linked node, which helped orient the reader to the linked information. Dependent variables included measures of cognitive load: self-report of mental effort, reading time, and event-related desynchronization percentage of alpha, beta and theta brain wave rhythms; and learning performance: a recall task, and tests of domain and structural knowledge. Results indicated that use of leads reduced brain wave activity that may reflect split attention and extraneous cognitive load, and improved domain and structural knowledge acquisition. Further, findings provide insights into differentiating the types of cognitive load apparent in hypertext-assisted learning environments. Use of EEG measures allowed examination of instantaneous cognitive load, which showed that leads may be influencing germane load—reducing mental burden associated with creating coherence between two linked node. The self-report of mental effort measure appears more closely associated with overall and intrinsic load.     

Comparison of NASA-TLX scale,modified Cooper–Harper scale and mean inter-beat interval as measures of pilot mental workload during simulated flight tasks

Abstract

The sensitivity of NASA-TLX scale, modified Cooper–Harper (MCH) scale and the mean inter-beat interval (IBI) of successive heart beats, as measures of pilot mental workload (MWL), were evaluated in a flight training device (FTD). Operational F/A-18C pilots flew instrument approaches with varying task loads. Pilots’ performance, subjective MWL ratings and IBI were measured. Based on the pilots’ performance, three performance categories were formed; high-, medium- and low-performance. Values of the subjective rating scales and IBI were compared between categories. It was found that all measures were able to differentiate most task conditions and there was a strong, positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated. While NASA-TLX, MCH and IBI have all been previously used to measure MWL, this study is the first one to investigate their association in a modern FTD, using a realistic flying mission and operational pilots.

Practitioner summary: NASA-TLX scale, MCH scale and the IBI were evaluated in a flight training device. All measures were able to differentiate most task conditions and there was a positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated.

Abbreviations: ANOVA: Analysis of Variance; ECG: Electrocardiograph; F/A: fighter/attack; ft: feet; FTD: flight training device; G: Gravity; km: kilometer; m: meter; m/s: meters per second; MWL: mental workload; MCH: modified cooper-harper; NASA-TLX: NASA Task Load Index; NM: Nautical Mile; NN: normal-to-normal; IBI: inter-beat interval; ILS: Instrument Landing System; RR: R-Wave to R-Wave; SD: standard deviation; TTP: tactics, techniques and procedures; WTSAT: Weapon Tactics and Situation Awareness Trainer     

The impact of cognitive workload on physiological arousal in young adult drivers: a field study and simulation validation

Physiological measures provide a continuous and relatively non-invasive method of characterising workload. The extent to which such measures provide sensitivity beyond that provided by driving performance metrics is more open to question. Heart rate and skin conductance were monitored during actual highway driving in response to systematically increased levels of cognitive demand using an auditory delayed digit recall task. The protocol was consistent with an earlier simulator study, providing an opportunity to assess the validity of physiological measures recorded during driving simulation. The pattern of change in heart rate with increased cognitive demand was highly consistent between field and simulator. The findings meet statistical criteria for both relative and absolute validity, although there was a trend for absolute levels to be higher under actual driving conditions. For skin conductance level, the pattern in both environments was also quite similar and a reasonable case for overall relative validity can be made.

Statement of Relevance: Growing complexity and multiple demands on modern drivers’ attention highlight the significance of determining whether physiological measures provide increased sensitivity in workload detection. Better understanding, including whether simulator assessments provide valid measures of real-world response patterns, has implications in evaluating and refining interface designs and for developing advanced workload managers.     

Tactile display and vibrotactile pattern recognition on the torso

A wirelessly controlled tactile display has been designed, fabricated and tested for use as a navigation aid. The display comprises a 4 × 4 array of vibrating motors that is mounted on a waist band and stimulates the skin across the lower back. Three types of electromechanical actuators were evaluated for use in the display; based on their mechanical performance and power requirements, two of these motors were then used to fabricate tactile displays. The performance of the displays and the wireless tactile control units was assessed experimentally by having subjects identify which of eight possible vibrotactile patterns was presented to the lower back. The results indicated that subjects could recognize the vibrotactile patterns with almost perfect accuracy and that there was no difference between the two types of motor used for the displays. Moreover, the ability to recognize the pattern of vibrotactile stimulation was superior on the back as compared to the forearm. A further experiment confirmed that the tactile display can be used as a navigation aid outdoors and that the vibrotactile patterns presented can be interpreted as directional or instructional cues with almost perfect accuracy.     

Air traffic control: Ocular metrics reflect cognitive complexity

The objective of the study was to evaluate effects of complexity on cognitive workload in a simulated air traffic control conflict detection task by means of eye movements recording. We manipulated two complexity factors, convergence angle and aircrafts minimum distance at closest approach, in a multidimensional workload assessment method based on psychophysiological, performance, and subjective measures. Conflict trials resulted more complex and time-consuming than no conflicts, requiring more frequent fixations and saccades. Moreover, large saccades showed reduced burst power with higher task complexity. A motion-based and a ratio-based strategy were suggested for conflicts and no conflicts on the basis of ocular metrics analysis: aircrafts differential speed and distance to convergence point at trial start were considered determinant for strategy adoption.Relevance to industryEye metrics measurement for online workload assessment enhances better identification of workload-inducing scenarios and adopted strategy for traffic management. System design, as well as air traffic control operators training programs, might benefit from on line workload measurement.     

An attempt to evaluate mental workload using wavelet transform of EEG

An attempt was made to evaluate mental workload using a wavelet transform of electroencephalographic (EEG) signals. Participants performed a continuous matching task at three levels of task difficulty. EEG signals during the task were recorded continuously from Fz, Cz, and Pz. The reaction time increased as the difficulty of the task increased. The percentage correct decreased as the task became more difficult. In accordance with this, the rating score on the NASA-Task Load Index tended to increase with increased task difficulty. The EEG signals were analyzed using wavelet transform to investigate time-frequency characteristics. The total power at theta, alpha, and beta frequency bands and the time that the maximum power appeared for the three frequency bands were extracted from the scalogram. Increasing cognitive task difficulty seems to delay the time at which the central nervous system works most actively. These measures were found to be sensitive indicators of mental workload and could differentiate three cognitive task loads (low, moderate, and high) with high precision. Actual or potential applications of this research include a method that is relatively quick and accurate, compared with traditional methods, for the evaluation of mental workload.     

Response Times for Auditory and Vibrotactile Directional Cues in Different Immersive Displays

ABSTRACT

Using multisensory signals in advanced driver assistance is continuously increasing as a way to increase the attention and reduce the reaction time. It is essential that driver assistance system is capable of providing directional cues to the driver to direct his attention to the sides of the car as usually the focus is in front of the car. These signals could be for blind spot information, navigation, lane departure warning, collision warning, etc. This study investigated the effect of auditory and vibrotactile on directional attention in driver assistance systems. Moreover, two types of immersive displays were used in the driving simulation, namely the Head Mounted Display (HMD) and CAVE display, to study the effect of the type of display on the human performance. Lane Chang Task was used to assess the attention by measuring the response time to directional information. Vibrotactile and Auditory cues induced equal response times, meanwhile, vibrotactile signal was significantly gained higher satisfaction than auditory.     

Cognitive engineering,cognitive augmentation,and information display

Abstract— From a perspective of cognitive engineering, this paper presents eight principles of good display design based on human cognition: (1) displays that divide attention will increase the cognitive load on working memory; (2) displays that continuously present the same information may compensate for the decay of short‐term memory; (3) displays that assist the viewer in mentally chunking information will decrease the cognitive load on working memory; (4) displays that present icons or images to direct attention to important visual stimuli will likely minimize inattentional blindness; (5) displays that help minimize attentional distractions and help focus attention on a given task will minimize the burden on working memory; (6) displays that present cues composed of strong singletons will capture and engage exogenous attention; (7) displays that help the viewer make a mental connection between an analogue and a target will assist in the induction of analogical reasoning; (8) displays that present over time multiple exposures consisting of statistical regularities may promote implicit learning and expertise development. Thus, information displays of the future can systematically engage high‐level human cognitive processes for purposes of improving human performance on a variety of tasks found in education, the military, and many other applications.     

Human information processing during physical exercise.

This study was designed to investigate how conditions of physical exercise affect human information processing. Sixteen subjects performed two information processing tasks (perception and decision) during two exercise conditions (endurance vs interval protocols) and during two control conditions (rest vs minimal load protocols). The control conditions required subjects either to perform the information processing tasks under resting conditions or while pedalling a bicycle ergometer at a minimal workload. Workload during the exercise protocols consisted of a fixed percentage of the subject's maximal workload. Each 40 min protocol consisted of five consecutive stages: practice, baseline, warming-up, exercise, and cooling-down, during which heart rate and ratings of perceived exertion were determined. In the perception task subjects had to identify a briefly presented row of three letters. In the decision task subjects had to indicate which of the outer numbers in a row of three digits was the larger. Results indicated that the two control protocols did not influence cognitive task performance; however, in the exercise protocols, increments in physical workload improved performance on the decision task and reduced performance on the perception task, while decrements in physical workload reduced performance on the decision task and improved performance on the perception task. Changes in mental task performance were not evident within protocol stages; only after stage transitions did changes in mental performance occur. We discussed possible theoretical approaches to explain these results and concluded that models advanced in the context of dual-task methodology seem most promising.     

Effect of physical workload and modality of information presentation on pattern recognition and navigation task performance by high-fit young males

Abstract

Many occupations require both physical exertion and cognitive task performance. Knowledge of any interaction between physical demands and modalities of cognitive task information presentation can provide a basis for optimising performance. This study examined the effect of physical exertion and modality of information presentation on pattern recognition and navigation-related information processing. Results indicated males of equivalent high fitness, between the ages of 18 and 34, rely more on visual cues vs auditory or haptic for pattern recognition when exertion level is high. We found that navigation response time was shorter under low and medium exertion levels as compared to high intensity. Navigation accuracy was lower under high level exertion compared to medium and low levels. In general, findings indicated that use of the haptic modality for cognitive task cueing decreased accuracy in pattern recognition responses.

Practitioner Summary: An examination was conducted on the effect of physical exertion and information presentation modality in pattern recognition and navigation. In occupations requiring information presentation to workers, who are simultaneously performing a physical task, the visual modality appears most effective under high level exertion while haptic cueing degrades performance.