Assessing mental workload in virtual reality based EOT crane operations: A multi-measure approach

BackgroundEye-movement metrics and subjective workload measures are extensively used to determine mental workload of participants. The aim of this study was to assess Electric overhead travelling (EOT) crane operators’ mental workload variability based on eye movement metrics such as fixation frequency, fixation duration, saccade duration, saccade amplitude, and fixation/saccade ratio during EOT crane operations in virtual reality (VR) based EOT crane simulator.MethodsA 2^k (k = 3) factorial experiment with factors namely, hazardous scenario, activity level, and trial was designed and conducted to demonstrate the proposed assessment approach. Throughout the experiment, we recorded the eye movements of 12 EOT crane operators of a steel industry of authors’ country. Post experiment, the National Aeronautics and Space Administration task load index (NASA-TLX) was adopted as a subjective workload measure and run time of task completion was recorded. Eye-movement metrics, subjective workload measure, run time were tested with multivariate analysis of variance (MANOVA), and three way repeated measure analysis of variance (ANOVA).ResultsAt the level of α = 0.05, the experimental factors significantly influence the means of eye movement metrics, subjective ratings and run time. There was also significant influence among their interactions. A positive correlation was also found for eye movements metrics with NASA-TLX and run time.ConclusionsEye movement metrics help in understanding the mental workload of participants unobtrusively and continuously. Analysis of subjective workload measure and run time along with eye-gaze analysis provide a deeper understanding on the pattern of mental workload.     

Mental workload measurement for emergency operating procedures in digital nuclear power plants

Mental workload is a major consideration for the design of emergency operation procedures (EOPs) in nuclear power plants. Continuous and objective measures are desired. This paper compares seven mental workload measurement methods (pupil size, blink rate, blink duration, heart rate variability, parasympathetic/sympathetic ratio, total power and (Goals, Operations, Methods, and Section Rules)-(Keystroke Level Model) GOMS-KLM-based workload index) with regard to sensitivity, validity and intrusiveness. Eighteen participants performed two computerised EOPs of different complexity levels, and mental workload measures were collected during the experiment. The results show that the blink rate is sensitive to both the difference in the overall task complexity and changes in peak complexity within EOPs, that the error rate is sensitive to the level of arousal and correlate to the step error rate and that blink duration increases over the task period in both low and high complexity EOPs. Cardiac measures were able to distinguish tasks with different overall complexity. The intrusiveness of the physiological instruments is acceptable. Finally, the six physiological measures were integrated using group method of data handling to predict perceived overall mental workload.

Practitioner Summary: The study compared seven measures for evaluating the mental workload with emergency operation procedure in nuclear power plants. An experiment with simulated procedures was carried out, and the results show that eye response measures are useful for assessing temporal changes of workload whereas cardiac measures are useful for evaluating the overall workload.     

Comparison of NASA-TLX scale,modified Cooper–Harper scale and mean inter-beat interval as measures of pilot mental workload during simulated flight tasks

Abstract

The sensitivity of NASA-TLX scale, modified Cooper–Harper (MCH) scale and the mean inter-beat interval (IBI) of successive heart beats, as measures of pilot mental workload (MWL), were evaluated in a flight training device (FTD). Operational F/A-18C pilots flew instrument approaches with varying task loads. Pilots’ performance, subjective MWL ratings and IBI were measured. Based on the pilots’ performance, three performance categories were formed; high-, medium- and low-performance. Values of the subjective rating scales and IBI were compared between categories. It was found that all measures were able to differentiate most task conditions and there was a strong, positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated. While NASA-TLX, MCH and IBI have all been previously used to measure MWL, this study is the first one to investigate their association in a modern FTD, using a realistic flying mission and operational pilots.

Practitioner summary: NASA-TLX scale, MCH scale and the IBI were evaluated in a flight training device. All measures were able to differentiate most task conditions and there was a positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated.

Abbreviations: ANOVA: Analysis of Variance; ECG: Electrocardiograph; F/A: fighter/attack; ft: feet; FTD: flight training device; G: Gravity; km: kilometer; m: meter; m/s: meters per second; MWL: mental workload; MCH: modified cooper-harper; NASA-TLX: NASA Task Load Index; NM: Nautical Mile; NN: normal-to-normal; IBI: inter-beat interval; ILS: Instrument Landing System; RR: R-Wave to R-Wave; SD: standard deviation; TTP: tactics, techniques and procedures; WTSAT: Weapon Tactics and Situation Awareness Trainer     

Physiological workload reactions to increasing levels of task difficulty

The sensitivity of physiological measures to mental workload was investigated in a flight simulator. Twelve pilots had to fly through a tunnel with varying levels of difficulty. Additionally, they had to perform a memory task with four levels of difficulty. The easiest memory task was combined with the easiest tunnel task and the most difficult memory task with the most difficult tunnel task. Between the tunnel tasks, subjects had to fly a pursuit task in which a target jet had to be followed. Rest periods before and after the experiment were used as a baseline for the physiological measures. Mental workload was measured with heart period, continuous blood pressure, respiration and eye blinks. Several respiratory parameters, heart rate variability, blood pressure variability and the gain between systolic blood pressure and heart period (modulus) were scored. All measures showed differences between rest and flight, and between the pursuit and the tunnel task. Only heart period was sensitive to difficulty levels in the tunnel task. Heart rate variability increased when respiratory activity around 0.10 Hz increased, which occurred often. The modulus was hardly influenced by respiration and therefore appears to be a better measure than heart rate variability. Among the respiratory parameters, the duration of a respiratory cycle was the most sensitive to changes in workload. The time in between two successive eye blinks (blink interval) increased and the blink duration decreased as more visual information had to be processed. Increasing the difficulty of the memory task led to a decrement in blink interval, probably caused by subvocal activity during rehearsal of target letters. The data show that physiological measures are sensitive to mental effort, whereas rating scales are sensitive to both mental effort and task difficulty.     

Mental workload and cognitive task automaticity: an evaluation of subjective and time estimation metrics

Abstract

The evaluation of mental workload is becoming increasingly important in system design and analysis. The present study examined the structure and assessment of mental workload in performing decision and monitoring tasks by focusing on two mental workload measurements: subjective assessment and time estimation. The task required the assignment of a series of incoming customers to the shortest of three parallel service lines displayed on a computer monitor. The subject was either in charge of the customer assignment (manual mode) or was monitoring an automated system performing the same task (automatic mode). In both cases, the subjects were required to detect the non-optimal assignments that they or the computer had made. Time pressure was manipulated by the experimenter to create fast and slow conditions. The results revealed a multi-dimensional structure of mental workload and a multi-step process of subjective workload assessment. The results also indicated that subjective workload was more influenced by the subject's participatory mode than by the factor of task speed. The time estimation intervals produced while performing the decision and monitoring tasks had significantly greater length and larger variability than those produced while either performing no other tasks or performing a well practised customer assignment task. This result seemed to indicate that time estimation was sensitive to the presence of perceptual/cognitive demands, but not to response related activities to which behavioural automaticity has developed.     

Pilot mental workload: how well do pilots really perform?

The purpose of this study was to investigate the effects of increasing mental demands on various aspects of aircrew performance. In particular, the robustness of the prioritization and allocation hierarchy of aviate–navigate–communicate was examined, a hierarchy commonly used within the aviation industry. A total of 42 trainee pilots were divided into three workload groups (low, medium, high) to complete a desktop, computer-based exercise that simulated combinations of generic flight deck activities: flight control manipulation, rule-based actions and higher level cognitive processing, in addition to Air Traffic Control instructions that varied in length from one chunk of auditory information to seven chunks. It was found that as mental workload and auditory input increased, participants experienced considerable difficulty in carrying out the primary manipulation task. A similar decline in prioritization was also observed. Moreover, when pilots were under a high mental workload their ability to comprehend more than two chunks of auditory data deteriorated rapidly.     

Cultural influences on the measurement of subjective mental workload

Cognitive ergonomics is well entrenched in North American and most European work environments, where systems and products are designed with the capabilities and limitations of the user in mind. A prominent technique for analysing task demands is subjective mental workload measurement. Subjective ratings of mental workload have been shown—in North American and European populations—to be reliable and sensitive to changes in mental workload. However, there is reason to think that cultural differences may affect subjective ratings. This study compared the performance and subjective mental workload ratings of Indonesian (n = 87) and Dutch (n = 88) students in the context of a hybrid memory/visual search task. Performance was comparable for the two groups, but the sensitivity of the subjective workload measures was better in the Dutch than in the Indonesian group. The results are interpreted in light of social axioms and cultural values.

Statement of Relevance: Now more than ever ergonomists must think beyond national borders in improving work processes. Globalisation requires that cultural differences be taken into account in developing and applying ergonomic techniques. This paper presents empirical work showing that cultural differences seriously impact the sensitivity of mental workload measures.     

Learning rate and subjective mental workload in five truck driving tasks

Both learning curve models and subjective mental workload are useful tools for determining the length of training for new workers and predicting future task performance. An experiment was designed to collect the task completion times and subjective mental workload of five driving tasks including (a) reverse into garage, (b) 3-point turn, (c) parallel parking, (d) S-curve and (e) up-down-hill. The results indicated that task completion times of truck driving can be predicted with a learning curve. Practice significantly reduced the mental workload rating. However, the novice trainees tended to have a more significant reduction because, compared to experienced trainees, they tended to give greater or lower workload scores than the experienced trainees before and after practice, respectively. The current study may not be complete enough to provide guidelines for a training programme, but it is adequate to suggest that learning rate and workload measure can serve as indexes for factoring in the individual differences.

Practitioner summary: Learning curves can be used to determine the length of training for new workers and performance standards for a particular task. Learning rate and mental workload were found to be important measures for comparing individual differences in order to better design a training programme. However, mental workload must be evaluated by experienced participants.     

Time perception as a workload measure in simulated car driving

In experimental studies using flight simulations subjects’ duration estimates have shown to be an effective indicator of cognitive task demands. In this study we wanted to find out whether subjective time perception could serve as a measure of cognitive workload during simulated car driving. Participants drove on a round course of a driving simulator consisting of three different environments with different levels of task demands. Drivers were required to perform a time-production task while driving the vehicle. Electrodermal activity and subjective ratings of mental workload (SWAT) were recorded simultaneously. The length of produced intervals increased significantly in more complex driving situations, as did electrodermal activity and subjective ratings of mental workload. Thus, time production is a valid indicator of cognitive involvement in simulated driving and could become a valid method to measure the current mental workload of car drivers in various traffic situations.     

Utilizing slope method as an alternative data analysis for functional near-infrared spectroscopy-derived cerebral hemodynamic responses

The purpose was to propose an alternative data analysis for functional near-infrared spectroscopy (fNIRS)-derived hemodynamics as a function of cortical activation changes. We evaluated hemodynamic responses from the prefrontal cortex region while 38 participants performed a cognitive task. The task consisted of an arithmetic calculation with three levels of complexity (i.e., easy, medium and difficult). These task-dependent hemodynamic responses were analyzed by the slope method (i.e. using a linear regression through the cognitive task) and were compared with the corresponding responses obtained with a traditional approach of the amplitude method. Subjective scales of task loading (assessed by DP15 and NASA-TLX) and behavioral outcomes (performance and reaction time) were also recorded in response to the task complexity. Results revealed that the proposed slope method allowed a better discrimination in terms of cortical activation among all levels of mental workload. There was no significant increase in cortical activation between the medium and difficult levels (p = .30; d = 0.09) with the amplitude method while the slope coefficient was sensitive to the different levels (p < .01; d = 0.32). These preliminary results from a large sample size demonstrated that the slope method appears suitable for discriminating the changes in cortical activation with respect to the mental workload.Relevance to industryIn this work, we proposed an optimum way of quantifying the mental workload of participants in terms of fNIRS-derived cerebral hemodynamic responses.     

The Sternberg memory search task as an index of pilot workload

Abstract

The rationale for the use of the Sternberg memory search task as a diagnostic measure of pilot workload is described, and seven investigations that have employed this task in flight simulators or aircraft environments are summarized. The details of two further flight simulator experiments in which workload is measured by an auditory Sternberg task are reported. These results indicate the diagnostic value of the task in discriminating between the perceptual/central processing and response demands of a holding pattern and an approach pattern, respectively. Perceptual and response load were greater in an approach phase, relative to a holding phase. Neither phase imposed a substantial central processing load. Based upon the findings of the studies described, the paper concludes with a set of recommendations for employing the Sternberg task in aviation environments. These recommendations emphasize the importance of information display and response procedures, the choice of particular memory sets, the presentation of data, and the need to avoid extremely difficult flight tasks.     

Air-to-ground training missions: a psychophysiological workload analysis

Psychophysiological measures are used to assess the workload of F4 Phantom aircraft pilots and weapon systems officers (WSOs) during air-to-ground training missions and during the performance of two levels of difficulty of a laboratory tracking task. The bombing range portion of the missions was associated with the highest pilot workload, while the WSO flying the aircraft was the highest workload segment for the WSOs. The pilots' data were found to have a wider range of values for the physiological measures than were found in the WSO data. The different levels of tracking task difficulty produced significant physiological effects but the range of values found for most of the flight segments were much greater. These data demonstrate that extrapolating laboratory data to the flight environment is risky at best. The various physiological measures were differentially sensitive to the different demands of the various flight segments.     

Learning while multitasking: short and long-term benefits of brain stimulation

Abstract

We employed a simulated production task that mimics the real-world skill acquisition required of operators working in control rooms of power plants to assess short and long-term effects of transcranial random noise stimulation (tRNS). tRNS has shown potential for enhancing learning and performance of cognitive skills. Forty subjects (24 female) learned how to execute the simulated production task during the training phase and were required to perform a secondary task during the skill acquisition phase while they received active (12?min) or sham tRNS on DLPFC. After 2 weeks they had to recall the task again without any stimulation. The results demonstrate that tRNS promoted better multitasking as reflected by better performance in a secondary task during and immediately after tRNS. However, 2 weeks later, beneficial effect of tRNS on retention was moderated by general mental ability. Particularly, tRNS benefited those with lower general mental ability.

Practitioner summary: By using a simulated production task, we assessed the effects of tRNS on learning and skill retention. The study indicates that neurostimulation can enhance the learning of multiple complex tasks. Moreover, it shows that retention of those tasks can be supported by neurostimulation, especially for those with lower general mental ability.     

Double trade-off curves with different cognitive processing combinations: testing the cancellation axiom of mental workload measurement theory.

The accomplishment model of average mental workload--a formal axiomatic measurement theory--was used as a basis for developing and testing secondary task indices of mental workload (H. A. Colle & G. B. Reid, 1997). Its cancellation axiom implies global sensitivity, which is an important theoretical and practical criterion for mental workload indices. Performance levels of different secondary tasks were empirically equated in mental workload and then used to test the cancellation axiom. Cognitive processing similarity--including orthographic, phonemic, and semantic processing of pairs of operator and secondary tasks--was manipulated in three experiments. Equivalencies between secondary tasks were independent of secondary-operator task similarity, consistent with the cancellation axiom and the global sensitivity of these secondary tasks. The results suggest that standardized secondary task techniques can be developed for the practical measurement of mental workload. Actual or potential applications of this research include the development of functionally useful and realistic secondary task measures of mental workload.     

Human interaction with robotic systems: performance and workload evaluations

Abstract

We first tested the effect of differing tactile informational forms (i.e. directional cues vs. static cues vs. dynamic cues) on objective performance and perceived workload in a collaborative human–robot task. A second experiment evaluated the influence of task load and informational message type (i.e. single words vs. grouped phrases) on that same collaborative task. In both experiments, the relationship of personal characteristics (attentional control and spatial ability) to performance and workload was also measured. In addition to objective performance and self-report of cognitive load, we evaluated different physiological responses in each experiment. Results showed a performance–workload association for directional cues, message type and task load. EEG measures however, proved generally insensitive to such task load manipulations. Where significant EEG effects were observed, right hemisphere amplitude differences predominated, although unexpectedly these latter relationships were negative. Although EEG measures were partially associated with performance, they appear to possess limited utility as measures of workload in association with tactile displays.

Practitioner Summary: As practitioners look to take advantage of innovative tactile displays in complex operational realms like human–robotic interaction, associated performance effects are mediated by cognitive workload. Despite some patterns of association, reliable reflections of operator state can be difficult to discern and employ as the number, complexity and sophistication of these respective measures themselves increase.     

Workload Management in Teleoperation of Unmanned Ground Vehicles: Effects of a Delay Compensation Aid on Human Operators’ Workload and Teleoperation Performance

ABSTRACT

Workload management is of critical concern in the teleoperation of unmanned vehicles because teleoperation is often employed in high-risk industries wherein high workload can lead to sub-optimal task performance and can harm human operators’ long-term well-being. This study aimed to assess the detrimental effects of time delays in teleoperation on operators’ workload and performance, and how a delay compensation aid mitigated such effects. We conducted a human-in-the-loop experiment with 36 participants using a dual-task teleoperation platform, where participants drove a simulated High Mobility Multipurpose Wheeled Vehicle (HMMWV) and performed a one-back memory task under three conditions: the delay condition, the delay with compensation aid condition, and the ideal no delay condition. A model-free predictor was used as the compensation aid. Results indicate that with a time delay of 0.8-s participants’ workload increased and performance degraded significantly. Moreover, the model-free predictor mitigated the detrimental effects of time delay on workload and task performance. Our findings suggest that participants are more sensitive in their perceived workload compared to the objective and physiological measures of workload. In addition, without any delay compensation algorithms, continuous teleoperation may not be ideal for operations with long time delays.     

Visual evoked potentials as indicators of the workload at visual display terminals

No physiological correlates are known for visual fatigue produced by prolonged viewing of visual display terminals (VDTs). The primary aim of the present study was to find a physiological correlate of a performance measure which is related to fatigue. The assumption was that a change in the state of fatigue expresses itself in fluctuations of accommodation. If indeed variations in accommodation are indicative of fatigue, then visual evoked potentials (VEPs) might be used for the objective estimation of fatigue, because pattern EPs are highly sensitive to the quality of the retinal image.

In a realistic visual task, the VEP to pattern onset and the performance to a binary task are recorded simultaneously. Both indicators of fatigue, the amplitude of the VEP and the reaction time, proved to be sensitive to the workload of the task. If accommodation effort was increased, then reaction time increased and VEP amplitude decreased. Thus, VEPs are an indicator of the workload of a visual task. This allows comparison of the fatiguing effects of various office settings on the basis of the pattern EP of a particular subject. However, because VEP amplitude and reaction time do not have the same relation amongst subjects, the pattern EP cannot be used for inter-individual comparison.     

Development and validation of a behavioural video coding scheme for detecting mental workload in manual assembly

Abstract

Manual assembly in the future Industry 4.0 workplace will put high demands on operators’ cognitive processing. The development of mental workload (MWL) measures therefore looms large. Physiological gauges such as electroencephalography (EEG) show promising possibilities, but still lack sufficient reliability when applied in the field. This study presents an alternative measure with a substantial ecological validity. First, we developed a behavioural video coding scheme identifying 11 assembly behaviours potentially revealing MWL being too high. Subsequently, we explored its validity by analysing videos of 24 participants performing a high and a low complexity assembly. Results showed that five of the behaviours identified, such as freezing and the amount of part rotations, significantly differed in occurrence and/or duration between the two conditions. The study hereby proposes a novel and naturalistic method that could help practitioners to map and redesign critical assembly phases, and researchers to enrich validation of MWL-measures through measurement triangulation.

Practitioner summary: Current physiological mental workload (MWL) measures still lack sufficient reliability when applied in the field. Therefore, we identified several observable assembly behaviours that could reveal MWL being too high. The results propose a method to map MWL by observing specific assembly behaviours such as freezing and rotating parts.

Abbreviations: MWL: mental workload; EEG: electroencephalography; fNIRS: functional near infrared spectroscopy; AOI: area of interest; SMI: SensoMotoric Instruments, ETG: Eye-Tracking Glasses; FPS: frames per second; BORIS: Behavioral Observation Research Interactive Software; IRR: inter-rater reliability; SWAT: Subjective Workload Assessment Technique; NASA-TLX: National Aeronautics and Space Administration Task Load Index; EL: emotional load; DSSQ: Dundee Stress State Questionnaire; PHL: physical load; SBO: Strategisch Basis Onderzoek     

Improving pilot mental workload classification through feature exploitation and combination: a feasibility study

Predicting high pilot mental workload is important to the United States Air Force because lives and aircraft have been lost due to errors made during periods of flight associated with mental overload and task saturation. Current research efforts use psychophysiological measures such as electroencephalography (EEG), cardiac, ocular, and respiration measures in an attempt to identify and predict mental workload levels. Existing classification methods successfully classify pilot mental workload using flight data for a single pilot on a given day, but are unsuccessful across different pilots and/or days. We demonstrate a small subset of combined and calibrated psychophysiological features collected from a single pilot on a given day that accurately classifies mental workload for a separate pilot on a different day. We achieve classification accuracy (CA) improvements over previous classifiers exceeding 80% while using significantly fewer features and dramatically reducing the CA variance. Without the need for EEG data, our feature combination and calibration scheme also radically reduces the raw data collection requirements, making data collection immensely easier to manage and spectacularly reducing computational processing requirements.     

Variations in ATC-Work Load as a Function of Variations in Cockpit Workload

Abstract

The relation between pilots‘ workload and radar controllers‘ workload was investigated. Pilots‘ workload served as the independent variable. It was assumed that a number of levels of workload can be distinguished in a standard airport, traffic circuit under visual flight rules. The dependent variable (controllers’ workload) was measured by response frequency on an auditory binary choice task. Results indicate a rise in radar controllers’ workload in a predicted direction for a number of pilots’ workload levels.