Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work

Subjective workload and situation awareness measures, such as the NASA task load index (TLX) and the situational awareness rating technique (SART), are frequently used in human–system evaluation. However, the interpretation of these ratings is debated. In this study, empirical evidence for the measures' theoretical assumptions was investigated by comparing operators' ratings collected immediately after performing a scenario and ratings collected after operators' acquisition through a video review of the scenario, knowledge of actual system states. Eighteen licensed control room operators participated in the simulator study, running 12 relatively challenging scenarios. It was found that the interpretation of TLX items involving introspection remained stable after operators acquired factual scenario knowledge, while the interpretation of items involving the perception of external events, such as situation awareness and performance, depended on the operators' scenario knowledge. The result shows that operators’ ratings could discriminate between mental effort, performance, frustration, and situation awareness. No clear evidence for the SART index as a measure of situation awareness was found. Instead, a subjective situation awareness measure developed for this study was distinct from workload and related to operator performance, showing that this type of measure warrants future investigation of its validity. The study findings help in developing measurement procedures and interpreting subjective measures. Finally, the study reveals that informing operators about the scenario can provide useful subjective ratings of situation awareness and performance. Future research should include procedures for how to inform participants adequately and efficiently in subjective assessments.     

A comparative evaluation of the wearable augmented reality-based data presentation interface and traditional methods for data entry tasks

Data entry is a ubiquitous task performed in today's offices. Persistent data entry is linked with high workload and fatigue due to poor ergonomic workplace design and poor posture. This study aims to alleviate data entry operators' workload and improve data entry performance by applying wearable augmented reality (AR) technology to data entry tasks. An AR-based interface was developed and used to present data to the participants, who entered the data in web-based data entry forms. A total of eighteen participants performed data entry tasks to evaluate the AR interface with traditional methods – extra desktop monitor and paper-based data presentation methods. Each method's performance was judged on the task completion time, typing error rate, workload, and usability. The usability and overall perceived workload while using an AR interface for data entry were similar to the traditional way of using paper, despite the additional burden due to the weight of the AR headset. AR interface did not perform better than the extra desktop monitor interface for usability and overall perceived workload. The results from this study can be utilized to design AR devices that are suited for data entry tasks.     

Interaction between an operator and the control desk at the control room of the railway traffic: A Serbian experience

The subject of this paper is the interaction between an operator and the control desk at the Railway Traffic Control Room (RTCR) in Nis, Serbia. The following methods were used for this research: operators' anthropometric measurement; determining of maximum force of the movements of operators' arms; an analysis of workload of the movements of operators' arms, heads and torsos, and an analysis of errors in operators' movements in response to visual cues. The paper presents the following results: static parameter of 20 anthropometric measures of 41 operators' bodies were measured; nine corresponding dimensions of particular body parts for 5th, 50th, and 95th percentile were calculated; 10 characteristic work angles were measured; five corresponding functional body parts were calculated; times and distances covered by operators' arms were measured; maximum forces used for these movements were calculated; positions and number of movements of arms, heads, and torsos of operators were determined, and movement errors and probability of movement errors in choice of direction, arms and cumulative conditional probability were calculated. The research was conducted with the aim of discovering determining factors of mutual synchronization of the operator and desk.     

Assessing mental workload in virtual reality based EOT crane operations: A multi-measure approach

BackgroundEye-movement metrics and subjective workload measures are extensively used to determine mental workload of participants. The aim of this study was to assess Electric overhead travelling (EOT) crane operators’ mental workload variability based on eye movement metrics such as fixation frequency, fixation duration, saccade duration, saccade amplitude, and fixation/saccade ratio during EOT crane operations in virtual reality (VR) based EOT crane simulator.MethodsA 2^k (k = 3) factorial experiment with factors namely, hazardous scenario, activity level, and trial was designed and conducted to demonstrate the proposed assessment approach. Throughout the experiment, we recorded the eye movements of 12 EOT crane operators of a steel industry of authors’ country. Post experiment, the National Aeronautics and Space Administration task load index (NASA-TLX) was adopted as a subjective workload measure and run time of task completion was recorded. Eye-movement metrics, subjective workload measure, run time were tested with multivariate analysis of variance (MANOVA), and three way repeated measure analysis of variance (ANOVA).ResultsAt the level of α = 0.05, the experimental factors significantly influence the means of eye movement metrics, subjective ratings and run time. There was also significant influence among their interactions. A positive correlation was also found for eye movements metrics with NASA-TLX and run time.ConclusionsEye movement metrics help in understanding the mental workload of participants unobtrusively and continuously. Analysis of subjective workload measure and run time along with eye-gaze analysis provide a deeper understanding on the pattern of mental workload.     

Impact of automated decision aids on performance,operator behaviour and workload in a simulated supervisory control task

In studies reporting automation effects on overall system performance and on the operator, the methods used to measure workload often did not appropriately reflect the complexity of this construct. The present study addresses the impact of automation on operator workload and behaviour in process control fault management. Workload effects were assessed with subjective, cardiovascular and secondary task performance indicators. Interactions with the interface of the process control simulation directed at gathering information and controlling the system were recorded. Automation made operators more efficient, allowing faster fault management with less information sampling and control actions. Subjective workload ratings were significantly lower in the assisted conditions as compared to manual, which was not reflected in cardiovascular and secondary task measures. Participants' information sampling activity did not differ between medium and high level of automation. Results suggest that participants paid constantly high attention to their task even with highly automated support.     

Effect of time pressure and target uncertainty on human operator performance and workload for autonomous unmanned aerial system

Autonomous unmanned aircraft systems (UAS) are being utilized at an increasing rate for a number of military applications. The role of a human operator differs from that of a pilot in a manned aircraft, and this new role creates a need for a shift in interface and task design in order to take advantage of the full potential of these systems. This study examined the effects of time pressure and target uncertainty on autonomous unmanned aerial vehicle operator task performance and workload. A 2 × 2 within subjects experiment design was conducted using Multi-Modal Immersive Intelligent Interface for Remote Operation (MIIIRO) software. The primary task was image identification, and secondary tasks consisted of responding to events encountered in typical UAS operations. Time pressure was found to produce a significant difference in subjective workload ratings as well as secondary task performance scores, while target uncertainty was found to produce a significant difference in the primary task performance scores. Interaction effects were also found for primary tasks and two of the secondary tasks. This study has contributed to the knowledge of UAS operation, and the factors which may influence performance and workload within the UAS operator. Performance and workload effects were shown to be elicited by time pressure. Relevance to industry: The research findings from this study will help the UAS community in designing human computer interface and enable appropriate business decisions for staffing and training, to improve system performance and reduce the workload.     

A graphic modeling and analysis tool for human fault diagnosis tasks

This study presents a graphic modeling and analysis tool for use in constructing an operator's mental model in fault diagnosis tasks. In most automatic and complicated process control systems, human fault diagnosis tasks have become increasingly complex and specialized. The system designer should consider the cognitive process of human operator to avert failure of implement action owing to a lack of compatibility between humans and aiding system interface. Here, an experiment is performed to investigate the nature of human fault diagnosis. A graphic modeling and analysis tool is then proposed to model the continuous process of human fault diagnosis. The approach proposed herein exploits both the line-chart and Petri nets to demonstrate the operator's thoughts and actions. Moreover, results in this study are integrated into an adaptive standard diagnosis model that can assess the operators' mental workload and accurately depict the interactions between human operator and aiding system.Relevance to industryAutomatic intelligent diagnosis systems can not provide satisfactory operating performance. Human diagnosticians are more effective than computer ones. Results in this study offer further insight into an operator behavior in graphic form and also how to design a better aiding system.     

Situation awareness measurement of an ecological interface designed to operator support during alarm floods

Alarm summary interfaces are lists of chronologically sorted alarms used in oil producing supervision programs for decades. When a great number of alarms are activated in a small interval of time – that is, an alarm flood – alarm summary ceases to be as useful to understand the process situation. New alarm systems and interfaces are necessary to support operators' decision when dealing with such alarm floods. Situational awareness is an important concept for process operators when making decisions. The application of Ecological Interface Design concepts can lead to better interface designs, especially in unanticipated situations. The aim of this work is to verify the usefulness of a new ecological alarm interface, called Advanced System of Intelligent Alarms (SAAI, the Portuguese acronym). SAAI displays alarms prioritised in real-time with relevant graphical information on process conditions. Professional operators' situational awareness is measured when using either the SAAI interface, or the alarm summary interface, connected to a dynamic simulator running a model of an oil production process. Measurement techniques used include objective and subjective rating approaches, and a proposed technique. Operators' situation awareness for SAAI interface has shown greater values under all techniques used, compared to alarm summary interface. Although the new interface implied that operators would have to look also at an extra monitor, results show that their situation awareness improved on average, and that situation awareness might be less impaired under alarm floods.Relevance to industrySituation awareness plays an important role in operators' decision-making process. Alarm systems are critical in complex process operation, and alarm floods may impair operators' decisions. A new alarm ecological interface designed to maintain operators' situational awareness under alarm floods may mean an improvement in operation safety and efficiency.     

Effects of three‐dimensional virtual reality and traditional training methods on mental workload and training performance

Industries will implement effective training programs to improve training performance, and an ideal training performance occurs under proper mental workload (MWL). Virtual reality (VR) has recently been widely utilized in training; however, only a few studies have investigated its effects on MWL and training performance simultaneously. The purpose of this study is to investigate the effects of VR training and traditional training methods, such as technical manuals (TM) and multimedia films (MF), on training performance and MWL. The results of the performance measurement show that VR training is considered the best training method compared to TM and MF, particularly in the case of complex tasks. The results of physiological measurements (GSR [galvanic skin response], LF% [low frequency], and LF/HF [high frequency] ratio) show a significant difference between reading TM and using computer (MF and VR), wherein the latter has a lower MWL. However, no significant difference in subjective MWL assessment (NASA‐TLX [task load index]) and HF% measurement is found.     

Air traffic control: Ocular metrics reflect cognitive complexity

The objective of the study was to evaluate effects of complexity on cognitive workload in a simulated air traffic control conflict detection task by means of eye movements recording. We manipulated two complexity factors, convergence angle and aircrafts minimum distance at closest approach, in a multidimensional workload assessment method based on psychophysiological, performance, and subjective measures. Conflict trials resulted more complex and time-consuming than no conflicts, requiring more frequent fixations and saccades. Moreover, large saccades showed reduced burst power with higher task complexity. A motion-based and a ratio-based strategy were suggested for conflicts and no conflicts on the basis of ocular metrics analysis: aircrafts differential speed and distance to convergence point at trial start were considered determinant for strategy adoption.Relevance to industryEye metrics measurement for online workload assessment enhances better identification of workload-inducing scenarios and adopted strategy for traffic management. System design, as well as air traffic control operators training programs, might benefit from on line workload measurement.     

Comparison of NASA-TLX scale,modified Cooper–Harper scale and mean inter-beat interval as measures of pilot mental workload during simulated flight tasks

Abstract

The sensitivity of NASA-TLX scale, modified Cooper–Harper (MCH) scale and the mean inter-beat interval (IBI) of successive heart beats, as measures of pilot mental workload (MWL), were evaluated in a flight training device (FTD). Operational F/A-18C pilots flew instrument approaches with varying task loads. Pilots’ performance, subjective MWL ratings and IBI were measured. Based on the pilots’ performance, three performance categories were formed; high-, medium- and low-performance. Values of the subjective rating scales and IBI were compared between categories. It was found that all measures were able to differentiate most task conditions and there was a strong, positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated. While NASA-TLX, MCH and IBI have all been previously used to measure MWL, this study is the first one to investigate their association in a modern FTD, using a realistic flying mission and operational pilots.

Practitioner summary: NASA-TLX scale, MCH scale and the IBI were evaluated in a flight training device. All measures were able to differentiate most task conditions and there was a positive correlation between NASA-TLX and MCH scale. An explicit link between IBI, NASA-TLX, MCH and performance was demonstrated.

Abbreviations: ANOVA: Analysis of Variance; ECG: Electrocardiograph; F/A: fighter/attack; ft: feet; FTD: flight training device; G: Gravity; km: kilometer; m: meter; m/s: meters per second; MWL: mental workload; MCH: modified cooper-harper; NASA-TLX: NASA Task Load Index; NM: Nautical Mile; NN: normal-to-normal; IBI: inter-beat interval; ILS: Instrument Landing System; RR: R-Wave to R-Wave; SD: standard deviation; TTP: tactics, techniques and procedures; WTSAT: Weapon Tactics and Situation Awareness Trainer     

A deep learning scheme for mental workload classification based on restricted Boltzmann machines

The mental workload (MWL) classification is a critical problem for quantitative assessment and analysis of operator functional state in many safety-critical situations with indispensable human–machine cooperation. The MWL can be measured by psychophysiological signals. In this work, we propose a novel restricted Boltzmann machine (RBM) architecture for MWL classification. In relation to this architecture, we examine two main issues: the optimal structure of RBM and selection of the most important EEG channels (electrodes) for MWL classification. The trial-and-error and entropy-based pruning methods are compared for the RBM structure identification. The degree of importance of EEG channels is calculated from the weights in a well-trained network in order to select the most relevant channels for classification task. Extensive comparative results showed that the selected EEG channels lead to accurate MWL classification across subjects.     

Double trade-off curves with different cognitive processing combinations: testing the cancellation axiom of mental workload measurement theory.

The accomplishment model of average mental workload--a formal axiomatic measurement theory--was used as a basis for developing and testing secondary task indices of mental workload (H. A. Colle & G. B. Reid, 1997). Its cancellation axiom implies global sensitivity, which is an important theoretical and practical criterion for mental workload indices. Performance levels of different secondary tasks were empirically equated in mental workload and then used to test the cancellation axiom. Cognitive processing similarity--including orthographic, phonemic, and semantic processing of pairs of operator and secondary tasks--was manipulated in three experiments. Equivalencies between secondary tasks were independent of secondary-operator task similarity, consistent with the cancellation axiom and the global sensitivity of these secondary tasks. The results suggest that standardized secondary task techniques can be developed for the practical measurement of mental workload. Actual or potential applications of this research include the development of functionally useful and realistic secondary task measures of mental workload.     

I spy with my AI: The effects of AI-based visual cueing on human operators’ performance and cognitive load in CCTV control rooms

The increased number of security cameras in modern cities has elevated the video-feed monitoring demands of closed-circuit television (CCTV) operators. As a result, new AI-driven support systems that leverage the power of computer vision algorithms have been deployed to facilitate the operators' work. However, to effectively design intuitive, AI-driven interfaces and validate their impact on the operators' performance, extensive user testing is required. To address this, we previously developed and tested a virtual reality (VR) control room that can be used to iteratively evaluate intelligent computer assistants and interfaces while operators are subjected to different cognitive load. In the present study, we use this VR environment and physiological markers (e.g., eye tracking measures) to investigate how AI-based visual cueing (i.e., pushing forward video streams on which detections are highlighted by rectangles drawn around targets) affects operator performance and cognitive load. Results suggest that support systems using such technology in a control room improve operators’ performance and decrease their cognitive load, as reflected by changes in pupil dilation and subjective reports irrespective of induced cognitive load.     

A neuroergonomic approach to evaluating mental workload in hypermedia interactions

Neuroergonomics could provide on-line methods for measuring mental effort while the operator interacts with hypermedia. We present an experimental study in which 28 participants interacted with a modified version of an existing Spanish e-commerce website in two searching tasks (Goal oriented shopping and Experiential shopping) that demand different amounts of cognitive resources. Mental workload was evaluated multidimensionally, using subjective rating, an interaction index, and eye-related indices. Eye movements and pupil diameter were recorded. The results showed visual scanning behaviour coincided with subjective test scores and performance data in showing a higher information processing load in Goal oriented shopping. However, pupil diameter was able to detect only the variation in user activation during the interaction task, a finding that replicates previous results on the validity of pupil size as an index of arousal. We conclude that a neuroergonomics approach could be a useful method for detecting variations in operators’ attentional states.

Relevance to industry

These results could provide important information for the development of a new attentional screening tool for the prevention of accidents in several application domains.     

A defeasible reasoning framework for human mental workload representation and assessment

Human mental workload (MWL) has gained importance in the last few decades as an important design concept. It is a multifaceted complex construct mainly applied in cognitive sciences and has been defined in many different ways. Although measuring MWL has potential advantages in interaction and interface design, its formalisation as an operational and computational construct has not sufficiently been addressed. This research contributes to the body of knowledge by providing an extensible framework built upon defeasible reasoning, and implemented with argumentation theory (AT), in which MWL can be better defined, measured, analysed, explained and applied in different human–computer interactive contexts. User studies have demonstrated how a particular instance of this framework outperformed state-of-the-art subjective MWL assessment techniques in terms of sensitivity, diagnosticity and validity. This in turn encourages further application of defeasible AT for enhancing the representation of MWL and improving the quality of its assessment.     

Effects of display design on performance in a simulated ship navigation environment

Although automation is playing an increasing role on the ship's bridge, empirical research on the effectiveness of alternative bridge designs is limited. In this paper, we describe an experimental study of the benefits of integrated information display, using a computerized simulation of a highly automated ship's bridge. The study compared three types of interface design, which presented radar and electronic chart information to the operator in different ways: (a) integrated display, (b) functionally-separate display, and (c) spatially-separate display. Effects were examined in relation to time on watch and scenario complexity. Following extensive training on the task, 39 participants were tested over a 4-h experimental session, during which they encountered a sequence of collision scenarios of varying complexity. Using a dual-task methodology, a range of measures of primary and secondary task performance were taken, together with assessment of information sampling behaviour and subjective operator state (workload, fatigue, anxiety and situation awareness). The results indicated slight navigational advantages of the integrated display over the two alternative display types, although it also incurred higher levels of operator cost, particularly fatigue. There were no marked effects of time on watch, but more complex scenarios were associated with impaired performance, increased workload and reduced situation awareness. Overall, the findings have suggested some benefits of integrating primary information sources in a ship's bridge environment. The study further confirms the value of experimental simulations as tools for investigating design issues for ship's bridge automation.     

Evaluation of hydraulic excavator Human–Machine Interface concepts using NASA TLX

This study evaluated newly proposed Human–Machine Interface (HMI) design concepts for improving the ergonomics of hydraulic excavators. The design concepts were based on an augmented interaction technique which involved the use of heads-up display (HUD) and coordinated control as HMI elements. Two alternative HMI designs were elaborated in order to separately evaluate the ergonomic impacts of the head-up display and the coordinated control by comparing them to the standard HMI design. The effectiveness of these three HMI designs in terms of the reduction of the operators' mental and physical workload were assessed by conducting experiments utilizing human subjects, ages 23–35 years. The National Aeronautics and Space Administration's Task Load Index (NASA TLX) method was used for collecting subjective workload scores based on a weighted average of ratings of six factors: Mental Demand, Physical Demand, Temporal Demand, Own Performance, Effort, and Frustration Level. The results showed that the type of HMI design affects different aspects of the operator's workload. Indeed, it showed how the proposed augmented interaction is an effective solution for reducing the ergonomic gaps in terms of mental workload, and to a lesser extent the physical workload, subjected by the standard HMI design.     

Mental workload and cognitive task automaticity: an evaluation of subjective and time estimation metrics

Abstract

The evaluation of mental workload is becoming increasingly important in system design and analysis. The present study examined the structure and assessment of mental workload in performing decision and monitoring tasks by focusing on two mental workload measurements: subjective assessment and time estimation. The task required the assignment of a series of incoming customers to the shortest of three parallel service lines displayed on a computer monitor. The subject was either in charge of the customer assignment (manual mode) or was monitoring an automated system performing the same task (automatic mode). In both cases, the subjects were required to detect the non-optimal assignments that they or the computer had made. Time pressure was manipulated by the experimenter to create fast and slow conditions. The results revealed a multi-dimensional structure of mental workload and a multi-step process of subjective workload assessment. The results also indicated that subjective workload was more influenced by the subject's participatory mode than by the factor of task speed. The time estimation intervals produced while performing the decision and monitoring tasks had significantly greater length and larger variability than those produced while either performing no other tasks or performing a well practised customer assignment task. This result seemed to indicate that time estimation was sensitive to the presence of perceptual/cognitive demands, but not to response related activities to which behavioural automaticity has developed.     

Mobile pupillometry in manual assembly: A pilot study exploring the wearability and external validity of a renowned mental workload lab measure

Human operators in the upcoming Industry 4.0 workplace will face accelerating job demands such as elevated cognitive complexity. Unobtrusive objective measures of mental workload (MWL) are therefore in high demand as indicated by both theory and practice. This pilot study explored the wearability and external validity of pupillometry, a MWL measure robustly validated in laboratory settings and now deployable in work settings demanding operator mobility. In an ecologically valid work environment, 21 participants performed two manual assemblies - one of low and one of high complexity - while wearing eye-tracking glasses for pupil size measurement. Results revealed that the device was perceived as fairly wearable in terms of physical and mental comfort. In terms of validity, no significant differences in mean pupil size were found between the assemblies even though subjective mental workload differed significantly. Exploratory analyses on the pupil size when attending to the assembly instructions only, were inconclusive. The present work suggests that current lab-based procedures might not be adequate yet for in-the-field mobile pupillometry. From a broader perspective, these findings also invite a more nuanced view on the current validity of lab-validated physiological MWL-measures when applied in real-life settings. We therefore conclude with some key insights for future development of mobile pupillometry.