Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants’ subjective feelings.     

Effects of different physical workload parameters on mental workload and performance

The design and evaluation of an occupational task should include an assessment of mental workload, since excessive levels of mental workload can cause errors or delayed information processing. Physically demanding work that is performed concurrently with a cognitive task may impact mental workload by impairing mental processing or decreasing performance. The primary objective of this study was to determine whether there is a differential effect of various types of physical activity on both mental workload and cognitive performance. Objective and subjective assessment tools (heart rate variability and visual analog scale) were used as indicators of mental workload, while correct responses during an arithmetic task reflected levels of performance. Thirty participants (ages 18-24 years) performed a combination of tasks inducing both physical and mental workload. Type of physical effort, frequency of movement, and force exertion level were manipulated to alter the workload associated with the physical activity. Changes in subjective ratings generally corresponded to changes in both performance on the arithmetic task and objective mental workload assessment. Some discrepancies occurred at the highest physical force exertion level as participants perceived an increase in effort to maintain the same level of performance. Further research is needed to determine the force exertion threshold, beyond which the physical effort required interferes with mental workload and/or cognitive performance.

Relevance to industry

Technological advancements have increased the requirement for many workers to execute cognitive tasks concurrently with physical activity. When designing and evaluating such situations it is important to determine the interactive effects of these activities. A simple, uni-dimensional tool is suggested as a screening tool to identify situations requiring excessive or increased mental workload that many degrade performance or place additional stress on the individual.     

Dynamic analysis of dorsal thermal images

A dynamic analysis was performed on thermal images of the dorsum of the foot. The psychophysiological effects of a facial massage by an esthetician were evaluated. First, the psychophysiological effects of facial massage were assessed using a proprietary stress test. The physiological indices measured were the α-wave power spectrum, the dorsal skin temperature variations, and the high frequency (HF) component of the heart-rate variability (HRV). A state-trait anxiety inventory (STAI), a profile of mood states (POMS), and the level of sensory awareness were administered to evaluate the psychological status. The aspects of the level of sensory awareness examined were comfort, awakening, and the effects of massage. We then assessed the stress responses according to thermal imaging of the dorsal side of the foot using infrared thermography.     

Mental workload measurement for emergency operating procedures in digital nuclear power plants

Mental workload is a major consideration for the design of emergency operation procedures (EOPs) in nuclear power plants. Continuous and objective measures are desired. This paper compares seven mental workload measurement methods (pupil size, blink rate, blink duration, heart rate variability, parasympathetic/sympathetic ratio, total power and (Goals, Operations, Methods, and Section Rules)-(Keystroke Level Model) GOMS-KLM-based workload index) with regard to sensitivity, validity and intrusiveness. Eighteen participants performed two computerised EOPs of different complexity levels, and mental workload measures were collected during the experiment. The results show that the blink rate is sensitive to both the difference in the overall task complexity and changes in peak complexity within EOPs, that the error rate is sensitive to the level of arousal and correlate to the step error rate and that blink duration increases over the task period in both low and high complexity EOPs. Cardiac measures were able to distinguish tasks with different overall complexity. The intrusiveness of the physiological instruments is acceptable. Finally, the six physiological measures were integrated using group method of data handling to predict perceived overall mental workload.

Practitioner Summary: The study compared seven measures for evaluating the mental workload with emergency operation procedure in nuclear power plants. An experiment with simulated procedures was carried out, and the results show that eye response measures are useful for assessing temporal changes of workload whereas cardiac measures are useful for evaluating the overall workload.     

Double trade-off curves with different cognitive processing combinations: testing the cancellation axiom of mental workload measurement theory.

The accomplishment model of average mental workload--a formal axiomatic measurement theory--was used as a basis for developing and testing secondary task indices of mental workload (H. A. Colle & G. B. Reid, 1997). Its cancellation axiom implies global sensitivity, which is an important theoretical and practical criterion for mental workload indices. Performance levels of different secondary tasks were empirically equated in mental workload and then used to test the cancellation axiom. Cognitive processing similarity--including orthographic, phonemic, and semantic processing of pairs of operator and secondary tasks--was manipulated in three experiments. Equivalencies between secondary tasks were independent of secondary-operator task similarity, consistent with the cancellation axiom and the global sensitivity of these secondary tasks. The results suggest that standardized secondary task techniques can be developed for the practical measurement of mental workload. Actual or potential applications of this research include the development of functionally useful and realistic secondary task measures of mental workload.     

Evaluation and prediction mental workload in user interface of maritime operations using eye response

Eye response measurement is one of the objective measure methods and useful for assessing of operators' mental workload (MWL). The main objectives of this paper are to consider the relationship between operators' MWL and eye responses in the task of operating marine engine interface. Also, an artificial neural network (ANN) model was developed to predict the operators' MWL based on integrating eye response data. Eye response indices (pupil dilation, blink rate, fixation rate, and saccadic rate) were recorded, and two subjective rating methods (The National Aeronautics and Space Administration's Task Load Index [NASA-TLX] and subjective workload assessment technique [SWAT]) were used for 27 participants. The results again confirm that the eye response is sensitive to MWL in workload levels of the task when using the interface control. The ANN model developed by measuring these indices can predict the operators' MWL with the determination coefficient (R²) of 0.971, 0.912 and 0.918 for training, validation, and testing, respectively. These results indicated that the ANN approach is quite accurate for the prediction of operators' MWL based on eye response indices.Relevance to industryThe developed model is expected to provide the operator with a reference value of their MWL by evaluating their physiological indices. This result might be applied for developing an intelligent prediction model in the actual work environment to inform or support the operator in a variety of ways. From this, the manager can organize the human resources for each task to sustain the appropriate MWL as well as to improve the work performance.     

Learning rate and subjective mental workload in five truck driving tasks

Both learning curve models and subjective mental workload are useful tools for determining the length of training for new workers and predicting future task performance. An experiment was designed to collect the task completion times and subjective mental workload of five driving tasks including (a) reverse into garage, (b) 3-point turn, (c) parallel parking, (d) S-curve and (e) up-down-hill. The results indicated that task completion times of truck driving can be predicted with a learning curve. Practice significantly reduced the mental workload rating. However, the novice trainees tended to have a more significant reduction because, compared to experienced trainees, they tended to give greater or lower workload scores than the experienced trainees before and after practice, respectively. The current study may not be complete enough to provide guidelines for a training programme, but it is adequate to suggest that learning rate and workload measure can serve as indexes for factoring in the individual differences.

Practitioner summary: Learning curves can be used to determine the length of training for new workers and performance standards for a particular task. Learning rate and mental workload were found to be important measures for comparing individual differences in order to better design a training programme. However, mental workload must be evaluated by experienced participants.     

External and internal attentional environments I. The utilization of cardiac deceleratory and acceleratory response data for evaluating differences in mental workload between machine-paced and self-paced work ∗

Previous evidence has suggested that self-paced (SP) task performance may constitute a higher mental workload than machine-paced (MP) performance. These differences in mental workload were thought to be due to the presence, when working SP, of an internal pacing mechanism serving to maintain the worker's rhythm. In MP tasks, this function would be maintained externally by the machine. The present investigation attempted to directly test this hypothesis. For this purpose, Lacey's psychophysiological model relating changes in heart rate (HR) to attentional demands was employed. Differences in cardiac deceleratory and acceleratory activity between MP and SP performance were evaluated for each of two tasks. In one task, the emphasis was predominantly on visual detection. Based on the suspected direction of attentional demands, this task was characterized as external. The other task required mental solution to arithmetic problems and was categorized accordingly as internal. Psychophysiological findings were consistent with Lacey's basic model and offered no support for the existence of an internal pacing mechanism under SP conditions. Instead, they suggested the presence of uncertainty factors reflecting higher mental workload during the MP performance of both tasks. Performance data, however, did not support the causal interpretation given by Lacey for his psychophysiological model, and were explained in terms of a complex interplay between HR level and HR change.     

Utilizing slope method as an alternative data analysis for functional near-infrared spectroscopy-derived cerebral hemodynamic responses

The purpose was to propose an alternative data analysis for functional near-infrared spectroscopy (fNIRS)-derived hemodynamics as a function of cortical activation changes. We evaluated hemodynamic responses from the prefrontal cortex region while 38 participants performed a cognitive task. The task consisted of an arithmetic calculation with three levels of complexity (i.e., easy, medium and difficult). These task-dependent hemodynamic responses were analyzed by the slope method (i.e. using a linear regression through the cognitive task) and were compared with the corresponding responses obtained with a traditional approach of the amplitude method. Subjective scales of task loading (assessed by DP15 and NASA-TLX) and behavioral outcomes (performance and reaction time) were also recorded in response to the task complexity. Results revealed that the proposed slope method allowed a better discrimination in terms of cortical activation among all levels of mental workload. There was no significant increase in cortical activation between the medium and difficult levels (p = .30; d = 0.09) with the amplitude method while the slope coefficient was sensitive to the different levels (p < .01; d = 0.32). These preliminary results from a large sample size demonstrated that the slope method appears suitable for discriminating the changes in cortical activation with respect to the mental workload.Relevance to industryIn this work, we proposed an optimum way of quantifying the mental workload of participants in terms of fNIRS-derived cerebral hemodynamic responses.     

Reality-based interaction affecting mental workload in virtual reality mental arithmetic training

ABSTRACT

The concept of digital game-based learning (DGBL) evolves rapidly together with technological enhancements of virtual reality (VR) and smart phones. However, the mental workload (MWL) that VR-training applications demand and motivational qualities originating from user experience (UX) should be identified in order to create effective and enjoyable training/learning challenges that fit with individual users’ capabilities. This study examined the effects of reality-based interaction (RBI) and VR on measures of student motivation and MWL, in a mental arithmetic game for secondary school pupils. In a randomised controlled trial with sixty school children, a mental arithmetic game was tested with three different interaction and two different presentation methods – VR RBI, VR head-mounted-display tapping and tablet flick-gesture. Results found a significant effect of RBI on MWL but no differences in enjoyment of training were found between VR-experience and tablet training-experience. In fact, adding the gaming-context to the mental arithmetic task created an enjoyable, motivating experience regardless of presentation or interaction-style.     

Measuring Mental Workload Using ERPs Based on FIR,ICA, and MARA

Mental workload is considered to be strongly linked to human performance, and the ability to measure it accurately is key for balancing human health and work. In this study, brain signals were elicited by mental arithmetic tasks of varying difficulty to stimulate different levels of mental workload. In addition, a finite impulse response (FIR) filter, independent component analysis (ICA), and multiple artifact rejection algorithms (MARAs) were used to filter event-related potentials (ERPs). Then, the data consisting of ERPs, subjective ratings of mental workload, and task performance, were analyzed through the use of variance and Spearman’s correlation during a simulated computer task. We found that participants responded faster and performed better in the easy task condition, followed by the medium and high-difficulty conditions, which verifies the validity of the ERP filtering. Moreover, larger P2 and P3 waveforms were evoked as the task difficulty increased, and a higher task difficulty elicited a more enhanced N300. Correlation analysis revealed a negative relationship between the amplitude of P3 and the subjective ratings, and a positive relationship between the P3 amplitude and accuracy. The results presented in this paper demonstrate that a combination of FIR, ICA, and MARA methods can filter ERPs in the non-invasive real-time measurement of workload. Additionally, frontocentral P2, N3, and parietal P3 components showed differences between genders. The proposed measurement of mental workload can be useful for real-time identification of mental states and can be applied to human–computer interaction in the future.     

Effects of three‐dimensional virtual reality and traditional training methods on mental workload and training performance

Industries will implement effective training programs to improve training performance, and an ideal training performance occurs under proper mental workload (MWL). Virtual reality (VR) has recently been widely utilized in training; however, only a few studies have investigated its effects on MWL and training performance simultaneously. The purpose of this study is to investigate the effects of VR training and traditional training methods, such as technical manuals (TM) and multimedia films (MF), on training performance and MWL. The results of the performance measurement show that VR training is considered the best training method compared to TM and MF, particularly in the case of complex tasks. The results of physiological measurements (GSR [galvanic skin response], LF% [low frequency], and LF/HF [high frequency] ratio) show a significant difference between reading TM and using computer (MF and VR), wherein the latter has a lower MWL. However, no significant difference in subjective MWL assessment (NASA‐TLX [task load index]) and HF% measurement is found.     

Evaluation and prediction of on‐line maintenance workload in nuclear power plants

This study evaluates engineers' mental workload while maintaining digital systems in nuclear power plants (NPPs). First, according to the factors affecting the mental workload, a questionnaire was designed to evaluate the mental workload of maintenance engineers at the Second NPP in Taiwan. Then 16 maintenance engineers from the Second NPP participated in the experiment survey. The results indicated that the mental workload was lower in maintaining digital systems than that in analog systems. Finally, a mental workload model based on the neural network technique was established to predict the mental workload of maintenance engineers in maintaining digital systems. Through predicting mental workload, the manager can organize the human resources for each daily task to sustain the appropriate mental workload as well as improve maintenance performance. © 2008 Wiley Periodicals, Inc.     

Effects of psychosocial and individual factors on physiological risk factors for upper extremity musculoskeletal disorders while typing

Psychosocial factors are hypothesized to contribute to work-related musculoskeletal disorder (WMSD) development, although previous research has been largely epidemiological or has focused primarily on the shoulders, back and neck. The objective of this study was to quantify the effects of mental workload and time pressure on perceived workload and physiological responses of the distal upper extremity. A total of 18 typists completed nine 5-min typing sessions representing three levels of time pressure and mental workload. Levels were manipulated by adjusting typing speed and by requiring participants to perform arithmetic tasks while typing. Outcomes were measured in muscle activation levels, wrist postures and movements, key strike force and subjective assessments of workload. In general, increased time pressure increased muscle activation, key strike force and wrist deviations; and increased mental workload increased key strike force. Mental workload and time pressure mediated physical risk factors during typing to increase WMSD risk for the distal upper extremity.     

Effects of psychosocial and individual factors on physiological risk factors for upper extremity musculoskeletal disorders while typing

Psychosocial factors are hypothesized to contribute to work-related musculoskeletal disorder (WMSD) development, although previous research has been largely epidemiological or has focused primarily on the shoulders, back and neck. The objective of this study was to quantify the effects of mental workload and time pressure on perceived workload and physiological responses of the distal upper extremity. A total of 18 typists completed nine 5-min typing sessions representing three levels of time pressure and mental workload. Levels were manipulated by adjusting typing speed and by requiring participants to perform arithmetic tasks while typing. Outcomes were measured in muscle activation levels, wrist postures and movements, key strike force and subjective assessments of workload. In general, increased time pressure increased muscle activation, key strike force and wrist deviations; and increased mental workload increased key strike force. Mental workload and time pressure mediated physical risk factors during typing to increase WMSD risk for the distal upper extremity.     

Air-to-ground training missions: a psychophysiological workload analysis

Psychophysiological measures are used to assess the workload of F4 Phantom aircraft pilots and weapon systems officers (WSOs) during air-to-ground training missions and during the performance of two levels of difficulty of a laboratory tracking task. The bombing range portion of the missions was associated with the highest pilot workload, while the WSO flying the aircraft was the highest workload segment for the WSOs. The pilots' data were found to have a wider range of values for the physiological measures than were found in the WSO data. The different levels of tracking task difficulty produced significant physiological effects but the range of values found for most of the flight segments were much greater. These data demonstrate that extrapolating laboratory data to the flight environment is risky at best. The various physiological measures were differentially sensitive to the different demands of the various flight segments.     

Muscle- and task-dependent responses to concurrent physical and mental workload during intermittent static work

Many workers experience combined physical and mental demands in their jobs, yet the contribution of these demands to the development of musculoskeletal disorders is unclear. The purpose of this study was to investigate muscle- and task-dependent responses to concurrent demands during intermittent static work. Twenty-four participants performed shoulder, wrist, and torso exertions at three levels of physical workload (PWL) in the absence (control) and presence (concurrent) of a mental arithmetic task. Compared to the control, concurrent demand conditions resulted in decreased muscle activity (4–9% decrease), increased cardiovascular load (2–4% increase), and impaired motor co-ordination (9–24% increase in force fluctuation). Furthermore, these outcomes were more prominent at higher PWL levels and within postural (shoulder and torso) muscles. Mental task performance exhibited greater interference with the physical task at low and high PWL levels. Thus, it may be important to consider these muscle- and task-specific interactions of concurrent demands during job design to address worker health and performance issues.

Practitioner Summary: Occupational tasks place both physical and mental demands on workers. These demands can adversely affect physiological responses and performance, and are muscle- and task-dependent. Findings from this research may facilitate the development of ergonomics interventions, such as task redesign and tool/workstation design, that may help reduce risk of workplace injuries.     

Assessment and monitoring of mental workload in subway train operations using physiological,subjective, and performance measures

Subway train operation is a complex, sociotechnical system that involves a variety of cognitively demanding tasks. The train operators are responsible for continuously monitoring the surrounding environment, maintaining awareness, processing information, and making decisions under risk. The resulting mental strain on operators can negatively affect their performance and the interaction of the human–machine system. The objective of this study was to evaluate if physiological, subjective, and performance measures could identify the level of mental workloads arising from routine and nonroutine operations in the subway system. A total of 11 subway train operators underwent different driving scenarios in a high‐fidelity simulator. The simulated tasks were divided into two categories: routine operations (preparing to drive and driving between stations without interruptions or emergencies) and nonroutine operation (responding to a tunnel fire, dealing with a high density of passengers, encountering a passenger/technician on the track, and responding to train failure). The mental workload was monitored and evaluated in these tasks using an electrocardiogram, subjective self‐rating scales, and driving performance. Both heart rate variability and performance measures (including reaction time and error rate) detected mental workload variations in the different operations. On the other hand, the subjective ratings (including NASA‐TLX) assessed the overall mental workload associated with a task, without explaining the mental demand variations within the task over time. Subway train drivers experienced different levels of mental workload during routine and nonroutine driving conditions. The findings of this study can be used to extract mental workload limits to optimize workload levels during train operations.     

Physical,mental, emotional,and subjective workload components in train drivers

This study, using 12 train drivers on a high speed track and 11 drivers on a mountain track, tried to differentiate between the physical, emotional, mental, and subjective workload components imposed on the drivers during work. With the simultaneous recording and on-line analysis of heart rate and physical activity, the emotional component in terms of the so-called additional heart rate was separated from the physical component. Mental workload was calculated by the heart rate variability and by shifts in the T-wave amplitude of the ECG. Speed of the train, mode of driving, and stress of the situation were rated by two observers who accompanied the drivers in the cabin. During speeds up to l00km/h as compared to standstills no heart rate changes occurred, but with speeds from l00km/h up to 200 km/h heart rate decreased indicating a monotony effect. However, heart rate variability, and T-wave amplitude indicated higher mental load during driving in most speed categories. Starting the train and coming to a halt showed greater emotional workload as compared to moving. Observer ratings of stress and subjective ratings of stress by the drivers revealed several discrepancies. Discrepancies were also seen between workload as indicated by the physiological parameters, and corresponding stress ratings by the observers or by the drivers.