Selection of input features across subjects for classifying crewmember workload using artificial neural networks

The issue of crewmember workload is important in complex system operation because operator overload leads to decreased mission effectiveness. Psychophysiological research on mental workload uses measures such as electroencephalogram (EEG), cardiac, eye-blink, and respiration measures to identify mental workload levels. This paper reports a research effort whose primary objective was to determine if one parsimonious set of salient psychophysiological features can be identified to accurately classify mental workload levels across multiple test subjects performing a multiple task battery. To accomplish this objective, a stepwise multivariate discriminant analysis heuristic and artificial neural network feature selection with a signal-to-noise ratio (SNR) are used. In general, EEG power in the 31-40-Hz frequency range and ocular input features appeared highly salient. The second objective was to assess the feasibility of a single model to classify mental workload across different subjects. A classification accuracy of 87% was obtained for seven independent validation subjects using neural network models trained with data from other subjects. This result provides initial evidence for the potential use of generalized classification models in multitask workload assessment.     

Towards safe and collaborative aerodrome operations: Assessing shared situational awareness for adverse weather detection with EEG-enabled Bayesian neural networks

Teams formulated by aviation professionals are essential in maintaining a safe and efficient aerodrome environment. Nonetheless, the shared situational awareness between the flight crews under adverse weather conditions might be impaired. This research aims to evaluate the impact of a proposed enhancement in communication protocol on cognitive workload and develop a human-centred classification model to identify hazardous meteorological conditions. Thirty groups of subjects completed four post-landing taxiing tasks under two visibility conditions (CAVOK/CAT IIIA) while two different communication protocols (presence/absence of turning direction information) were adopted by the air traffic control officer (ATCOs). Electroencephalography (EEG) and the NASA Task Load Index were respectively used to reflect the pilot’s mental state and to evaluate the pilot’s mental workload subjectively. Results indicated that impaired visibility increases the subjective workload significantly, while the inclusion of turning direction information in the ATCO’s instruction would not significantly intensify their cognitive workload. Mutual information was used to quantitatively assess the shared situational awareness between the pilot flying and the pilot monitoring. Finally, this research proposes a human-centred approach to identify potentially hazardous weather conditions from EEG power spectral densities with Bayesian neural networks (BNN). The classification model has outperformed other baseline algorithms with an accuracy of 66.5%, an F1 score of 61.4%, and an area under the ROC of 0.749. Using the concept of explainable AI with Shapley Additive Explanations (SHAP) values, the exploration of latent mental patterns formulates novel knowledge to gain insights into the vital physiological indicators of the pilots in response to different scenarios from the BNN model. In the long term, the model facilitates the decision regarding the necessity of providing automation and decision-making aids to pilots.     

基于注意力机制的多分支LSTM心理负荷评估模型

充分获取脑电信号的有效特征已成为心理负荷评估亟待解决的问题.提出一种多分支LSTM和注意力机制相结合的多分类网络框架.首先,此网络在对脑电信号做切片处理后,采用多分支LSTM网络提取切片中的时间特征;然后,利用注意力机制对所提取的时间特征进行权重参数优化;最后,通过softmax层输出心理负荷评估结果.通过消融实验和对比实验对模型进行验证.结果 表明,此网络无论在二分类任务还是多分类任务中的表现均优于现有先进网络.     

A deep learning scheme for mental workload classification based on restricted Boltzmann machines

The mental workload (MWL) classification is a critical problem for quantitative assessment and analysis of operator functional state in many safety-critical situations with indispensable human–machine cooperation. The MWL can be measured by psychophysiological signals. In this work, we propose a novel restricted Boltzmann machine (RBM) architecture for MWL classification. In relation to this architecture, we examine two main issues: the optimal structure of RBM and selection of the most important EEG channels (electrodes) for MWL classification. The trial-and-error and entropy-based pruning methods are compared for the RBM structure identification. The degree of importance of EEG channels is calculated from the weights in a well-trained network in order to select the most relevant channels for classification task. Extensive comparative results showed that the selected EEG channels lead to accurate MWL classification across subjects.     

Operator functional state classification using multiple psychophysiological features in an air traffic control task

We studied 2 classifiers to determine their ability to discriminate among 4 levels of mental workload during a simulated air traffic control task using psychophysiological measures. Data from 7 air traffic controllers were used to train and test artificial neural network and stepwise discriminant classifiers. Very high levels of classification accuracy were achieved by both classifiers. When the 2 task difficulty manipulations were tested separately, the percentage correct classifications were between 84% and 88%. Feature reduction using saliency analysis for the artificial neural networks resulted in a mean of 90% correct classification accuracy. Considering the data as a 2-class problem, acceptable load versus overload, resulted in almost perfect classification accuracies, with mean percentage correct of 98%. In applied situations, the most important distinction among operator functional states would be to detect mental overload situations. These results suggest that psychophysiological data are capable of such discriminations with high levels of accuracy. Potential applications of this research include test and evaluation of new and modified systems and adaptive aiding.     

A Method for Classification and Evaluation of Pilot’s Mental States Based on CNN

How to accurately recognize the mental state of pilots is a focus in civil aviation safety. The mental state of pilots is closely related to their cognitive ability in piloting. Whether the cognitive ability meets the standard is related to flight safety. However, the pilot's working state is unique, which increases the difficulty of analyzing the pilot's mental state. In this work, we proposed a Convolutional Neural Network (CNN) that merges attention to classify the mental state of pilots through electroencephalography (EEG). Considering the individual differences in EEG, semi-supervised learning based on improved K-Means is used in the model training to improve the generalization ability of the model. We collected the EEG data of 12 pilot trainees during the simulated flight and compared the method in this paper with other methods on this data. The method in this paper achieved an accuracy of 86.29%, which is better than 4D-aNN and HCNN etc. Negative emotion will increase the probability of fatigue appearing, and emotion recognition is also meaningful during the flight. Then we conducted experiments on the public dataset SEED, and our method achieved an accuracy of 93.68%. In addition, we combine multiple parameters to evaluate the results of the classification network on a more detailed level and propose a corresponding scoring mechanism to display the mental state of the pilots directly.     

Psychophysiological evaluation of seafarers to improve training in maritime virtual simulator

Over the years, safety in maritime industries has been reinforced by many state-of-the-art technologies. However, the accident rate hasn’t dropped significantly with the advanced technology onboard. The main cause of this phenomenon is human errors which drive researchers to study human factors in the maritime domain. One of the key factors that contribute to human performance is their mental states such as cognitive workload and stress. In this paper, we propose and implement an Electroencephalogram (EEG)-based psychophysiological evaluation system to be used in maritime virtual simulators for monitoring, training and assessing the seafarers. The system includes an EEG processing part, visualization part, and an evaluation part. By using the processing part of the system, different brain states including cognitive workload and stress can be identified from the raw EEG data recorded during maritime exercises in the simulator. By using the visualization part, the identified brain states, raw EEG signals, and videos recorded during the maritime exercises can be synchronized and displayed together. By using the evaluation part of the system, an indicative recommendation on “pass”, “retrain”, or “fail” of the seafarers’ performance can be obtained based on the EEG-based cognitive workload and stress recognition. Detailed analysis of the demanding events in the maritime tasks is provided by the system for each seafarer that could be used to improve their training. A case study is presented using the proposed system. EEG data from 4 pilots were recorded when they were performing maritime tasks in the simulator. The data are processed and evaluated. The results show that one pilot gets a “pass” recommendation, one pilot gets a “retrain” recommendation, and the other two get “fail” results regarding their performance in the simulator.     

Investigating pilot performance using mixed-modality simulated data link

OBJECTIVE: General aviation (GA) pilot performance utilizing a mixed-modality simulated data link was objectively evaluated based on the time required in accessing, understanding, and executing data link commands. Additional subjective data were gathered on workload, situation awareness (SA), and preference. BACKGROUND: Research exploring mixed-modality data link integration to the single-pilot GA cockpit is lacking, especially with respect to potential effects on safety. METHODS: Sixteen visual flight rules (VFR)-rated pilots participated in an experiment using a flight simulator equipped with a mixed-modality data link. Data link modalities were text display, synthesized speech, digitized speech, and synthesized speech/text combination. Flight conditions included VFR (unlimited ceiling and visibility) or marginal VFR flight conditions (clouds 2,800 ft above ground level, 3-mile visibility). RESULTS: Statistically significant differences were found in pilot performance, mental workload, and SA across the data link modalities. Textual data link resulted in increased time and workload as compared with the three speech-type data link conditions, which did not differ. SA measures indicated higher performance with textual and digitized speech data link conditions. CONCLUSION: Textual data link can be significantly enhanced for single-pilot GA operations by the addition of a speech component. APPLICATION: Potential applications include operational safety in future GA systems that incorporate data link for use by a single pilot and guidance in the development of flight performance objectives for these systems.     

Monitoring of mental workload levels during an everyday life office-work scenario

Personal and ubiquitous healthcare applications offer new opportunities to prevent long-term health damage due to increased mental workload by continuously monitoring physiological signs related to prolonged high workload and providing just-in-time feedback. In order to achieve a quantification of mental load, different load levels that occur during a workday have to be discriminated. In this work, we present how mental workload levels in everyday life scenarios can be discriminated with data from a mobile ECG logger by incorporating individual calibration measures. We present an experiment design to induce three different levels of mental workload in calibration sessions and to monitor mental workload levels in everyday life scenarios of seven healthy male subjects. Besides the recording of ECG data, we collect subjective ratings of the perceived workload with the NASA Task Load Index (TLX), whereas objective measures are assessed by collecting salivary cortisol. According to the subjective ratings, we show that all participants perceived the induced load levels as intended from the experiment design. The heart rate variability (HRV) features under investigation can be classified into two distinct groups. Features in the first group, representing markers associated with parasympathetic nervous system activity, show a decrease in their values with increased workload. Features in the second group, representing markers associated with sympathetic nervous system activity or predominance, show an increase in their values with increased workload. We employ multiple regression analysis to model the relationship between relevant HRV features and the subjective ratings of NASA-TLX in order to predict the mental workload levels during office-work. The resulting predictions were correct for six out of the seven subjects. In addition, we compare the performance of three classification methods to identify the mental workload level during office-work. The best results were obtained with linear discriminant analysis (LDA) that yielded a correct classification for six out of the seven subjects. The k-nearest neighbor algorithm (k-NN) and the support vector machine (SVM) resulted in a correct classification of the mental workload level during office-work for five out of the seven subjects.     

Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants’ subjective feelings.     

基于改进EEGNet的n-back任务脑电信号识别

在人机交互的过程中,脑力负荷过高是产生操作错误的重要因素,现阶段基于脑电信号具有时间分辨率高和便携性好的特点,常用于脑力负荷的评估.近几年来深度学习的快速发展也使得其广泛应用在脑电领域并取得了比传统的机器学习更加优异的效果, n-back任务可通过设定不同的n值来诱发不同程度的脑力负荷.由此设计了基于视觉和听觉的n-back的范式来避免维度单一,同时还提出一种新的卷积神经网络模型,使用64通道的eego脑电设备采集数据经eeglab预处理后用于该模型的训练.在测试集上与EEGNet, FBCNet, ShallowConNet的性能进行对比,其提出的新模型在分类准确率有较为明显的提升,使得该研究在脑力负荷的评估尤其在多维度n-back任务的分类上具有一定应用潜力.     

Air-to-ground training missions: a psychophysiological workload analysis

Psychophysiological measures are used to assess the workload of F4 Phantom aircraft pilots and weapon systems officers (WSOs) during air-to-ground training missions and during the performance of two levels of difficulty of a laboratory tracking task. The bombing range portion of the missions was associated with the highest pilot workload, while the WSO flying the aircraft was the highest workload segment for the WSOs. The pilots' data were found to have a wider range of values for the physiological measures than were found in the WSO data. The different levels of tracking task difficulty produced significant physiological effects but the range of values found for most of the flight segments were much greater. These data demonstrate that extrapolating laboratory data to the flight environment is risky at best. The various physiological measures were differentially sensitive to the different demands of the various flight segments.     

多特征融合的脑电情绪分类

为进一步探究不同类型特征互补性对脑电情绪分类的影响,提出一种基于多特征融合的脑电情绪分类新方法。对预处理后的脑电信号进行DE、MST和SampEn特征提取,采用双样本T检验去除冗余筛选出最优特征并融合,采用SVM分类模型来识别不同的情绪状态。在SEED-Ⅳ数据集上的实验结果表明,单一特征中DE的平均分类准确率最高（77.86%）,而融合非线性SampEn特征与功能连接MST属性后平均分类准确率得到进一步提升（84.58%）,不同时间段采集的数据上重测实验则证明了该方法的有效性与稳定性。     

Perceived vs. measured effects of advanced cockpit systems on pilot workload and error: Are pilots' beliefs misaligned with reality?

Four types of advanced cockpit systems were tested in an in-flight experiment for their effect on pilot workload and error. Twelve experienced pilots flew conventional cockpit and advanced cockpit versions of the same make and model airplane. In both airplanes, the experimenter dictated selected combinations of cockpit systems for each pilot to use while soliciting subjective workload measures and recording any errors that pilots made. The results indicate that the use of a GPS navigation computer helped reduce workload and errors during some phases of flight but raised them in others. Autopilots helped reduce some aspects of workload in the advanced cockpit airplane but did not appear to reduce workload in the conventional cockpit. Electronic flight and navigation instruments appeared to have no effect on workload or error. Despite this modest showing for advanced cockpit systems, pilots stated an overwhelming preference for using them during all phases of flight.     

Clustering technique-based least square support vector machine for EEG signal classification

This paper presents a new approach called clustering technique-based least square support vector machine (CT-LS-SVM) for the classification of EEG signals. Decision making is performed in two stages. In the first stage, clustering technique (CT) has been used to extract representative features of EEG data. In the second stage, least square support vector machine (LS-SVM) is applied to the extracted features to classify two-class EEG signals. To demonstrate the effectiveness of the proposed method, several experiments have been conducted on three publicly available benchmark databases, one for epileptic EEG data, one for mental imagery tasks EEG data and another one for motor imagery EEG data. Our proposed approach achieves an average sensitivity, specificity and classification accuracy of 94.92%, 93.44% and 94.18%, respectively, for the epileptic EEG data; 83.98%, 84.37% and 84.17% respectively, for the motor imagery EEG data; and 64.61%, 58.77% and 61.69%, respectively, for the mental imagery tasks EEG data. The performance of the CT-LS-SVM algorithm is compared in terms of classification accuracy and execution (running) time with our previous study where simple random sampling with a least square support vector machine (SRS-LS-SVM) was employed for EEG signal classification. We also compare the proposed method with other existing methods in the literature for the three databases. The experimental results show that the proposed algorithm can produce a better classification rate than the previous reported methods and takes much less execution time compared to the SRS-LS-SVM technique. The research findings in this paper indicate that the proposed approach is very efficient for classification of two-class EEG signals.     

Effects of modes of cockpit automation on pilot performance and workload in a next generation flight concept of operation

The objective of this study was to compare the effects of various forms of advanced cockpit automation for flight planning on pilot performance and workload under a futuristic concept of operation. A lab experiment was conducted in which airline pilots flew simulated tailored arrivals to an airport using three modes of automation (MOAs), including a control‐display unit (CDU) to the aircraft flight management system, an enhanced CDU (CDU+), and a continuous descent approach (CDA) tool. The arrival scenario required replanning to avoid convective activity and was constrained by a minimum fuel requirement at the initial approach fix. The CDU and CDU+ modes allowed for point‐by‐point path planning or selection among multiple standard arrivals, respectively. The CDA mode completely automated the route replanning for pilots. It was expected that the higher‐level automation would significantly reduce pilot workload and improve overall flight performance. In general, results indicated that the MOAs influenced pilot performance and workload responses according to hypotheses. This study provides new knowledge about the relationship of cockpit automation and interface features with pilot performance and workload in a novel next generation–style flight concept of operation. © 2012 Wiley Periodicals, Inc.     

External and internal attentional environments I. The utilization of cardiac deceleratory and acceleratory response data for evaluating differences in mental workload between machine-paced and self-paced work ∗

Previous evidence has suggested that self-paced (SP) task performance may constitute a higher mental workload than machine-paced (MP) performance. These differences in mental workload were thought to be due to the presence, when working SP, of an internal pacing mechanism serving to maintain the worker's rhythm. In MP tasks, this function would be maintained externally by the machine. The present investigation attempted to directly test this hypothesis. For this purpose, Lacey's psychophysiological model relating changes in heart rate (HR) to attentional demands was employed. Differences in cardiac deceleratory and acceleratory activity between MP and SP performance were evaluated for each of two tasks. In one task, the emphasis was predominantly on visual detection. Based on the suspected direction of attentional demands, this task was characterized as external. The other task required mental solution to arithmetic problems and was categorized accordingly as internal. Psychophysiological findings were consistent with Lacey's basic model and offered no support for the existence of an internal pacing mechanism under SP conditions. Instead, they suggested the presence of uncertainty factors reflecting higher mental workload during the MP performance of both tasks. Performance data, however, did not support the causal interpretation given by Lacey for his psychophysiological model, and were explained in terms of a complex interplay between HR level and HR change.     

The influence of training level on manual flight in connection to performance,scan pattern,and task load

This work focuses on the analysis of pilots’ performance during manual flight operations in different stages of training and their influence on gaze strategy. The secure and safe operation of air traffic is highly dependent on the individual performances of the pilots. Before becoming a pilot, he/she has to acquire a broad set of skills by training to pass all the necessary qualification and licensing standards. A basic skill for every pilot is manual control operations, which is a closed-loop control process with several cross-coupled variables. Even with increased automation in the cockpit, the manual control operations are essential for every pilot as a last resort in the event of automation failure. A key element in the analysis of manual flight operations is the development over time in relation to performance and visual perception. An experiment with 28 participants (including 11 certified pilots) was conducted in a Boeing 737 simulator. For defined flight phases, the dynamic time warping method was applied to evaluate the performance for selected criteria, and eye-tracking methodology was utilized to analyze the gaze-pattern development. The manipulation of workload and individual experience influences the performance and the gaze pattern at the same time. Findings suggest that the increase of workload has an increased influence on pilots depending on the flight phase. Gaze patterns from experienced pilots provide insights into the training requirements of both novices and experts. The connection between workload, performance and gaze pattern is complex and needs to be analyzed under as many differing conditions. The results imply the necessity to evaluate manual flight operations with respect to more flight phases and a detailed selection of performance indications.

     

Operator functional state classification using least-square support vector machine based recursive feature elimination technique

This paper proposed two psychophysiological-data-driven classification frameworks for operator functional states (OFS) assessment in safety-critical human-machine systems with stable generalization ability. The recursive feature elimination (RFE) and least square support vector machine (LSSVM) are combined and used for binary and multiclass feature selection. Besides typical binary LSSVM classifiers for two-class OFS assessment, two multiclass classifiers based on multiclass LSSVM-RFE and decision directed acyclic graph (DDAG) scheme are developed, one used for recognizing the high mental workload and fatigued state while the other for differentiating overloaded and base-line states from the normal states. Feature selection results have revealed that different dimensions of OFS can be characterized by specific set of psychophysiological features. Performance comparison studies show that reasonable high and stable classification accuracy of both classification frameworks can be achieved if the RFE procedure is properly implemented and utilized.