Situational awareness ability and cognitive skills training in a complex real-world task

Successful performance in complex dynamic environments depends on domain-dependent factors, such as situational awareness (SA). Underlying SA in a domain are domain-independent cognitive abilities in perception, memory, attention and executive control. Individuals with lower underlying ability perform relatively poorly in complex dynamic real-world tasks. The first experiment examined whether cognitive skills training could overcome limitations in underlying SA ability that impact on complex dynamic task performance. Participants were taught a mix of cognitive management strategies (e.g. divided and focused attention and visual search) in a simulated air traffic control task. A second experiment investigated the link between underlying SA ability, TRACON and SAGAT, a widely used measure of domain-specific SA. In a third experiment, the focus was on encouraging participants to plan ahead and consider the interrelations of elements (aircraft) in the environment. Whilst both training methods ameliorated the negative impact that lower SA ability had on complex dynamic task performance, the results of the third study indicated that this may have been achieved through improved planning behaviour. Finally, participants with higher underlying SA ability performed well irrespective of training condition.     

Predictive capability of cognitive ability and cognitive style for spaceflight emergency operation performance

This study explores the effects of cognitive ability (information seeking, inference, spatial recognition, attention span, and attention allocation) and cognitive style (active-reflective, sensing-intuitive, visual-verbal, and sequential-global) on task performance of simulated spaceflight emergency operations that require judgment and operation on a Chinese spaceflight instrument board and the possible interaction effect with training experience. The performance criteria included task completion time and number of human errors. It was found that inference ability, spatial recognition ability, and attention span had significant effects on task completion time, while attention allocation ability had significant effect on the number of error. The participants with a sequential cognitive style made significantly fewer errors than those with a global cognitive style. Training experience significantly decreased task completion time. The participants with sequential cognitive style learnt faster than those with global cognitive style in the spaceflight instrument operations. With increasing training experience, the predictive capability of cognitive ability on performance decreased, whereas the predictive capability of the sequential-global cognitive style on performance increased.     

Situation awareness and attention allocation measures for quantifying telepresence experiences in teleoperation

This study assessed the utility of measures of situation awareness (SA) and attention allocation for quantifying telepresence, the sense of being present at a remote site, in a teleoperation task scenario. Attention and SA have been identified as cognitive constructs potentially underlying telepresence. The motivation for this research was to establish an objective measure of telepresence and investigate the relationship between telepresence and teleoperation performance. Twenty‐four research participants performed a virtual ordnance disposal task at varying levels of difficulty (LODs). The task involved locating, identifying, and disposing of virtual land mines in an outdoor environment using a simulated remote‐control rover with a robotic arm. Performance, SA, and attention allocation were recorded along with subjective assessments of telepresence. Results demonstrated LOD effects on performance and telepresence. Regression analysis revealed LOD and attention to explain significant portions of the variance in telepresence. Results of the study provide further evidence that telepresence may share a relationship with performance, and that cognitive constructs, such as attention and SA, may serve as alternative, objective measures of telepresence. © 2004 Wiley Periodicals, Inc. Hum Factors Man 14: 51–67, 2004.     

Does situation awareness add to the validity of cognitive tests?

OBJECTIVE: Does adding situation awareness (SA) to a battery of cognitive tests improve prediction? BACKGROUND: Identifying variables that predict skilled performance in a complex task aids in understanding the nature of skill and also aids in the selection of operators to perform that task. SA is thought to be an important predictor of performance. SA is often thought to be based on underlying cognitive mechanisms. METHOD: Three performance measures taken from the Federal Aviation Administration's (FAA's) Air Traffic Scenarios Test, the low-fidelity simulation component of the FAA's controller selection battery, were used as criterion variables in a hierarchical regression. After predicting performance based on a battery of cognitive (e.g., intelligence, working memory, spatial memory) and noncognitive tests (e.g., cognitive style, personality, demographics), we added measures of SA. RESULTS: SA did provide increases in prediction, but only when measured with the Situation Present Assessment Method, an on-line query method. When the same questions were asked off line, SA did not enter the model in two cases and improved prediction by only 2% in the third. CONCLUSION: Thus, some measures of SA do show incremental validity, even against a backdrop of a large number of cognitive variables. APPLICATION: On-line measures of SA can be a worthwhile addition to standard batteries of tests used to predict performance in cognitively oriented industrial tasks.     

The predictive qualities of operator characteristics for process control performance: the influence of personality and cognitive variables

This article examines the relationship between operator characteristics and process control performance. Thirty-nine trainee operators participated in a 4-h training session of a simulated process control task and a testing session in which various system faults had to be managed. Cognitive ability, cognitive flexibility, self-efficacy and personality traits were measured as operator characteristics. Cognitive ability related positively to system control performance but not to diagnostic performance. Participants with low cognitive flexibility performed best on system control, whereas participants with high cognitive flexibility performed best on diagnostic performance. A hierarchical regression analysis revealed that cognitive ability, cognitive flexibility and declarative knowledge accounted for about 30% of the variability of system control. The findings suggest that consideration of cognitive ability and cognitive flexibility be increased in personnel selection for complex work environments.     

Enhancing human indoor cognitive map development and wayfinding performance with immersive augmented reality-based navigation systems

Augmented reality (AR) is an interactive experience where computer-generated perceptual information is superimposed into the real-world environment. Most existing research in AR-based wayfinding has focused on the technological aspects of developing AR-based software or devices to realize navigation. No previous investigations have focused on understanding the impact of immersive augmented reality (IAR)–based systems on human wayfinding performance from the cognitive perspective. Aimed at investigating the influence of IAR-based systems on people’s cognitive map development and their subsequent wayfinding performance as well as the effect of using three-dimensional (3D) layout models in IAR environments in addition to superimposed guideposts, an experiment was carried out in a building with a complex floor plan. A total of 54 university students were evenly divided into three groups: a control group with no IAR assistance, a second group using an IAR-based navigation system that includes only superimposed guideposts, and a third group using an IAR-based navigation system that includes both guideposts and a 3D layout model. Each participant was asked to conduct a spatial exploration task in the environment, sketch a floor map based on their spatial cognition, and perform a wayfinding task to find eight specific locations in the building. An analysis of the participants’ performance and responses to a number of self-evaluation questionnaires collected in the experiment indicates that IAR technology can help people develop their cognitive maps more effectively and can substantially improve their wayfinding performance with a much lower workload. A second finding is that adding a 3D layout model can enhance the effect of an IAR-based navigation system in terms of cognitive map development. The findings from this research extend the existing knowledge about IAR-based navigation and further verify that AR technology has the potential to reduce human workload for cognitive tasks. The results also could support its more effective application in various scenarios that require assisted wayfinding and cognitive map training, such as emergency evacuation drills.     

The relationship between monochronicity,polychronicity and individual characteristics

With the increasing complexity of control rooms and the information explosion, effective multitasking is now desired. Monochronicity and polychronicity, which describe a person's ability to do one thing and many things at a time, respectively, have been studied for a long time. However, it is not clear these abilities are related to various individual characteristics. Forty-eight Chinese participants were tested on their perception, memory, judgement, attention ability and cognitive style. They also performed a task that required search and calculation under three conditions of unpaced, paced and paced with sequencing. There were significant differences in the performance and strategy between monochronic and polychronic individuals in the selective attention test. Monochronic individuals focused their attention on the primary task and achieved higher performance. Polychronic individuals had somewhat better total performance in more than one task under time-constrained conditions. The results clearly indicate that an individual's time use behaviours ought to be considered in training and control scenarios to account for differences among people.     

Cognitive diversity and team performance in a complex multiple task environment

This article examines the multiple effects of cognitive diversity in teams operating complex human-machine-systems. The study employed a PC-based multiple-task environment, called the Cabin Air Management System, which models a process control task in the operational context of a spacecraft's life support system. Two types of cognitive diversity were examined: system understanding and team specialization. System understanding referred to the depth of understanding team members were given during training (low-level procedure-oriented vs. high level knowledge-oriented training). Team specialization referred to the degree to which knowledge about system fault scenarios was distributed between team members (specialized vs. non-specialized). A total of 72 participants took part in the study. After having received 4.5 h of training on an individual basis, participants completed a 1-h experimental session, in which they worked in two-person teams on a series of fault scenarios of varying difficulty. Measures were taken of primary and secondary task performance, system intervention and information sampling strategies, system knowledge, subjective operator state, communication patterns and conflict. The results provided evidence for the benefits of cognitive diversity with regard to system understanding. This manifested itself in better primary task performance and more efficient manual system control. No advantages were found for cognitive diversity with regard to specialization. There was no effect of cognitive diversity on intra-team conflict, with conflict levels generally being very low. The article concludes with a discussion of the implications of the findings for the engineering of cognitive diversity in teams operating complex human-machine-systems.     

Cognitive diversity and team performance in a complex multiple task environment

This article examines the multiple effects of cognitive diversity in teams operating complex human-machine-systems. The study employed a PC-based multiple-task environment, called the Cabin Air Management System, which models a process control task in the operational context of a spacecraft's life support system. Two types of cognitive diversity were examined: system understanding and team specialization. System understanding referred to the depth of understanding team members were given during training (low-level procedure-oriented vs. high level knowledge-oriented training). Team specialization referred to the degree to which knowledge about system fault scenarios was distributed between team members (specialized vs. non-specialized). A total of 72 participants took part in the study. After having received 4.5 h of training on an individual basis, participants completed a 1-h experimental session, in which they worked in two-person teams on a series of fault scenarios of varying difficulty. Measures were taken of primary and secondary task performance, system intervention and information sampling strategies, system knowledge, subjective operator state, communication patterns and conflict. The results provided evidence for the benefits of cognitive diversity with regard to system understanding. This manifested itself in better primary task performance and more efficient manual system control. No advantages were found for cognitive diversity with regard to specialization. There was no effect of cognitive diversity on intra-team conflict, with conflict levels generally being very low. The article concludes with a discussion of the implications of the findings for the engineering of cognitive diversity in teams operating complex human-machine-systems.     

The effect of auditory distraction on the useful field of view in hearing impaired individuals and its implications for driving

This study assessed whether the increased demand of listening in hearing impaired individuals exacerbates the detrimental impact of auditory distraction on a visual task (useful field of view test), relative to normally hearing listeners. Auditory distraction negatively affects this visual task, which is linked with various driving performance outcomes. Hearing impaired and normally hearing participants performed useful field of view testing with and without a simultaneous listening task. They also undertook a cognitive test battery. For all participants, performing the visual and auditory tasks together reduced performance on each respective test. For a number of subtests, hearing impaired participants showed poorer visual task performance, though not to a statistically significant extent. Hearing impaired participants were significantly poorer at a reading span task than normally hearing participants and tended to score lower on the most visually complex subtest of the visual task in the absence of auditory task engagement. Useful field of view performance is negatively affected by auditory distraction, and hearing loss may present further problems, given the reductions in visual and cognitive task performance suggested in this study. Suggestions are made for future work to extend this study, given the practical importance of the findings.     

How Does Software Visualization Contribute to Software Comprehension? A Grounded Theory Approach

Despite their ability to synthesize vast amounts of information, software visualization tools are not widely adopted in the software engineering industry. In an effort to investigate the underlying reasons, we conducted a usability study to investigate the affordances of software visualization techniques for the maintenance of complex software systems. Expert programmers were asked to carry out programming tasks with or without using a software visualization tool while their screens and eye gaze patterns were recorded. Statistical analysis of task performance data showed that participants who used the software visualization tool outperformed the control group in terms of task completion time and accuracy. However, quantitative analysis of performance measures did not reveal in what ways software visualizations contributed to this improvement. In an effort to identify the cognitive strategies that underlie this quantitative performance difference, process models grounded in qualitative analysis of eye-tracking data were constructed. The process models indicated that software visualizations guided the subjects in the experiment group toward following specific software comprehension strategies, which account for the difference observed in task performance data.     

Assessing situation awareness in multitasking supervisory control using success rate of self-terminating search

The purpose of this study is to investigate measuring situation awareness (SA) by using the success rate of self-terminating visual search (STVS) in a multitasking supervisory control environment. Currently, many researchers have focused on developing unobtrusive SA metrics in a multitasking environment. Although some of the advanced eye-tracking metrics can estimate the operator's SA during a task, certain methodological issues remain for measuring SA in a supervisory control and multitasking environment. In this study, participants performed a distributed control task with the multi-attribute task battery. To measure the participant's SA, situation awareness global assessment technique (SAGAT) was used during the experiment. SA performance was compared to the success rate of STVS. An eye-head integrated tracking system was used to measure the rate of STVS based on the participants' eye movements and head positions in different monitoring conditions. The results showed that task-switching activity was influenced by salient features in a display. It causes a significant difference on SA between different monitoring conditions, and the difference is also monitored in the success rate of STVS. Furthermore, the study found that the attention level in visual searching influences situational awareness in a supervisory control environment. The findings from this study suggest that the success rate of STVS could be used as a new metric for measuring SA related to the visual perception of the elements in a multitasking supervisory control environment.     

Comparative study of three training methods for enhancing process control performance: Emphasis shift training,situation awareness training,and drill and practice

Three training methods to improve attention management skills in process control were compared. Forty students from technical disciplines participated in a five-hour module of emphasis shift training (EST), EST combined with situation awareness training (EST/SA), and drill and practice (D&P) on a simulated process control task. Participants were then tested three times for 45 min each (immediately after training, two weeks after training, and six weeks after training) for system control performance and diagnostic performance on familiar and nonfamiliar fault states. D&P led to superior diagnostic performance on familiar system faults. EST/SA training supported the diagnosis of novel system faults. EST was less effective than expected for system control performance. Implications for training design in process control are discussed.     

Effects of training on short- and long-term skill retention in a complex multiple-task environment

The paper reports the results of an experiment on the performance and retention of a complex task. This was a computer-based simulation of the essential elements of a spacecraft's life support system. It allowed the authors to take a range of measures, including primary and secondary task performance, system intervention and information sampling strategies, mental model structure, and subjective operator state. The study compared the effectiveness of two methods of training, based on low level (procedure-based) and high level (system-based) understanding. Twenty-five participants were trained extensively on the task, then given a 1-h testing session. A second testing session was carried out 8 months after the first (with no intervening practice) with 17 of the original participants. While training had little effect on control performance, there were considerable effects on system management strategies, as well as in structure of operator's mental model. In the second testing session, the anticipated general performance decrement did not occur, though for complex faults there was an increase in selectivity towards the primary control task. The relevance of the findings for training and skill retention in real work environments is discussed in the context of a model of compensatory control.     

Individual differences in response to cognitive training: Using a multi-modal,attentionally demanding game-based intervention for older adults

The effectiveness of a game-based cognitive training intervention on multiple abilities was assessed in a sample of 39 older adults aged 60–77. The intervention task was chosen based on a cognitive task analysis designed to determine the attentional and multi-modal demands of the game. Improvements on a measure of attention were found for the intervention group compared to controls. Furthermore, for the intervention group only, initial ability scores predicted improvements on both tests of attention and spatial orientation. These results suggest cognitive training may be more effective for those initially lower in ability.     

Effects of training on short- and long-term skill retention in a complex multiple-task environment

The paper reports the results of an experiment on the performance and retention of a complex task. This was a computer-based simulation of the essential elements of a spacecraft's life support system. It allowed the authors to take a range of measures, including primary and secondary task performance, system intervention and information sampling strategies, mental model structure, and subjective operator state. The study compared the effectiveness of two methods of training, based on low level (procedure-based) and high level (system-based) understanding. Twenty-five participants were trained extensively on the task, then given a 1-h testing session. A second testing session was carried out 8 months after the first (with no intervening practice) with 17 of the original participants. While training had little effect on control performance, there were considerable effects on system management strategies, as well as in structure of operator's mental model. In the second testing session, the anticipated general performance decrement did not occur, though for complex faults there was an increase in selectivity towards the primary control task. The relevance of the findings for training and skill retention in real work environments is discussed in the context of a model of compensatory control.     

Situation awareness and driving performance in a simulated navigation task

The objective of this study was to identify task and vehicle factors that may affect driver situation awareness (SA) and its relationship to performance, particularly in strategic (navigation) tasks. An experiment was conducted to assess the effects of in-vehicle navigation aids and reliability on driver SA and performance in a simulated navigation task. A total of 20 participants drove a virtual car and navigated a large virtual suburb. They were required to follow traffic signs and navigation directions from either a human aid via a mobile phone or an automated aid presented on a laptop. The navigation aids operated under three different levels of information reliability (100%, 80% and 60%). A control condition was used in which each aid presented a telemarketing survey and participants navigated using a map. Results revealed perfect navigation information generally improved driver SA and performance compared to unreliable navigation information and the control condition (task-irrelevant information). In-vehicle automation appears to mediate the relationship of driver SA to performance in terms of operational and strategic (navigation) behaviours. The findings of this work support consideration of driver SA in the design of future vehicle automation for navigation tasks.     

Comparison of a brain-based adaptive system and a manual adaptable system for invoking automation

OBJECTIVE: Two experiments are presented examining adaptive and adaptable methods for invoking automation. BACKGROUND: Empirical investigations of adaptive automation have focused on methods used to invoke automation or on automation-related performance implications. However, no research has addressed whether performance benefits associated with brain-based systems exceed those in which users have control over task allocations. METHOD: Participants performed monitoring and resource management tasks as well as a tracking task that shifted between automatic and manual modes. In the first experiment, participants worked with an adaptive system that used their electroencephalographic signals to switch the tracking task between automatic and manual modes. Participants were also divided between high- and low-reliability conditions for the system-monitoring task as well as high- and low-complacency potential. For the second experiment, participants operated an adaptable system that gave them manual control over task allocations. RESULTS: Results indicated increased situation awareness (SA) of gauge instrument settings for individuals high in complacency potential using the adaptive system. In addition, participants who had control over automation performed more poorly on the resource management task and reported higher levels of workload. A comparison between systems also revealed enhanced SA of gauge instrument settings and decreased workload in the adaptive condition. CONCLUSION: The present results suggest that brain-based adaptive automation systems may enhance perceptual level SA while reducing mental workload relative to systems requiring user-initiated control. APPLICATION: Potential applications include automated systems for which operator monitoring performance and high-workload conditions are of concern.     

Modeling and analysis of a dynamic judgment task using a lens modelapproach

We investigated performance in a complex, dynamic decision-making task. Individual judgment performance in a command and control environment was modeled as linear combinations of environmental cue values using a lens model approach. Examination of the judgment models indicated that participants had similar judgment policies, while correlational and error analyzes indicated that performance differences were due to participants' abilities to execute judgment strategies rather than their knowledge of the task environment. The research demonstrated how the lens model approach can be extended to account for dynamic aspects of decision making in complex environments, through the use of individual, time dependent environmental models for each participant. Additionally, the research showed that a lens model approach is useful for characterizing factors in individual performance in complex judgment tasks. Empirically, this research suggests that training on task knowledge should be supplemented by training which focuses on the consistent execution of judgment strategies     

The impact of engineering information format on task performance: Gaze scanning pattern analysis

The emergence of new visualization technologies such as Virtual Reality (VR) and Augmented Reality (AR) had been widely implemented in the Architecture, Engineering, and Construction (AEC) industry. Although cumulative evidence pointed out a positive impact of these visualization technologies on construction task performance, there is still an obvious disagreement on the benefits or implications of these new visualization technologies, due to the lack of understanding of the mechanisms in which the visualization affects cognitive processes related to information processing. To obtain more evidence, this paper presents a human-subject experiment (n = 90) to investigate the impact of information format on the performance of an industrial pipeline maintenance task. The investigation centers around how different engineering information formats affect the attention patterns as a potential explanation for the changes in performance. A between-group experiment design was used where the participants were randomly assigned to one of the three groups (2D group, 3D group, and VR group) depending on what type of information was given to review the pipe operation instruction. After the review session, the participants were asked to perform the operation task in the virtual environment based on their memory. The results showed that the 3D and VR groups outperformed the 2D group in task performance. The analysis of eye-tracking data further indicated that the information format significantly changed the gaze scanning pattern when participants were reviewing the operational instructions. We also found that the task performance was correlated with eye-tracking features including gaze movement and pupil dilation. Our findings provided more evidence about the mechanisms in which new visualization technologies affect the attention patterns, helped resolve the current disagreement within the literature. In addition, a prediction model was proposed to use eye-tracking features to predict construction task performance.