Detecting automation failures in a simulated supervisory control environment

Abstract

The goal of this research was to determine how individuals perform and allocate their visual attention when monitoring multiple automated displays that differ in automation reliability. Ninety-six participants completed a simulated supervisory control task where each automated display had a different level of reliability (namely 70%, 85% and 95%). In addition, participants completed a high and low workload condition. The performance data revealed that (1) participants’ failed to detect automation misses approximately 2.5 times more than automation false alarms, (2) participants’ had worse automation failure detection in the high workload condition and (3) participant automation failure detection remained mostly static across reliability. The eye tracking data revealed that participants spread their attention relatively equally across all three of the automated displays for the duration of the experiment. Together, these data support a system-wide trust approach as the default position of an individual monitoring multiple automated displays.

Practitioner Summary: Given the rapid growth of automation throughout the workforce, there is an immediate need to better understand how humans monitor multiple automated displays concurrently. The data in this experiment support a system-wide trust approach as the default position of an individual monitoring multiple automated displays.

Abbreviations: DoD: Department of Defense; UA: unmanned aircraft; SCOUT: Supervisory Control Operations User Testbed; UAV: unmanned aerial vehicle; AOI: areas of interest     

Effects of computer self-efficacy and system reliability on user interaction with decision support systems

Computer self-efficacy (CSE) has been identified as a major determinant of computer-related ability and usage in organizational contexts. However, there has been little research on the impact of CSE on the use of high level computerized systems such as decision support systems. In order to examine this relationship, participants completed a visual search task with the assistance of an automated decision support system which varied in reliability from moderately reliable (70%) to highly reliable (90%). After completion of the task, a median split was performed on participants’ CSE scores to divide participants into high-CSE and low-CSE groups. We examined the relationship between CSE and trust and utilization of the system. High-CSE participants trusted the system more, complied with it more and generated significantly more hits than low-CSE participants, particularly on trials in which the aid was highly reliable. This indicated that high-CSE levels led to a better ability to gauge the true capabilities of the system. However, all participants uniformly underestimated the true reliability of the system at both levels of CSE. This study provides insight into the potential significance of personality factors in shaping human–technology interaction and has implications for designers of automated systems.     

The role of trust in automation reliance

A recent and dramatic increase in the use of automation has not yielded comparable improvements in performance. Researchers have found human operators often underutilize (disuse) and overly rely on (misuse) automated aids (Parasuraman and Riley, 1997). Three studies were performed with Cameron University students to explore the relationship among automation reliability, trust, and reliance. With the assistance of an automated decision aid, participants viewed slides of Fort Sill terrain and indicated the presence or absence of a camouflaged soldier. Results from the three studies indicate that trust is an important factor in understanding automation reliance decisions. Participants initially considered the automated decision aid trustworthy and reliable. After observing the automated aid make errors, participants distrusted even reliable aids, unless an explanation was provided regarding why the aid might err. Knowing why the aid might err increased trust in the decision aid and increased automation reliance, even when the trust was unwarranted. Our studies suggest a need for future research focused on understanding automation use, examining individual differences in automation reliance, and developing valid and reliable self-report measures of trust in automation.     

Social stress and performance in human-machine interaction: a neglected research field

Abstract

Given the increasing capabilities of highly automated systems, the article argues for a need to address the issue of social stress in human-machine interaction. It suggests a classification system of subordinate concepts found in the research literature under the heading of social stress. A review of the literature revealed a paucity of studies examining the effects of social stressors on performance. In particular, the review showed a shortage of experimental lab-based work, needed to establish clear cause-effect relationships. The article examined the suitability of different social stressors for lab-based research, not only when humans are the source of stress but also in so-called hybrid teams where social stress is caused by machine agents. The review shows that a closer link is needed between the separate literature on social stress and automation. Finally, three mechanisms are proposed that may predict how social stress may affect performance: ‘blank-out’-mechanism, ‘rumination’-mechanism, and ‘increased-motivation’-mechanism.

Practitioner summary: Theories of ergonomics and human factors may benefit from better integration of research and theoretical work in the domain of social stress. This is due to the increasing capabilities of machines to induce social stress.

Abbreviations: HMI: human-machine interaction; TSST: trier social stress test     

The impact of false warnings on partial and full lane departure warnings effectiveness and acceptance in car driving

In the past, lane departure warnings (LDWs) were demonstrated to improve driving behaviours during lane departures but little is known about the effects of unreliable warnings. This experiment focused on the influence of false warnings alone or in combination with missed warnings and warning onset on assistance effectiveness and acceptance. Two assistance unreliability levels (33 and 17%) and two warning onsets (partial and full lane departure) were manipulated in order to investigate interaction. Results showed that assistance, regardless unreliability levels and warning onsets, improved driving behaviours during lane departure episodes and outside of these episodes by favouring better lane-keeping performances. Full lane departure and highly unreliable warnings, however, reduced assistance efficiency. Drivers’ assistance acceptance was better for the most reliable warnings and for the subsequent warnings. The data indicate that imperfect LDWs (false warnings or false and missed warnings) further improve driving behaviours compared to no assistance.

Practitioner Summary: This study revealed that imperfect lane departure warnings are able to significantly improve driving performances and that warning onset is a key element for assistance effectiveness and acceptance. The conclusion may be of particular interest for lane departure warning designers.     

Coastal definition using Landsat data†

Abstract

An automated technique is described, for extracting land-water boundaries from Landsat MSS images by means of ‘density contour-threading’ of the band 7 data. The resulting coastlines, etc. can then be transformed to fit any map projection—in particular the British National Grid—with the aid of selected ground control points.     

Control of posture during tasks representing common work-related postures – a reliability study

Assessment of control of posture using a task battery that represents work-related postural conditions is highly recommended for providing a comprehensive understanding of collective postural demands. However, dearth of evidence exists on the reliability of a task battery, thus precluding its use as an outcome measure in field research. This study investigated the intrasession reliability and systematic variation of force plate derived centre of pressure (COP) measures obtained during repeated performance of a task battery (lifting task, limits of stability and bipedal and unipedal stance). COP signals obtained during each task performance were processed to derive various time-domain COP measures. Statistical analyses revealed that 13 of the 19 COP measures displayed excellent relative (ICC_(2,3) ≥ 0.75) and acceptable absolute reliability (SEM%: ≤ 10). Although COP measures displayed systematic variation, the differences were less or equal to the measurement error, except COP measures of unipedal stance and limits of stability. The chosen task battery is reliable and can be used for comprehensive evaluation of control of posture, in both field and laboratory research.

Practitioner Summary: Repeated evaluation of multiple tasks together sequentially could introduce measurement variability. This study investigated intrasession reliability of a task battery representing common work-related postures. The chosen task battery was found to be reliable with acceptable measurement error and can be used in field research settings for evaluation of control of posture.     

Examining endplate fatigue failure during cyclic compression loading with variable and consistent peak magnitudes using a force weighting adjustment approach: an in vitro study

Abstract

Repetitive movement is common in many occupational contexts. Therefore, cumulative load is a widely recognised risk factor for lowback injury. This study quantified the effect of force weighting factors on cumulative load estimates and injury prediction during cyclic loading. Forty-eight porcine cervical spine motion segments were assigned to experimental groups that differed by average peak compression magnitude (30%, 50% and 70% of predicted tolerance) and amplitude variation (consistent, variable). Cyclic loading was performed at a frequency of 0.5?Hz until fatigue failure occurred. Weighting factors were determined and applied instantaneously. Inclusion of weighting factors resulted in statistically similar cumulative load estimates at injury between variable and consistent loading (p?>?.071). Further, survivorship was generally greater when the peak compression magnitude was consistent compared to variable. These results emphasise the importance of weighting factors as an equalisation tool for the evaluation of cumulative low back loading exposures in occupational contexts.

Practitioner summary: Weighting factors can equalise the risk of injury based on compression magnitude. When weighted, the cumulative compression was similar between consistent and variable cyclic loading protocols, despite being significantly different when unweighted and having similar injury rates. Therefore, assessing representative occupational exposures without evaluating task performance variability may underestimate injury risk.

Abbreviations: FSU: functional spinal unit; UCT: ultimate compression tolerance     

Burnout among pilots: psychosocial factors related to happiness and performance at simulator training

Abstract

In this study among airline pilots, we aim to uncover the work characteristics (job demands and resources) and the outcomes (job crafting, happiness and simulator training performance) that are related to burnout for this occupational group. Using a large sample of airline pilots, we showed that 40% of the participating pilots experience high burnout. In line with Job Demands-Resources theory, job demands were detrimental for simulator training performance because they made pilots more exhausted and less able to craft their job, whereas job resources had a favourable effect because they reduced feelings of disengagement and increased job crafting. Moreover, burnout was negatively related to pilots’ happiness with life. These findings highlight the importance of psychosocial factors and health for valuable outcomes for both pilots and airlines.

Practitioner Summary: Using an online survey among the members of a European pilots’ professional association, we examined the relationship between psychosocial factors (work characteristics, burnout) and outcomes (simulator training performance, happiness). Forty per cent of the participating pilots experience high burnout. Job demands were detrimental, whereas job resources were favourable for simulator training performance/happiness.

Twitter text: 40% of airline pilots experience burnout and psychosocial work factors and burnout relate to performance at pilots’ simulator training.     

Human factors assessment of conflict resolution aid reliability and time pressure in future air traffic control

Though it has been reported that air traffic controllers' (ATCos') performance improves with the aid of a conflict resolution aid (CRA), the effects of imperfect automation on CRA are so far unknown. The main objective of this study was to examine the effects of imperfect automation on conflict resolution. Twelve students with ATC knowledge were instructed to complete ATC tasks in four CRA conditions including reliable, unreliable and high time pressure, unreliable and low time pressure, and manual conditions. Participants were able to resolve the designated conflicts more accurately and faster in the reliable versus unreliable CRA conditions. When comparing the unreliable CRA and manual conditions, unreliable CRA led to better conflict resolution performance and higher situation awareness. Surprisingly, high time pressure triggered better conflict resolution performance as compared to the low time pressure condition. The findings from the present study highlight the importance of CRA in future ATC operations.

Practitioner Summary: Conflict resolution aid (CRA) is a proposed automation decision aid in air traffic control (ATC). It was found in the present study that CRA was able to promote air traffic controllers' performance even when it was not perfectly reliable. These findings highlight the importance of CRA in future ATC operations.     

DenseSPH-YOLOv5: An automated damage detection model based on DenseNet and Swin-Transformer prediction head-enabled YOLOv5 with attention mechanism

Objective.Computer vision-based up-to-date accurate damage classification and localization are of decisive importance for infrastructure monitoring, safety, and the serviceability of civil infrastructure. Current state-of-the-art deep learning (DL)-based damage detection models, however, often lack superior feature extraction capability in complex and noisy environments, limiting the development of accurate and reliable object distinction.Method.To this end, we present DenseSPH-YOLOv5, a real-time DL-based high-performance damage detection model where DenseNet blocks have been integrated with the backbone to improve in preserving and reusing critical feature information. Additionally, convolutional block attention modules (CBAM) have been implemented to improve attention performance mechanisms for strong and discriminating deep spatial feature extraction that results in superior detection under various challenging environments. Moreover, an additional feature fusion layers and a Swin-Transformer Prediction Head (SPH) have been added leveraging advanced self-attention mechanism for more efficient detection of multiscale object sizes and simultaneously reducing the computational complexity.Results.Evaluating the model performance in large-scale Road Damage Dataset (RDD-2018), at a detection rate of 62.4 FPS, DenseSPH-YOLOv5 obtains a mean average precision (mAP) value of 85.25%, F1-score of 81.18%, and precision (P) value of 89.51% outperforming current state-of-the-art models.Significance.The present research provides an effective and efficient damage localization model addressing the shortcoming of existing DL-based damage detection models by providing highly accurate localized bounding box prediction. Current work constitutes a step towards an accurate and robust automated damage detection system in real-time in-field applications.     

What makes a maker: the motivation for the maker movement in ICT

ABSTRACT

The maker movement is a cultural trend that focuses on an individual’s ability to be a creator of things using technology. This study provides an exploratory analysis of the motivations of making and community participation in the maker movement. This study conducted a field study based on motivation theory using a survey methodology that involved 93 individual makers from seven online maker communities. The analysis results show that different motives have a positive effect on makers in different ways: extrinsic, status opportunity motivation affects making itself while use-value motivation affects community participation. Intrinsic motivation does not have a significant relationship with either participation or making. Furthermore, community participation is an important factor in making as it influences both learning and performance, while making only influences performance. These results can help to enlarge the understanding of makers and contribute to the growth of the maker movement.     

Human–automation interaction for multiple robot control: the effect of varying automation assistance and individual differences on operator performance

In a human–automation interaction study, automation assistance level (AL) was investigated for its effects on operator performance in a dynamic, multi-tasking environment. Participants supervised a convoy of manned and unmanned vehicles traversing a simulated environment in three AL conditions, while maintaining situation awareness and identifying targets. Operators’ situation awareness, target detection performance, workload and individual differences were evaluated. Results show increasing AL generally improved task performance and decreased perceived workload, however, differential effects due to operator spatial ability and perceived attentional control were found. Eye-tracking measures were useful in parsing out individual differences that subjective measures did not detect. At the highest AL, participants demonstrated potentially complacent behaviour, indicating task disengagement.

Practitioner Summary: The effect of varying automation assistance level (AL) on operator performance on multiple tasks were examined in a within-subjects experiment. Findings indicated a moderate AL improved performance, while higher levels encouraged complacent behaviour. Effects due to individual differences suggest that effective AL depends on the underlying characteristics of the operator.     

Software verification and graph similarity for automated evaluation of students’ assignments

ContextThe number of students enrolled in universities at standard and on-line programming courses is rapidly increasing. This calls for automated evaluation of students assignments.ObjectiveWe aim to develop methods and tools for objective and reliable automated grading that can also provide substantial and comprehensible feedback. Our approach targets introductory programming courses, which have a number of specific features and goals. The benefits are twofold: reducing the workload for teachers, and providing helpful feedback to students in the process of learning.MethodFor sophisticated automated evaluation of students’ programs, our grading framework combines results of three approaches (i) testing, (ii) software verification, and (iii) control flow graph similarity measurement. We present our tools for software verification and control flow graph similarity measurement, which are publicly available and open source. The tools are based on an intermediate code representation, so they could be applied to a number of programming languages.ResultsEmpirical evaluation of the proposed grading framework is performed on a corpus of programs written by university students in programming language C within an introductory programming course. Results of the evaluation show that the synergy of proposed approaches improves the quality and precision of automated grading and that automatically generated grades are highly correlated with instructor-assigned grades. Also, the results show that our approach can be trained to adapt to teacher’s grading style.ConclusionsIn this paper we integrate several techniques for evaluation of student’s assignments. The obtained results suggest that the presented tools can find real-world applications in automated grading.     

An iterative multiple sampling method for intrusion detection

ABSTRACT

Threats to network security increase with growing volumes and velocity of data across networks, and they present challenges not only to law enforcement agencies, but to businesses, families and individuals. The volume, velocity and veracity of shared data across networks entail accurate and reliable automated tools for filtering out useful from malicious, noisy or irrelevant data. While data mining and machine learning techniques have widely been adopted within the network security community, challenges and gaps in knowledge extraction from data have remained due to insufficient data sources on attacks on which to test the algorithms accuracy and reliability. We propose a data-flow adaptive approach to intrusion detection based on high-dimensional cyber-attacks data. The algorithm repeatedly takes random samples from an inherently bi-modal, high-dimensional dataset of 82,332 observations on 25 numeric and two categorical variables. Its main idea is to capture subtle information resulting from reduced data dimension of a large number of malicious flows and by iteratively estimating roles played by individual variables in construction of key components. Data visualization and numerical results provide a clear separation of a set of variables associated with attack types and show that component-dominating parameters are crucial in monitoring future attacks.     

Workload Management in Teleoperation of Unmanned Ground Vehicles: Effects of a Delay Compensation Aid on Human Operators’ Workload and Teleoperation Performance

ABSTRACT

Workload management is of critical concern in the teleoperation of unmanned vehicles because teleoperation is often employed in high-risk industries wherein high workload can lead to sub-optimal task performance and can harm human operators’ long-term well-being. This study aimed to assess the detrimental effects of time delays in teleoperation on operators’ workload and performance, and how a delay compensation aid mitigated such effects. We conducted a human-in-the-loop experiment with 36 participants using a dual-task teleoperation platform, where participants drove a simulated High Mobility Multipurpose Wheeled Vehicle (HMMWV) and performed a one-back memory task under three conditions: the delay condition, the delay with compensation aid condition, and the ideal no delay condition. A model-free predictor was used as the compensation aid. Results indicate that with a time delay of 0.8-s participants’ workload increased and performance degraded significantly. Moreover, the model-free predictor mitigated the detrimental effects of time delay on workload and task performance. Our findings suggest that participants are more sensitive in their perceived workload compared to the objective and physiological measures of workload. In addition, without any delay compensation algorithms, continuous teleoperation may not be ideal for operations with long time delays.     

基于循环谱的多循环频率加权合作频谱检测

在接入主用户授权频段之前,认知用户需要检测该频段是否处于空闲状态,以免干扰主用户通信.利用主用户信号和噪声的不同谱相关特性,研究了基于循环谱的频谱检测方法.将主用户非零循环频率上的接收信号循环谱幅度作为检测统计量,给出了判决准则和检测方法.利用主用户信号在不同循环频率下不同程度的循环平稳特征,多个循环频率之间通过加权迭代合作来提高检测结果的可信度;并通过蒙特卡罗仿真方法验证该方法的可行性.仿真结果表明:加权迭代合作可有效实现频谱检测,且检测性能优于等权合作检测;通过合理选择信号采样点数、循环频率个数、迭代次数进行合作检测既可有效提高检测概率,又能保证检测灵敏度.     

Age differences in trust and reliance of a medication management system

The present study examined age differences in trust and reliance of an automated decision aid. In Experiment 1, older and younger participants performed a simple mathematical task concurrent with a simulated medication management task. The decision aid was designed to facilitate medication management, but with varying reliability. Trust, self-confidence and usage of the aid were measured. The results indicated that older adults had greater trust in the aid and were less confident in their performance, but they did not calibrate trust differently than younger adults. In Experiment 2, a variant of the same task was used to investigate whether older adults are subject to over-reliance on the automation. Differences in omission and commission errors were examined. The results indicated that older adults were more reliant on the decision aid and committed more automation-related errors. A signal detection analyses indicated that older adults were less sensitive to automation failures. Results are discussed with respect to the perceptual and cognitive factors that influence age differences in the use of fallible automation.     

不确定飞行控制系统中断故障检测与分离

研究不确定飞行控制系统执行器中断故障检测与分离问题,同时设计了状态反馈控制器和检测器,在保证闭环控制系统稳定的前提下,通过设计的检测器对系统状态进行重组以产生残差进而检测执行器的中断故障。此外,通过设计一组分离器,可以确定出执行器发生故障的位置。最后,通过研究一个飞行控制系统模型验证了所提出方法的有效性。     

Automated volcanic hot-spot detection based on FY-4A/AGRI infrared data

ABSTRACT

A new algorithm (FYVOLC) for the automated detection of volcanic hot spots using data obtained from China’s new-generation satellite FY-4A is developed and tested. FYVOLC improves the detection ability of the Volcanic Anomaly SofTware (VAST) algorithm by incorporating a Normalized Brightness Temperature Difference Index (NBTDI) to reduce the influence of ‘cold’ cloud. In addition, FYVOLC introduces a mid-infrared brightness temperature criterion to identify volcanic hot spots by making calculations based on the image itself without artificially determining any parameters. To test the volcanic hot-spot detection performance of FYVOLC, FY-4A Advanced Geostationary Radiation Imager data were used for eruptions from four volcanoes: Mayon Volcano in the Philippines (25–26 January 2018), and Bromo (1–2 September 2018), Lawu (3–4 September 2018), and Soputan volcanoes (3–4 October 2018) in Indonesia. A total of 147 images of the above four volcanoes were used, and the results obtained using the FYVOLC algorithm were compared with those from three existing volcanic hot-spot detection algorithms: the simplified contextual, VAST, and HOTSAT algorithms. It is shown that the simplified contextual and VAST algorithms are prone to generating false alerts (with a maximum false alert rate of up to 41% and 48%, respectively), whereas the simplified contextual and HOTSAT algorithms are prone to missing hot-spot pixels (with a maximum miss rate of up to 71% and 54%, respectively). The FYVOLC algorithm has the best detection accuracy owing to the adopted NBTDI and image-based mid-infrared brightness temperature criterion. The maximum false alert rate of FYVOLC is 12%, and the maximum miss rate is 11%. By analysing the thermal anomaly time-series of the 147 images, it was found that the detection results of FYVOLC are basically consistent with the actual hot spots, except for some images that were strongly affected by cloud cover. This study is the first to realize the automated detection of volcanic hot spots and monitor temporally dynamic thermal phenomena based on FY-4A satellite data. The results have significance for the continuing development of global volcanic early-warning systems and for the dynamic monitoring of volcanoes after eruptions.