Misuse of automated decision aids: Complacency, automation bias and the impact of training experience

The present study investigates automation misuse based on complacency and automation bias in interacting with a decision aid in a process control system. The effect of a preventive training intervention which includes exposing participants to rare automation failures is examined. Complacency is reflected in an inappropriate checking and monitoring of automated functions. In interaction with automated decision aids complacency might result in commission errors, i.e., following automatically generated recommendations even though they are false. Yet, empirical evidence proving this kind of relationship is still lacking. A laboratory experiment (N=24) was conducted using a process control simulation. An automated decision aid provided advice for fault diagnosis and management. Complacency was directly measured by the participants’ information sampling behavior, i.e., the amount of information sampled in order to verify the automated recommendations. Possible commission errors were assessed when the aid provided false recommendations. The results provide clear evidence for complacency, reflected in an insufficient verification of the automation, while commission errors were associated with high levels of complacency. Hence, commission errors seem to be a possible, albeit not an inevitable consequence of complacency. Furthermore, exposing operators to automation failures during training significantly decreased complacency and thus represents a suitable means to reduce this risk, even though it might not avoid it completely. Potential applications of this research include the design of training protocols in order to prevent automation misuse in interaction with automated decision aids.     

The role of trust in automation reliance

A recent and dramatic increase in the use of automation has not yielded comparable improvements in performance. Researchers have found human operators often underutilize (disuse) and overly rely on (misuse) automated aids (Parasuraman and Riley, 1997). Three studies were performed with Cameron University students to explore the relationship among automation reliability, trust, and reliance. With the assistance of an automated decision aid, participants viewed slides of Fort Sill terrain and indicated the presence or absence of a camouflaged soldier. Results from the three studies indicate that trust is an important factor in understanding automation reliance decisions. Participants initially considered the automated decision aid trustworthy and reliable. After observing the automated aid make errors, participants distrusted even reliable aids, unless an explanation was provided regarding why the aid might err. Knowing why the aid might err increased trust in the decision aid and increased automation reliance, even when the trust was unwarranted. Our studies suggest a need for future research focused on understanding automation use, examining individual differences in automation reliance, and developing valid and reliable self-report measures of trust in automation.     

The effect of individual differences in working memory in older adults on performance with different degrees of automated technology

Abstract

A leading hypothesis to explain older adults’ overdependence on automation is age-related declines in working memory. However, it has not been empirically examined. The purpose of the current experiment was to examine how working memory affected performance with different degrees of automation in older adults. In contrast to the well-supported idea that higher degrees of automation, when the automation is correct, benefits performance but higher degrees of automation, when the automation fails, increasingly harms performance, older adults benefited from higher degrees of automation when the automation was correct but were not differentially harmed by automation failures. Surprisingly, working memory did not interact with degree of automation but did interact with automation correctness or failure. When automation was correct, older adults with higher working memory ability had better performance than those with lower abilities. But when automation was incorrect, all older adults, regardless of working memory ability, performed poorly.

Practitioner Summary: The design of automation intended for older adults should focus on ways of making the correctness of the automation apparent to the older user and suggest ways of helping them recover when it is malfunctioning.     

Detecting automation failures in a simulated supervisory control environment

Abstract

The goal of this research was to determine how individuals perform and allocate their visual attention when monitoring multiple automated displays that differ in automation reliability. Ninety-six participants completed a simulated supervisory control task where each automated display had a different level of reliability (namely 70%, 85% and 95%). In addition, participants completed a high and low workload condition. The performance data revealed that (1) participants’ failed to detect automation misses approximately 2.5 times more than automation false alarms, (2) participants’ had worse automation failure detection in the high workload condition and (3) participant automation failure detection remained mostly static across reliability. The eye tracking data revealed that participants spread their attention relatively equally across all three of the automated displays for the duration of the experiment. Together, these data support a system-wide trust approach as the default position of an individual monitoring multiple automated displays.

Practitioner Summary: Given the rapid growth of automation throughout the workforce, there is an immediate need to better understand how humans monitor multiple automated displays concurrently. The data in this experiment support a system-wide trust approach as the default position of an individual monitoring multiple automated displays.

Abbreviations: DoD: Department of Defense; UA: unmanned aircraft; SCOUT: Supervisory Control Operations User Testbed; UAV: unmanned aerial vehicle; AOI: areas of interest     

A comparison of adaptive and adaptable automation under different levels of environmental stress

The effectiveness of different forms of adaptive and adaptable automation was examined under low- and high-stress conditions, in the form of different levels of noise. Thirty-six participants were assigned to one of the three types of variable automation (adaptive event-based, adaptive performance-based and adaptable serving as a control condition). Participants received 3 h of training on a simulation of a highly automated process control task and were subsequently tested during a 4-h session under noise exposure and quiet conditions. The results for performance suggested no clear benefits of one automation control mode over the other two. However, it emerged that participants under adaptable automation adopted a more active system management strategy and reported higher levels of self-confidence than in the two adaptive control modes. Furthermore, the results showed higher levels of perceived workload, fatigue and anxiety for performance-based adaptive automation control than the other two modes. PRACTITIONER SUMMARY: This study compared two forms of adaptive automation (where the automated system flexibly allocates tasks between human and machine) with adaptable automation (where the human allocates the tasks). The adaptable mode showed marginal advantages. This is of relevance, given that this automation mode may also be easier to design.     

Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults

This study examined the use of deliberately anthropomorphic automation on younger and older adults' trust, dependence and performance on a diabetes decision-making task. Research with anthropomorphic interface agents has shown mixed effects in judgments of preferences but has rarely examined effects on performance. Meanwhile, research in automation has shown some forms of anthropomorphism (e.g. etiquette) have effects on trust and dependence on automation. Participants answered diabetes questions with no-aid, a non-anthropomorphic aid or an anthropomorphised aid. Trust and dependence in the aid was measured. A minimally anthropomorphic aide primarily affected younger adults' trust in the aid. Dependence, however, for both age groups was influenced by the anthropomorphic aid. Automation that deliberately embodies person-like characteristics can influence trust and dependence on reasonably reliable automation. However, further research is necessary to better understand the specific aspects of the aid that affect different age groups. Automation that embodies human-like characteristics may be useful in situations where there is under-utilisation of reasonably reliable aids by enhancing trust and dependence in that aid. Practitioner Summary: The design of decision-support aids on consumer devices (e.g. smartphones) may influence the level of trust that users place in that system and their amount of use. This study is the first step in articulating how the design of aids may influence user's trust and use of such systems.     

A multi-level analysis of the effects of age and gender stereotypes on trust in anthropomorphic technology by younger and older adults

Previous research has shown that gender stereotypes, elicited by the appearance of the anthropomorphic technology, can alter perceptions of system reliability. The current study examined whether stereotypes about the perceived age and gender of anthropomorphic technology interacted with reliability to affect trust in such technology. Participants included a cross-section of younger and older adults. Through a factorial survey, participants responded to health-related vignettes containing anthropomorphic technology with a specific age, gender, and level of past reliability by rating their trust in the system. Trust in the technology was affected by the age and gender of the user as well as its appearance and reliability. Perceptions of anthropomorphic technology can be affected by pre-existing stereotypes about the capability of a specific age or gender.     

Human performance in a multiple-task environment: Effects of automation reliability on visual attention allocation

Diagnostic automation has been posited to alleviate the high demands of multiple-task environments; however, mixed effects have been found pertaining to performance aid success. To better understand these effects, attention allocation must be studied directly. We developed a multiple-task environment to study the effects of automation on visual attention. Participants interacted with a system providing varying levels of automation and automation reliability and then were transferred to a system with no support. Attention allocation was measured by tracking the number of times each task was viewed. We found that participants receiving automation allocated their time according to the task frequency and that tasks that benefited most from automation were most harmed when it was removed. The results suggest that the degree to which automation affects multiple-task performance is dependent on the relative attributes of the tasks involved. Moreover, there is an inverse relationship between support and cost when automation fails.     

Driver-centred vehicle automation: using network analysis for agent-based modelling of the driver in highly automated driving systems

To the average driver, the concept of automation in driving infers that they can become completely ‘hands and feet free’. This is a common misconception, however, one that has been shown through the application of Network Analysis to new Cruise Assist technologies that may feature on our roads by 2020. Through the adoption of a Systems Theoretic approach, this paper introduces the concept of driver-initiated automation which reflects the role of the driver in highly automated driving systems. Using a combination of traditional task analysis and the application of quantitative network metrics, this agent-based modelling paper shows how the role of the driver remains an integral part of the driving system implicating the need for designers to ensure they are provided with the tools necessary to remain actively in-the-loop despite giving increasing opportunities to delegate their control to the automated subsystems.

Practitioner Summary: This paper describes and analyses a driver-initiated command and control system of automation using representations afforded by task and social networks to understand how drivers remain actively involved in the task. A network analysis of different driver commands suggests that such a strategy does maintain the driver in the control loop.     

浅谈水厂自动化中存在的问题及解决措施

结合对宜兴水务集团水厂及泵站自控设备的管理经验,对水厂自动化从设计、设备和管理等方面进行探讨并提出解决措施。     

Pilot performance and preference for short cycles of automation in adaptive function allocation

The present experiment examined pilot response to the rapid cycling of automation. The experiment was conducted using a multi-task simulation environment consisting of tracking, fuel management, and system monitoring sub-tasks. Monitoring and fuel management sub-tasks were performed manually in all conditions. The tracking sub-task cycled between manual and automated control at fixed intervals of either 15, 30 or 60 sec. These cycle times were completely crossed with three levels of tracking difficulty giving nine within-subject conditions which lasted 5 min each. Performance was measured on each of the sub-tasks, as was pilot fatigue level and subjective workload for the respective conditions. Results indicated that both difficulty and cycle duration significantly affected tracking performance which was degraded with task difficulty and longer cycle times. Fuel management and system monitoring performance were unaffected by tracking difficulty and automation duration. However, a subsequent analysis was conducted using the 15 sec period immediately following each automation episode as a ‘window’ of performance. A different pattern of results was observed. Tracking performance was similarly affected by difficulty, but was no longer affected by cycle duration. Furthermore, fuel management error indicated a trend toward better performance in low difficulty conditions. Results illustrate micro trade-offs within sub-tasks and macro trade-offs between sub-tasks. Overall, the results support the contention that excessively short cycles of automation prove disruptive to performance in multi-task conditions.     

The effects of errors on system trust, self-confidence, and the allocation of control in route planning

The concept of trust is believed by some to compensate for feelings of uncertainty. Therefore, trust is considered to be crucial in people's decision to rely on a complex automated system to perform tasks for them. This experiment aimed to study the effects of errors on control allocation, and the mediating role of trust and self-confidence in the domain of route planning. Using a computer-based route planner, participants completed 10 route-planning trials in manual mode, and 10 in automatic mode, allowing participants to become equally experienced in operating both modes. During these so-called fixed trials, the numbers of errors in automatic as well as manual mode were systematically varied. Subsequently, participants completed six free trials, during which they were free to choose between modes. Our results showed that high automation error rates (AERs) decreased levels of system trust compared to low AERs. Conversely, high manual error rates (MERs) resulted in lower levels of self-confidence compared to low MERs, although to a lesser extent. Moreover, the difference between measures of trust and self-confidence proved to be highly predictive of the number of times automatic mode was selected during the six free trials. Additionally, results suggest a fundamental bias to trust one's own abilities over those of the system. Finally, evidence indicating a relationship between trust and self-confidence is discussed.     

Effects of modes of cockpit automation on pilot performance and workload in a next generation flight concept of operation

The objective of this study was to compare the effects of various forms of advanced cockpit automation for flight planning on pilot performance and workload under a futuristic concept of operation. A lab experiment was conducted in which airline pilots flew simulated tailored arrivals to an airport using three modes of automation (MOAs), including a control‐display unit (CDU) to the aircraft flight management system, an enhanced CDU (CDU+), and a continuous descent approach (CDA) tool. The arrival scenario required replanning to avoid convective activity and was constrained by a minimum fuel requirement at the initial approach fix. The CDU and CDU+ modes allowed for point‐by‐point path planning or selection among multiple standard arrivals, respectively. The CDA mode completely automated the route replanning for pilots. It was expected that the higher‐level automation would significantly reduce pilot workload and improve overall flight performance. In general, results indicated that the MOAs influenced pilot performance and workload responses according to hypotheses. This study provides new knowledge about the relationship of cockpit automation and interface features with pilot performance and workload in a novel next generation–style flight concept of operation. © 2012 Wiley Periodicals, Inc.     

基于模糊理论的可信计算信任评估方法研究

信任链是可信计算的关键技术之一,如何对其表达和评估是研究热点。针对可信计算环境中影响信任评价的各种复杂因素以及信任关系的不确定性和动态性,提出一种基于模糊理论的可信计算信任评估方法。引入历史度量记录和时间衰减因子以构建直接信任度,建立粒度为5的信任等级空间,并给出间接信任度的模糊评判方法。运用改进的Einstein算子定义相似度函数,在此基础上给出信任链上实体的信任模糊推理和评估过程。方法把模糊推理和信任传递相结合,可以全面地评估信任链上实体的信任度。模拟实验表明,与同类方法相比较,所提方法对恶意评估具有较好的抵抗能力,且评估结果的可信性和可靠性更好,是一种可信计算信任评估的新方法。     

Exploring the effects of trust,task interdependence and virtualness on knowledge sharing in teams

The sharing of knowledge within teams is critical to team functioning. However, working with team members who are in different locations (i.e. in virtual teams) may introduce communication challenges and reduce opportunities for rich interactions, potentially affecting knowledge sharing and its outcomes. Therefore, using questionnaire‐based data, this study examined the potential effects of different aspects of virtuality on a knowledge‐sharing model. Social exchange theory was used to develop a model relating trust to knowledge sharing and knowledge sharing to team effectiveness. The moderating effects of virtuality and task interdependence on these relationships were examined. A strong positive relationship was found between trust and knowledge sharing for all types of teams (local, hybrid and distributed), but the relationship was stronger when task interdependence was low, supporting the position that trust is more critical in weak structural situations. Knowledge sharing was positively associated with team effectiveness outcomes; however, this relationship was moderated by team imbalance and hybrid structures, such that the relationship between sharing and effectiveness was weaker. Organizations should therefore avoid creating unbalanced or hybrid virtual teams.     

The mediating role of knowledge transfer and the effects of client-consultant mutual trust on the performance of enterprise implementation projects

Enterprise systems represent a strategic asset that enables an organization to integrate business processes and data throughout its organization. Firms often rely on consultants to transfer implementation knowledge to their employees and expect successful project outcome; however, widespread project failures raise concerns regarding whether the use of consultants are effective. This study examines whether mutual trust affects knowledge transfer effectiveness which then impacts project outcome. Building on psychological contract, knowledge management, and trust literature, the results of this matched-pair, field survey suggest knowledge transfer mediates the relationship between mutual trust and project outcome. Implications for research and practice are discussed.     

Web服务中主观信任和客观声誉的研究

随着Web服务的出现,对软件服务和服务组合的安全可靠的运行使得服务提供方、服务请求方以及代理之间的相互信任变得尤为重要。首先给出信任和声誉的定义以及它们之间的区别和联系,由此提出一个用于度量软件服务间信任关系的信誉评估模型——WSTR。给出了更加灵活的计算直接信任度方法,并且将声誉引入推荐信任的合成中,利用现有的节点作为声誉存储实体,更加能够体现信任的主观性和客观性的结合。仿真结果表明声誉度能够客观地反映出真实的实际情况。     

Trust in automation: Part I. Theoretical issues in the study of trust and human intervention in automated systems

Abstract

Today many systems are highly automated. The human operator's role in these systems is to supervise the automation and intervene to take manual control when necessary. The operator's choice of automatic or manual control has important consequences for system performance, and therefore it is important to understand and optimize this decision process. One important determinant of operators' choice of manual or automatic control may be their degree of trust in the automation. However, there have been no experimental tests of this hypothesis until recently, nor is there a model of human trust in machines to form a theoretical foundation for empirical studies. In this paper a model of human trust in machines is developed, taking models of trust between people as a starting point, and extending them to the human-machine relationship. The resulting model defines human trust in machines and specifies how trust changes with experience on a system, providing a framework for experimental research on trust and human intervention in automated systems.     

Reconfigurable control of Hammerstein systems after actuator failures: stability,tracking, and performance

A new method for the reconfigurable control of stable Hammerstein systems with sector-bounded static nonlinear input characteristics subject to actuator failures is described. It aims at the recovery of the nominal stability, setpoint tracking, disturbance rejection and performance properties by the reconfigured closed-loop system. This article extends the virtual actuator from linear systems to Hammerstein systems and provides sufficient linear matrix inequality conditions for closed-loop stability, and a corresponding synthesis algorithm. It is shown that the approach is robust against uncertainties of the static input nonlinearity in a small-gain sense, and universal in a certain sense. Feasible setpoints for the reconfigured closed-loop system are characterised, and infeasible setpoints are projected to feasible ones. An extension guarantees minimum performance loss. The method is successfully experimentally evaluated using a system of interconnected tanks.     

A systematic literature review on intelligent automation: Aligning concepts from theory,practice, and future perspectives

With the recent developments in robotic process automation (RPA) and artificial intelligence (AI), academics and industrial practitioners are now pursuing robust and adaptive decision making (DM) in real-life engineering applications and automated business workflows and processes to accommodate context awareness, adaptation to environment and customisation. The emerging research via RPA, AI and soft computing offers sophisticated decision analysis methods, data-driven DM and scenario analysis with regard to the consideration of decision choices and provides benefits in numerous engineering applications. The emerging intelligent automation (IA) – the combination of RPA, AI and soft computing – can further transcend traditional DM to achieve unprecedented levels of operational efficiency, decision quality and system reliability. RPA allows an intelligent agent to eliminate operational errors and mimic manual routine decisions, including rule-based, well-structured and repetitive decisions involving enormous data, in a digital system, while AI has the cognitive capabilities to emulate the actions of human behaviour and process unstructured data via machine learning, natural language processing and image processing. Insights from IA drive new opportunities in providing automated DM processes, fault diagnosis, knowledge elicitation and solutions under complex decision environments with the presence of context-aware data, uncertainty and customer preferences. This sophisticated review attempts to deliver the relevant research directions and applications from the selected literature to the readers and address the key contributions of the selected literature, IA’s benefits, implementation considerations, challenges and potential IA applications to foster the relevant research development in the domain.