Applying telepresence and augmented reality to teleoperate field robots

In many real-world teleoperation applications like mining, advanced teleoperation techniques will be needed in the near future. Full telepresence provides the best possible resources for controlling a moving machine. Unfortunately, full telepresence is expensive to implement and may be technically motivated only when really needed.In this paper different complexity of telepresence systems are compared. All systems were installed on the same vehicle by using the same teleoperation system by configuring it differently. Different kinds of tasks like corridor driving, loading, etc. were performed with all systems by several users. The comparison was made by timing the executions, registering errors, and interviewing the operators.     

Gaze direction based vehicle teleoperation method with omnidirectional image stabilization and automatic body rotation control

In robot teleoperation, a robot works as a physical agent at a remote site for a robot operator. There are mainly two tasks in robot teleoperation using camera images: environment recognition using visual information and robot control according to the recognition. In this paper, we propose a gaze direction based vehicle teleoperation method with an omnidirectional image stabilization and an automatic body rotation control. In the proposed method, we manage above two tasks in the same manner that are usually treated separately. This method is an intuitive vehicle teleoperation method where an operator do not need to have concern about vehicle body orientations and can absorb differences of vehicle driving mechanisms. That is, this method frees an operator from being bothered from controlling a vehicle and the operator can concentrate on where he/she intends to go. This method mainly consists of two technologies: an omnidirectional image stabilization technology and automatic body rotation control. The conducted experiments show effectiveness of the proposed method.     

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

     

力反馈水下遥操作系统稳定性分析

在遥操作系统中, 为了提高操作者的交互感知能力, 通常将与环境的接触力反馈回主端. 然而, 反馈力会导 致主端机器人的诱导运动, 使得主端机器人发出不准确的运动指令, 甚至导致闭环系统不稳定. 此外, 在作业过程 中, 水下机械臂不可避免地受到洋流和海浪带来的外部干扰. 针对带有接触力反馈的水下遥操作系统, 本文设计了 一种补偿主端位置漂移的辅助系统, 构造了有限时间干扰观测器, 并讨论了观测时间内系统状态的有界性, 进一步 证明了整个闭环系统是指数输入–状态稳定的. 仿真结果表明, 该方法能够保证变时延和外部干扰条件下水下遥操 作系统跟踪性能.     

Advanced Interfaces for Vehicle Teleoperation: Collaborative Control, Sensor Fusion Displays, and Remote Driving Tools

We are working to make vehicle teleoperation accessible to all users, novices and experts alike. In our research, we are developing a new control model for teleoperation, sensor-fusion displays and a suite of remote driving tools. Our goal is to build a framework which enables humans and robots to communicate, to exchange ideas and to resolve differences. In short, to develop systems in which humans and robots work together and jointly solve problems.     

Teleoperation based on the hidden robot concept

Overlaying classical teleoperation control schemes based on a bilateral master-slave coupling, a teleoperation architecture designed in a general teleworking context is proposed. In this scheme, the executing machine is perceptually and functionally hidden to the operator by means of an intermediate functional representation between a real remote world and man. As any executing machine, and more particularly a robot, will be replaced by man, the image of the robot will not appear in the intermediate representation. This principle is thus named: “the hidden robot concept.” In this approach, the teleoperation problem is divided into two main parts: 1) choosing the appropriate intermediate representation and determining its interaction and relation with man and 2) building the relations and transformations between the intermediate representation and the real remote environment. The constituents of this teleoperator are outlined in this paper and an experiment validating this concept is presented     

Workload Management in Teleoperation of Unmanned Ground Vehicles: Effects of a Delay Compensation Aid on Human Operators’ Workload and Teleoperation Performance

ABSTRACT

Workload management is of critical concern in the teleoperation of unmanned vehicles because teleoperation is often employed in high-risk industries wherein high workload can lead to sub-optimal task performance and can harm human operators’ long-term well-being. This study aimed to assess the detrimental effects of time delays in teleoperation on operators’ workload and performance, and how a delay compensation aid mitigated such effects. We conducted a human-in-the-loop experiment with 36 participants using a dual-task teleoperation platform, where participants drove a simulated High Mobility Multipurpose Wheeled Vehicle (HMMWV) and performed a one-back memory task under three conditions: the delay condition, the delay with compensation aid condition, and the ideal no delay condition. A model-free predictor was used as the compensation aid. Results indicate that with a time delay of 0.8-s participants’ workload increased and performance degraded significantly. Moreover, the model-free predictor mitigated the detrimental effects of time delay on workload and task performance. Our findings suggest that participants are more sensitive in their perceived workload compared to the objective and physiological measures of workload. In addition, without any delay compensation algorithms, continuous teleoperation may not be ideal for operations with long time delays.     

PD-like controller for delayed bilateral teleoperation of wheeled robots

This paper proposes a proportional derivative (PD)-like controller applied to the delayed bilateral teleoperation of wheeled robots with force feedback in face of asymmetric and varying-time delays. In contrast to bilateral teleoperation of manipulator robots, in these systems, there is a mismatch between the models of the master and slave (mobile robot), problem that is approached in this work, where the system stability is analysed. From this study, it is possible to infer the control parameters, depending on the time delay, necessary to assure stability. Finally, the performance of the delayed teleoperation system is evaluated through tests where a human operator drives a 3D simulator as well as a mobile robot for pushing objects.     

Investigating multiple human performance measures in teleoperation task: A translation task in a tele‐sandblasting maintenance system

In sandblasting tasks for complex steel structure maintenance, teleoperation is required to keep humans away from occupational risk and hazard. On the other hand, teleoperation typically degrades system‐human performances, resulting in poor product quality and must be designed such that the performances remain as high as possible. However, designing the teleoperation system regarding to a single performance measure may lead to an improper design. In this article, we propose two novel loss‐function‐based human‐performance measures to incorporate with a widely used performance measure, movement time, to thoroughly represent performance: unfinished surface and damaged surface. We aim to investigate the effects of two main design parameters, viewing distance and path width. The results show that only path width is significant for overall performances. Furthermore, the effect of gender is significant such that men outperform women in cleaning the surface. Finally, the optimal setting conditions are suggested to achieve their optimal performances.     

Bilateral teleoperation through the Internet

This paper proposes a stable control structure for the bilateral teleoperation of robots through Internet. The problem is motivated by the increasing use of the Internet as a communication channel. Internet has a time-varying delay which depends on factors such as congestion, bandwidth and distance. In this work, we propose a control structure for the teleoperation of a manipulator robot with force feedback. Such a control structure includes state controllers (placed on the local and remote sites) and a time-delay compensation, which modifies the delayed position command generated by the human operator using the force that he feels in such a delayed moment and the current force between the slave and the remote environment. In addition, the proposed control scheme is designed considering a model of the communication channel. Finally, experiments of bilateral teleoperation of robots through Intranet and Internet are shown to test the performance and stability of the designed teleoperation system.     

Teleoperation of robot arm with position measurement via angle-pixel characteristic and visual supporting function

In this paper, a teleoperation system of a robot arm with position measurement function and visual supporting function is developed. The working robot arm is remotely controlled by the manual operation of the human operator and the autonomous control via visual servo. The visual servo employs the template matching technique. The position measurement is realized using a stereo camera based on the angle-pixel characteristic. The visual supporting function to give the human operator useful information about the teleoperation is also provided. The usefulness of the proposed teleoperation system is confirmed through experiments using an industrial articulated robot arm.     

基于波变量补偿的阻抗匹配时延双边遥操作

通信时延是遥操作系统中固有的问题,它会严重影响遥操作的性能,降低系统的稳定性和跟踪性。基于无源理论的波变量法可以保证遥操作系统在任意时延下稳定,是解决时延问题的一个重要方法。然而,波变量法带来的波反射会阻扰有用信号的传输,降低了主从端信号的跟踪性,严重时甚至会导致整个系统振荡。针对这一问题,提出了一种基于波变量补偿的阻抗匹配双边遥操作系统结构,旨在减少波反射,提高操作者的临场感和系统的跟踪性。通过仿真实验,结果表明所提方法能够保证固定时延条件下遥操作系统的稳定性,并具有较好的跟踪性。     

Stable Delayed Bilateral Teleoperation of Mobile Manipulators

This paper proposes a control scheme for a stable teleoperation of non‐holonomic mobile manipulator robots. This configuration presents high‐coupled dynamics and motion redundancy. The problem approached in this work is the teleoperation of the end effector velocity of the Mobile Manipulator, while system redundancy is used to achieve secondary control objectives. We considered variable asymmetric time delays as well as non‐passive models of operator and environment. From this study, it is possible to infer the control parameters, depending on the time delay, in order to assure stability. Finally, the performance of the delayed teleoperation system is evaluated through simulations of human‐in‐the‐loop internet teleoperation.     

Teleoperation User Interfaces for Mining Robotics

This paper presents a brief discussion of the requirements for user interfaces for teleoperation of mining vehicles and systems. Current commercial interfaces are relatively simple, evolving from line-of-sight remote control systems with the addition of video displays. This level of sophistication has nonetheless allowed teleoperation to be a viable and profitable technique. CSIRO Exploration and Mining research into interface design currently concentrates on fusion of non-visual feedback to the operator and efficient presentation of such information. Case studies based on user interfaces for two experimental teleoperated mining-related systems, the Numbat mine emergency response vehicle and remote highwall mining control systems are described. In both cases the purpose of the user interface is to satisfy the major client requirements for mining systems of robustness, reliability and ease of use by site operational and support personnel.     

外骨骼式遥操作主手设计及主从异构映射算法研究

为了更好地促进机器人适应复杂的遥操作任务,开发了能够精确获取人体上肢运动信息的外骨骼式遥操作主手,并通过异构映射算法,实现对6自由度协作机械臂的遥操作．首先,基于人体仿生结构,设计了可穿戴式8自由度外骨骼主手（臂部7自由度和手部1自由度）;其次,通过改进的D-H（Denavit-Hartenberg）方法建立遥操作系统的运动学模型,基于Matlab的机器人工具箱进行了工作空间仿真,并设计主从异构映射算法;最后,实验验证外骨骼主手在遥操作系统中的可操作性,以及工作空间异构映射算法的可行性．实验表明,外骨骼主手能够控制从端机械手臂,且保证末端位置和姿态一致,可在大范围工作空间内复现人体上肢精细运动,主从跟随误差达2 mm,工作空间类似于直径1.08 m的半球形．因此,可穿戴式的外骨骼主手使操作者能更加直观地参与到遥操作系统当中,辅助操作者更加高效地完成精细复杂任务．     

Adaptive Inverse Dynamics Four-Channel Control of Uncertain Nonlinear Teleoperation Systems

Most of the methods to date on bilateral control of nonlinear teleoperation systems lead to nonlinear and coupled closed-loop dynamics, even in the ideal case of perfect knowledge of the master, the slave, the human operator and the environment. Consequently, the transparency of these closed-loop systems is difficult to study. In comparison, inverse dynamics controllers can deal with the nonlinear terms in the dynamics in a way that, in the ideal case, the closed-loop systems become linear and decoupled. In this paper, for multi-d.o.f. nonlinear teleoperation systems with uncertainties, adaptive inverse dynamics controllers are incorporated into the four-channel bilateral teleoperation control framework. The resulting controllers do not need exact knowledge of the dynamics of the master, the slave, the human operator or the environment. A Lyapunov analysis is presented to prove the transparency of the teleoperation system. Simulations are also presented to show the effectiveness of the proposed approach.     

SoTCM: a scene-oriented task complexity metric for gaze-supported teleoperation tasks

Recent developments in human–robot interaction (HRI) research have heightened the need to incorporate indirect human signals to implicitly facilitate intuitive human–guided interactions. Eye-gaze has been widely used nowadays as an input interface in multi-modal teleoperation scenarios due to their advantage in revealing human intentions and forthcoming actions. However, to date, there has been no discussion about how the structure of the environment, that the human is interacting with, could affect the complexity of the teleoperation task. In this paper, a new metric named “Scene-oriented Task Complexity Metric” (SoTCM) is proposed to estimate the complexity of a certain scene that is involved in eye-gaze-supported teleoperation tasks. The proposed SoTCM objectively estimates the effort that could be exerted by the human operator in terms of the expected time required to point at all the informative locations retrieved from the scene under discussion. The developed SoTCM depends on both the density and distribution of the informative locations in the scene, while incorporates the eye movement behavior found in the psychology literature. The proposed SoTCM is subjectively validated by using the time-to-complete index in addition to the standard (NASA-TLX) workload measure in eight varying structure scenes. Results confirmed a significant relation between SoTCM and the measured task workload which endorses the applicability of using SoTCM in predicting scene complexities and subsequently the task workload in advance.     

MAS体系结构在遥操作系统中的应用研究

设计了一个基于多智能体系统MAS结构的遥操作系统框架模型，它结合离散事件状态DES控制模型，可用于在目前广泛采用的将虚拟现实与自主智能系统相结合来克服时延影响的遥操作系统中，解决现场环境的几何学、动力学模型参数未知或不准确引起的相关问题。并以在遥操作系统中的直升飞机作为执行端为例，说明该系统框架模型的具体应用和可用性。     

Absolute stability analysis of sampled-data scaled bilateral teleoperation systems

Stability of a bilateral teleoperation system may be jeopardized by controller discretization, which has been shown to involve energy leaks. This paper proposes a novel approach to analyzing the absolute stability of sampled-data bilateral teleoperation systems consisting of discrete-time controllers and continuous-time master, slave, operator, and environment. The proposed stability analysis permits scaling and delay in the master and the slave positions and forces. The absolute stability conditions reported here impose bounds on the gains of the discrete-time controller, the damping terms of the master and the slave, and the sampling time. A design-related application of these results is in proper selection of various control parameters and the sampling rate for stable teleoperation under discrete-time control. To explore the trade-off between the control gains and the sampling time, it is studied that how large sampling times, which require low control gains for maintaining stability, can lead to unacceptable teleoperation transparency and human task performance in a teleoperated switching task. This shows that the effect of sampling time must be taken into account because neglecting it (as in the absolute stability literature) undermines both stability and transparency of teleoperation. The resulting absolute stability condition has been verified via experiments with two Phantom Omni robots.