Gaze direction based vehicle teleoperation method with omnidirectional image stabilization and automatic body rotation control

In robot teleoperation, a robot works as a physical agent at a remote site for a robot operator. There are mainly two tasks in robot teleoperation using camera images: environment recognition using visual information and robot control according to the recognition. In this paper, we propose a gaze direction based vehicle teleoperation method with an omnidirectional image stabilization and an automatic body rotation control. In the proposed method, we manage above two tasks in the same manner that are usually treated separately. This method is an intuitive vehicle teleoperation method where an operator do not need to have concern about vehicle body orientations and can absorb differences of vehicle driving mechanisms. That is, this method frees an operator from being bothered from controlling a vehicle and the operator can concentrate on where he/she intends to go. This method mainly consists of two technologies: an omnidirectional image stabilization technology and automatic body rotation control. The conducted experiments show effectiveness of the proposed method.     

面向任务的虚拟环境和人机交互方法研究

基于虚拟现实技术的人机交互方法为面向任务的机器人远程作业研究提供了新的思 想和思路，增强了机器人系统的遥控作业能力．本文对机器人系统中虚拟环境和人机交互方 法研究进行了综述，对虚拟环境中的示教和遥操作进行了评析，并探讨了未来遥操作机器人 系统作业的发展趋势．     

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Robotic teleoperation, i.e., manipulating remote robotic systems at a distance, has gained its popularity in various industrial applications, including construction operations. The key to a successful teleoperation robot system is the delicate design of the human-robot interface that helps strengthen the human operator’s situational awareness. Traditional human-robot interface for robotic teleoperation is usually based on imagery data (e.g., video streaming), causing the limited field of view (FOV) and increased cognitive burden for processing additional spatial information. As a result, 3D scene reconstruction methods based on point cloud models captured by scanning technologies (e.g., depth camera and LiDAR) have been explored to provide immersive and intuitive feedback to the human operator. Despite the added benefits of applying reconstructed 3D scenes in telerobotic systems, challenges still present. Most 3D reconstruction methods utilize raw point cloud data due to the difficulty of real-time model rendering. The significant size of point cloud data makes the processing and transfer between robots and human operators difficult and slow. In addition, most reconstructed point cloud models do not contain physical properties such as weight and colliders. A more enriched control mechanism based on physics engine simulations is impossible. This paper presents an intelligent robot teleoperation interface that collects, processes, transfers, and reconstructs the immersive scene model of the workspace in Virtual Reality (VR) and enables intuitive robot controls accordingly. The proposed system, Telerobotic Operation based on Auto-reconstructed Remote Scene (TOARS), utilizes a deep learning algorithm to automatically detect objects in the captured scene, along with their physical properties, based on the point cloud data. The processed information is then transferred to the game engine where rendered virtual objects replace the original point cloud models in the VR environment. TOARS is expected to significantly improve the efficiency of 3D scene reconstruction and situational awareness of human operators in robotic teleoperation.     

预估控制下的实时网络遥操作移动机器人

构建了能使操作者通过Internet远程实时控制的移动机器人系统．为了补偿网络时延和抵消其对遥操作系统的影响，基于我们以前提出的改进型Smith预估器原理，采用了预估控制策略．为了保证系统稳定性和透明性，基于主从端的传感器信息交换，设计了一个动态模型管理器，其中模型和力反馈误差调节通过模糊控制实现．除了力反馈外，为了增强遥操作的实时性，引入了预估的虚拟显示．为了精确地预测网络时延，提出了一个新颖的时钟同步算法．为了降低时延抖动，结合我们提出的两个算法，实现了数据缓冲策略．最后，通过长距离的网络遥操作实验验证了系统和控制策略的实用性和有效性．     

Force reflecting teleoperation with adaptive impedance control

Experimentation and a survey of the literature clearly show that contact stability in a force reflecting teleoperation system requires high levels of damping on the master robot. However, excessive damping increases the energy required by an operator for commanding motion. The objective of this paper is to describe a new force reflecting teleoperation methodology that reduces operator energy requirements without sacrificing stability. We begin by describing a new approach to modeling and identifying the remote environment of the teleoperation system. We combine a conventional multi-input, multi-output recursive least squares (MIMO-RLS) system identification, identifying in real-time the remote environment impedance, with a discretized representation of the remote environment. This methodology generates a time-varying, position-dependent representation of the remote environment dynamics. Next, we adapt the target impedance of the master robot with respect to the dynamic model of the remote environment. The environment estimation and impedance adaptation are executed simultaneously and in real time. We demonstrate, through experimentation, that this approach significantly reduces the energy required by an operator to execute remote tasks while simultaneously providing sufficient damping to ensure contact stability.     

Bilateral teleoperation through the Internet

This paper proposes a stable control structure for the bilateral teleoperation of robots through Internet. The problem is motivated by the increasing use of the Internet as a communication channel. Internet has a time-varying delay which depends on factors such as congestion, bandwidth and distance. In this work, we propose a control structure for the teleoperation of a manipulator robot with force feedback. Such a control structure includes state controllers (placed on the local and remote sites) and a time-delay compensation, which modifies the delayed position command generated by the human operator using the force that he feels in such a delayed moment and the current force between the slave and the remote environment. In addition, the proposed control scheme is designed considering a model of the communication channel. Finally, experiments of bilateral teleoperation of robots through Intranet and Internet are shown to test the performance and stability of the designed teleoperation system.     

Teleoperation of Mobile Robots With Time-Varying Delay

This paper proposes a stable control scheme for teleoperation of mobile robots with visual feedback in presence of time-varying delay. The proposed control scheme is based on using a model of the human operator to combine (on the remote site) the velocity command generated by the human operator in a delayed time instant, the received information (which stimulates the operator) in such moment, and the current state of the remote site to set the velocity reference, which is applied to the mobile robot. In addition, the proposed control scheme does not modify the information sent from the remote site to the local site. On the other hand, the proposed scheme uses estimated parameters of the human operator and a fictitious force, which is calculated using data fusion from ultrasonic sensors and optical flow field taken from panoramic images. To illustrate the performance and stability of the proposed control structure, several teleoperation experiments between Argentina and Brazil linked via Internet are shown.     

Preface

This paper discusses the problems in teleoperation systems for a mobile robot and the utilization of a virtual world in such systems. In order to achieve smooth operation of the mobile robot through a communication link, we should consider time delays in data transfer. To compensate for the incomplete data sets, the virtual images can be generated by computer graphics when the information on the working environment can be acquired beforehand. In this paper, we construct a teleoperation system with a virtual world. The performance of the system is examined through experiments with actual mobile robots which show that the virtual robot can be operated by an operator in almost the same manner as the teleoperated real robot. In an experimental environment with a second moving robot, we can keep the status of the second robot under perfect control and operate the first robot with no interference.     

Bilateral teleoperation of a group of mobile robots for cooperative tasks

A visual and force feedback-based teleoperation scheme is proposed for cooperative tasks. The bilateral teleoperation system includes a haptic device, an overhead camera and a group of wheeled robots. The commands of formation and average velocities of the multiple robots are generated by the operator through the haptic device. The state of the multiple robots and the working environment is sent to the human operator. The received information contains the feedback force through the haptic device and visual information returned by a depth camera. The feedback force based on the difference between the desired and actual average velocities is presented. The wave variable method is employed in the bilateral teleoperation of multiple mobile robots with time delays. The effectiveness of the bilateral teleoperation system is demonstrated by experiments. The robots in the slave side are able to follow the commands from the master side to interact with the environments, including moving in different formations and pushing a box. The results show that the scheme enables the operator to manipulate a group of robots to complete cooperative tasks freely.     

Augmented reality-based robot teleoperation system using RGB-D imaging and attitude teaching device

Augmented reality (AR)-based programming using the demonstration method has been widely studied. However, studies on AR-based programming for remote robots are lacking because of the limitation of human–computer interaction. This paper proposes an AR-based robot teleoperation system and method using RGB-D imaging and an attitude teaching device. By sending the color and depth images of the remote robot environment to the local side, the operators can complete the teleoperation of the robot at the local side. First, the operators select key positions on the motion path of the robot endpoint from color images via a mouse, and the computer calculates the 3D coordinates of these key points in the robot base coordinate system to complete the position teaching process. In the robot attitude teaching process, the AR technology is used to superimpose the virtual robot model onto the color images of the robot teleoperation environment, so as to make the virtual robot endpoint to move along the teaching path. An operator can use the portable attitude teaching device designed in this study to control the robot movement parameters, such as the attitude and motion speed, during the movement of the virtual robot. After the position and attitude teaching processes, the robot movement trajectory can be generated. To make the base coordinate system of the virtual model consistent with that of the physical robot, we propose an online AR registration method, which does not require manually placing the AR registration marker. The proposed AR-based robot teleoperation system can quickly and easily complete robot teleoperation at the local side.     

Enhanced teleoperation performance using hybrid control and virtual fixture

To develop secure, natural and effective teleoperation, the perception of the slave plays a key role for the interaction of a human operator with the environment. By sensing slave information, the human operator can choose the correct operation in a process during the human–robot interaction. This paper develops an integrated scheme based on a hybrid control and virtual fixture approach for the telerobot. The human operator can sense the slave interaction condition and adjust the master device via the surface electromyographic signal. This hybrid control method integrates the proportional-derivative control and the variable stiffness control, and involves the muscle activation at the same time. It is proposed to quantitatively analyse the human operator's control demand to enhance the control performance of the teleoperation system. In addition, due to unskilful operation and muscle physiological tremor of the human operator, a virtual fixture method is developed to ensure accuracy of operation and to reduce the operation pressure on the human operator. Experimental results demonstrated the effectiveness of the proposed method for the teleoperated robot.     

基于领域知识的3维动态场景目标识别和配准

为了增强遥操作系统的临场感,使操作者更好地融入远程工作环境,提出了基于领域知识的3维动态场景目标识别和配准算法.首先,通过离线解析和切分虚拟样机CAD（计算机辅助设计）模型构建包含多视角点云特征和装配约束的领域知识库.其次,通过动态采集场景点云并计算CVFH（clustered viewpoint feature histogram）和FPFH（fast point feature histogram）特征,利用领域知识库中多视角点云特征和点云CVFH特征进行比对实现目标识别,并由FPFH特征经过两步配准实现目标姿态确定.最后,利用装配约束知识库实现以遥操作机器人工作状态变化为驱动的指导信息的精确配准和实时推送.实验结果表明,该系统不但可以有效地指导远程机器人完成维修操作,还可以提高遥操作的精度和效率.     

Human-assisted virtual reality for a magnetic-haptic micromanipulation platform

This paper deals with the development of a virtual reality interface (VRI) for a magnetic-haptic micromanipulation platform (MHMP) (Mehrtash et?al. in IEEE/ASME Trans Mechatron 16(3):459–469, 2011). Our previously developed MHMP has shown a great deal of promise in non-contact micromanipulations. This micromanipulation platform concerns the integration of magnetic actuation technology and a bilateral macro–micro teleoperation. The MHMP has two separate stations: one magnetic microrobotic station and one haptic. The magnetic microrobotic station manipulates micro-sized objects based on the commands from the haptic station. The haptic station uses bilateral communication with the magnetic microrobotic station to allow a human operator the feeling of a micro-domain environment. In this paper, we report a VRI that enables human operators to improve their skills in using the MHMP, before carrying out an actual dexterous task. The VRI is made up of three main components: a haptic station, a simulation engine, and a display unit. The haptic station provides the operator with the force/torque information from virtual or remote environments, and is also used to recognize the operator’s hand motion command. Dynamical computation and control system modeling have been carried out on the simulation engine. Based on the real-time computation, this engine, as the heart of the system, provides force applied to the operator’s hand and the microrobot’s position for the haptic station and the display unit, respectively. The display unit employs 3D computer graphics to demonstrate the micromanipulation tasks and environments. The VRI is also developed in such a way that it can be separately used in parallel with the MHMP for the 3D visualization of a real task by providing multiple virtual viewports. This paper introduces the configuration of the proposed VRI, and reports the result of a preliminary experiment using micromanipulation investigation for validation.     

A New Difficulty Index for Teleoperated Robots Driving through Obstacles

Teleoperation allows humans to reach environments that would otherwise be too difficult or dangerous. The distance between the human operator and remote robot introduces a number of issues that can negatively impact system performance including degraded and delayed information exchange between the robot and human. Some operation scenarios and environments can tolerate these degraded conditions, while others cannot. However, little work has been done to investigate how factors such as communication delay, automation, and environment characteristics interact to affect teleoperation system performance. This paper presents results from a user study analyzing the effects of teleoperation factors including communication delay, autonomous assistance, and environment layout on user performance. A mobile robot driving task is considered in which subjects drive a robot to a goal location around obstacles as quickly (minimize time) and safely (avoid collisions) as possible. An environment difficulty index (ID) is defined in the paper and is shown to be able to predict the average time it takes for the human to drive the robot to a goal location with different obstacle configurations. The ID is also shown to predict the path chosen by the human better than travel time along that path.     

基于Internet多操作者多机器人的遥操作系统的研究

近年来基于Internet多操作者多机器人MOMR(Multi-Operator-Multi-Robot)协作成 为许多学者关注的对象．相对于单操作者单机器人SOSR(Single-Operator-Single-Robot )遥操作任务的单一性和局限性，MOMR在群体作业如：设备维护、建筑建造、外科手术等许 多场合具有明显的优势．对于危险或难以到达的环境，MOMR遥操作系统的控制成为一个主要 问题． 本文综述了基于Internet 的MOMR系统的结构、概念以及国内外的研究现状，并讨论了MOMR 系统存在的主要问题和各种协调(或协作)控制方法．     

Control gain adaptation in virtual reality mediated human–telerobot interaction

The Internet connects millions of computers worldwide, and provides a new potential working environment for remote‐controlled telerobotic systems. The main limitation of using the Internet in this application is random delays between communicating nodes, which can cause disturbances in human–machine interaction and affect telepresence experiences. This is particularly important in systems integrating virtual reality technology to present interfaces. Telepresence, or the sense of presence in a remote environment, hypothetically is positively related to teleoperation task performance. This research evaluated the effect of constant and random network (communication) delays on remote‐controlled telerover performance, operator workload, and telepresence experiences. The research also assessed the effect of using a system gain adaptation algorithm to offset the negative impact of communication delays on the various response measures. It was expected that with gain adaptation, system stability, performance, and user telepresence experiences would improve with a corresponding decrease in workload. Results indicated that gain adaptation had a significant effect on the performance measures. The study demonstrated that gain adaptation could reduce deterioration in telepresence experiences and improve user performance in teleoperated and telerobotic control. © 2005 Wiley Periodicals, Inc. Hum Factors Man 15: 259–274, 2005.     

Reinforcement Learning and Robust Control for Robot Compliance Tasks

The complexity in planning and control of robot compliance tasks mainly results from simultaneous control of both position and force and inevitable contact with environments. It is quite difficult to achieve accurate modeling of the interaction between the robot and the environment during contact. In addition, the interaction with the environment varies even for compliance tasks of the same kind. To deal with these phenomena, in this paper, we propose a reinforcement learning and robust control scheme for robot compliance tasks. A reinforcement learning mechanism is used to tackle variations among compliance tasks of the same kind. A robust compliance controller that guarantees system stability in the presence of modeling uncertainties and external disturbances is used to execute control commands sent from the reinforcement learning mechanism. Simulations based on deburring compliance tasks demonstrate the effectiveness of the proposed scheme.     

An operator interface for teleprogramming employing synthetic fixtures

In the teleprogramming system an operator is presented with a virtual reality representation of a remote environment. The operator's interaction within that virtual environment is observed and translated into a sequence of robot program instructions for transmission to, and execution by, a robot at the remote site. In this paper we focus on operator interaction with the master station of the teleprogramming system. The use of synthetic fixtures to provide force and visual clues to assist the operator in performing tasks with speed and precision is discussed. It is suggested that, at least in some situations, it is both necessary and desirable to trade off realism for improved task performance. The difficulty of coping with exceptional conditions and, in particular, uncertainty in the world model used to generate the virtual environment is described and the operator interface for diagnosing and resolving errors is presented. An overview is also given of both the hardware and software used to implement master station for the teleprogramming system.     

基于视觉/力觉辅助的遥操作系统研究与实现

针对多变未知的作业环境以及复杂繁多的作业任务,设计一种基于视觉/力觉辅助作业的遥操作系统.首先,利用Kinect摄像头采集作业现场的3D点云数据,结合虚拟现实技术,在计算机上将虚拟机械臂模型叠加到真实环境中,同时显示虚拟模型与现场信息,提高操作人员的现场感知能力;根据获取的障碍物和目标物体位置信息,建立人工势场,增强操作者控制远端机械臂进行避障及接近目标物体操作的能力,完成环绕和避障任务.该方案在7自由度Schunk机械臂上得到了实现.结果表明,此方案可以很好地弥补虚拟环境缺乏现场信息的缺点,提高了遥操作系统的可靠性.     

Forbidden region virtual fixtures from streaming point clouds

Forbidden region virtual fixtures protect objects from unwanted contact with a robot. In this paper, we propose a method for creating forbidden region haptic virtual fixtures for teleoperation from streaming point clouds obtained by an RGB-D camera. Upon violating the protected area, the operator receives force feedback that opposes motion inside the forbidden region. Three architectures for creating virtual fixtures are presented and their advantages and disadvantages are described. The proposed methods have the ability to implement constraints and can handle dynamic environments in real-time. The effectiveness of the methods is demonstrated in experiments with a surgical robot.