Teleoperation Control of a Position‐Based Impedance Force Controlled Mobile Robot by Neural Network Learning: Experimental Studies

Effective haptic performance in teleoperation control systems can be achieved by solving two major problems: the time‐delay in communication channels and the transparency of force control. The time‐delay in communication channels causes poor performance and even instability in a system. The transparency of force feedback is important for an operator to improve the performance of a given task. This article suggests a possible solution for these two problems through the implementation of a teleoperation control system between the master haptic device and the slave mobile robot. Regulation of the contact force in the slave mobile robot is achieved by introducing a position‐based impedance force control scheme in the slave robot. The time‐delay problem is addressed by forming a Smith predictor configuration in the teleoperation control environment. The configuration of the Smith predictor structure takes the time‐delay term out of the characteristic equation in order to make the system stable when the system model is given a priori. Since the Smith predictor is formulated from exact linear modeling, a neural network is employed to identify and model the slave robot system as a nonlinear model estimator. Simulation studies of several control schemes are performed. Experimental studies are conducted to verify the performance of the proposed control scheme by regulating the contact force of a mobile robot through the master haptic device.     

Transparency analysis and delay compensation scheme for haptic-based networked virtual environments

Haptic-based NVEs (networked virtual environments) with CS (client/server) communication architectures support better consistency but induce larger end-to-end delays than those with P2P (peer-to-peer) communication architectures. Unfortunately, large delay severely deteriorates the transparency (i.e., reality) of haptic interaction. To improve the haptic interaction quality for haptic-based NVEs with CS communication architectures, in this paper the degradation of haptic interaction quality is analyzed according to network delays. Based on the analysis, the maximum allowable delay bound is predicted and unrealistic force feedback caused by the network delay is compensated. Experimental results confirm that the proposed analysis provides an acceptable quantification method about haptic interaction quality and that the proposed delay-compensation scheme effectively improves haptic interaction quality with respect to network delays.     

Teleoperation of Mobile Robots With Time-Varying Delay

This paper proposes a stable control scheme for teleoperation of mobile robots with visual feedback in presence of time-varying delay. The proposed control scheme is based on using a model of the human operator to combine (on the remote site) the velocity command generated by the human operator in a delayed time instant, the received information (which stimulates the operator) in such moment, and the current state of the remote site to set the velocity reference, which is applied to the mobile robot. In addition, the proposed control scheme does not modify the information sent from the remote site to the local site. On the other hand, the proposed scheme uses estimated parameters of the human operator and a fictitious force, which is calculated using data fusion from ultrasonic sensors and optical flow field taken from panoramic images. To illustrate the performance and stability of the proposed control structure, several teleoperation experiments between Argentina and Brazil linked via Internet are shown.     

Human-centered control scheme for delayed bilateral teleoperation of mobile robots

Teleoperation task performance strongly depends on how well the human operator’s commands are executed. In this paper, we propose a control scheme for delayed bilateral teleoperation of mobile robots that considers user’s commands execution in order to achieve a high-performance teleoperation system in some important aspects like time to complete the task, safety, and operator dependence. We describe some evaluation metrics that allow us to address these aspects and a quantitative metric is proposed and incorporated in the control scheme to compensate wrong commands. A force feedback is applied to the master at the local site as a haptic cue. In addition, the system stability is analyzed taking into consideration the master and remote robot dynamic models and the asymmetric time-varying delays of the communication channel. Multiple human-in-the-loop simulations were carried out and the results of the evaluation metrics were discussed. Additionally, we present experiments where a user teleoperates a mobile robot via the Internet connection between Argentina and Italy.     

预估控制下的实时网络遥操作移动机器人

构建了能使操作者通过Internet远程实时控制的移动机器人系统．为了补偿网络时延和抵消其对遥操作系统的影响，基于我们以前提出的改进型Smith预估器原理，采用了预估控制策略．为了保证系统稳定性和透明性，基于主从端的传感器信息交换，设计了一个动态模型管理器，其中模型和力反馈误差调节通过模糊控制实现．除了力反馈外，为了增强遥操作的实时性，引入了预估的虚拟显示．为了精确地预测网络时延，提出了一个新颖的时钟同步算法．为了降低时延抖动，结合我们提出的两个算法，实现了数据缓冲策略．最后，通过长距离的网络遥操作实验验证了系统和控制策略的实用性和有效性．     

Bilateral teleoperation through the Internet

This paper proposes a stable control structure for the bilateral teleoperation of robots through Internet. The problem is motivated by the increasing use of the Internet as a communication channel. Internet has a time-varying delay which depends on factors such as congestion, bandwidth and distance. In this work, we propose a control structure for the teleoperation of a manipulator robot with force feedback. Such a control structure includes state controllers (placed on the local and remote sites) and a time-delay compensation, which modifies the delayed position command generated by the human operator using the force that he feels in such a delayed moment and the current force between the slave and the remote environment. In addition, the proposed control scheme is designed considering a model of the communication channel. Finally, experiments of bilateral teleoperation of robots through Intranet and Internet are shown to test the performance and stability of the designed teleoperation system.     

Experimental evaluation of haptic data communication with prediction

The transmission of haptic data is relatively challenging in multimedia communication. In this research study, the methods are presented for exploiting the properties of human haptic perception for data reduction of haptic data transmission. Packet-switched communication of haptic data is characterized by high packet rates on the communication channel. The quality of the internet-based haptic tele-control/tele-presence systems is highly dependent on the quality of the communication channel between the operator and the remote site, and on the delay jitter in the data exchange. The proposed research work is evaluated experimentally using a Geomagic Touch (previously PHANTOM Sensable Omni) haptic device with a sphere as a virtual model. Four experiments were conducted to evaluate the proposed research study. In the first experiment the JND Weber’s law is applied on sent force values only, while in the second experiment, the force calculation algorithm has been modified to include human movement velocity. The third experiment discusses the use of JND on the sent velocity values. The evaluation of the human’s perception shows that the proposed modification to the basic dead-band approach highly reduces the number of sent packets with minimal disturbance in haptic feeling. Further enhancements using prediction techniques have also been introduced in the fourth experimental evaluation. The linear predictions are added to the above proposed reduction methods. Combining the dead-band approach with a fast, configurable and accurate prediction algorithm enables a significant reduction in the amount of data sent across the network. The reduction is estimated to be 85%, while preserving the original data structure.     

Remote Tactile Transmission with Time Delay for Robotic Master–Slave Systems

This study develops a method to compensate for the communication time delay for tactile transmission systems. For transmitting tactile information from remote sites, the communication time delay degrades the validity of feedback. However, so far time delay compensation methods for tactile transmissions have yet to be proposed. For visual or force feedback systems, local models of remote environments were adopted for compensating the communication delay. The local models cancel the perceived time delay in sensory feedback signals by synchronizing them with the users' operating movements. The objectives of this study are to extend the idea of the local model to tactile feedback systems and develop a system that delivers tactile roughness of textures from remote environments to the users of the system. The local model for tactile roughness is designed to reproduce the characteristic cutaneous deformations, including vibratory frequencies and amplitudes, similar to those that occur when a human finger scans rough textures. Physical properties in the local model are updated in real-time by a tactile sensor installed on the slave-side robot. Experiments to deliver the perceived roughness of textures were performed using the developed system. The results showed that the developed system can deliver the perceived roughness of textures. When the communication time delay was simulated, it was confirmed that the developed system eliminated the time delay perceived by the operators. This study concludes that the developed local model is effective for remote tactile transmissions.     

基于视觉反馈和预测仿真的Internet机器人遥操作

随着网络技术的发展,基于Internet的机器人遥操作因其广泛的应用前景而成为研究的热点,如何消除Internet引入的复杂时延对控制系统的影响是解决问题的关键.该文在分析Internet两点间时延规律的基础上,设计并实现了一个基于视觉反馈和预测仿真的机器人遥操作系统(Telemanipulator,Tman),三维仿真服务器模拟出操作现场,远程用户根据视觉反馈和本机的三维仿真及时做出控制决策;同时,为了消除时延可能导致的超调所带来的安全隐患,采取远程控制结合本地自主控制的模式,由本地服务器完成最后的小区域内的精细操作.实验结果表明仿真预测和控制模式切换有效地补偿了时延,保证了控制系统的鲁棒性和安全性.     

Position referenced force augmentation in teleoperated hydraulic manipulators operating under delayed and lossy networks: A pilot study

Position error between motions of the master and slave end-effectors is inevitable as it originates from hard-to-avoid imperfections in controller design and model uncertainty. Moreover, when a slave manipulator is controlled through a delayed and lossy communication channel, the error between the desired motion originating from the master device and the actual movement of the slave manipulator end-effector is further exacerbated. This paper introduces a force feedback scheme to alleviate this problem by simply guiding the operator to slow down the haptic device motion and, in turn, allows the slave manipulator to follow the desired trajectory closely. Using this scheme, the master haptic device generates a force, which is proportional to the position error at the slave end-effector, and opposite to the operator’s intended motion at the master site. Indeed, this force is a signal or cue to the operator for reducing the hand speed when position error, due to delayed and lossy network, appears at the slave site. Effectiveness of the proposed scheme is validated by performing experiments on a hydraulic telemanipulator setup developed for performing live-line maintenance. Experiments are conducted when the system operates under both dedicated and wireless networks. Results show that the scheme performs well in reducing the position error between the haptic device and the slave end-effector. Specifically, by utilizing the proposed force, the mean position error, for the case presented here, reduces by at least 92% as compared to the condition without the proposed force augmentation scheme. The scheme is easy to implement, as the only required on-line measurement is the angular displacement of the slave manipulator joints.     

Synchronization control for physics-based collaborative virtual environments with shared haptics

In this paper, we propose a synchronization scheme to achieve a high level of consistency in peer-to-peer-based shared virtual environments (SVEs), as well as to display natural and realistic motions of virtual objects, in collaborative haptic tasks. The synchronization scheme utilizes an advanced feedback controller to compensate for the state error between geographically separated sites with a significant amount of time delay. It is designed using the mathematical model of a two-user SVE manipulating a freely moving object represented as a mass with damping resistance, with a haptic interface. Thanks to feedback control theory of time delay systems, the controller is shown to result in closed-loop stability and is be robust to perturbations in the time delay. Together with the synchronization control, a recovery filter is also designed and integrated so as to preserve the natural behavior of the synchronized object, which is, otherwise, affected by the feedback control action. In addition to verifying the theoretical results, two experiments using real Internet and local area network communications are carried out. These tests clearly support the validity of the analyses and demonstrate the applicability of the synchronization scheme.     

High-fidelity bilateral teleoperation systems and the effect of multimodal haptics.

In master-slave teleoperation applications that deal with a delicate and sensitive environment, it is important to provide haptic feedback of slave/environment interactions to the user's hand as it improves task performance and teleoperation transparency (fidelity), which is the extent of telepresence of the remote environment available to the user through the master-slave system. For haptic teleoperation, in addition to a haptics-capable master interface, often one or more force sensors are also used, which warrant new bilateral control architectures while increasing the cost and the complexity of the teleoperation system. In this paper, we investigate the added benefits of using force sensors that measure hand/master and slave/environment interactions and of utilizing local feedback loops on the teleoperation transparency. We compare the two-channel and the four-channel bilateral control systems in terms of stability and transparency, and study the stability and performance robustness of the four-channel method against nonidealities that arise during bilateral control implementation, which include master-slave communication latency and changes in the environment dynamics. The next issue addressed in the paper deals with the case where the master interface is not haptics capable, but the slave is equipped with a force sensor. In the context of robotics-assisted soft-tissue surgical applications, we explore through human factors experiments whether slave/environment force measurements can be of any help with regard to improving task performance. The last problem we study is whether slave/environment force information, with and without haptic capability in the master interface, can help improve outcomes under degraded visual conditions.     

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

     

A teleoperation system for micro positioning with haptic feedback

This paper presents the research work on a 1 Degree of Freedom (DOF) macro-micro teleoperation system which enables human operator to perform complex task in micro environment such as cell insertion with the capability of haptic feedback. To reach submicron resolution, a nano-motion piezo actuator was used as the slave robot and a servo DC motor was used as the master robot. Force sensors were implemented at both ends for haptic feedback and a microscope equipped with camera was employed for real-time visual feedback. The hysteresis nonlinearity of the piezo motor was modeled using LuGre friction model and compensated for. A Sliding Mode Based Impedance Controller (SMBIC) was designed at the slave side to ensure position tracking while an impedance force controller was designed at the master side to ascertain tracking of the force. Control parameters were chosen based on Llewellyn stability criteria such that the entire system stays stable against parameter uncertainties and constant time delay. The experimental results demonstrated capability of the proposed control frameworks in desirable tracking of the position and force signals while the entire system remained stable. The results of this study can be used for complex tasks in micron environment such as cell insertion.     

Bilateral teleoperation of a group of mobile robots for cooperative tasks

A visual and force feedback-based teleoperation scheme is proposed for cooperative tasks. The bilateral teleoperation system includes a haptic device, an overhead camera and a group of wheeled robots. The commands of formation and average velocities of the multiple robots are generated by the operator through the haptic device. The state of the multiple robots and the working environment is sent to the human operator. The received information contains the feedback force through the haptic device and visual information returned by a depth camera. The feedback force based on the difference between the desired and actual average velocities is presented. The wave variable method is employed in the bilateral teleoperation of multiple mobile robots with time delays. The effectiveness of the bilateral teleoperation system is demonstrated by experiments. The robots in the slave side are able to follow the commands from the master side to interact with the environments, including moving in different formations and pushing a box. The results show that the scheme enables the operator to manipulate a group of robots to complete cooperative tasks freely.     

小型化柔性触觉再现装置

提出了一种小型化的具有实时触觉反馈作用的人机接口装置的设计方法,以控制弹性梁的有效变形长度为核心,保证了触觉再现的实时性和准确性,从而可以在操作者指端大范围再现虚拟对象或远程对象的刚度。介绍了该小型化人机触觉再现接口装置的硬件设计和软件设计。该实时触觉再现接口具有尺寸小、刚度再现范围大,可方便与鼠标相结合等特点,可广泛应用于遥操作机器人和虚拟现实领域。     

Simultaneous remote haptic collaboration for assembling tasks

Stand-alone virtual environments (VEs) using haptic devices have proved useful for assembly/disassembly simulation of mechanical components. Nowadays, collaborative haptic virtual environments (CHVEs) are also emerging. A new peer-to-peer collaborative haptic assembly simulator (CHAS) has been developed whereby two users can simultaneously carry out assembly tasks using haptic devices. Two major challenges have been addressed: virtual scene synchronization (consistency) and the provision of a reliable and effective haptic feedback. A consistency-maintenance scheme has been designed to solve the challenge of achieving consistency. Results show that consistency is guaranteed. Furthermore, a force-smoothing algorithm has been developed which is shown to improve the quality of force feedback under adverse network conditions. A range of laboratory experiments and several real trials between Labein (Spain) and Queen’s University Belfast (Northern Ireland) have verified that CHAS can provide an adequate haptic interaction when both users perform remote assemblies (assembly of one user’s object with an object grasped by the other user). Moreover, when collisions between grasped objects occur (dependent collisions), the haptic feedback usually provides satisfactory haptic perception. Based on a qualitative study, it is shown that the haptic feedback obtained during remote assemblies with dependent collisions can continue to improve the sense of co-presence between users with regard to only visual feedback.     

Hybrid client-server architecture and control techniques for collaborative product development using haptic interfaces

In this paper, a collaborative product development and prototyping framework is proposed by using distributed haptic interfaces along with deformable objects modeling. Collaborative Virtual Environment (CVE) is a promising technique for industrial product development and virtual prototyping. Network control problems such as network traffic and network delay in communication have greatly limited collaborative virtual environment applications. The problems become more difficult when high-update-rate haptic interfaces and computation intensive deformable objects modeling are integrated into CVEs for intuitive manipulation and enhanced realism. A hybrid network architecture is proposed to balance the computational burden of haptic rendering and deformable object simulation. Adaptive artificial time compensation is used to reduce the time discrepancy between the server and the client. Interpolation and extrapolation approaches are used to synchronize graphic and haptic data transmitted over the network. The proposed techniques can be used for collaborative product development, virtual assembly, remote product simulation and other collaborative virtual environments where both haptic interfaces and deformable object models are involved.     

Priority-based haptic event filtering for transmission and error control in networked virtual environments

A data filtering scheme is proposed for transmission and error control of haptic events in haptic-based network virtual environments; this scheme is called as priority-based haptic event filtering. Because a high update rate of approximately 1 kHz is required for haptic rendering, sophisticated transmission rate control and reduction schemes are necessary for the haptic events. Although existing schemes can reduce the transmission rate without any perception impairment, they are very sensitive to packet losses. In this paper, we prioritize the haptic events according to the delay and loss effects. Utilizing the proposed haptic event prioritization, the proposed filtering scheme adapts the transmission rate and updates the predicted loss rate according to the current network state. Our simulation and experiment results confirm that the proposed scheme can effectively select important haptic events and guarantee an improved haptic interaction quality over a bandwidth-limited lossy network than existing transmission schemes tailored for networked haptics.