Dynamic Visual Tracking for Manipulators Using an Uncalibrated Fixed Camera

This paper presents a new controller for controlling a number of feature points on a robot manipulator to trace desired trajectories specified on the image plane of a fixed camera. It is assumed that the intrinsic and extrinsic parameters of the camera are not calibrated. A new adaptive algorithm is developed to estimate the unknown parameters online, based on three original ideas. First, we use the pseudoinverse of the depth-independent interaction matrix to map the image errors onto the joint space of the manipulator. By eliminating the depths in the interaction matrix, we can linearly parameterize the closed-loop dynamics of the manipulator. Second, to guarantee the existence of the pseudoinverse, the adaptive algorithm introduces a potential force to drive the estimated parameters away from the values that result in a singular Jacobian matrix. Third, to ensure that the estimated parameters are convergent to their true values up to a scale, we combine the Slotine-Li method with an online algorithm for minimizing the error between the estimated projections and real image coordinates of the feature points. We have proved asymptotic convergence of the image errors to zero by the Lyapunov theory based on the nonlinear robot dynamics. Experiments have been carried out to verify the performance of the proposed controller.     

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine–Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.      

微小无人直升机自适应视觉伺服

针对摄像机未标定和特征点坐标未知的情况, 本文提出一种新颖的基于图像的无人直升机自适应视觉伺服方法. 控制器是基于反推法设计的, 但是和已有的基于反推法的视觉伺服不同的是, 它利用与深度无关矩阵将图像误差映射到执行器空间, 从而可以避免估计特征点的深度. 这种设计方法可以线性化未知的摄像机参数和特征点坐标, 所以能方便地设计自适应算法来在线估计这些未知参数, 同时为了保证图像误差收敛和避免估计参数收敛至零解而引入了两个势函数. 利用Lyapunov方法证明了基于非线性动力学的控制器的稳定性, 并给出了仿真验证.     

Adaptive visual servoing for the robot manipulator with extreme learning machine and reinforcement learning

In this study, a novel image-based visual servo (IBVS) controller for robot manipulators is investigated using an optimized extreme learning machine (ELM) algorithm and an offline reinforcement learning (RL) algorithm. First of all, the classical IBVS method and its difficulties in accurately estimating the image interaction matrix and avoiding the singularity of pseudo-inverse are introduced. Subsequently, an IBVS method based on ELM and RL is proposed to solve the problem of the singularity of the pseudo-inverse solution and tune adaptive servo gain, improving the servo efficiency and stability. Specifically, the ELM algorithm optimized by particle swarm optimization (PSO) was used to approximate the pseudo-inverse of the image interaction matrix to reduce the influence of camera calibration errors. Then, the RL algorithm was adopted to tune the adaptive visual servo gain in continuous space and improve the convergence speed. Finally, comparative simulation experiments on a 6-DOF robot manipulator were conducted to verify the effectiveness of the proposed IBVS controller.     

软体机器人手眼视觉/形状混合控制

当末端带有相机的连续型软体机器人进行作业时,由于避障、安全性等多方面因素,既需要末端相机－机器人系统的视觉伺服,也需要机器人的整体形状控制．针对这个问题,本文提出了一种软体机器人手眼视觉/形状混合控制方法．该方法无需知道空间特征点的3维坐标,只需给定特征点在末端相机像平面的期望像素坐标和软体机器人的期望形状就可达到控制目的．建立了软体机器人的运动学模型,利用该模型,结合深度无关交互矩阵自适应手眼视觉控制和软体机器人形状控制,提出了一种混合控制律,并用李亚普诺夫稳定性理论对该控制律进行证明．仿真和实验的结果均表明,末端相机特征点像素坐标和形状可以收敛到期望值．     

Nonsingular Terminal Sliding Mode Control of Robot Manipulators Using Fuzzy Wavelet Networks

This paper presents an adaptive nonsingular terminal sliding mode (NTSM) tracking control design for robotic systems using fuzzy wavelet networks. Compared with linear hyperplane-based sliding control, terminal sliding mode controller can provide faster convergence and higher precision control. Therefore, a terminal sliding controller combined with the fuzzy wavelet network, which can accurately approximate unknown dynamics of robotic systems by using an adaptive learning algorithm, is an attractive control approach for robots. In addition, the proposed learning algorithm can on-line tune parameters of dilation and translation of fuzzy wavelet basis functions and hidden-to-output weights. Therefore, a robust control law is used to eliminate uncertainties including the inevitable approximation errors resulted from the finite number of fuzzy wavelet basis functions. The proposed controller requires no prior knowledge about the dynamics of the robot and no off-line learning phase. Moreover, both tracking performance and stability of the closed-loop robotic system can be guaranteed by Lyapunov theory. Finally, the effectiveness of the fuzzy wavelet network-based control approach is illustrated through comparative simulations on a six-link robot manipulator     

Navigation function-based visual servo control

In this paper, the mapping between the desired camera feature vector and the desired camera pose (i.e., the position and orientation) is investigated to develop a measurable image Jacobian-like matrix. An image-space path planner is then proposed to generate a desired image trajectory based on this measurable image Jacobian-like matrix and an image-space navigation function (NF) (i.e., a special potential field function) while satisfying rigid body constraints. An adaptive, homography-based visual servo tracking controller is then developed to navigate the position and orientation of a camera held by the end-effector of a robot manipulator to a goal position and orientation along the desired image-space trajectory while ensuring the target points remain visible (i.e., the target points avoid self-occlusion and remain in the field-of-view (FOV)) under certain technical restrictions. Due to the inherent nonlinear nature of the problem and the lack of depth information from a monocular system, a Lyapunov-based analysis is used to analyze the path planner and the adaptive controller. Simulation results are provided to illustrate the performance of the proposed approach.     

Visual servoing tracking control of uncalibrated manipulators with a moving feature point

The paper is concerned with the problem of uncalibrated visual servoing robots tracking a dynamic feature point along with the desired trajectory. A nonlinear observer and a nonlinear controller are proposed, which allow the considered uncalibrated visual servoing robotic system to fulfil the desired tracking task. Based on this novel control method, a dynamic feature point with unknown motion parameters can be tracked effectively along with the desired trajectory, even with multiple uncertainties existing in the camera, the kinematics and the manipulator dynamics. By the Lyapunov theory, asymptotic convergence of the image errors to zero with the proposed control scheme is rigorously proven. Simulations have been conducted to verify the performance of the proposed control scheme. The results demonstrated good convergence of the image errors.     

Design of robotic visual servo control based on neural network and genetic algorithm

A new visual servo control scheme for a robotic manipulator is presented in this paper, where a back propagation (BP) neural network is used to make a direct transition from image feature to joint angles without requiring robot kinematics and camera calibration. To speed up the convergence and avoid local minimum of the neural network, this paper uses a genetic algorithm to find the optimal initial weights and thresholds and then uses the BP algorithm to train the neural network according to the data given. The proposed method can effectively combine the good global searching ability of genetic algorithms with the accurate local searching feature of BP neural network. The Simulink model for PUMA560 robot visual servo system based on the improved BP neural network is built with the Robotics Toolbox of Matlab. The simulation results indicate that the proposed method can accelerate convergence of the image errors and provide a simple and effective way of robot control.     

Direct adaptive control of manipulators in cartesian space

A new adaptive-control scheme for direct control of manipulator end effector to achieve trajectory tracking in Cartesian space is developed in this article. The control structure is obtained from linear multivariable theory and is composed of simple feedforward and feedback controllers and an auxiliary input. The direct adaptation laws are derived from model reference adaptive control theory and are not based on parameter estimation of the robot model. The utilization of adaptive feedforward control and the inclusion of auxiliary input are novel features of the present scheme and result in improved dynamic performance over existing adaptive control schemes. The adaptive controller does not require the complex mathematical model of the robot dynamics or any knowledge of the robot parameters or the payload, and is computationally fast for on-line implementation with high sampling rates. The control scheme is applied to a two-link manipulator for illustration.     

Enhanced direct adaptive fuzzy control of robot manipulators

In this article, an enhanced direct adaptive fuzzy robot controller is developed to overcome problems of high‐frequency oscillations across the boundary of the constraint set and large control signals. The direct adaptive fuzzy robot control algorithm employs tracking errors of the joint motion to drive the parameter adaptation. The predominant concern of the adaptation law is to reduce the tracking errors, and closed‐loop stability is ensured by appending a supervisory controller. This adaptive controller, appended with the supervisory controller, does not require the exact robot dynamics, but only the boundary of the dynamics. Theoretical results and simulation studies on a two‐link robot manipulator show that by modifying the activation function of the supervisory controller, the enhanced direct adaptive fuzzy robot controller is as robust as before and the problems of high‐frequency oscillations across the boundary of the constraint set and large control signals are alleviated. ©1999 John Wiley & Sons, Inc.     

On-line genetic algorithm-based fuzzy-neural sliding mode controller using improved adaptive bound reduced-form genetic algorithm

In this article, a novel on-line genetic algorithm-based fuzzy-neural sliding mode controller trained by an improved adaptive bound reduced-form genetic algorithm is developed to guarantee robust stability and good tracking performance for a robot manipulator with uncertainties and external disturbances. A general sliding manifold, which can be non-linear or time varying, is used to construct a sliding surface and reduce control law chattering. In this article, the sliding surface is used to derive a genetic algorithm-based fuzzy-neural sliding mode controller. To identify structured system dynamics, a B-spline membership function fuzzy-neural network, which is trained by the improved genetic algorithm, is used to approximate the regressor of the robot manipulator. The sliding mode control with a general sliding surface plays the role of a compensator when the fuzzy-neural network does not approximate the dynamics regressor of the robot manipulator well in the transient period. The adjustable parameters of the fuzzy-neural network are tuned by the improved genetic algorithm, which, with the use of the sequential-search-based crossover point method and the single gene crossover, converges quickly to near-optimal parameter values. Simulation results show that the proposed genetic algorithm-based fuzzy-neural sliding mode controller is effective and yields superior tracking performance for robot manipulators.     

Uncalibrated Eye-to-Hand Visual Servoing Using Inverse Fuzzy Models

A new uncalibrated eye-to-hand visual servoing based on inverse fuzzy modeling is proposed in this paper. In classical visual servoing, the Jacobian plays a decisive role in the convergence of the controller, as its analytical model depends on the selected image features. This Jacobian must also be inverted online. Fuzzy modeling is applied to obtain an inverse model of the mapping between image feature variations and joint velocities. This approach is independent from the robot's kinematic model or camera calibration and also avoids the necessity of inverting the Jacobian online. An inverse model is identified for the robot workspace, using measurement data of a robotic manipulator. This inverse model is directly used as a controller. The inverse fuzzy control scheme is applied to a robotic manipulator performing visual servoing for random positioning in the robot workspace. The obtained experimental results show the effectiveness of the proposed control scheme. The fuzzy controller can position the robotic manipulator at any point in the workspace with better accuracy than the classic visual servoing approach.     

Neural network-based adaptive controller design of roboticmanipulators with an observer

A neural network (NN)-based adaptive controller with an observer is proposed for the trajectory tracking of robotic manipulators with unknown dynamics nonlinearities. It is assumed that the robotic manipulator has only joint angle position measurements. A linear observer is used to estimate the robot joint angle velocity, while NNs are employed to further improve the control performance of the controlled system through approximating the modified robot dynamics function. The adaptive controller for robots with an observer can guarantee the uniform ultimate bounds of the tracking errors and the observer errors as well as the bounds of the NN weights. For performance comparisons, the conventional adaptive algorithm with an observer using linearity in parameters of the robot dynamics is also developed in the same control framework as the NN approach for online approximating unknown nonlinearities of the robot dynamics. Main theoretical results for designing such an observer-based adaptive controller with the NN approach using multilayer NNs with sigmoidal activation functions, as well as with the conventional adaptive approach using linearity in parameters of the robot dynamics are given. The performance comparisons between the NN approach and the conventional adaptation approach with an observer is carried out to show the advantages of the proposed control approaches through simulation studies     

A Robust Adaptive Terminal Sliding Mode Control for Rigid Robotic Manipulators

In this paper, a robust adaptive terminal sliding mode controller is developed for n-link rigid robotic manipulators with uncertain dynamics. An MIMO terminal sliding mode is defined for the error dynamics of a closed loop robot control system, and an adaptive mechanism is introduced to estimate the unknown parameters of the upper bounds of system uncertainties in the Lyapunov sense. The estimates are then used as controller parameters so that the effects of uncertain dynamics can be eliminated and a finite time error convergence in the terminal sliding mode can be guaranteed. Also, a useful bounded property of the derivative of the inertial matrix is explored, the convergence rate of the terminal sliding variable vector is investigated, and an experiment using a five bar robotic manipulator is carried out in support of the proposed control scheme.     

含驱动器动力学的移动机器人编队自适应控制

针对含有驱动器及编队动力学的多非完整移动机器人编队控制问题,基于领航者-跟随者[l-ψ]控制结构,通过反步法设计了一种将运动学控制器与驱动器输入电压控制器相结合的新型控制策略。采用径向基神经网络（RBFNN）对跟随者及领航者动力学非线性不确定部分进行在线估计,并通过自适应鲁棒控制器对神经网络建模误差进行补偿。该方法不但解决了移动机器人编队控制的参数与非参数不确定性问题,同时也确保了机器人编队在期望队形下对指定轨迹的跟踪;基于Lyapunov方法的设计过程,保证了控制系统的稳定与收敛;仿真结果表明了该方法的有效性。     

Robust visual tracking control system of a mobile robot based on a dual-Jacobian visual interaction model

This paper presents a novel design of a robust visual tracking control system, which consists of a visual tracking controller and a visual state estimator. This system facilitates human–robot interaction of a unicycle-modeled mobile robot equipped with a tilt camera. Based on a novel dual-Jacobian visual interaction model, a robust visual tracking controller is proposed to track a dynamic moving target. The proposed controller not only possesses some degree of robustness against the system model uncertainties, but also tracks the target without its 3D velocity information. The visual state estimator aims to estimate the optimal system state and target image velocity, which is used by the visual tracking controller. To achieve this, a self-tuning Kalman filter is proposed to estimate interesting parameters and to overcome the temporary occlusion problem. Furthermore, because the proposed method is fully working in the image space, the computational complexity and the sensor/camera modeling errors can be reduced. Experimental results validate the effectiveness of the proposed method, in terms of tracking performance, system convergence, and robustness.     

移动机器人自适应视觉伺服镇定控制

对有单目视觉的移动机器人系统,提出了一种自适应视觉伺服镇定控制算法;在缺乏深度信息传感器并且摄像机外参数未知的情况下,该算法利用视觉反馈实现了移动机器人位置和姿态的渐近稳定.由于机器人坐标系与摄像机坐标系之间的平移外参数(手眼参数)是未知的,本文利用静态特征点的位姿变化特性,建立移动机器人在摄像机坐标系下的运动学模型.然后,利用单应矩阵分解的方法得到了可测的角度误差信号,并结合2维图像误差信号,通过一组坐标变换,得到了系统的开环误差方程.在此基础之上,基于Lyapunov稳定性理论设计了一种自适应镇定控制算法.理论分析、仿真与实验结果均证明了本文所设计的单目视觉控制器在摄像机外参数未知的情况下,可以使移动机器人渐近稳定到期望的位姿.     

Further results on adaptive iterative learning control of robot manipulators

Based on a combination of a PD controller and a switching type two-parameter compensation force, an iterative learning controller with a projection-free adaptive algorithm is presented in this paper for repetitive control of uncertain robot manipulators. The adaptive iterative learning controller is designed without any a priori knowledge of robot parameters under certain properties on the dynamics of robot manipulators with revolute joints only. This new adaptive algorithm uses a combined time-domain and iteration-domain adaptation law allowing to guarantee the boundedness of the tracking error and the control input, in the sense of the infinity norm, as well as the convergence of the tracking error to zero, without any a priori knowledge of robot parameters. Simulation results are provided to illustrate the effectiveness of the learning controller.     

Nonlinear robust adaptive Cartesian impedance control of UAVs equipped with a robot manipulator

In this paper, a new nonlinear robust adaptive impedance controller is addressed for Unmanned Aerial Vehicles (UAVs) equipped with a robot manipulator that physically interacts with environment. A UAV equipped with a robot manipulator is a novel system that can perform different tasks instead of human being in dangerous and/or inaccessible environments. The objective of the proposed robust adaptive controller is control of the UAV and its robotic manipulator’s end-effector impedance in Cartesian space in order to have a stable physical interaction with environment. The proposed controller is robust against parametric uncertainties in the nonlinear dynamics model of the UAV and the robot manipulator. Moreover, the controller has robustness against the bounded force sensor inaccuracies and bounded unstructured modeling (nonparametric) uncertainties and/or disturbances in the system. Tracking performance and stability of the system are proved via Lyapunov stability theorem. Using simulations on a quadrotor UAV equipped with a three-DOF robot manipulator, the effectiveness of the proposed robust adaptive impedance controller is investigated in the presence of the force sensor error, and parametric and non-parametric uncertainties.