Uncalibrated visual servoing of robots using a depth-independent interaction matrix

This paper presents a new adaptive controller for image-based dynamic control of a robot manipulator using a fixed camera whose intrinsic and extrinsic parameters are not known. To map the visual signals onto the joints of the robot manipulator, this paper proposes a depth-independent interaction matrix, which differs from the traditional interaction matrix in that it does not depend on the depths of the feature points. Using the depth-independent interaction matrix makes the unknown camera parameters appear linearly in the closed-loop dynamics so that a new algorithm is developed to estimate their values on-line. This adaptive algorithm combines the Slotine-Li method with on-line minimization of the errors between the real and estimated projections of the feature points on the image plane. Based on the nonlinear robot dynamics, we prove asymptotic convergence of the image errors to zero by the Lyapunov theory. Experiments have been conducted to verify the performance of the proposed controller. The results demonstrated good convergence of the image errors.     

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine–Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.      

微小无人直升机自适应视觉伺服

针对摄像机未标定和特征点坐标未知的情况, 本文提出一种新颖的基于图像的无人直升机自适应视觉伺服方法. 控制器是基于反推法设计的, 但是和已有的基于反推法的视觉伺服不同的是, 它利用与深度无关矩阵将图像误差映射到执行器空间, 从而可以避免估计特征点的深度. 这种设计方法可以线性化未知的摄像机参数和特征点坐标, 所以能方便地设计自适应算法来在线估计这些未知参数, 同时为了保证图像误差收敛和避免估计参数收敛至零解而引入了两个势函数. 利用Lyapunov方法证明了基于非线性动力学的控制器的稳定性, 并给出了仿真验证.     

Adaptive visual servoing for the robot manipulator with extreme learning machine and reinforcement learning

In this study, a novel image-based visual servo (IBVS) controller for robot manipulators is investigated using an optimized extreme learning machine (ELM) algorithm and an offline reinforcement learning (RL) algorithm. First of all, the classical IBVS method and its difficulties in accurately estimating the image interaction matrix and avoiding the singularity of pseudo-inverse are introduced. Subsequently, an IBVS method based on ELM and RL is proposed to solve the problem of the singularity of the pseudo-inverse solution and tune adaptive servo gain, improving the servo efficiency and stability. Specifically, the ELM algorithm optimized by particle swarm optimization (PSO) was used to approximate the pseudo-inverse of the image interaction matrix to reduce the influence of camera calibration errors. Then, the RL algorithm was adopted to tune the adaptive visual servo gain in continuous space and improve the convergence speed. Finally, comparative simulation experiments on a 6-DOF robot manipulator were conducted to verify the effectiveness of the proposed IBVS controller.     

Neural network-based adaptive controller design of roboticmanipulators with an observer

A neural network (NN)-based adaptive controller with an observer is proposed for the trajectory tracking of robotic manipulators with unknown dynamics nonlinearities. It is assumed that the robotic manipulator has only joint angle position measurements. A linear observer is used to estimate the robot joint angle velocity, while NNs are employed to further improve the control performance of the controlled system through approximating the modified robot dynamics function. The adaptive controller for robots with an observer can guarantee the uniform ultimate bounds of the tracking errors and the observer errors as well as the bounds of the NN weights. For performance comparisons, the conventional adaptive algorithm with an observer using linearity in parameters of the robot dynamics is also developed in the same control framework as the NN approach for online approximating unknown nonlinearities of the robot dynamics. Main theoretical results for designing such an observer-based adaptive controller with the NN approach using multilayer NNs with sigmoidal activation functions, as well as with the conventional adaptive approach using linearity in parameters of the robot dynamics are given. The performance comparisons between the NN approach and the conventional adaptation approach with an observer is carried out to show the advantages of the proposed control approaches through simulation studies     

A fuzzy cerebellar model articulation controller based visual servo system for robot

This paper presents a novel online learning visual servo controller integrating the FCMAC with proportion controller for the control of position of manipulator end-effector. Since the FCMAC has good learning capability and fast learning speed, and can save much computer memory space by fuzzy processing of input space division and memory unit activation, it is used to develop an adaptive control law by learning the relationship between the image feature errors and manipulator input, and the aim of online learning of the FCMAC is to minimize the output of proportion controller. Furthermore, the FCMAC has no need for models of robot manipulator and image feature extraction, so that the capability of proposed controller for tasks under uncertain environment can be improved. Finally, the proposed controller is proved to be effective by the experiment, and compared with BP neural network.     

Navigation function-based visual servo control

In this paper, the mapping between the desired camera feature vector and the desired camera pose (i.e., the position and orientation) is investigated to develop a measurable image Jacobian-like matrix. An image-space path planner is then proposed to generate a desired image trajectory based on this measurable image Jacobian-like matrix and an image-space navigation function (NF) (i.e., a special potential field function) while satisfying rigid body constraints. An adaptive, homography-based visual servo tracking controller is then developed to navigate the position and orientation of a camera held by the end-effector of a robot manipulator to a goal position and orientation along the desired image-space trajectory while ensuring the target points remain visible (i.e., the target points avoid self-occlusion and remain in the field-of-view (FOV)) under certain technical restrictions. Due to the inherent nonlinear nature of the problem and the lack of depth information from a monocular system, a Lyapunov-based analysis is used to analyze the path planner and the adaptive controller. Simulation results are provided to illustrate the performance of the proposed approach.     

具有H_∞跟踪性能的机器人自适应PID控制

针对非线性不确定机器人系统的轨迹跟踪控制问题,提出一种鲁棒自适应PID控制算法.该控制器由主控制器和监督控制器组成.主控制器以常规PID控制为基础,基于滑模控制思想设计PID参数的自适应律,根据误差实时修正PID参数.基于Lyapunov函数设计的监督控制器补偿自适应PID控制器与理想控制器之间的差异,使系统具有设定的H_∞的跟踪性能.最后,两关节机器人的仿真实验结果表明了算法的有效性.

Abstract：

A robust adaptive PID control algorithm is proposed for trajectory tracking of robot manipulators with nonlinear uncertainties.The controller is composed of a main controller and a supervisory controller.The main controller is designed based on the traditional PID controller.The parameters of the PID controller are updated online according to the system running errors with the adaptation law based on the sliding mode control.The supervisory controller is proposed to compensate the error between the adaptive PID controller and the ideal controller in the sense of the Lyapunov function with the specified H_∞ tracking performance.Finally, the simulation results based on a two-joint robot manipulator show the effectiveness of the presented controller.     

Autonomous robot calibration using vision technology

Unlike the traditional robot calibration methods, which need external expensive calibration apparatus and elaborate setups to measure the 3D feature points in the reference frame, a vision-based self-calibration method for a serial robot manipulator, which only requires a ground-truth scale in the reference frame, is proposed in this paper. The proposed algorithm assumes that the camera is rigidly attached to the robot end-effector, which makes it possible to obtain the pose of the manipulator with the pose of the camera. By designing a manipulator movement trajectory, the camera poses can be estimated up to a scale factor at each configuration with the factorization method, where a nonlinear least-square algorithm is applied to improve its robustness. An efficient approach is proposed to estimate this scale factor. The great advantage of this self-calibration method is that only image sequences of a calibration object and a ground-truth length are needed, which makes the robot calibration procedure more autonomous in a dynamic manufacturing environment. Simulations and experimental studies on a PUMA 560 robot reveal the convenience and effectiveness of the proposed robot self-calibration approach.     

Enhanced direct adaptive fuzzy control of robot manipulators

In this article, an enhanced direct adaptive fuzzy robot controller is developed to overcome problems of high‐frequency oscillations across the boundary of the constraint set and large control signals. The direct adaptive fuzzy robot control algorithm employs tracking errors of the joint motion to drive the parameter adaptation. The predominant concern of the adaptation law is to reduce the tracking errors, and closed‐loop stability is ensured by appending a supervisory controller. This adaptive controller, appended with the supervisory controller, does not require the exact robot dynamics, but only the boundary of the dynamics. Theoretical results and simulation studies on a two‐link robot manipulator show that by modifying the activation function of the supervisory controller, the enhanced direct adaptive fuzzy robot controller is as robust as before and the problems of high‐frequency oscillations across the boundary of the constraint set and large control signals are alleviated. ©1999 John Wiley & Sons, Inc.     

图书馆机器人机械手参数自整定模糊PID控制器设计

采用PC/104系列板卡设计了一款嵌入式图书馆机器人气动机械手控制器,对机械手的参数自整定模糊PID控制算法进行了重点探讨,根据模糊子集的隶属度赋值表和模糊逻辑规则,查模糊矩阵表得出修正参数,完成对PID参数的在线自校正.用Microsoft eMbedded Visual C++(EVC)编程实现了图书取放气动机械手的智能控制,给出了控制软件算法流程及关键部分实现方法.用阶跃、正弦等典型输入信号做系统仿真,实验结果表明气动机械手能够快速、稳定、几乎无误差地跟踪系统给定值.所提出的系统设计方法对类似领域具有普遍适用性.     

移动机器人自适应视觉伺服镇定控制

对有单目视觉的移动机器人系统,提出了一种自适应视觉伺服镇定控制算法;在缺乏深度信息传感器并且摄像机外参数未知的情况下,该算法利用视觉反馈实现了移动机器人位置和姿态的渐近稳定.由于机器人坐标系与摄像机坐标系之间的平移外参数(手眼参数)是未知的,本文利用静态特征点的位姿变化特性,建立移动机器人在摄像机坐标系下的运动学模型.然后,利用单应矩阵分解的方法得到了可测的角度误差信号,并结合2维图像误差信号,通过一组坐标变换,得到了系统的开环误差方程.在此基础之上,基于Lyapunov稳定性理论设计了一种自适应镇定控制算法.理论分析、仿真与实验结果均证明了本文所设计的单目视觉控制器在摄像机外参数未知的情况下,可以使移动机器人渐近稳定到期望的位姿.     

Prediction Error Based Adaptive Jacobian Tracking of Robots With Uncertain Kinematics and Dynamics

In this note, we have proposed a prediction error based adaptive Jacobian controller to solve the problem of concurrent adaptation to both uncertain kinematics and dynamics. This controller is composed of a modified computed torque controller and two cushion floor least-square estimators, and in ideal case of perfect knowledge of the robot parameters it leads to linear and decoupled error dynamics. The kinematic and dynamic parameters adaptations are driven by prediction errors. Using input-output stability analysis, we show that the end-effector motion tracking errors converge asymptotically. We have also derived an alternative adaptive Jacobian controller that does not require the invertibility of the estimated inertia matrix. Simulation results are presented to show the performance of the proposed controllers.      

Robust control of robot manipulators with fast load adaptation

Many adaptive robot controllers have been proposed in the literature to solve manipulator trajectory tracking problems for high-speed operations in the presence of parameter uncertainties. However, most of these controllers stem from the applications of the existing adaptive control theory, which is traditionally focused on tracking slowly time-varying parameters. In fact, manipulator dynamics have fast transient processes for high-speed operations and load changes are abrupt. These observations motivate the present research to incorporate change detection techniques into self-tuning schemes for tracking abrupt load variations and achieving fast load adaptation. To this end, a robustly global stabilizing controller for a robot model with parametric and non-parametric uncertainies is developed based on the Lyapunov second method, and it is then made adaptive via the self-tuning regulator concept. The two-model approach to online change detection in load is used and the estimation algorithm is reinitialized once load changes are detected. This allows a much faster adaptive identification of load parameters than the ordinary forgetting factor approach. Simulation results demonstrate that the proposed controller achieves better tracking accuracy than the existing adaptive and non-adaptive controllers.     

Uncalibrated Eye-to-Hand Visual Servoing Using Inverse Fuzzy Models

A new uncalibrated eye-to-hand visual servoing based on inverse fuzzy modeling is proposed in this paper. In classical visual servoing, the Jacobian plays a decisive role in the convergence of the controller, as its analytical model depends on the selected image features. This Jacobian must also be inverted online. Fuzzy modeling is applied to obtain an inverse model of the mapping between image feature variations and joint velocities. This approach is independent from the robot's kinematic model or camera calibration and also avoids the necessity of inverting the Jacobian online. An inverse model is identified for the robot workspace, using measurement data of a robotic manipulator. This inverse model is directly used as a controller. The inverse fuzzy control scheme is applied to a robotic manipulator performing visual servoing for random positioning in the robot workspace. The obtained experimental results show the effectiveness of the proposed control scheme. The fuzzy controller can position the robotic manipulator at any point in the workspace with better accuracy than the classic visual servoing approach.     

Adaptive manipulator control: A case study

The author's previous work (1986, 1987) utilized the particular structure of manipulator dynamics to develop a simple, globally convergent adaptive controller for manipulator trajectory control problems. After summarizing the basic algorithm, they demonstrate the approach on a high-speed two-degree-of-freedom semi-direct-drive robot. They show that the dynamic parameters of the manipulator, assumed to be initially unknown, can be estimated within the first half second of a typical run, and that accordingly, the manipulator trajectory can be precisely controlled. These experimental results demonstrate that the adaptive controller enjoys essentially the same level of robustness to unmodeled dynamics as a PD (proportional and differential) controller, yet achieves much better tracking accuracy than either PD or computed-torque schemes. Its superior performance for high-speed operations, in the presence of parametric and nonparametric uncertainties, and its relative computational simplicity, make it an attractive option both for addressing complex industrial tasks, and for simplifying high-level programming of more standard operations     

Adaptive inverse dynamics control of robots with uncertain kinematics and dynamics

It has been about two decades since the first globally convergent adaptive tracking controller was derived for robots with dynamic uncertainties. However, not until recently has the problem of concurrent adaptation to both the kinematic and dynamic uncertainties found its solution. This adaptive controller belongs to passivity-based control. Though passivity-based controllers have many attractive properties, in general, they are not able to guarantee the uniform performance of the robot over the entire workspace. Even in the ideal case of perfect knowledge of the manipulator parameters, the closed-loop system remains nonlinear and coupled. Thus the closed-loop tracking performance is difficult to quantify, while the inverse dynamics controllers can overcome these deficiencies. Therefore, in this work, we will develop a new adaptive Jacobian tracking controller based on the inverse manipulator dynamics. Using the Lyapunov approach, we have proved that the end-effector motion tracking errors converge asymptotically to zero. Simulation results are presented to show the performance of the proposed controller.     

Synchronized adaptive control for coordinating manipulators with time‐varying actuator constraint and uncertain dynamics

This paper addresses the control issue for cooperative visual servoing manipulators on strongly connected graph with communication delays, in which case that the uncertain robot dynamics and kinematics, uncalibrated camera model, and actuator constraint are simultaneously considered. An adaptive cooperative image‐based approach is established to overcome the control difficulty arising from nonlinear coupling between visual model and robot agents. To estimate the coupled camera‐robot parameters, a novel adaptive strategy is developed and its superiority mainly lies in the containment of both individual image‐space errors and the synchronous errors among networked robots; thus, the cooperative performance is significantly strengthened. Moreover, the proposed cooperative controller with a Nussbaum‐type gain is implemented to both globally stabilize the closed‐loop systems and realize the synchronization control objective under the existence of unknown and time‐varying actuator constraint. Finally, simulations are carried out to validate the developed approach.     

基于2D三焦点张量的移动机器人视觉伺服镇定控制

针对单目视觉移动机器人系统, 本文提出了一种基于二维三焦点张量(2D trifocal tensor, 2DTT)的视觉伺服镇定控制方法. 具体而言, 首先描述了2D三焦点张量的导出过程, 并给出了基于图像特征点的估计方法. 在此基础上根据2D三焦点张量的元素, 设计了一种反馈线性化控制器以实现机器人的位置镇定, 以及一种比例控制器来实现姿态镇定, 因而在场景信息与平移信息均未知情况下完成了移动机器人的视觉镇定控制. 通过理论分析证明了本文设计的镇定控制算法具有指数收敛性能. 相比现有方法, 这种基于2D 三焦点张量的方法在图像特征识别方面具有更强的鲁棒性, 并且在平面场景与立体场景情况下均适用. 最后利用仿真与实验结果验证了本文提出的视觉伺服方法的优良性能.     

Model Predictive Control for Constrained Image‐Based Visual Servoing in Uncalibrated Environments

This paper presents a novel approach for image‐based visual servoing (IBVS) of a robotic system by considering the constraints in the case when the camera intrinsic and extrinsic parameters are uncalibrated and the position parameters of the features in 3‐D space are unknown. Based on the model predictive control method, the robotic system's input and output constraints, such as visibility constraints and actuators limitations, can be explicitly taken into account. Most of the constrained IBVS controllers use the traditional image Jacobian matrix, the proposed IBVS scheme is developed by using the depth‐independent interaction matrix. The unknown parameters can appear linearly in the prediction model and they can be estimated by the identification algorithm effectively. In addition, the model predictive control determines the optimal control input and updates the estimated parameters together with the prediction model. The proposed approach can simultaneously handle system constraints, unknown camera parameters and depth parameters. Both the visual positioning and tracking tasks can be achieved desired performances. Simulation results based on a 2‐DOF planar robot manipulator for both the eye‐in‐hand and eye‐to‐hand camera configurations are used to demonstrate the effectiveness of the proposed method.