Visual servoing of redundant manipulator with Jacobian matrix estimation using self-organizing map

Vision based redundant manipulator control with a neural network based learning strategy is discussed in this paper. The manipulator is visually controlled with stereo vision in an eye-to-hand configuration. A novel Kohonen’s self-organizing map (KSOM) based visual servoing scheme has been proposed for a redundant manipulator with 7 degrees of freedom (DOF). The inverse kinematic relationship of the manipulator is learned using a Kohonen’s self-organizing map. This learned map is shown to be an approximate estimate of the inverse Jacobian, which can then be used in conjunction with the proportional controller to achieve closed loop servoing in real-time. It is shown through Lyapunov stability analysis that the proposed learning based servoing scheme ensures global stability. A generalized weight update law is proposed for KSOM based inverse kinematic control, to resolve the redundancy during the learning phase. Unlike the existing visual servoing schemes, the proposed KSOM based scheme eliminates the computation of the pseudo-inverse of the Jacobian matrix in real-time. This makes the proposed algorithm computationally more efficient. The proposed scheme has been implemented on a 7 DOF PowerCube? robot manipulator with visual feedback from two cameras.     

Visual motor control of a 7 DOF robot manipulator using a fuzzy SOM network

A fuzzy self-organizing map (SOM) network is proposed in this paper for visual motor control of a 7 degrees of freedom (DOF) robot manipulator. The inverse kinematic map from the image plane to joint angle space of a redundant manipulator is highly nonlinear and ill-posed in the sense that a typical end-effector position is associated with several joint angle vectors. In the proposed approach, the robot workspace in image plane is discretized into a number of fuzzy regions whose center locations and fuzzy membership values are determined using a Fuzzy C-Mean (FCM) clustering algorithm. SOM network then learns the inverse kinematics by on-line by associating a local linear map for each cluster. A novel learning algorithm has been proposed to make the robot manipulator to reach a target position. Any arbitrary level of accuracy can be achieved with a number of fine movements of the manipulator tip. These fine movements depend on the error between the target position and the current manipulator position. In particular, the fuzzy model is found to be better as compared to Kohonen self-organizing map (KSOM) based learning scheme proposed for visual motor control. Like existing KSOM learning schemes, the proposed scheme leads to a unique inverse kinematic solution even for a redundant manipulator. The proposed algorithms have been successfully implemented in real-time on a 7 DOF PowerCube robot manipulator, and results are found to concur with the theoretical findings.     

Autonomous robotic capture of non-cooperative target using visual servoing and motion predictive control

This paper presents a framework for autonomous capture operation of a non-cooperative mobile target in a 3-dimensional workspace using a robotic manipulator with visual servoing. The visual servoing with an eye-in-hand configuration is based on motion predictive control using Kalman filter for the on-line state and parameter estimation of the target. A transitional decision making process is developed to guide the robotic manipulator between the different phases of the capture operation by employing a custom metric that translates visual misalignments between the end-effector and the target into a guidance measurement. These phases include the target acquisition and approach stage and the alignment and capture phase. Experiments have been carried out on a custom designed and built robotic manipulator with 6 degrees of freedom. The objective is to evaluate the performance of the proposed motion predictive control scheme for the autonomous capturing task and to demonstrate the robustness of the proposed control scheme in the presence of noise and unexpected disturbances in vision system, sensory-motor coordination and constraints for the execution in real environments. Experimental results of the visual servoing control scheme integrated with the motion predictive Kalman filter indicate the feasibility and applicability of the proposed control scheme. It shows that when the target motion is properly predicted, the tracking and capture performance has been improved significantly.     

Model Predictive Control for Constrained Image‐Based Visual Servoing in Uncalibrated Environments

This paper presents a novel approach for image‐based visual servoing (IBVS) of a robotic system by considering the constraints in the case when the camera intrinsic and extrinsic parameters are uncalibrated and the position parameters of the features in 3‐D space are unknown. Based on the model predictive control method, the robotic system's input and output constraints, such as visibility constraints and actuators limitations, can be explicitly taken into account. Most of the constrained IBVS controllers use the traditional image Jacobian matrix, the proposed IBVS scheme is developed by using the depth‐independent interaction matrix. The unknown parameters can appear linearly in the prediction model and they can be estimated by the identification algorithm effectively. In addition, the model predictive control determines the optimal control input and updates the estimated parameters together with the prediction model. The proposed approach can simultaneously handle system constraints, unknown camera parameters and depth parameters. Both the visual positioning and tracking tasks can be achieved desired performances. Simulation results based on a 2‐DOF planar robot manipulator for both the eye‐in‐hand and eye‐to‐hand camera configurations are used to demonstrate the effectiveness of the proposed method.     

Neural network Reinforcement Learning for visual control of robot manipulators

It is known that most of the key problems in visual servo control of robots are related to the performance analysis of the system considering measurement and modeling errors. In this paper, the development and performance evaluation of a novel intelligent visual servo controller for a robot manipulator using neural network Reinforcement Learning is presented. By implementing machine learning techniques into the vision based control scheme, the robot is enabled to improve its performance online and to adapt to the changing conditions in the environment. Two different temporal difference algorithms (Q-learning and SARSA) coupled with neural networks are developed and tested through different visual control scenarios. A database of representative learning samples is employed so as to speed up the convergence of the neural network and real-time learning of robot behavior. Moreover, the visual servoing task is divided into two steps in order to ensure the visibility of the features: in the first step centering behavior of the robot is conducted using neural network Reinforcement Learning controller, while the second step involves switching control between the traditional Image Based Visual Servoing and the neural network Reinforcement Learning for enabling approaching behavior of the manipulator. The correction in robot motion is achieved with the definition of the areas of interest for the image features independently in both control steps. Various simulations are developed in order to present the robustness of the developed system regarding calibration error, modeling error, and image noise. In addition, a comparison with the traditional Image Based Visual Servoing is presented. Real world experiments on a robot manipulator with the low cost vision system demonstrate the effectiveness of the proposed approach.     

基于自抗扰控制器的机器人无标定手眼协调

研究机器人无标定手眼协调问题.分析了图像空间到机器人操作空间之间的非线性映 射关系,并把非线性的映射关系看成是系统的未建模动态.基于自抗扰控制器思想,通过对系统 未建模动态和外扰的补偿,完成了不依赖于任务的无标定手眼协调控制器的设计,实现了广泛意 义的机器人无标定手眼协调控制.仿真和实验结果表明了该方法的有效性.     

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

This paper presents a novel approach for image-based visual servoing of a robot manipulator with an eye-in-hand camera when the camera parameters are not calibrated and the 3-D coordinates of the features are not known. Both point and line features are considered. This paper extends the concept of depth-independent interaction (or image Jacobian) matrix, developed in earlier work for visual servoing using point features and fixed cameras, to the problem using eye-in-hand cameras and point and line features. By using the depth-independent interaction matrix, it is possible to linearly parameterize, by the unknown camera parameters and the unknown coordinates of the features, the closed-loop dynamics of the system. A new algorithm is developed to estimate unknown parameters online by combining the Slotine–Li method with the idea of structure from motion in computer vision. By minimizing the errors between the real and estimated projections of the feature on multiple images captured during motion of the robot, this new adaptive algorithm can guarantee the convergence of the estimated parameters to the real values up to a scale. On the basis of the nonlinear robot dynamics, we proved asymptotic convergence of the image errors by the Lyapunov theory. Experiments have been conducted to demonstrate the performance of the proposed controller.      

基于图像矩与神经网络的机器人四自由度视觉伺服

针对传统的视觉伺服方法中图像几何特征的标记、提取与匹配过程复杂且通用性差等问题,本文提出了一种基于图像矩的机器人四自由度(4DOF)视觉伺服方法.首先建立了眼在手系统中图像矩与机器人位姿之间的非线性增量变换关系,为利用图像矩进行机器人视觉伺服控制提供了理论基础,然后在未对摄像机与手眼关系进行标定的情况下,利用反向传播(BP)神经网络的非线性映射特性设计了基于图像矩的机器人视觉伺服控制方案,最后用训练好的神经刚络进行了视觉伺服跟踪控制.实验结果表明基于本文算法可实现0.5 mm的位置与0.5°的姿态跟踪精度,验证了算法的的有效性与较好的伺服性能.     

Intelligent control of a robot manipulator by visual feedback

We present a new approach to visual feedback control using image-based visual servoing with stereo vision. In order to control the position and orientation of a robot with respect to an object, a new technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only at around a desired location, but also at other locations. The suggested technique can guide a robot manipulator to the desired location without needing such a priori knowledge as the relative distance to the desired location or a model of the object, even if the initial positioning error is large. We describes a model of stereo vision and how to generate feedback commands. The performance of the proposed visual servoing system is illustrated by a simulation and by experimental results, and compared with the conventional method for an assembling robot. This work was presented in part at the Fourth International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–22, 1999     

机械手的模糊逆模型鲁棒控制

提出一种基于模糊聚类和滑动模控制的模糊逆模型控制方法,并将其应用于动力学 方程未知的机械手轨迹控制.首先,采用C均值聚类算法构造两关节机械手的高木-关野 (T-S)模糊模型,并由此构造模糊系统的逆模型.然后,在提出的模糊逆模型控制结构中, 离散时间滑动模控制和时延控制(TDC)用于补偿模糊建模误差和外扰动,保证系统的全局 稳定性并改进其动态和稳态性能.系统的稳定性和轨迹误差的收敛性可以通过稳定性定理来 证明.最后,以两关节机械手的轨迹跟随控制为例,揭示了该设计方法的控制性能.     

Visual Servoing Path Planning for Cameras Obeying the Unified Model

Abstract

This paper proposes a path planning visual servoing strategy for a class of cameras that includes conventional perspective cameras, fisheye cameras and catadioptric cameras as special cases. Specifically, these cameras are modeled by adopting a unified model recently proposed in the literature and the strategy consists of designing image trajectories for eye-in-hand robotic systems that allow the robot to reach a desired location while satisfying typical visual servoing constraints. To this end, the proposed strategy introduces the projection of the available image features onto a virtual plane and the computation of a feasible image trajectory through polynomial programming. Then, the computed image trajectory is tracked by using an image-based visual servoing controller. Experimental results with a fisheye camera mounted on a 6-d.o.f. robot arm are presented in order to illustrate the proposed strategy.     

Visual servoing tracking control of uncalibrated manipulators with a moving feature point

The paper is concerned with the problem of uncalibrated visual servoing robots tracking a dynamic feature point along with the desired trajectory. A nonlinear observer and a nonlinear controller are proposed, which allow the considered uncalibrated visual servoing robotic system to fulfil the desired tracking task. Based on this novel control method, a dynamic feature point with unknown motion parameters can be tracked effectively along with the desired trajectory, even with multiple uncertainties existing in the camera, the kinematics and the manipulator dynamics. By the Lyapunov theory, asymptotic convergence of the image errors to zero with the proposed control scheme is rigorously proven. Simulations have been conducted to verify the performance of the proposed control scheme. The results demonstrated good convergence of the image errors.     

Fuzzy-Basis-Function-Network-Based $H_infty$ Tracking Control for Robotic Manipulators Using Only Position Feedback

This paper presents an H _infin fuzzy output-feedback tracking-control scheme for robotic manipulators without measuring joint velocities. The developed controller and observer are based on a fuzzy basis function network (FBFN), which is employed to approximate nonlinear functions in the dynamics of controller and observer. The FBFN-based observer that estimates joint velocities can remove the needs of full-state measurements. According to the inevitable approximation errors and external disturbances, an H _infin auxiliary control signal is used to suppress the effects of the uncertainties. Moreover, all parameters of the fuzzy basis functions (FBFs) and FBF-to-output weights can be tuned online. The proposed controller requires no prior knowledge about the dynamics of the robot manipulator and no offline learning phase. Finally, comparative simulations on a three-link robot manipulator are provided to illustrate the tracking performance of the H _infin FBFN-based output-feedback control approach.     

A visual-inertial servoing method for tracking object with two landmarks and an inertial measurement unit

This paper introduces a robot visual-inertial tracking algorithm for a robot manipulator intended to track an object using inertial sensors incorporated into the object. To create this algorithm, the inertial Jacobian is first newly defined in order to show the relationship between an angle set velocity vector of the object and the angular velocity vector of the robot tip. Then, the inertial Jacobian is combined with the conventional image Jacobian. Therefore, the proposed algorithm requires only two landmarks with the help of an inertial measurement unit to track a moving object with six degrees of freedom, while at least three landmarks are required in the conventional stereo visual servoing algorithm. Further, the possession of a multi-rate controller allows the integration system to out-perform conventional systems in the tracking of an object’s attitude change. A suggested application of the proposed method is tracking and selection of a container from a shipping vessel that is being affected by large waves. Simulations and experiments were conducted to verify the feasibility of the proposed methodology.     

具有变结构控制器的无定标视觉伺服*

从控制的角度出发,提出了一种模型无关的无定标视觉伺服控制方法.在该方法中不需要机器人及摄像机模型,图像雅克比矩阵的计算采用最小二乘估计,机器人系统采用变结构的控制理论设计控制器;而后用李亚普诺夫方法对其进行了稳定性分析,结果证明系统能够渐近稳定.仿真实验证明了算法的有效性.     

一种新的双环结构机器人无标定自抗扰视觉伺服控制方法

在研究基于自抗扰控制器的机器人无标定视觉伺服方法的基础上,提出了一种新的双环结构机器人无标定自抗扰视觉伺服控制方法．内环采用Kalman滤波算法进行图像雅可比矩阵的在线辨识,可较好地逼近真实模型;外环采用自抗扰控制器,利用非线性观测器实时估计系统相对于当前估计模型的总扰动,并在控制中加以动态补偿．针对六自由度工业机器人进行了二维运动目标的跟踪实验,实验结果表明了该方法的可行性和有效性．     

Dynamic visual servo control of a 4-axis joint tool to track image trajectories during machining complex shapes

A large part of the new generation of computer numerical control systems has adopted an architecture based on robotic systems. This architecture improves the implementation of many manufacturing processes in terms of flexibility, efficiency, accuracy and velocity. This paper presents a 4-axis robot tool based on a joint structure whose primary use is to perform complex machining shapes in some non-contact processes. A new dynamic visual controller is proposed in order to control the 4-axis joint structure, where image information is used in the control loop to guide the robot tool in the machining task. In addition, this controller eliminates the chaotic joint behavior which appears during tracking of the quasi-repetitive trajectories required in machining processes. Moreover, this robot tool can be coupled to a manipulator robot in order to form a multi-robot platform for complex manufacturing tasks. Therefore, the robot tool could perform a machining task using a piece grasped from the workspace by a manipulator robot. This manipulator robot could be guided by using visual information given by the robot tool, thereby obtaining an intelligent multi-robot platform controlled by only one camera.     

Stable neurovisual servoing for robot manipulators

In this paper, we propose a stable neurovisual servoing algorithm for set-point control of planar robot manipulators in a fixed-camera configuration an show that all the closed-loop signals are uniformly ultimately bounded (UUB) and converge exponentially to a small compact set. We assume that the gravity term and Jacobian matrix are unknown. Radial basis function neural networks (RBFNNs) with online real-time learning are proposed for compensating both gravitational forces and errors in the robot Jacobian matrix. The learning rule for updating the neural network weights, similar to a back propagation algorithm, is obtained from a Lyapunov stability analysis. Experimental results on a two degrees of freedom manipulator are presented to evaluate the proposed controller.     

Applications of neural networks for coordinate transformations in robotics

The use of artificial neural networks is investigated for application to trajectory control problems in robotics. The relative merits of position versus velocity control is considered and a control scheme is proposed in which neural networks are used as static maps (trained off-line) to compute the inverse of the manipulator Jacobian matrix. A proof of the stability of this approach is offered, assuming bounded errors in the static map. A representative two-link robot is investigated using an artificial neural network which has been trained to compute the components of the inverse of the Jacobian matrix. The controller is implemented in the laboratory and its performance compared to a similar controller with the analytical inverse Jacobian matrix.     

不需要标定系统模型的"眼在手上"视觉伺服控制技术

工业实践中不可能精确地标定摄像机和机器人模型，但当前的视觉伺服控制都需要标定系统模型．针对这一现象，提出一种新颖的、能应用于“眼在手上”视觉伺服控制结构的动态无标定的视觉伺服控制算法，无需标定摄像机和机器人运动学模型即可跟踪运动物体，通过将非线性目标函数最小化，以视觉信息跟踪动态图像．针对目前视觉伺服控制系统中“眼在手上”系统的复合雅克比矩阵随每个时间增量的变化无法计算的现象，提出了对每一时间增量时刻的图像雅克比矩阵的变化做出估计的方法，仿真实验证明了上述方法的正确性和有效性．