Moment feature-based three-dimensional tracking using two high-speed vision systems

When considering real-world applications of robot control with visual servoing, both three-dimensional (3-D) information and a high feedback rate are required. We have developed a 3-D target-tracking system with a 1-ms feedback rate using two high-speed vision systems called Column Parallel Vision (CPV) systems. To obtain 3-D information, such as position, orientation and shape parameters of the target object, a feature-based algorithm has been introduced using moment feature values extracted from vision systems for a spheroidal object model. Also, we propose a new 3-D self-windowing method to extract the target in 3-D space using epipolar geometry, which is an extension of the conventional self-windowing method in 2-D images.     

Visual servoing of redundant manipulator with Jacobian matrix estimation using self-organizing map

Vision based redundant manipulator control with a neural network based learning strategy is discussed in this paper. The manipulator is visually controlled with stereo vision in an eye-to-hand configuration. A novel Kohonen’s self-organizing map (KSOM) based visual servoing scheme has been proposed for a redundant manipulator with 7 degrees of freedom (DOF). The inverse kinematic relationship of the manipulator is learned using a Kohonen’s self-organizing map. This learned map is shown to be an approximate estimate of the inverse Jacobian, which can then be used in conjunction with the proportional controller to achieve closed loop servoing in real-time. It is shown through Lyapunov stability analysis that the proposed learning based servoing scheme ensures global stability. A generalized weight update law is proposed for KSOM based inverse kinematic control, to resolve the redundancy during the learning phase. Unlike the existing visual servoing schemes, the proposed KSOM based scheme eliminates the computation of the pseudo-inverse of the Jacobian matrix in real-time. This makes the proposed algorithm computationally more efficient. The proposed scheme has been implemented on a 7 DOF PowerCube? robot manipulator with visual feedback from two cameras.     

基于遗传算法模式匹配的机器人实时视觉伺服

对于机器人手臂来讲 ,对工作环境的识别是完成一个智能任务的最重要的问题之一 .因为这种智能可以使它工作在一个变化的环境中 .本文提出了一种新的机器人手臂的控制策略 ,可以利用视觉信息来指导机器人的手臂在它的工作空间中捡起一个已知形状但任意位置和方向的物体 .在对物体的搜索过程中 ,利用基于视觉闭环的视觉伺服来完成对机器人手臂的运动控制 .本系统利用遗传算法 (Genetic Algorithm ,GA)和模式匹配技术完成对搜索空间的搜索并获得了良好的结果 .本文完成了对带有两连杆手臂的视觉伺服系统的仿真 ,仿真结果证明了算法的有效性     

机器人视觉伺服综述

系统论述了机器人视觉伺服发展的历史和现状。从不同角度对机器人视觉控制系统进行分类,重点介绍了基位置的视觉伺服系统和基于图像的视觉伺服系统。对人工神经网络在机器人视觉伺服方面的应用情况作了介绍。讨论了视觉伺服中图像特征的选择问题。对机器人视觉所涉及的前沿问题进行阐述,并指出了目前研究中所存在的问题及今后发展方向。     

Neural Control for Driving a Mobile Robot Integrating Stereo Vision Feedback

This paper proposes a neural control integrating stereo vision feedback for driving a mobile robot. The proposed approach consists in synthesizing a suitable inverse optimal control to avoid solving the Hamilton Jacobi Bellman equation associated to nonlinear system optimal control. The mobile robot dynamics is approximated by an identifier using a discrete-time recurrent high order neural network, trained with an extended Kalman filter algorithm. The desired trajectory of the robot is computed during navigation using a stereo camera sensor. Simulation and experimental result are presented to illustrate the effectiveness of the proposed control scheme.     

The use of active deformable models in model-based robotic visual servoing

This paper presents a new approach for visual tracking and servoing in robotics. We introduce deformable active models as a powerful means for tracking a rigid or semi-rigid (possibly partially occluded) object in movement within the manipulator's workspace. Deformable models imitate, in real-time, the dynamic behavior of elastic structures. These computer-generated models are designed to capture the silhouette of objects with well-defined boundaries, in terms of image gradient. By means of an eye-in-hand robot arm configuration, the desired motion of the end-effector is computed with the objective of keeping the target's position and shape invariant with respect to the camera frame. Optimal estimation and control techniques (LQG regulator) have been successfully implemented in order to deal with noisy measurements provided by our vision sensor. Experimental results are presented for the tracking of a rigid or semi-rigid object. The experiments performed in a real-time environment show the effectiveness and robustness of the proposed method for servoing tasks based on visual feedback.     

Image-Based Visual Servoing for Nonholonomic Mobile Robots Using Epipolar Geometry

We present an image-based visual servoing strategy for driving a nonholonomic mobile robot equipped with a pinhole camera toward a desired configuration. The proposed approach, which exploits the epipolar geometry defined by the current and desired camera views, does not need any knowledge of the 3-D scene geometry. The control scheme is divided into two steps. In the first, using an approximate input-output linearizing feedback, the epipoles are zeroed so as to align the robot with the goal. Feature points are then used in the second translational step to reach the desired configuration. Asymptotic convergence to the desired configuration is proven, both in the calibrated and partially calibrated case. Simulation and experimental results show the effectiveness of the proposed control scheme     

A real-time fuzzy reasoning based control system for catching a moving goldfish

Designing a real-time visual tracking system to catch a goldfish is a complex task because of the large amount of streaming video data that must be transmitted and processed immediately when tracking the goldfish. Usually, building such visual servoing systems requires the application of high-cost specialized hardware and the development of complicated visual control software. In this paper, a novel low-cost, real-time visual servo control system is presented. The system uses stereo vision consisting of two calibrated cameras to acquire images of the goldfish, and applies the continuously adaptive mean shift (CAMSHIFT) vision-tracking algorithm to provide feedback of a fish’s real-time position at a high frame rate. It then employs a 5-axis robot manipulator controlled by a fuzzy reasoning system to catch the fish. This visual tracking and servoing system is less sensitive to lighting influences and thus performs more efficiently. Experiments with the proposed method yielded very good results, as the system’s real-time 3D vision successfully tracked two fish and guided the manipulator, which has a net attached to its end effector, to catch one of them.     

A new formulation of visual servoing based on cylindrical coordinate system

Image-based visual servoing is a flexible and robust technique to control a robot and guide it to a desired position only by using two-dimensional visual data. However, it is well known that the classical visual servoing based on the Cartesian coordinate system has one crucial problem, that the camera moves backward at infinity, in case that the camera motion from the initial to desired poses is a pure rotation of 1800 around the optical axis. This paper proposes a new formulation of visual servoing, based on a cylindrical coordinate system that can shift the position of the origin. The proposed approach can interpret from a pure rotation around an arbitrary axis to the proper camera rotational motion. It is shown that this formulation contains the classical approach based on the Cartesian coordinate system as an extreme case with the origin located at infinity. Furthermore, we propose a decision method of the origin-shift parameters by estimating a rotational motion from the differences between initial and desired image-plane positions of feature points.     

具有深度自适应估计的视觉伺服优化

在手眼机器人视觉伺服中,如何确定机器人末端摄像机移动的速度和对物体的深度进行有效的估计还没有较好的解决方法.本文采用一般模型法,通过求解最优化控制问题来设计摄像机的速度,同时,利用物体初始及期望位置的深度估计值,提出了一种自适应估计的算法对物体的深度进行估计,给出了深度变化趋势,实现了基于图像的定位控制.该方法能够使机器人在工作空间范围内从任一初始位置出发到达期望位置,实现了系统的全局渐近稳定且不需要物体的几何模型及深度的精确值.最后给出的仿真实例表明了本方法的有效性.     

Robust auto tool change for industrial robots using visual servoing

This work presents an automated solution for tool changing in industrial robots using visual servoing and sliding mode control. The robustness of the proposed method is due to the control law of the visual servoing, which uses the information acquired by a vision system to close a feedback control loop. Furthermore, sliding mode control is simultaneously used in a prioritised level to satisfy the constraints typically present in a robot system: joint range limits, maximum joint speeds and allowed workspace. Thus, the global control accurately places the tool in the warehouse, but satisfying the robot constraints. The feasibility and effectiveness of the proposed approach is substantiated by simulation results for a complex 3D case study. Moreover, real experimentation with a 6R industrial manipulator is also presented to demonstrate the applicability of the method for tool changing.     

GA-pattern matching-based manipulator control system for real-time visual servoing

_—In robotic applications, tasks of picking and placing are the most fundamental ones. Also, for a robot manipulator, the recognition of its working environment is one of the most important issues to do intelligent tasks, since this aptitude enables it to work in a variable environment. This paper presents a new control strategy for robot manipulators, which utilizes visual information to direct the manipulator in its working space, to pick up an object of known shape, but with arbitrary position and orientation. During the search for an object to be picked up, vision-based control by closed-loop feedback, referred to as visual servoing, is performed to obtain the motion control of the manipulator hand. The system employs a genetic algorithm (GA) and a pattern matching technique to explore the search space and exploit the best solutions by this search technique. The control strategy utilizes the found results of GA-pattern matching in every step of GA evolution to direct the manipulator towards the target object. We named this control strategy step-GA-evnlution. This control method can be applied for manipulator real-time visual servoing and solve its path planning problem in real-time, i.e. in order for the manipulator to adapt the execution of the task by visual information during the process execution. Simulations have been performed, using a two-link planar manipulator and three image models, in order to find which one is the best for real-time visual servoing and the results show the effectiveness of the control method.     

A subsumptive, hierarchical, and distributed vision-based architecture for smart robotics

We present a distributed vision-based architecture for smart robotics that is composed of multiple control loops, each with a specialized level of competence. Our architecture is subsumptive and hierarchical, in the sense that each control loop can add to the competence level of the loops below, and in the sense that the loops can present a coarse-to-fine gradation with respect to vision sensing. At the coarsest level, the processing of sensory information enables a robot to become aware of the approximate location of an object in its field of view. On the other hand, at the finest end, the processing of stereo information enables a robot to determine more precisely the position and orientation of an object in the coordinate frame of the robot. The processing in each module of the control loops is completely independent and it can be performed at its own rate. A control Arbitrator ranks the results of each loop according to certain confidence indices, which are derived solely from the sensory information. This architecture has clear advantages regarding overall performance of the system, which is not affected by the "slowest link," and regarding fault tolerance, since faults in one module does not affect the other modules. At this time we are able to demonstrate the utility of the architecture for stereoscopic visual servoing. The architecture has also been applied to mobile robot navigation and can easily be extended to tasks such as "assembly-on-the-fly."     

A review on vision-based control of flexible manipulators

In this review, recent developments in the field of flexible robot arm control using visual servoing are reviewed. In comparison to rigid robots, the end-effector position of flexible links cannot be obtained precisely enough with respect to position control using kinematic information and joint variables. To solve the task here the use of a vision sensor (camera) system, visual servoing is proposed to realize the task of control flexible manipulators with improved quality requirements. The paper is organized as follows: the visual servoing architectures will be reviewed for rigid robots first. The advantages, disadvantages, and comparisons between different approaches of visual servoing are carried out. The using of visual servoing to control flexible robot is addressed next. Open problems such as state variables estimation as well as the combination of different sensor properties as well as some application-oriented points related to flexible robot are discussed in detail.     

The UJI librarian robot

This paper describes the UJI librarian robot, a mobile manipulator that is able to autonomously locate a book in an ordinary library, and grasp it from a bookshelf, by using eye-in-hand stereo vision and force sensing. The robot is only provided with the book code, a library map and some knowledge about its logical structure and takes advantage of the spatio-temporal constraints and regularities of the environment by applying disparate techniques such as stereo vision, visual tracking, probabilistic matching, motion estimation, multisensor-based grasping, visual servoing and hybrid control, in such a way that it exhibits a robust and dependable performance. The system has been tested, and experimental results show how it is able to robustly locate and grasp a book in a reasonable time without human intervention.     

Visual Servoing Path Planning for Cameras Obeying the Unified Model

Abstract

This paper proposes a path planning visual servoing strategy for a class of cameras that includes conventional perspective cameras, fisheye cameras and catadioptric cameras as special cases. Specifically, these cameras are modeled by adopting a unified model recently proposed in the literature and the strategy consists of designing image trajectories for eye-in-hand robotic systems that allow the robot to reach a desired location while satisfying typical visual servoing constraints. To this end, the proposed strategy introduces the projection of the available image features onto a virtual plane and the computation of a feasible image trajectory through polynomial programming. Then, the computed image trajectory is tracked by using an image-based visual servoing controller. Experimental results with a fisheye camera mounted on a 6-d.o.f. robot arm are presented in order to illustrate the proposed strategy.     

Navigation system including associative memory

A new adaptive linear robot control system for a robot work cell that can visually track and intercept stationary and moving objects undergoing arbitrary motion anywhere along its predicted trajectory within the robot's workspace is presented in this paper. The proposed system was designed by integrating a stationary monocular CCD camera with off-the-shelf frame grabber and an industrial robot operation into a single application on the MATLAB platform. A combination of the model based object recognition technique and a learning vector quantization network is used for classifying stationary objects without overlapping. The optical flow technique and the MADALINE network are used for determining the target trajectory and generating the predicted robot trajectory based on visual servoing, respectively. The necessity of determining a model of the robot, camera, all the stationary and moving objects, and environment is eliminated. The location and image features of these objects need not be preprogrammed, marked and known before, and any change in a task is possible without changing the robot program. After the learning process on the robot, it is shown that the KUKA robot is capable of tracking and intercepting both stationary and moving objects at an optimal rendezvous point on the conveyor accurately in real-time.     

一种基于图像的移动机器人泊位方法

提出一种新的移动机器人泊位方法. 该方法采用一幅预先采集的参考图像定义机器人的期望泊位状 态（期望的位置和方向）,利用尺度不变特征变换（SIFT）算法和基于双向BBF 的特征匹配算法实现泊位站当前图 像与参考图像之间的匹配以获取视觉反馈信息,基于极点伺服策略根据参考图像准直机器人,采用质心跟踪法防止 目标图像出视场,采用RANSAC 算法求解当前图像与参考图像间的仿射变换,利用一个末段控制策略实现精确泊 位．本文方法不需要环境模型或人工标记．室内环境下的实验结果证实了该方法的有效性．     

曲线焊缝跟踪的视觉伺服协调控制

为实现工业机器人自动跟踪曲线焊缝,提出了协调焊枪运动和视觉跟踪的视觉伺服控制方法．建立了特征点的数学模型,并在此基础上确定机器人运动的旋转轴．设计了一种双层结构的模糊视觉伺服控制器,通过动态确定控制量有效范围来保证图像特征存在于视场中．为准确确定有效范围,设计了带模型动态补偿的Kalman滤波器．曲线焊缝的自动跟踪实验验证了所提方法的有效性．