Stereo Tracking and Servoing for Space Applications

This paper proposes a real-time, robust and efficient three-dimensional (3-D) model-based tracking algorithm for visual servoing. A virtual visual servoing approach is used for 3-D tracking. This method is similar to more classical nonlinear pose computation techniques. Robustness is obtained by integrating an M-estimator into the virtual visual control law via an iteratively reweighted least-squares implementation. The presented approach is also extended to the use of multiple cameras. Results show the method to be robust to occlusion, changes in illumination and mistracking.     

First results in omni-directional visual servoing

In this paper, we describe a visual servoing system developed as a human-robot interface to drive a mobile robot toward any chosen target. An omni-directional camera is used to get the 360° of field of view and an efficient tracking technique is developed to track the target. The use of the omni-directional geometry eliminates many of the problems common in visual tracking and makes the use of visual servoing a practical alternative for robot-human interaction. The experiments demonstrate that it is an effective and robust way to guide a robot. In particular, the experiments show robustness of the tracker to loss of template, vehicle motion, and change in scale and orientation.     

Real-time markerless tracking for augmented reality: the virtual visual servoing framework

Tracking is a very important research subject in a real-time augmented reality context. The main requirements for trackers are high accuracy and little latency at a reasonable cost. In order to address these issues, a real-time, robust, and efficient 3D model-based tracking algorithm is proposed for a "video see through" monocular vision system. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose. In this paper, nonlinear pose estimation is formulated by means of a virtual visual servoing approach. In this context, the derivation of point-to-curves interaction matrices are given for different 3D geometrical primitives including straight lines, circles, cylinders, and spheres. A local moving edges tracker is used in order to provide real-time tracking of points normal to the object contours. Robustness is obtained by integrating an M-estimator into the visual control law via an iteratively reweighted least squares implementation. This approach is then extended to address the 3D model-free augmented reality problem. The method presented in this paper has been validated on several complex image sequences including outdoor environments. Results show the method to be robust to occlusion, changes in illumination, and mistracking.     

Effects of camera calibration errors on static-eye and hand-eye visual servoing

Effects of camera calibration errors for the point-to-point task are investigated in static-eye and hand-eye visual servoing realized with position-based and image-based control laws. For these four configurations, the effect of uncertainty on intrinsic and extrinsic parameters is analyzed. The results show local stability for all configurations under small calibration errors. However, a steady-state error is found in the hand-eye position-based configuration. Simulations have been carried out in order to confirm the theoretical results and evaluate the effects of the uncertainty in terms of the stability region. Another contribution of the paper consists of providing a method for estimating the stability region robust against uncertainty directions for the static-eye position-based case with uncertainty on the camera centers.     

Guidance of an ultrasound probe by visual servoing

A new visual servoing technique based on two-dimensional (2-D) ultrasound (US) image is proposed in order to control the motion of an US probe held by a medical robot. In opposition to a standard camera which provides a projection of the three-dimensional (3-D) scene to a 2-D image, US information is strictly in the observation plane of the probe and consequently visual servoing techniques have to be adapted. In this paper the coupling between the US probe and a motionless crossed string phantom used for probe calibration is modeled. Then a robotic task is developed which consists of positioning the US image on the intersection point of the crossed string phantom while moving the probe to different orientations. The goal of this task is to optimize the procedure of spatial parameter calibration of 3-D US systems.     

Multi-Sensor-Based Control Strategy for Initiating and Maintaining Interaction Between a Robot and a Human

This paper presents a multi-sensor-based control strategy allowing a mobile robot to safely navigate with respect to a given human being. Two sensors are embedded in our robot: a vision system that is able to detect and track the person of interest, and a RFID antennas belt that can locate the tag worn by the latter. Thus, our control strategy will be built using image features (when the user is visible) and RFID information (when not). In the first case, a robust visual servoing control will be designed, while in the second case a suitable RFID controller will be proposed. Experimental results demonstrate the efficiency of the proposed control strategy.     

Achieving high-precision laparoscopic manipulation through adaptive force control

In this paper, we present a new solution to laparoscopic manipulation based on forcefeedback control. This method allows us to both explicitely control the forces applied to the patient through the trocar and to precisely control the position of the surgical instrument. It does not require any geometrical model of the operative environment nor any fine robot base placement prior to the instrument insertion. Different adaptive control strategies involving different kinds of sensory equipments are proposed. These strategies are experimentally validated on a laboratory apparatus. An experiment is also presented where a laparoscope held by the robot's arm tracks a target through visual servoing.     

Improved eye-vergence visual servoing system in longitudinal direction with RM-GA

Visual servoing is a control method to manipulate the motion of the robot using visual information, which aims to realize “working while watching.” However, the visual servoing towards moving target with hand–eye cameras fixed at hand is inevitably affected by hand dynamical oscillation. To overcome this defect of the hand–eye fixed camera system, an eye-vergence system has been put forward, where the pose of the cameras could be rotated to observe the target object. The visual servoing controllers of hand and eye-vergence are installed independently, so that it can observe the target object at the center of camera images through eye-vergence function. In this research, genetic algorithm (GA) is used as a pose tracking method, which is called “Real-Time Multi-step GA(RM-GA),” solves on-line optimization problems for 3D visual servoing. The performances of real-time object tracking using eye-vergence system and “RM-GA” method have been examined, and also the pose tracking accuracy has been verified.     

A real-time fuzzy reasoning based control system for catching a moving goldfish

Designing a real-time visual tracking system to catch a goldfish is a complex task because of the large amount of streaming video data that must be transmitted and processed immediately when tracking the goldfish. Usually, building such visual servoing systems requires the application of high-cost specialized hardware and the development of complicated visual control software. In this paper, a novel low-cost, real-time visual servo control system is presented. The system uses stereo vision consisting of two calibrated cameras to acquire images of the goldfish, and applies the continuously adaptive mean shift (CAMSHIFT) vision-tracking algorithm to provide feedback of a fish’s real-time position at a high frame rate. It then employs a 5-axis robot manipulator controlled by a fuzzy reasoning system to catch the fish. This visual tracking and servoing system is less sensitive to lighting influences and thus performs more efficiently. Experiments with the proposed method yielded very good results, as the system’s real-time 3D vision successfully tracked two fish and guided the manipulator, which has a net attached to its end effector, to catch one of them.     

Automatic robotic tasks in unstructured environments using an image path tracker

This paper describes a new method to perform automatic tasks with a robot in an unstructured environment. A task to replace a blown light bulb in a streetlamp is described to show that this method works properly. In order to perform this task correctly, the robot is positioned by tracking secure previously defined paths. The robot, using an eye-in-hand configuration on a visual servoing scheme and a force sensor, is able to interact with its environment due to the fact that the path tracking is performed with time-independent behaviour. The desired path is expressed in the image space. However, the proposed method obtains a correct tracking not only in the image, but also in the 3D space. This method solves the problems of the previously proposed time-independent tracking systems based on visual servoing, such as the specification of the desired tracking velocity, less oscillating behaviour and a correct tracking in the 3D space when high velocities are used. The experiments shown in this paper demonstrate the necessity of time-independent behaviour in tracking and the correct performance of the system.     

A general approach for sensibility analysis in visual servoing

In an experimental robotic context, it is very important to study the robustness of visual servoing control laws with respect to uncertainties that arise on the camera intrinsic and extrinsic parameters. Here we propose a general approach allowing us to analyze the sensibility of such control laws with respect to the camera parameter uncertainties. We only focus on sensibility aspects and do not address the stability analysis problem. The presented approach is validated on several examples which consider uncertainties on focal length, camera optical centers and measurement noise.     

Visual Control of Robots with Delayed Images

This research develops a control scheme for visual servoing that explicitly takes into account the delay introduced by image acquisition and processing. For this purpose, a predictor block, i.e., an estimator that predicts several samples ahead of time, is properly included in the scheme. The proposed approach is analytically analyzed in terms of dynamics and steady-state errors, and compared to previous approaches. Furthermore, several simulations are comparatively shown in order to illustrate the benefits and limitations of the proposed control scheme. Finally, some experimental results using a turntable and a 3-d.o.f. Cartesian robot are provided in order to validate the analytical and simulation results.     

Optimal Camera Trajectory under Visibility Constraint in Visual Servoing: A Variational Approach

The principal deficiency of an image-based servo is that the induced three-dimensional (3-D) trajectories are not optimal and sometimes, especially when the displacements to realize are large, these trajectories are not physically valid, leading to the failure of the servoing process. In this paper, we adress the problem of generating trajectories of some image features that correspond to optimal 3-D trajectories in order to control efficiently a robotic system using an image-based control strategy. First, a collineation path between given the start and end points is obtained, and then the trajectories of the image features are derived. Path planning is formulated as a variational problem that allows us to consider simultaneously optimality and inequality constraints (visibility). A numerical method is employed for solving the path planning problem in the variational form.     

Developing robust vision modules for microsystems applications

In this work, several robust vision modules are developed and implemented for fully automated micromanipulation. These are autofocusing, object and end-effector detection, real-time tracking and optical system calibration modules. An image based visual servoing architecture and a path planning algorithm are also proposed based on the developed vision modules. Experimental results are provided to assess the performance of the proposed visual servoing approach in positioning and trajectory tracking tasks. Proposed path planning algorithm in conjunction with visual servoing imply successful micromanipulation tasks.     

Visual Object Tracking and Servoing Control of a Nano-Scale Quadrotor: System,Algorithms, and Experiments

There are two main trends in the development of unmanned aerial vehicle(UAV)technologies:miniaturization and intellectualization,in which realizing object tracking capabilities for a nano-scale UAV is one of the most challenging problems.In this paper,we present a visual object tracking and servoing control system utilizing a tailor-made 38 g nano-scale quadrotor.A lightweight visual module is integrated to enable object tracking capabilities,and a micro positioning deck is mounted to provide accurate pose estimation.In order to be robust against object appearance variations,a novel object tracking algorithm,denoted by RMCTer,is proposed,which integrates a powerful short-term tracking module and an efficient long-term processing module.In particular,the long-term processing module can provide additional object information and modify the short-term tracking model in a timely manner.Furthermore,a positionbased visual servoing control method is proposed for the quadrotor,where an adaptive tracking controller is designed by leveraging backstepping and adaptive techniques.Stable and accurate object tracking is achieved even under disturbances.Experimental results are presented to demonstrate the high accuracy and stability of the whole tracking system.     

Virtual world explorations by using topological and semantic knowledge

This paper is dedicated to virtual world exploration techniques. Automatic camera control is important in many fields as computational geometry, visual servoing, robot motion, graph drawing, etc. The paper introduces a high-level camera controlling approach in virtual environments. The proposed method is related to real-time 3D scene exploration and is made of two steps. In the first step, a set of good viewpoints is chosen to give the user a maximum knowledge of the scene. The second step uses the viewpoints to compute a camera path between them. Finally, we define a notion of semantic distance between objects of the scene to improve the approach.     

荷载不确定移动机器人视觉伺服系统鲁棒预测控制

考虑具有可见性约束和执行器约束的载荷不确定移动机器人视觉伺服系统,提出一种鲁棒视觉伺服预测控制策略.首先将该移动机器人视觉伺服系统建模为关于视觉伺服误差和驱动的不确定系统.其次,对约束的视觉伺服误差子系统,设计基于半正定规划的速度规划预测控制算法.该算法分为离线计算和在线调度两个部分,降低预测控制算法的在线计算量.而对...     

Pose Estimation using Point and Line Correspondences

The problem of a real-time pose estimation between a 3D scene and a single camera is a fundamental task in most 3D computer vision and robotics applications such as object tracking, visual servoing, and virtual reality. In this paper we present two fast methods for estimating the 3D pose using 2D to 3D point and line correspondences. The first method is based on the iterative use of a weak perspective camera model and forms a generalization of DeMenthon's method (1995) which consists of determining the pose from point correspondences. In this method the pose is iteratively improved with a weak perspective camera model and at convergence the computed pose corresponds to the perspective camera model. The second method is based on the iterative use of a paraperspective camera model which is a first order approximation of perspective. We describe in detail these two methods for both non-planar and planar objects. Experiments involving synthetic data as well as real range data indicate the feasibility and robustness of these two methods. We analyse the convergence of these methods and we conclude that the iterative paraperspective method has better convergence properties than the iterative weak perspective method. We also introduce a non-linear optimization method for solving the pose problem.     

Visual servoing of redundant manipulator with Jacobian matrix estimation using self-organizing map

Vision based redundant manipulator control with a neural network based learning strategy is discussed in this paper. The manipulator is visually controlled with stereo vision in an eye-to-hand configuration. A novel Kohonen’s self-organizing map (KSOM) based visual servoing scheme has been proposed for a redundant manipulator with 7 degrees of freedom (DOF). The inverse kinematic relationship of the manipulator is learned using a Kohonen’s self-organizing map. This learned map is shown to be an approximate estimate of the inverse Jacobian, which can then be used in conjunction with the proportional controller to achieve closed loop servoing in real-time. It is shown through Lyapunov stability analysis that the proposed learning based servoing scheme ensures global stability. A generalized weight update law is proposed for KSOM based inverse kinematic control, to resolve the redundancy during the learning phase. Unlike the existing visual servoing schemes, the proposed KSOM based scheme eliminates the computation of the pseudo-inverse of the Jacobian matrix in real-time. This makes the proposed algorithm computationally more efficient. The proposed scheme has been implemented on a 7 DOF PowerCube? robot manipulator with visual feedback from two cameras.     

Vision resolvability for visually servoed manipulation

This article introduces a sensor placement measure called vision resolvability. The measure provides a technique for estimating the relative ability of various visual sensors, including monocular systems, stereo pairs, multi-baseline stereo systems, and 3D rangefinders, to accurately control visually manipulated objects. The resolvability ellipsoid illustrates the directional nature of resolvability, and can be used to direct camera motion and adjust camera intrinsic parameters in real-time so that the servoing accuracy of the visual servoing system improves with camera-lens motion. The Jacobian mapping from task space to sensor space is derived for a monocular system, a stereo pair with parallel optical axes, and a stereo pair with perpendicular optical axes. Resolvability ellipsoids based on these mappings for various sensor configurations are presented. Visual servoing experiments demonstrate that vision resolvability can be used to direct camera-lens motion to increase the ability of a visually servoed manipulator to precisely servo objects. © 1996 John Wiley & Sons, Inc.