Fusing force and vision feedback for manipulating deformable objects

This article describes a framework that fuses vision and force feedback for the control of highly deformable objects. Deformable active contours, or snakes, are used to visually observe changes in object shape over time. Finite‐element models of object deformations are used with force feedback to predict expected visually observed deformations. Our approach improves the performance of large, complex deformable contour tracking over traditional computer vision tracking techniques. This same approach of combining deformable active contours with finite‐element material models is modified so that a vision sensor, i.e., a charge‐coupled device (CCD) camera, can be used as a force sensor. By visually tracking changes in contours on the object, material deflections can be transformed into applied stress estimates through finite element modeling. Therefore, internal object stresses due to object manipulation can be visually observed and controlled, thus creating a framework for deformable object manipulation. A pinhole camera model is used to accomplish vision and force sensor feedback assimilation from these two sensing modalities during manipulation. © 2001 John Wiley & Sons, Inc.     

Manipulating Deformable Linear Objects: Fuzzy-Based Active Vibration Damping Skill

Human can handle a deformable object and damp its vibration with recognized skill. However, for an industrial robot, handling a deformable object with acute vibration is often a difficult task. This paper addresses the problem of active damping skill for handling deformable linear objects (DLOs) by using a strategy inspired from human manipulation skills. The strategy is illustrated by several rules, which are explained by a fuzzy and a P controller. A proportional-integral-derivative (PID) controller is also employed to explain the rules as a comparison. The interpretations from controllers are translated into high level commands in a robotic language V+. A standard industrial robot with a force/torque sensor mounted on the wrist was employed to demonstrate the skill. Experimental results showed the fuzzy based damping skill is quite effective and stable even without any previous acknowledge of the deformable linear objects.Category (5)     

Haptic perception of shape and hollowness of deformable objects using the anthrobot‐III robot hand

This article presents a methodology for the haptic perception of contour shapes of almost planar objects grasped by a five‐fingered robot hand as well as the detection of any object cavity. The originality of our approach resides in (1) finding the reaction force patterns at the fingertips of a five‐fingered robot hand that grasps different deformable objects (forward problem) and (2) using these contact force patterns to find the shapes of grasped objects (inverse problem) and (3) to determine material defects such as holes in an object with identified shape. Contact force patterns are generated in the forward problem by the finite element method (FEM) and the shape identification in the inverse problem is realized by a supervised neural network architecture using the backpropagation algorithm. Following shape identification, detection of holes is performed by clustering actual and prototypical contact force patterns using the self‐organizing feature maps of neural gas networks as an unsupervised hole‐screening method. ©1999 John Wiley & Sons, Inc.     

In-hand recognition and manipulation of elastic objects using a servo-tactile control strategy

Grasping and manipulating objects with robotic hands depend largely on the features of the object to be used. Especially, features such as softness and deformability are crucial to take into account during the manipulation tasks. Indeed, positions of the fingers and forces to be applied by the robot hand when manipulating an object must be adapted to the caused deformation. For unknown objects, a previous recognition stage is usually needed to get the features of the object, and the manipulation strategies must be adapted depending on that recognition stage. To obtain a precise control in the manipulation task, a complex object model is usually needed and performed, for example using the Finite Element Method. However, these models require a complete discretization of the object and they are time-consuming for the performance of the manipulation tasks. For that reason, in this paper a new control strategy, based on a minimal spring model of the objects, is presented and used for the control of the robot hand. This paper also presents an adaptable tactile-servo control scheme that can be used in in-hand manipulation tasks of deformable objects. Tactile control is based on achieving and maintaining a force value at the contact points which changes according to the object softness, a feature estimated in an initial recognition stage.     

Autonomous Shape Control of a Deformable Object by Multiple Manipulators

Shape control of a deformable object by a robotic system is a challenging problem because of the difficulty of imposing shape change by a finite number actuation points to an essentially infinite dimensional object. In this paper, a new approach to shape changing of deformable objects by a system of manipulators is presented. First, an integrated dynamic equation of motion for a system of multiple manipulators handling a deformable object is developed. The initial and the final shapes of the deformable object are specified by curves that represent the boundary of the object. We design an optimization-based planner that minimizes an energy-like criterion to determine the locations of the contact points on the desired curve representing the final shape of the object. The motion of each manipulator is controlled independently without any communication between them. Finally we design a robust controller for shape changing that can work in the presence of modeling uncertainty. The simulation results demonstrate the efficacy of the proposed method.     

Intelligent Learning for Deformable Object Manipulation

The majority of manipulation systems are designed with the assumption that the objects being handled are rigid and do not deform when grasped. This paper addresses the problem of robotic grasping and manipulation of 3-D deformable objects, such as rubber balls or bags filled with sand. Specifically, we have developed a generalized learning algorithm for handling of 3-D deformable objects in which prior knowledge of object attributes is not required and thus it can be applied to a large class of object types. Our methodology relies on the implementation of two main tasks. Our first task is to calculate deformation characteristics for a non-rigid object represented by a physically-based model. Using nonlinear partial differential equations, we model the particle motion of the deformable object in order to calculate the deformation characteristics. For our second task, we must calculate the minimum force required to successfully lift the deformable object. This minimum lifting force can be learned using a technique called iterative lifting. Once the deformation characteristics and the associated lifting force term are determined, they are used to train a neural network for extracting the minimum force required for subsequent deformable object manipulation tasks. Our developed algorithm is validated with two sets of experiments. The first experimental results are derived from the implementation of the algorithm in a simulated environment. The second set involves a physical implementation of the technique whose outcome is compared with the simulation results to test the real world validity of the developed methodology.     

Robust object manipulation for tactile-based blind grasping

Tactile-based blind grasping addresses realistic robotic grasping in which the hand only has access to proprioceptive and tactile sensors. The robotic hand has no prior knowledge of the object/grasp properties, such as object weight, inertia, and shape. There exists no manipulation controller that rigorously guarantees object manipulation in such a setting. Here, a robust control law is proposed for object manipulation in tactile-based blind grasping. The analysis ensures semi-global asymptotic and exponential stability in the presence of model uncertainties and external disturbances that are neglected in related work. Simulation and hardware results validate the effectiveness of the proposed approach.     

Using linked volumes to model object collisions, deformation,cutting, carving, and joining

In volume graphics, objects are represented by arrays or clusters of sampled 3D data. A volumetric object representation is necessary in computer modeling whenever interior structure affects an object's behavior or appearance. However, existing volumetric representations are not sufficient for modeling the behaviors expected in applications such as surgical simulation, where interactions between both rigid and deformable objects and the cutting, tearing, and repairing of soft tissues must be modeled in real time. Three-dimensional voxel arrays lack the sense of connectivity needed for complex object deformation, while finite element models and mass-spring systems require substantially reduced geometric resolution for interactivity and they can not be easily cut or carved interactively. This paper discusses a linked volume representation that enables physically realistic modeling of object interactions such as: collision detection, collision response, 3D object deformation, and interactive object modification by carving, cutting, tearing, and joining. The paper presents a set of algorithms that allow interactive manipulation of linked volumes that have more than an order of magnitude more elements and considerably more flexibility than existing methods. Implementation details, results from timing tests, and measurements of material behavior are presented     

Robust Shape Control of Deformable Objects Using Model-Based Techniques

Shape control of a deformable object by a robotic system is a challenging problem because of the difficulty of imposing shape change by a finite number of actuation points to an essentially infinite-dimensional object. In this paper, a new approach to shape changing of deformable objects by a system of manipulators is presented. First, an integrated dynamic equation of motion for a system of multiple manipulators handling a deformable object is developed. A shape correspondence between the initial contact points of the multiple manipulators on a deformable object and a two-dimensional curve that represents the final desired shape is determined. A shape Jacobian that contains the local shape information of the desired shape of the object is formulated and is introduced into the control law. We develop a shape estimator with a second-order dynamics that is used to estimate the curve parameters corresponding to the end-effector position in each time step as the initial object is deformed to its desired final shape. Finally, we design a robust controller for the shape changing task that can work in the presence of modeling uncertainty. The simulation results demonstrate the efficacy of the proposed method.     

gripper

Grasping of objects has been a challenging task for robots. The complex grasping task can be defined as object contact control and manipulation subtasks. In this paper, object contact control subtask is defined as the ability to follow a trajectory accurately by the fingers of a gripper. The object manipulation subtask is defined in terms of maintaining a predefined applied force by the fingers on the object. A sophisticated controller is necessary since the process of grasping an object without a priori knowledge of the object's size, texture, softness, gripper, and contact dynamics is rather difficult. Moreover, the object has to be secured accurately and considerably fast without damaging it. Since the gripper, contact dynamics, and the object properties are not typically known beforehand, an adaptive critic neural network (NN)-based hybrid position/force control scheme is introduced. The feedforward action generating NN in the adaptive critic NN controller compensates the nonlinear gripper and contact dynamics. The learning of the action generating NN is performed on-line based on a critic NN output signal. The controller ensures that a three-finger gripper tracks a desired trajectory while applying desired forces on the object for manipulation. Novel NN weight tuning updates are derived for the action generating and critic NNs so that Lyapunov-based stability analysis can be shown. Simulation results demonstrate that the proposed scheme successfully allows fingers of a gripper to secure objects without the knowledge of the underlying gripper and contact dynamics of the object compared to conventional schemes.     

A Deformable Object Tracking Algorithm Based on the Boundary Element Method that is Robust to Occlusions and Spurious Edges

The manipulation of deformable objects is an important problem in robotics and arises in many applications including biomanipulation, microassembly, and robotic surgery. For some applications, the robotic manipulator itself may be deformable. Vision-based deformable object tracking can provide feedback for these applications. Computer vision is a logical sensing choice for tracking deformable objects because the large amount of data that is collected by a vision system allows many points within the deformable object to be tracked simultaneously. This article introduces a template based deformable object tracking algorithm, based on the boundary element method, that is able to track a wide range of deformable objects. The robustness of this algorithm to occlusions and to spurious edges in the source image is also demonstrated. A robust error measure is used to handle the problem of occlusion and an improved edge detector based on the Canny edge operator is used to suppress spurious edges. This article concludes by quantifying the performance increase provided by the robust error measure and the robust edge detector. The performance of the algorithm is also demonstrated through the tracking of a sequence of cardiac MRI images.     

An algorithm based on bidirectional searching and geometric constrained sampling for automatic manipulation planning in aircraft cable assembly

Robotic manipulation of deformable linear objects has potential application in aircraft cable assembly. However, it is difficult to be implemented in real tasks due to requiring prediction of the object’s deformation and obstacle-free manipulation planning with high efficiency. Aiming at exploring automatic assembly planning for aircraft cables assembly in narrow cabins with obstacles, this paper proposes a novel planning algorithm named RRT-BwC (Bi-direction with Constrain). Firstly, formulation of the question and the manipulation objects are presented with geometric definitions. Then a bi-RRT-tree searching method is developed to design the planner for overcoming obstacles in the high dimensional planning space. The numerical distance between configurations in the cable shape space are defined to measure the demands for their transition in consideration of the manipulation logics. And the sampling zone constrains to the shape configuration nodes are also discussed. Finally, the algorithm presents a valid manipulation sequence for robotic manipulation. The functionality and performance of the improved RRT approach are demonstrated with a simulated real-world problem of aircraft cable assembly, exhibiting computation efficiency promotion.     

Shape-based retrieval of video objects

The increasing availability of object-based video content requires new technologies for automatically extracting and matching of the low level features of arbitrarily shaped video. This paper proposes methods for shape retrieval of arbitrarily shaped video objects. Our methods take into account not only the still shape features but also the shape deformations that may occur in an object's lifespan. We compute the shape similarity of video objects by comparing the similarity of their representative temporal instances. We also describe motion of a video object via describing the deformations in an object's shape. Experimental results show that our proposed methods offer very good retrieval performance and match closely with the human ranking.     

Slippage control in soft finger grasping and manipulation

Handling objects with robotic soft fingers without considering the odds of slippage are not realistic. Grasping and manipulation algorithms have to be tested under such conditions for evaluating their robustness. In this paper, a dynamic analysis of rigid object manipulation with slippage control is studied using a two-link finger with soft hemispherical tip. Dependency on contact forces applied by a soft finger while grasping a rigid object is examined experimentally. A power-law model combined with a linear viscous damper is used to model the elastic behavior and damping effect of the soft tip, respectively. In order to obtain precise dynamic equations governing the system, two second-order differential equations with variable coefficients have been designed to describe the different possible states of the contact forces accordingly. A controller is designed based on the rigid fingertip model using the concept of feedback linearization for each phase of the system dynamics. Numerical simulations are used to evaluate the performance of the controller. The results reveal that the designed controller shows acceptable performance for both soft and rigid finger manipulation in reducing and canceling slippage. Furthermore, simulations indicate that the applied force in the soft finger manipulation is considerably less than the rigid “one.”.     

Shape Morphing-Based Control of Robotic Visual Servoing

We present an approach for controlling robotic interactions with objects, using synthetic images generated by morphing shapes. In particular, we attempt the problem of positioning an eye-in-hand robotic system with respect to objects in the workspace for grasping and manipulation. In our formulation, the grasp position (and consequently the approach trajectory of the manipulator), varies with each object. The proposed solution to the problem consists of two parts. First, based on a model-based object recognition framework, images of the objects taken at the desired grasp pose are stored in a database. The recognition and identification of the grasp position for an unknown input object (selected from the family of recognizable objects) occurs by morphing its contour to the templates in the database and using the virtual energy spent during the morph as a dissimilarity measure. In the second step, the images synthesized during the morph are used to guide the eye-in-hand system and execute the grasp. The proposed method requires minimal calibration of the system. Furthermore, it conjoins techniques from shape recognition, computer graphics, and vision-based robot control in a unified engineering amework. Potential applications range from recognition and positioning with respect to partially-occluded or deformable objects to planning robotic grasping based on human demonstration.     

机器人多指操作的递阶控制

为机器人多指协调操作建立一递阶控制系统.给定一操作任务,任务规划器首先生成一系列物体的运动速度;然后,协调运动规划器根据期望的物体运动速度生成期望的手指运动速度和期望的抓取姿态变化;同时,抓取力规划器为平衡作用在物体上的外力,根据当前的抓取姿态,生成各手指所需的抓取力;最后,系统将手指的期望运动速度与为实现期望抓取力而生成的顺应速度合并,并通过手指的逆雅可比转化为手指关节运动速度后,由手指的关节级运动控制器实现手指的运动和抓取力的控制.该控制方法已成功应用于香港科技大学(HKUST)灵巧手控制系统的开发.实验证明该方法不仅能完成物体轨迹的跟踪控制任务,而且能完成物体对环境的力控制和力与速度的混合控制.     

机器人多指操作的递阶控制1)

This paper presents a hierarchical control system for robot multifingered coordinate manipulation. Given a manipulation,the task planner generates a sequence of object's motion velocities at first,and then generates for coordinate motion the desired velocities of finger's motion and desired orientation change of the grasped object according to the desired velocities of object's motion.At the same time,the force planner generates the grasp forces on the fingers in order to resist the external forces on the object,according to the grasp posture.Finally,the system generates a result compliance velocity from both the desired finger's velocities and desired grasp forces,and transfers it into joint velocites through the finger's inverse Jacobian.Then the controller of joint motion implements the control of both forces and velocities for the fingers.The approach has been applied to the development of control system HKUST dexterous hand successfully.Experiment results show that it is not only possible to trail and control the object's track,but also possible to realize force control and the hybrid control of both forces and velocities through this method.     

The role of model-based segmentation in the recovery of volumetricparts from range data

We present a method for segmenting and estimating the shape of 3D objects from range data. The technique uses model views, or aspects, to constrain the fitting of deformable models to range data. Based on an initial region segmentation of a range image, regions are grouped into aspects corresponding to the volumetric parts that make up an object. The qualitative segmentation of the range image into a set of volumetric parts not only captures the coarse shape of the parts, but qualitatively encodes the orientation of each part through its aspect. Knowledge of a part's coarse shape, its orientation, as well as the mapping between the faces in its aspect and the surfaces on the part provides strong constraints on the fitting of a deformable model (supporting both global and local deformations) to the data. Unlike previous work in physics-based deformable model recovery from range data, the technique does not require presegmented data. Furthermore, occlusion is handled at segmentation time and does not complicate the fitting process, as only 3D points known to belong to a part participate in the fitting of a model to the part. We present the approach in detail and apply it to the recovery of objects from range data     

A real-time strategy for dexterous manipulation: Fingertips motion planning,force sensing and grasp stability

This paper presents a global strategy for object manipulation with the fingertips with an anthropomorphic dexterous hand: the LMS Hand of the ROBIOSS team from PPRIME Institute in Poitiers (France). Fine manipulation with the fingertips requires to compute on one hand, finger motions able to produce the desired object motion and on the other hand, it is necessary to ensure object stability with a real time scheme for the fingertip force computation. In the literature, lot of works propose to solve the stability problem, but most of these works are grasp oriented; it means that the use of the proposed methods are not easy to implement for online computation while the grasped object is moving inside the hand. Also simple real time schemes and experimental results with full-actuated mechanical hands using three fingers were not proposed or are extremely rare. Thus we wish to propose in a same strategy, a robust and simple way to solve the fingertip path planning and the fingertip force computation. First, finger path planning is based on a geometric approach, and on a contact modelling between the grasped object and the finger. And as force sensing is required for force control, a new original approach based on neural networks and on the use of tendon-driven joints is also used to evaluate the normal force acting on the finger distal phalanx. And an efficient algorithm that computes fingertip forces involved is presented in the case of three dimensional object grasps. Based on previous works, those forces are computed by using a robust optimization scheme.In order to validate this strategy, different grasps and different manipulation tasks are presented and detailed with a simulation software, SMAR, developed by the PPRIME Institute. And finally experimental results with the real hand illustrate the efficiency of the whole approach.