Digital twin-enabled grasp outcomes assessment for unknown objects using visual-tactile fusion perception

Humans can instinctively predict whether a given grasp will be successful through visual and rich haptic feedback. Towards the next generation of smart robotic manufacturing, robots must be equipped with similar capabilities to cope with grasping unknown objects in unstructured environments. However, most existing data-driven methods take global visual images and tactile readings from the real-world system as input, making them incapable of predicting the grasp outcomes for cluttered objects or generating large-scale datasets. First, this paper proposes a visual-tactile fusion method to predict the results of grasping cluttered objects, which is the most common scenario for grasping applications. Concretely, the multimodal fusion network (MMFN) uses the local point cloud within the gripper as the visual signal input, while the tactile signal input is the images provided by two high-resolution tactile sensors. Second, collecting data in the real world is high-cost and time-consuming. Therefore, this paper proposes a digital twin-enabled robotic grasping system to collect large-scale multimodal datasets and investigates how to apply domain randomization and domain adaptation to bridge the sim-to-real transfer gap. Finally, extensive validation experiments are conducted in physical and virtual environments. The experimental results demonstrate the effectiveness of the proposed method in assessing grasp stability for cluttered objects and performing zero-shot sim-to-real policy transfer on the real robot with the aid of the proposed migration strategy.     

Few-experiential learning system of robotic picking task with selective dual-arm grasping

Recently, robots are introduced to warehouses and factories for automation and are expected to execute dual-arm manipulation as human does and to manipulate large, heavy and unbalanced objects. We focus on target picking task in the cluttered environment and aim to realize a robot picking system which the robot selects and executes proper grasping motion from single-arm and dual-arm motion. In this paper, we propose a few-experiential learning-based target picking system with selective dual-arm grasping. In our system, a robot first learns grasping points and object semantic and instance label with automatically synthesized dataset. The robot then executes and collects grasp trial experiences in the real world and retrains the grasping point prediction model with the collected trial experiences. Finally, the robot evaluates candidate pairs of grasping object instance, strategy and points and selects to execute the optimal grasping motion. In the experiments, we evaluated our system by conducting target picking task experiments with a dual-arm humanoid robot Baxter in the cluttered environment as warehouse.     

杂乱场景下小物体抓取检测研究

目的 杂乱场景下的物体抓取姿态检测是智能机器人的一项基本技能。尽管六自由度抓取学习取得了进展,但先前的方法在采样和学习中忽略了物体尺寸差异,导致在小物体上抓取表现较差。方法 提出了一种物体掩码辅助采样方法,在所有物体上采样相同的点以平衡抓取分布,解决了采样点分布不均匀问题。此外,学习时采用多尺度学习策略,在物体部分点云上使用多尺度圆柱分组以提升局部几何表示能力,解决了由物体尺度差异导致的学习抓取操作参数困难问题。通过设计一个端到端的抓取网络,嵌入了提出的采样和学习方法,能够有效提升物体抓取检测性能。结果 在大型基准数据集GraspNet-1Billion上进行评估,本文方法取得对比方法中的最优性能,其中在小物体上的抓取指标平均提升了7%,大量的真实机器人实验也表明该方法具有抓取未知物体的良好泛化性能。结论 本文聚焦于小物体上的抓取,提出了一种掩码辅助采样方法嵌入到提出的端到端学习网络中,并引入了多尺度分组学习策略提高物体的局部几何表示,能够有效提升在小尺寸物体上的抓取质量,并在所有物体上的抓取评估结果都超过了对比方法。     

Pick-and-verify: verification-based highly reliable picking system for various target objects in clutter

The reliability of picking task for various objects in clutter, as measured on the Amazon Picking Challenge, is far from the expectations of automation companies. Even if the best-performed team, who run object detection before picking the object, had picked a wrong object in the competition. In this paper, we propose a practical method to compose a highly reliable picking system with verification-based approach to reduce the rate of wrong picking and raise the reliability of picking ordered objects. In our approach, which we call pick-and-verify, the robot recognizes object twice: in clutter scene to detect the target and in hand after picking an object with less time loss and rise of reliability of picking the target. For grasping the detected object we do not assume its pose and it is actually the target object, instead, we adopt vision-based grasp planning for vacuum gripper with sensed 3-D point cloud. With the presented approach, the reliability of picking target objects raised 50%, and the score in the APC2015 competition has been improved to be close to the best-performed team by picking 9 out of 12 objects in 10 min with the same hardware in our previous system.     

Static grip selection for robot-based automated assembly systems

In this article we present an algorithmic approach to determine the suitable grasp of an object in an automated assembly environment. The algorithm is based on the available object surfaces and the initial and final task constraints and gripper characteristics. If the imposed task and gripper constraints do not allow a possible grasp, intermediate motions may need to be made to reorient the part. Once a set of possible grasps which statisfy task and gripper constraints are found, the stability of each grasp is analyzed using screw theory. An optimal grasp is one which minimizes the grasping forces over the possible set of grasps. Results utilizing our methodology are presented. Our method can be interfaced with CAD database such as a solid modelling system based on boundary representation for automatic selection of grasping configurations.     

抓取任务中的融差控制方法

依据“融差性思维”,提出了无需精确感知依旧可以在一定范围内有效工作的融差控制方法。具体分析了融差抓取方法如何运用相同控制量实现不同抓取任务的工作原理,这一原理使得融差抓取方法在面对一大类抓取任务时,不需要知道物体的具体参数,只需要知道这一大类物体的边界条件。进一步分析了融差抓取方法在欠驱动手爪上的适用性,并发现了欠驱动手爪的局限性。实验表明,在控制量设定不变的情况下,依据融差抓取方法,柔性手爪可以抓住且不抓坏宽度范围为5～45 mm的嫩豆腐,且能够成功抓取宽度范围为5～60 mm的硬质长方体;弹簧关节欠驱动手爪可以抓住且不抓坏宽度范围为20～40 mm的嫩豆腐,且能够成功抓取宽度范围为5～60 mm的硬质长方体。这体现了融差抓取方法的通用性和欠驱动手爪在抓取柔性物体时的局限性。最后,展示了柔性手爪使用融差抓取方法在桌面抓取应用中以简单的控制策略成功抓取不同形状、不同材质的物体。这充分说明了融差抓取方法不依赖于精确的对象感知及物体模型,能够简化控制策略。     

Grasping the not-so-obvious: vision-based object handling for industrial applications

This article reports on several industrial applications of a visually guided system for robot grasping using an inexpensive two-finger gripper. In all cases, the robot uses visual information as input and is able to reason about the shapes of objects in a scene in order to decide the best stable grasp online. The first version of this system was able to grasp rectangular parts in arbitrary positions in the scene and was successfully deployed. New applications in industry have been addressed that must cope with the cost, time, and reliability requirements imposed by the industrial process. The results show that the capabilities of the underlying methodology make it feasible to deal with more complex shapes, even a priori unknown, opening up new possibilities within industrial domains, such as the food industry, that has traditionally not been fully automated due to a large variability in the shaper of the objects to be handled.     

Perception and Grasping of Object Parts from Active Robot Exploration

Like humans, robots that need semantic perception and accurate estimation of the environment can increase their knowledge through active interaction with objects. This paper proposes a novel method for 3D object modeling for a robot manipulator with an eye-in-hand laser range sensor. Since the robot can only perceive the environment from a limited viewpoint, it actively manipulates a target object and generates a complete model by accumulation and registration of partial views. Three registration algorithms are investigated and compared in experiments performed in cluttered environments with complex rigid objects made of multiple parts. A data structure based on proximity graph, that encodes neighborhood relations in range scans, is also introduced to perform efficient range queries. The proposed method for 3D object modeling is applied to perform task-level manipulation. Indeed, once a complete model is available the object is segmented into its constituent parts and categorized. Object sub-parts that are relevant for the task and that afford a grasping action are identified and selected as candidate regions for grasp planning.     

A strategy for grasping unknown objects based on co-planarity and colour information

In this work, we describe and evaluate a grasping mechanism that does not make use of any specific object prior knowledge. The mechanism makes use of second-order relations between visually extracted multi-modal 3D features provided by an early cognitive vision system. More specifically, the algorithm is based on two relations covering geometric information in terms of a co-planarity constraint as well as appearance based information in terms of co-occurrence of colour properties. We show that our algorithm, although making use of such rather simple constraints, is able to grasp objects with a reasonable success rate in rather complex environments (i.e., cluttered scenes with multiple objects).Moreover, we have embedded the algorithm within a cognitive system that allows for autonomous exploration and learning in different contexts. First, the system is able to perform long action sequences which, although the grasping attempts not being always successful, can recover from mistakes and more importantly, is able to evaluate the success of the grasps autonomously by haptic feedback (i.e., by a force torque sensor at the wrist and proprioceptive information about the distance of the gripper after a gasping attempt). Such labelled data is then used for improving the initially hard-wired algorithm by learning. Moreover, the grasping behaviour has been used in a cognitive system to trigger higher level processes such as object learning and learning of object specific grasping.     

A variable compliance, soft gripper

Autonomous grasping is an important but challenging task and has therefore been intensively addressed by the robotics community. One of the important issues is the ability of the grasping device to accommodate varying object shapes in order to form a stable, multi-point grasp. Particularly in the human environment, where robots are faced with a vast set of objects varying in shape and size, a versatile grasping device is highly desirable. Solutions to this problem have often involved discrete continuum structures that typically comprise of compliant sections interconnected with mechanically rigid parts. Such devices require a more complex control and planning of the grasping action than intrinsically compliant structures which passively adapt to complex shapes objects. In this paper, we present a low-cost, soft cable-driven gripper, featuring no stiff sections, which is able to adapt to a wide range of objects due to its entirely soft structure. Its versatility is demonstrated in several experiments. In addition, we also show how its compliance can be passively varied to ensure a compliant but also stable and safe grasp.     

Visibility-based spatial reasoning for object manipulation in cluttered environments

In this paper, we present visibility-based spatial reasoning techniques for real-time object manipulation in cluttered environments. When a robot is requested to manipulate an object, a collision-free path should be determined to access, grasp, and move the target object. This often requires processing of time-consuming motion planning routines, making real-time object manipulation difficult or infeasible, especially in a robot with a high DOF and/or in a highly cluttered environment. This paper places special emphasis on developing real-time motion planning, in particular, for accessing and removing an object in a cluttered workspace, as a local planner that can be integrated with a general motion planner for improved overall efficiency. In the proposed approach, the access direction of the object to grasp is determined through visibility query, and the removal direction to retrieve the object grasped by the gripper is computed using an environment map. The experimental results demonstrate that the proposed approach, when implemented by graphics hardware, is fast and robust enough to manipulate 3D objects in real-time applications.     

A new strategy combining empirical and analytical approaches for grasping unknown 3D objects

This paper proposes a novel strategy for grasping 3D unknown objects in accordance with their corresponding task. We define the handle or the natural grasping component of an object as the part chosen by humans to pick up this object. When humans reach out to grasp an object, it is generally in the aim of accomplishing a task. Thus, the chosen grasp is quite related to the object task. Our approach learns to identify object handles by imitating humans. In this paper, a new sufficient condition for computing force-closure grasps on the obtained handle is also proposed. Several experiments were conducted to test the ability of the algorithm to generalize to new objects. They also show the adaptability of our strategy to the hand kinematics.     

三维立体视觉机械臂智能抓取分类系统的开发

采用工业相机、工业投影机、普通摄像头、计算机和机械臂开发了一套具有三维立体视觉的机械臂智能抓取分类系统。该系统采用自编软件实现了对工业相机、工业投影机的自动控制和同步，通过前期研究提出的双波长条纹投影三维形貌测量法获取了物体的高度信息，结合opencv技术和普通摄像头获取的物体二维平行面信息，实现了物体的自动识别和分类；利用串口通信协议，将上述处理后的数据传送至机械臂，系统进行几何姿态解算，实现了智能抓取，并能根据抓手上压力反馈自动调节抓手张合程度，实现自适应抓取。经实验证明该系统能通过自带的快速三维形貌获取装置实现准确、快速的抓取工作范围内的任意形状的物体并实现智能分类。     

A soft gripper with variable stiffness inspired by pangolin scales,toothed pneumatic actuator and autonomous controller

Aiming at combining compliant covering and rigid lifting to the object grasping, this paper presents the design principle of a variable stiffness soft gripper and carries out its structural design, gripper fabrication and controller development. The proposed soft finger is composed of a variable stiffness layer and a pneumatic driven layer. The variable stiffness layer is inspired by the pangolins whose scales are flexible in daily activities and become tough when being threatened by predators. A toothed pneumatic actuator is designed to supply power with increased stiffness. The three-finger soft gripper is fabricated by 3D printing and molding of super elastic material. The tests for verifying grasping capability and variable stiffness are implemented. Experimental results show that the gripper can grasp a large variety of objects and achieve enhanced stiffness. The stiffness of the gripper is more than twice higher than the pneumatic gripper without variable stiffness structure. Finally, the control system for autonomous grasping is developed. The control block is divided into the actuation layer, information processing layer and user interface layer. According to the grasping process, the feedback signals in the information processing layer are collected by sensors. A safe grasping assessment is added to the control scheme for changing the gripper stiffness autonomously, which differs from the traditional soft gripper controller. The proposed soft gripper has variable stiffness, enhanced pneumatic input, autonomous control system. Therefore, it has great potential to be applied in the unstructured environment for effective, adaptable and safe object grasping.     

A strategy for fast grasping of unknown objects using partial shape information from range sensors

In this paper, we present a strategy for fast grasping of unknown objects based on the partial shape information from range sensors for a mobile robot with a parallel-jaw gripper. The proposed method can realize fast grasping of an unknown object without needing complete information of the object or learning from grasping experience. Information regarding the shape of the object is acquired by a 2D range sensor installed on the robot at an inclined angle to the ground. Features for determining the maximal contact area are extracted directly from the partial shape information of the unknown object to determine the candidate grasping points. Note that since the shape and mass are unknown before grasping, a successful and stable grasp cannot be in fact guaranteed. Thus, after performing a grasping trial, the mobile robot uses the 2D range sensor to judge whether the object can be lifted. If a grasping trial fails, the mobile robot will quickly find other candidate grasping points for another trial until a successful and stable grasp is realized. The proposed approach has been tested in experiments, which found that a mobile robot with a parallel-jaw gripper can successfully grasp a wide variety of objects using the proposed algorithm. The results illustrate the validity of the proposed algorithm in term of the grasping time.     

A three-dimensional criterion for the determination of optimal grip points

Multifingered grasping has aroused remarkable interest because it makes possible the manipulation of objects of different shapes and sizes. However, manipulating and picking up objects in unstructured environments requires accurate contact-point selection. Generally, such processes are subject to external forces which are difficult to predict and may change during task execution.In this paper, an optimization criterion is proposed which is meant to select the optimal grip points in a three-dimensional problem for any number of contact points. This method may be applied to three-dimensional objects of any shape (curved or polygonal) and does not require that the external forces acting on the object be known. A grasp quality index is presented which has been obtained by minimizing the grasping forces required to balance a generalized external disturbance. The optimization criterion has led to the formulation of a single optimization problem with non-linear constraints. Finally, the paper presents the results obtained in searches for the optimal grip points on some two- and three-dimensional objects.     

Robust optimization with applications to design of context specific robot solutions

This paper presents an investigation of five optimization algorithms for simulation-based optimization for robotic tasks, where robust solutions are required. We evaluate the optimization methods on three use cases. The use cases involve using a robot for handling meat, optimizing gripper design for aligning objects and optimizing gripper design for table picking in cluttered scenes. We use dynamic simulations to model the use cases, where the most important physical aspects are captured. We have a focus on the robustness with respect to crucial system uncertainties, which is important in an industrial setting. The choice of parameterization and objective scores is also discussed since this choice has some impact on the performance of the optimization algorithms. For all problems, we find feasible solutions ready for real world testing, and overall the optimization method RBFopt has the best performance in terms of finding robust solutions within the fewest amount of simulations.     

Adaptive neuro-fuzzy prediction of grasping object weight for passively compliant gripper

The development of universal grippers able to pick up unfamiliar objects of widely varying shapes and surfaces is a very challenging task. Passively compliant underactuated mechanisms are one way to obtain the gripper which could accommodate to any irregular and sensitive grasping objects. The purpose of the underactuation is to use the power of one actuator to drive the open and close motion of the gripper. The fully compliant mechanism has multiple degrees of freedom and can be considered as an underactuated mechanism. This paper presents a new design of the adaptive underactuated compliant gripper with distributed compliance. The optimal topology of the gripper structure was obtained by iterative finite element method (FEM) optimization procedure. The main points of this paper are in explanation of a new sensing capability of the gripper for grasping and lifting up the gripping objects. Since the sensor stress depends on weight of the grasping object it is appropriate to establish a prediction model for estimation of the grasping object weight in relation to sensor stress. A soft computing based prediction model was developed. In this study an adaptive neuro-fuzzy inference system (ANFIS) was used as soft computing methodology to conduct prediction of the grasping objects weight. The training and checking data for the ANFIS network were obtained by FEM simulations.     

Input Displacement Neuro-fuzzy Control and Object Recognition by Compliant Multi-fingered Passively Adaptive Robotic Gripper

The requirement for new flexible adaptive grippers is the ability to detect and recognize objects in their environments. It is known that robotic manipulators are highly nonlinear systems, and an accurate mathematical model is difficult to obtain, thus making it difficult make decision strategies using conventional techniques. Here, an adaptive neuro fuzzy inference system (ANFIS) for controlling input displacement and object recognition of a new adaptive compliant gripper is presented. The grasping function of the proposed adaptive multi-fingered gripper relies on the physical contact of the finger with an object. This design of the each finger has embedded sensors as part of its structure. The use of embedded sensors in a robot gripper gives the control system the ability to control input displacement of the gripper and to recognize particular shapes of the grasping objects. Fuzzy based controllers develop a control signal according to grasping object shape which yields on the firing of the rule base. The selection of the proper rule base depending on the situation can be achieved by using an ANFIS strategy, which becomes an integrated method of approach for the control purposes. In the designed ANFIS scheme, neural network techniques are used to select a proper rule base, which is achieved using the back propagation algorithm. The simulation results presented in this paper show the effectiveness of the developed method.