Towards cognitive grasping: modeling of unknown objects and its corresponding grasp types

This paper describes an intuitive approach for a cognitive grasp of a robot. The cognitive grasp means the chain of processes that make a robot to learn and execute a grasping method for unknown objects like a human. In the learning step, a robot looks around a target object to estimate the 3D shape and understands the grasp type for the object through a human demonstration. In the execution step, the robot correlates an unknown object to one of known grasp types by comparing the shape similarity of the target object based on previously learned models. For this cognitive grasp, we mainly deal with two functionalities such as reconstructing an unknown 3D object and classifying the object by grasp types. In the experiment, we evaluate the performance of object classification according to the grasp types for 20 objects via human demonstration.     

Digital twin-enabled Graduation Intelligent Manufacturing System for fixed-position assembly islands

The layout of fixed-position assembly islands is widely used in the heavy equipment industry, where the product remains at one assembly island for its entire assembly period, while required workers, equipment, and materials are moved to the island according to the assembly plan. Such layout is not only suitable for producing bulky or fragile products, but also offers considerable flexibility and competitive operational efficiency for products with medium variety and volumes. However, due to inherent complexity of the product, sophisticated assembly operations heavily rely on skilled operators, and the complexity and uncertainty are high and amplified by such massive manual interventions as well as the unique routing patterns of the fixed-position assembly process. Aiming at reducing the complexity and uncertainty, this paper introduces a digital twin-enabled Graduation Intelligent Manufacturing System (DT-GiMS) for fixed-position assembly islands. Inspired by the success of graduation ceremony, an assembly system-Graduation Manufacturing System (GMS) is proposed for fixed-position assembly islands, in which job tickets, setup tickets, operation tickets, and logistics tickets are designed to organize the production activities. Following the concept of digital twin, unified digital representations with appropriate sets of information are created at object level, product level, and system level, respectively. Through Internet of Things (IoT), smart gateway, Web 3D and industrial wearable technologies, vital information including identity, status, geometric model, and production process can be captured and mapped in physical space, and converged and synchronized with their digital representations in twin (cloud) space on a real-time basis. The overall framework of DT-GiMS is presented with physical layer, digital layer, and service layer. Real-time convergence and synchronization among them ensure that right resources are allocated and utilized to the right activities at the right time with enhanced visibility. Considering customer demand and production capacity constraints, real-time ticket pool management mechanisms are proposed to manage production activities in a near-optimal way under DT-GiMS. With the support of cloud-based services provided in service layer in DT-GiMS, managers could easily make production decisions, and onsite operators could efficiently complete their daily tasks with nearly error-free operations with enhanced visibility. A demonstrative case is carried out to verify the effectiveness of the proposed concept and approach.     

A strategy for fast grasping of unknown objects using partial shape information from range sensors

In this paper, we present a strategy for fast grasping of unknown objects based on the partial shape information from range sensors for a mobile robot with a parallel-jaw gripper. The proposed method can realize fast grasping of an unknown object without needing complete information of the object or learning from grasping experience. Information regarding the shape of the object is acquired by a 2D range sensor installed on the robot at an inclined angle to the ground. Features for determining the maximal contact area are extracted directly from the partial shape information of the unknown object to determine the candidate grasping points. Note that since the shape and mass are unknown before grasping, a successful and stable grasp cannot be in fact guaranteed. Thus, after performing a grasping trial, the mobile robot uses the 2D range sensor to judge whether the object can be lifted. If a grasping trial fails, the mobile robot will quickly find other candidate grasping points for another trial until a successful and stable grasp is realized. The proposed approach has been tested in experiments, which found that a mobile robot with a parallel-jaw gripper can successfully grasp a wide variety of objects using the proposed algorithm. The results illustrate the validity of the proposed algorithm in term of the grasping time.     

Motion estimation of 3-D objects using multisensor data fusion

This article presents an approach to estimate the general 3-D motion of a polyhedral object using multiple sensor data some of which may not provide sufficient information for the estimation of object motion. Motion can be estimated continuously from each sensor through the analysis of the instantaneous state of an object. The instantaneous state of an object is specified by the rotation, which is defined by a rotation axis and rotation angle, and the displacement of the center of rotation. We have introduced a method based on Moore-Penrose pseudoinverse theory to estimate the instantaneous state of an object, and a linear feedback estimation algorithm to approach the motion estimation. The motion estimated from each sensor is fused to provide more accurate and reliable information about the motion of an unknown object. The techniques of multisensor data fusion can be categorized into three methods: averaging, decision, and guiding. We present a fusion algorithm which combines averaging and decision. With the assumption that the motion is smooth, our approach can handle the data sequences from multiple sensors with different sampling times. We can also predict the next immediate object position and its motion. The simulation results show our proposed approach is advantageous in terms of accuracy, speed, and versatility.     

OCOG: A common grasp computation algorithm for a set of planar objects

This paper addresses the problem of defining a simple End-Effector design for a robotic arm that is able to grasp a given set of planar objects. The OCOG (Objects COmmon Grasp search) algorithm proposed in this paper searches for a common grasp over the set of objects mapping all possible grasps for each object that satisfy force closure and quality criteria by taking into account the external wrenches (forces and torque) applied to the object. The mapped grasps are represented by feature vectors in a high-dimensional space. This feature vector describes the design of the gripper. A database is generated for all possible grasps for each object in the feature vector space. A search algorithm is then used for intersecting all possible grasps over all parts and finding a common grasp suitable for all objects. The search algorithm utilizes the kd-tree index structure for representing the database of the sets of feature vectors. The kd-tree structure enables an efficient and low cost nearest-neighbor search for common vectors between the sets. Each common vector found (feature vector) is the grasp configuration for a group of objects, which implies the future end-effector design. The final step classifies the grasps found to subsets of the objects, according to the common vectors found. Simulations and experiments are presented for four objects to validate the feasibility of the proposed algorithm. The algorithm will be useful for standardization of end-effector design and reducing its engineering time.     

Deep learning-based method for vision-guided robotic grasping of unknown objects

Nowadays, robots are heavily used in factories for different tasks, most of them including grasping and manipulation of generic objects in unstructured scenarios. In order to better mimic a human operator involved in a grasping action, where he/she needs to identify the object and detect an optimal grasp by means of visual information, a widely adopted sensing solution is Artificial Vision. Nonetheless, state-of-art applications need long training and fine-tuning for manually build the object’s model that is used at run-time during the normal operations, which reduce the overall operational throughput of the robotic system. To overcome such limits, the paper presents a framework based on Deep Convolutional Neural Networks (DCNN) to predict both single and multiple grasp poses for multiple objects all at once, using a single RGB image as input. Thanks to a novel loss function, our framework is trained in an end-to-end fashion and matches state-of-art accuracy with a substantially smaller architecture, which gives unprecedented real-time performances during experimental tests, and makes the application reliable for working on real robots. The system has been implemented using the ROS framework and tested on a Baxter collaborative robot.     

A few-shot learning framework for planar pushing of unknown objects

Intelligent Service Robotics - Robot planar pushing is one of the primitive elements of non-prehensile manipulation skills and has been widely studied as an alternative solution to complex...     

No-reference image quality assessment using fusion metric

This paper presents a fusion featured metric for no-reference image quality assessment of natural images. Natural images exhibit strong statistical properties across the visual contents such as leading edge, high dimensional singularity, scale invariance, etc. The leading edge represents the strong presence of continuous points, whereas high singularity conveys about non-continuous points along the curves. Both edges and curves are equally important in perceiving the natural images. Distortions to the image affect the intensities of these points. The change in the intensities of these key points can be measured using SIFT. However, SIFT tends to ignore certain points such as the points in the low contrast region which can be identified by curvelet transform. Therefore, we propose a fusion of SIFT key points and the points identified by curvelet transform to model these changes. The proposed fused feature metric is computationally efficient and light on resources. The neruofuzzy classifier is employed to evaluate the proposed feature metric. Experimental results show a good correlation between subjective and objective scores for public datasets LIVE, TID2008, and TID2013.

     

Design and kinetostatic modeling of a compliant gripper for grasp and autonomous release of objects

A gripper with an embedded compliant bistable mechanism (BM) for gripping and autonomous release of objects is developed. Due to adhesion forces, objects might stick to the end effector of a gripper upon release, where a shaking operation may be employed to release the objects. Vibration of the end effector induced by an impact pestle adjacent to the shuttle mass of the BM may achieve autonomous release of objects adhered to the end effector. Gripping and autonomous release of objects are accomplished when the BM moves between its two stable equilibrium positions. An analytical model is developed to predict the kinetostatic behavior of the BM and to assist in the design of the gripper.     

Constraint fusion for recognition and localization of articulated objects

This paper presents a method for localization and interpretation of modeled objects that is general enough to cover articulated and other types of constrained models. The flexibility between the components of the model is expressed as spatial constraints that are fused into the pose estimation during the interpretation process. The constraint fusion assists in obtaining a precise and stable pose of each of the object's components and in finding the correct interpretation. The proposed method can handle any constraint (including inequalities) between any number of different components of the model. The framework is based on Kalman filtering.     

Efficient 3D object perception and grasp planning for mobile manipulation in domestic environments

In this article, we describe efficient methods for tackling everyday mobile manipulation tasks that require object pick-up. In order to achieve real-time performance in complex environments, we focus our approach on fast yet robust solutions. For 3D perception of objects on planar surfaces, we develop scene segmentation methods that process depth images in real-time at high frame rates. We efficiently plan feasible, collision-free grasps for the segmented objects directly from the perceived point clouds to achieve fast execution times. We evaluate our approaches quantitatively in lab experiments and also report on the successful integration of our methods in public demonstrations at RoboCup@Home competitions in 2011 and 2012.     

An efficient method for fully automatic 3D digitization of unknown objects

Our goal is to develop a complete and automatic scanning strategy with minimum prior information about the object shape. We aim to establish a methodology for the automation of the 3D digitization process. The paper presents a novel approach to determine the Next Best View (NBV) for an efficient reconstruction of highly accurate 3D models. Our method is based on the classification of the acquired surfaces into Well Visible and Barely Visible combined with a best view selection algorithm based on mean shift, which avoids unreachable positions. Our approach is applicable to all kinds of range sensors. To prove the efficiency and the robustness of our method, test objects are first scanned manually by experts in 3D digitization from the VECTEO company. The comparison of results between manual and automatic scanning shows that our method is very efficient and faster than trained experts. The 3D models of the different objects are obtained with a strongly reduced number of acquisitions while moving efficiently the ranging device. The obtained results prove the effectiveness and the versatility of our 3D reconstruction approach for industrial applications.     

空间目标位置误差未知的碰撞概率算法

针对航天器和空间碎片位置误差未知的情况,提出一种空间目标碰撞概率的计算方法。该方法基于正交投影变换将碰撞概率中的三维问题转化到二维进行求解,并利用变量置换的方法计算空间目标碰撞概率。讨论了位置误差椭球形状和大小变化对碰撞概率的影响。根据实际碰撞事例验证了其有效性。     

A strategy for grasping unknown objects based on co-planarity and colour information

In this work, we describe and evaluate a grasping mechanism that does not make use of any specific object prior knowledge. The mechanism makes use of second-order relations between visually extracted multi-modal 3D features provided by an early cognitive vision system. More specifically, the algorithm is based on two relations covering geometric information in terms of a co-planarity constraint as well as appearance based information in terms of co-occurrence of colour properties. We show that our algorithm, although making use of such rather simple constraints, is able to grasp objects with a reasonable success rate in rather complex environments (i.e., cluttered scenes with multiple objects).Moreover, we have embedded the algorithm within a cognitive system that allows for autonomous exploration and learning in different contexts. First, the system is able to perform long action sequences which, although the grasping attempts not being always successful, can recover from mistakes and more importantly, is able to evaluate the success of the grasps autonomously by haptic feedback (i.e., by a force torque sensor at the wrist and proprioceptive information about the distance of the gripper after a gasping attempt). Such labelled data is then used for improving the initially hard-wired algorithm by learning. Moreover, the grasping behaviour has been used in a cognitive system to trigger higher level processes such as object learning and learning of object specific grasping.     

Model-reduced fault detection for multi-rate sensor fusion with unknown inputs

In multi-sensor fusion, it is hard to guarantee that all sensors work at the single sampling rate, especially in the distributive and/or heterogeneous case, and fault detection (FD) in multi-rate sensor fusion may face the existence of unknown inputs (UIs) in complex environment. Meanwhile, model reduction often refers to propose a possible lower-dimensional model to replace the original model without adding significant error in practical applications. By the fact that FD in dynamic systems should only focus on the fault-related controllability and observability characteristics, it is a good idea to obtain the fault-related controllable and observable subsystem via system decomposition (i.e., model reduction) for FD. Such a kind of model reduction not only guarantee the FD performance, but also reduce the system dimensions. To this end, we propose the model-reduced fault detection (MRFD) problem for multi-rate sensor fusion subject to UIs and faults imposed on the actuator and sensors. Our aim is to design a fast and computation-effective FD scheme based on the reduced model. We use the singular decomposition for UI decoupling, and then obtain the fault-related subsystem via controllability and observability decomposition. And then the multi-rate observer (MRO) with causality constraints is designed. Different from the traditional observer used for FD, the proposed MRO outputs the fault-related partial state estimate as soon as any a sensor measurement is received, resulting in fast and computation-effective FD. Furthermore, conditions for the existence of a stable MRO, fault-to-state controllability, and fault detectability are explored. A simulation example for simplified longitudinal flight control system and method comparison with the existing multi-rate FD algorithms show the effectiveness of the proposed MRFD method.     

Viewpoint optimization for aiding grasp synthesis algorithms using reinforcement learning

Grasp synthesis for unknown objects is a challenging problem as the algorithms are expected to cope with missing object shape information. This missing information is a function of the vision sensor viewpoint. The majority of the grasp synthesis algorithms in literature synthesize a grasp by using one single image of the target object and making assumptions on the missing shape information. On the contrary, this paper proposes the use of robot's depth sensor actively: we propose an active vision methodology that optimizes the viewpoint of the sensor for increasing the quality of the synthesized grasp over time. By this way, we aim to relax the assumptions on the sensor's viewpoint and boost the success rates of the grasp synthesis algorithms. A reinforcement learning technique is employed to obtain a viewpoint optimization policy, and a training process and automated training data generation procedure are presented. The methodology is applied to a simple force-moment balance-based grasp synthesis algorithm, and a thousand simulations with five objects are conducted with random initial poses in which the grasp synthesis algorithm was not able to obtain a good grasp with the initial viewpoint. In 94% of these cases, the policy achieved to find a successful grasp.     

Research on multi-camera information fusion method for intelligent perception

Multimedia Tools and Applications - In this paper, the Gaussian Mixture Model and Mean Shift algorithm are used to detect and track moving objects in the visual perception network composed of...     

An effective method for grasp planning on objects with complex geometry combining human experience and analytical approach

Science China Information Sciences - In this paper, an effective method for identifying the graspable components of objects with complex geometry is proposed for grasp planning based on human...