Learning object deformation models for robot motion planning

In this paper, we address the problem of robot navigation in environments with deformable objects. The aim is to include the costs of object deformations when planning the robot’s motions and trade them off against the travel costs. We present our recently developed robotic system that is able to acquire deformation models of real objects. The robot determines the elasticity parameters by physical interaction with the object and by establishing a relation between the applied forces and the resulting surface deformations. The learned deformation models can then be used to perform physically realistic finite element simulations. This allows the planner to evaluate robot trajectories and to predict the costs of object deformations. Since finite element simulations are time-consuming, we furthermore present an approach to approximate object-specific deformation cost functions by means of Gaussian process regression. We present two real-world applications of our motion planner for a wheeled robot and a manipulation robot. As we demonstrate in real-world experiments, our system is able to estimate appropriate deformation parameters of real objects that can be used to predict future deformations. We show that our deformation cost approximation improves the efficiency of the planner by several orders of magnitude.     

Markerless human–robot interface for dual robot manipulators using Kinect sensor

Remote teleoperation of robot manipulators is often necessary in unstructured, dynamic, and dangerous environments. However, the existing mechanical and other contacting interfaces require unnatural, or hinder natural, human motions. At present, the contacting interfaces used in teleoperation for multiple robot manipulators often require multiple operators. Previous vision-based approaches have only been used in the remote teleoperation for one robot manipulator as well as require the special quantity of illumination and visual angle that limit the field of application. This paper presents a noncontacting Kinect-based method that allows a human operator to communicate his motions to the dual robot manipulators by performing double hand–arm movements that would naturally carry out an object manipulation task. This paper also proposes an innovative algorithm of over damping to solve the problem of error extracting and dithering due to the noncontact measure. By making full use of the human hand–arm motion, the operator would feel immersive. This human–robot interface allows the flexible implementation of the object manipulation task done in collaboration by dual robots through the double hand–arm motion by one operator.     

Experience-based imitation using RNNPB

Robot imitation is a useful and promising alternative to robot programming. Robot imitation involves two crucial issues. The first is how a robot can imitate a human whose physical structure and properties differ greatly from its own. The second is how the robot can generate various motions from finite programmable patterns (generalization). This paper describes a novel approach to robot imitation based on its own physical experiences. We considered the target task of moving an object on a table. For imitation, we focused on an active sensing process in which the robot acquires the relation between the object's motion and its own arm motion. For generalization, we applied the RNNPB (recurrent neural network with parametric bias) model to enable recognition/generation of imitation motions. The robot associates the arm motion which reproduces the observed object's motion presented by a human operator. Experimental results proved the generalization capability of our method, which enables the robot to imitate not only motion it has experienced, but also unknown motion through nonlinear combination of the experienced motions.     

Object–object interaction affordance learning

This paper presents a novel object–object affordance learning approach that enables intelligent robots to learn the interactive functionalities of objects from human demonstrations in everyday environments. Instead of considering a single object, we model the interactive motions between paired objects in a human–object–object way. The innate interaction-affordance knowledge of the paired objects are learned from a labeled training dataset that contains a set of relative motions of the paired objects, human actions, and object labels. The learned knowledge is represented with a Bayesian Network, and the network can be used to improve the recognition reliability of both objects and human actions and to generate proper manipulation motion for a robot if a pair of objects is recognized. This paper also presents an image-based visual servoing approach that uses the learned motion features of the affordance in interaction as the control goals to control a robot to perform manipulation tasks.     

An obstacle avoidance algorithm for robot manipulators based on decision-making force

Obstacle avoidance is a significant skill not only for mobile robots but also for robot manipulators working in unstructured environments. Various algorithms have been proposed to solve off-line planning and on-line adaption problems. However, it is still not able to ensure safety and flexibility in complex scenarios. In this paper, a novel obstacle avoidance algorithm is proposed to improve the robustness and flexibility. The method contains three components: A closed-loop control system is used to filter the preplanned trajectory and ensure the smoothness and stability of the robot motion; the dynamic repulsion field is adopted to fulfill the robot with primitive obstacle avoidance capability; to mimic human’s complex obstacle avoidance behavior and instant decision-making mechanism, a parametrized decision-making force is introduced to optimize all the feasible motions. The algorithms were implemented in planar and spatial robot manipulators. The comparative results show the robot can not only track the task trajectory smoothly but also avoid obstacles in different configurations.     

Recognition, prediction, and planning for assisted teleoperation of freeform tasks

The approach of inferring user’s intended task and optimizing low-level robot motions has promise for making robot teleoperation interfaces more intuitive and responsive. But most existing methods assume a finite set of candidate tasks, which limits a robot’s functionality. This paper proposes the notion of freeform tasks that encode an infinite number of possible goals (e.g., desired target positions) within a finite set of types (e.g., reach, orient, pick up). It also presents two technical contributions to help make freeform UIs possible. First, an intent predictor estimates the user’s desired task, and accepts freeform tasks that include both discrete types and continuous parameters. Second, a cooperative motion planner continuously updates the robot’s trajectories to achieve the inferred tasks by repeatedly solving optimal control problems. The planner is designed to respond interactively to changes in the indicated task, avoid collisions in cluttered environments, handle time-varying objective functions, and achieve high-quality motions using a hybrid of numerical and sampling-based techniques. The system is applied to the problem of controlling a 6D robot manipulator using 2D mouse input in the context of two tasks: static target reaching and dynamic trajectory tracking. Simulations suggest that it enables the robot to reach intended targets faster and to track intended trajectories more closely than comparable techniques.     

A hierarchical long short term safety framework for efficient robot manipulation under uncertainty

Safe and efficient robot manipulation in uncertain clustered environments has been recognized to be a key element of future intelligent industrial robots. Unlike traditional robots that work in structured and deterministic environments, intelligent industrial robots need to operate in dynamically changing and stochastic environments with limited computation resources. This paper proposed a hierarchical long short term safety system (HLSTS), where the upper layer contains a long term planner for global reference trajectory generation and the lower layer contains a short term planner for real-time emergent safety maneuvers. Additionally, a hierarchical coordinator is proposed to enable smooth coordination of the two layers by compensating the communication delay through trajectory modification. The theoretical results verify that the long term planner can always find a feasible trajectory (feasibility guarantee); and the short term planner can guarantee safety in the probabilistic sense. The proposed architecture is validated in industrial settings in both simulations and real robot experiments, where the robot is interacting with randomly moving obstacles while performing a goal reaching task. Experimental results demonstrate that the proposed HLSTS framework not only guarantees safety but also improves task efficiency.     

Bilateral remote teaching and autonomous task execution with task progress feedback

This paper describes an approach to estimating the progress in a task executed by a humanoid robot and to synthesizing motion based on the current progress so that the robot can achieve the task. The robot observes a human performing whole body motion for a specific task, and encodes these motions into a hidden Markov model (HMM). The current observation is compared with the motion generated by the HMM, and the task progress can be estimated during the robot performing the motion. The robot subsequently uses the estimate of the task progress to generate a motion appropriate to the current situation with the feedback rule. We constructed a bilateral remote control system with humanoid robot HRP-4 and haptic device Novint Falcon, and we made the humanoid robot push a button. Ten trial motions of pushing a button were recorded for the training data. We tested our proposed approach on the autonomous execution of the pushing motion by the humanoid robot, and confirmed the effectiveness of our task progress feedback method.     

Robot introspection through learned hidden Markov models

In this paper we describe a machine learning approach for acquiring a model of a robot behaviour from raw sensor data. We are interested in automating the acquisition of behavioural models to provide a robot with an introspective capability. We assume that the behaviour of a robot in achieving a task can be modelled as a finite stochastic state transition system.Beginning with data recorded by a robot in the execution of a task, we use unsupervised learning techniques to estimate a hidden Markov model (HMM) that can be used both for predicting and explaining the behaviour of the robot in subsequent executions of the task. We demonstrate that it is feasible to automate the entire process of learning a high quality HMM from the data recorded by the robot during execution of its task.The learned HMM can be used both for monitoring and controlling the behaviour of the robot. The ultimate purpose of our work is to learn models for the full set of tasks associated with a given problem domain, and to integrate these models with a generative task planner. We want to show that these models can be used successfully in controlling the execution of a plan. However, this paper does not develop the planning and control aspects of our work, focussing instead on the learning methodology and the evaluation of a learned model. The essential property of the models we seek to construct is that the most probable trajectory through a model, given the observations made by the robot, accurately diagnoses, or explains, the behaviour that the robot actually performed when making these observations. In the work reported here we consider a navigation task. We explain the learning process, the experimental setup and the structure of the resulting learned behavioural models. We then evaluate the extent to which explanations proposed by the learned models accord with a human observer's interpretation of the behaviour exhibited by the robot in its execution of the task.     

机器人多指操作的递阶控制

为机器人多指协调操作建立一递阶控制系统.给定一操作任务,任务规划器首先生 成一系列物体的运动速度;然后,协调运动规划器根据期望的物体运动速度生成期望的手指 运动速度和期望的抓取姿态变化;同时,抓取力规划器为平衡作用在物体上的外力,根据当前 的抓取姿态,生成各手指所需的抓取力;最后,系统将手指的期望运动速度与为实现期望抓取 力而生成的顺应速度合并,并通过手指的逆雅可比转化为手指关节运动速度后,由手指的关 节级运动控制器实现手指的运动和抓取力的控制.该控制方法已成功应用于香港科技大学 (HKUST)灵巧手控制系统的开发.实验证明该方法不仅能完成物体轨迹的跟踪控制任务, 而且能完成物体对环境的力控制和力与速度的混合控制.     

Direct Virtual-hand Interface in Robot Assembly Programming

For a long time, robot assembly programming has been produced in two environments: on-line and off-line. On-line robot programming uses the actual robot for the experiments performing a given task; off-line robot programming develops a robot program in either an autonomous system with a high-level task planner and simulation or a 2D graphical user interface linked to other system components. This paper presents a whole hand interface for more easily performing robotic assembly tasks in the virtual tenvironment. The interface is composed of both static hand shapes (states) and continuous hand motions (modes). Hand shapes are recognized as discrete states that trigger the control signals and commands, and hand motions are mapped to the movements of a selected instance in real-time assembly. Hand postures are also used for specifying the alignment constraints and axis mapping of the hand-part coordinates. The basic virtual-hand functions are constructed through the states and modes developing the robotic assembly program. The assembling motion of the object is guided by the user immersed in the environment to a path such that no collisions will occur. The fine motion in controlling the contact and ending position/orientation is handled automatically by the system using prior knowledge of the parts and assembly reasoning. One assembly programming case using this interface is described in detail in the paper.     

A Human Aware Mobile Robot Motion Planner

Robot navigation in the presence of humans raises new issues for motion planning and control when the humans must be taken explicitly into account. We claim that a human aware motion planner (HAMP) must not only provide safe robot paths, but also synthesize good, socially acceptable and legible paths. This paper focuses on a motion planner that takes explicitly into account its human partners by reasoning about their accessibility, their vision field and their preferences in terms of relative human-robot placement and motions in realistic environments. This planner is part of a human-aware motion and manipulation planning and control system that we aim to develop in order to achieve motion and manipulation tasks in the presence or in synergy with humans.     

Physical simulation of land vehicles with obstacle avoidance and various terrain interactions

Programming the motions of an autonomous planetary robot moving in an hostile and hazardous environment is a complex task which requires both the construction of nominal motion plans and the anticipation as far as possible of the effects of the interactions existing between the vehicle and the terrain. In this paper we show how physical models and dynamic simulation tools can be used for amending and completing a nominal motion plan provided by a classical geometrical path planner. The purpose of our physical modeller-simulator is to anticipate the dynamic behaviour of the vehicle while executing the nominal motion plan. Then the obtained simulation results can be used to assess and optimize the nominal motion plan. In the first part, we outline the physical models that have been used for modelling the different types of vehicle, of terrain and of vehicle-surface interactions. Then we formulate the motion planning problem through the definition of two basic abstract constructions derived from physical model: the concept of generalized obstacle and the concept of physical target. We show with various examples how it is possible, when using this method, to solve the locomotion problem and the obstacle avoidance problem simultaneously and, furthermore, to provide the human operator with a true force feedback gestural control over the simulated robot.     

A three-dimensional machine-vision approach for automatic robot programming

This research investigates a novel robot-programming approach that applies machine-vision techniques to generate a robot program automatically. The hand motions of a demonstrator are initially recorded as a long sequence of images using two CCD cameras. Machine-vision techniques are then used to recognize the hand motions in three-dimensional space including open, closed, grasp, release and move. The individual hand feature and its corresponding hand position in each sample image is translated to robot's manipulator-level instructions. Finally a robot plays back the task using the automatically generated program.A robot can imitate the hand motions demonstrated by a human master using the proposed machine-vision approach. Compared with the traditional leadthrough and structural programming-language methods, the robot's user will not have to physically move the robot arm through the desired motion sequence and learn complicated robot-programming languages. The approach is currently focused on the classification of hand features and motions of a human arm and, therefore, is restricted to simple pick-and-place applications. Only one arm of the human master can be presented in the image scene, and the master must not wear long-sleeved clothes during demonstration to prevent false identification. Analysis and classification of hand motions in a long sequence of images are time-consuming. The automatic robot programming currently developed is performed off-line.     

An articulated limb motion planner for optimized movement

Task level animation of articulated figures, such as the human body, requires the ability to generate collision-free goal-directed motion of individual limbs in the presence of obstacles. This paper describes a new articulated limb motion planner for goal-directed point-to-point reaching motions. The produced motion avoids obstacles while optimizing an objective function. This two-phase algorithm uses heuristic guided Monte Carlo techniques to create a consistent underlying paradigm. The first phase consists of an existing potential field based random path planner which generates a population of candidate paths. This initial population is fed into the second phase, a genetic algorithm, which iteratively refines the population as it optimizes with respect to the objective function. The refinement process works on the principle of path coherency, the idea that a family of closely related collision-free paths lies in the vicinity of a given collision-free path. This paper focuses on seven different optimization functions. Optimized trajectories produced by the new motion planner are compared to those generated solely by the random path planner. The presented algorithm is flexible in that a wide range of objective functions can be optimized. Applications of the algorithm include task level animation, ergonomics and robotics.     

Multi-Agent System-Based Fuzzy Controller Design with Genetic Tuning for a Mobile Manipulator Robot in the Hand Over Task

This paper presents an application of the multi-agent system approach to a service mobile manipulator robot that interacts with a human during an object delivery and hand-over task in two dimensions. The base, elbow and shoulder of the robot are identified as three different agents, and are controlled using fuzzy control. The control variables of the controllers are linear velocity of the base, angular velocity of the elbow, and angular velocity of the shoulder. Main inputs to the system are the horizontal and vertical distances between the human and robot hands. These are input to all three agents. In developing the fuzzy control rules, effective delivery and avoidance of contact with humans, not to cause physical damage, are considered. The membership functions of the fuzzy controllers are tuned by using genetic algorithms. In tuning, the performance is calculated considering the distance deviation from the direct path, time spent to reach the human hand and energy consumed by the actuators. The proposed multi-agent system structure based on fuzzy control for the object delivery task succeeded in both effective and safe delivery.     

Time-space Viewpoint Planning for Guard Robot with Chance Constraint

This paper presents a novel movement planning algorithm for a guard robot in an indoor environment, imitating the job of human security. A movement planner is employed by the guard robot to continuously observe a certain person. This problem can be distinguished from the person following problem which continuously follows the object. Instead, the movement planner aims to reduce the movement and the energy while keeping the target person under its visibility. The proposed algorithm exploits the topological features of the environment to obtain a set of viewpoint candidates, and it is then optimized by a cost-based set covering problem. Both the robot and the target person are modeled using geodesic motion model which considers the environment shape. Subsequently, a particle model-based planner is employed, considering the chance constraints over the robot visibility, to choose an optimal action for the robot. Simulation results using 3D simulator and experiments on a real environment are provided to show the feasibility and effectiveness of our algorithm.

     

Asymmetric Velocity Moderation for human-safe robot control

With the increasing physical proximity of human–robot interaction, ensuring that robots do not harm surrounding humans has become crucial. Therefore, we propose asymmetric velocity moderation as a low-level controller for robotic systems to enforce human-safe motions. While our method prioritizes human safety, it also maintains the robot’s efficiency. Our proposed method restricts the robot’s speed according to (1) the displacement vector between human and robot, and (2) the robot’s velocity vector. That is to say, both the distance and the relative direction of movement are taken into account to restrict the robot’s motion. Through real-robot and simulation experiments using simplified HRI scenarios and dangerous situations, we demonstrate that our method is able to maintain the robot’s efficiency without undermining human safety.     

Few-experiential learning system of robotic picking task with selective dual-arm grasping

Recently, robots are introduced to warehouses and factories for automation and are expected to execute dual-arm manipulation as human does and to manipulate large, heavy and unbalanced objects. We focus on target picking task in the cluttered environment and aim to realize a robot picking system which the robot selects and executes proper grasping motion from single-arm and dual-arm motion. In this paper, we propose a few-experiential learning-based target picking system with selective dual-arm grasping. In our system, a robot first learns grasping points and object semantic and instance label with automatically synthesized dataset. The robot then executes and collects grasp trial experiences in the real world and retrains the grasping point prediction model with the collected trial experiences. Finally, the robot evaluates candidate pairs of grasping object instance, strategy and points and selects to execute the optimal grasping motion. In the experiments, we evaluated our system by conducting target picking task experiments with a dual-arm humanoid robot Baxter in the cluttered environment as warehouse.