Learning Object Affordances: From Sensory--Motor Coordination to Imitation

Affordances encode relationships between actions, objects, and effects. They play an important role on basic cognitive capabilities such as prediction and planning. We address the problem of learning affordances through the interaction of a robot with the environment, a key step to understand the world properties and develop social skills. We present a general model for learning object affordances using Bayesian networks integrated within a general developmental architecture for social robots. Since learning is based on a probabilistic model, the approach is able to deal with uncertainty, redundancy, and irrelevant information. We demonstrate successful learning in the real world by having an humanoid robot interacting with objects. We illustrate the benefits of the acquired knowledge in imitation games.     

Towards cognitive grasping: modeling of unknown objects and its corresponding grasp types

This paper describes an intuitive approach for a cognitive grasp of a robot. The cognitive grasp means the chain of processes that make a robot to learn and execute a grasping method for unknown objects like a human. In the learning step, a robot looks around a target object to estimate the 3D shape and understands the grasp type for the object through a human demonstration. In the execution step, the robot correlates an unknown object to one of known grasp types by comparing the shape similarity of the target object based on previously learned models. For this cognitive grasp, we mainly deal with two functionalities such as reconstructing an unknown 3D object and classifying the object by grasp types. In the experiment, we evaluate the performance of object classification according to the grasp types for 20 objects via human demonstration.     

Robotic grasping and manipulation through human visuomotor learning

A major goal of robotics research is to develop techniques that allow non-experts to teach robots dexterous skills. In this paper, we report our progress on the development of a framework which exploits human sensorimotor learning capability to address this aim. The idea is to place the human operator in the robot control loop where he/she can intuitively control the robot, and by practice, learn to perform the target task with the robot. Subsequently, by analyzing the robot control obtained by the human, it is possible to design a controller that allows the robot to autonomously perform the task. First, we introduce this framework with the ball-swapping task where a robot hand has to swap the position of the balls without dropping them, and present new analyses investigating the intrinsic dimension of the ball-swapping skill obtained through this framework. Then, we present new experiments toward obtaining an autonomous grasp controller on an anthropomorphic robot. In the experiments, the operator directly controls the (simulated) robot using visual feedback to achieve robust grasping with the robot. The data collected is then analyzed for inferring the grasping strategy discovered by the human operator. Finally, a method to generalize grasping actions using the collected data is presented, which allows the robot to autonomously generate grasping actions for different orientations of the target object.     

Adaptation technique for integrating genetic programming and reinforcement learning for real robots

We propose an integrated technique of genetic programming (GP) and reinforcement learning (RL) to enable a real robot to adapt its actions to a real environment. Our technique does not require a precise simulator because learning is achieved through the real robot. In addition, our technique makes it possible for real robots to learn effective actions. Based on this proposed technique, we acquire common programs, using GP, which are applicable to various types of robots. Through this acquired program, we execute RL in a real robot. With our method, the robot can adapt to its own operational characteristics and learn effective actions. In this paper, we show experimental results from two different robots: a four-legged robot "AIBO" and a humanoid robot "HOAP-1." We present results showing that both effectively solved the box-moving task; the end result demonstrates that our proposed technique performs better than the traditional Q-learning method.     

Interleaving Planning and Robot Execution for Asynchronous User Requests

ROGUE is an architecture built on a real robot which provides algorithms for the integration of high-level planning, low-level robotic execution, and learning. ROGUE addresses successfully several of the challenges of a dynamic office gopher environment. This article presents the techniques for the integration of planning and execution.ROGUE uses and extends a classical planning algorithm to create plans for multiple interacting goals introduced by asynchronous user requests. ROGUE translates the planner';s actions to robot execution actions and monitors real world execution. ROGUE is currently implemented using the PRODIGY4.0 planner and the Xavier robot. This article describes how plans are created for multiple asynchronous goals, and how task priority and compatibility information are used to achieve appropriate efficient execution. We describe how ROGUE communicates with the planner and the robot to interleave planning with execution so that the planner can replan for failed actions, identify the actual outcome of an action with multiple possible outcomes, and take opportunities from changes in the environment.ROGUE represents a successful integration of a classical artificial intelligence planner with a real mobile robot.     

Dyna-Q-based vector direction for path planning problem of autonomous mobile robots in unknown environments

Reinforcement learning (RL) is a popular method for solving the path planning problem of autonomous mobile robots in unknown environments. However, the primary difficulty faced by learning robots using the RL method is that they learn too slowly in obstacle-dense environments. To more efficiently solve the path planning problem of autonomous mobile robots in such environments, this paper presents a novel approach in which the robot’s learning process is divided into two phases. The first one is to accelerate the learning process for obtaining an optimal policy by developing the well-known Dyna-Q algorithm that trains the robot in learning actions for avoiding obstacles when following the vector direction. In this phase, the robot’s position is represented as a uniform grid. At each time step, the robot performs an action to move to one of its eight adjacent cells, so the path obtained from the optimal policy may be longer than the true shortest path. The second one is to train the robot in learning a collision-free smooth path for decreasing the number of the heading changes of the robot. The simulation results show that the proposed approach is efficient for the path planning problem of autonomous mobile robots in unknown environments with dense obstacles.     

Bayesian learning of tool affordances based on generalization of functional feature to estimate effects of unseen tools

To address the problem of estimating the effects of unknown tools, we propose a novel concept of tool representation based on the functional features of the tool. We argue that functional features remain distinctive and invariant across different tools used for performing similar tasks. Such a representation can be used to estimate the effects of unknown tools that share similar functional features. To learn the usages of tools to physically alter the environment, a robot should be able to reason about its capability to act, the representation of available tools, and effect of manipulating tools. To enable a robot to perform such reasoning, we present a novel approach, called Tool Affordances, to learn bi-directional causal relationships between actions, functional features and the effects of tools. A Bayesian network is used to model tool affordances because of its capability to model probabilistic dependencies between data. To evaluate the learnt tool affordances, we conducted an inference test in which a robot inferred suitable functional features to realize certain effects (including novel effects) from the given action. The results show that the generalization of functional features enables the robot to estimate the effects of unknown tools that have similar functional features. We validate the accuracy of estimation by error analysis.     

Learning action models for the improved execution of navigation plans

Most state-of-the-art navigation systems for autonomous service robots decompose navigation into global navigation planning and local reactive navigation. While the methods for navigation planning and local navigation themselves are well understood, the plan execution problem, the problem of how to generate and parameterize local navigation tasks from a given navigation plan is largely unsolved.

This paper describes how a robot can autonomously learn to execute navigation plans. We formalize the problem as a Markov Decision Process (MDP) and derive a decision theoretic action selection function from it. The action selection function employs models of the robot’s navigation actions, which are autonomously acquired from experience using neural networks or regression tree learning algorithms. We show, both in simulation and on an RWI B21 mobile robot, that the learned models together with the derived action selection function achieve competent navigation behavior.     

Planning-Space Shift Motion Generation: Variable-space Motion Planning Toward Flexible Extension of Body Schema

To improve the flexibility of robotic learning, it is important to realize an ability to generate a hierarchical structure. This paper proposes a learning framework which can dynamically change the planning space depending on the structure of tasks. Synchronous motion information is utilized to generate ??modes?? and hierarchical structure of the controller is constructed based on the modes. This enables efficient planning and control in low-dimensional planning space, though the dimension of the total state space is in general very high. Three types of object manipulation tasks are tested as applications, where an object is found and used as a tool (or as a part of the body) to extend the ability of the robot. The proposed framework is expected to be a basic learning model to account for body schema acquisition including tool affordances.     

Learning how to grasp based on neural network retraining

Humans have an incredible capacity to manipulate objects using dextrous hands. A large number of studies indicate that robot learning by demonstration is a promising strategy to improve robotic manipulation and grasping performance. Concerning this subject we can ask: How does a robot learn how to grasp? This work presents a method that allows a robot to learn new grasps. The method is based on neural network retraining. With this approach we aim to enable a robot to learn new grasps through a supervisor. The proposed method can be applied for 2D and 3D cases. Extensive object databases were generated to evaluate the method performance in both 2D and 3D cases. A total of 8100 abstract shapes were generated for 2D cases and 11700 abstract shapes for 3D cases. Simulation results with a computational supervisor show that a robotic system can learn new grasps and improve its performance through the proposed HRH (Hopfield-RBF-Hopfield) grasp learning approach.     

Object–object interaction affordance learning

This paper presents a novel object–object affordance learning approach that enables intelligent robots to learn the interactive functionalities of objects from human demonstrations in everyday environments. Instead of considering a single object, we model the interactive motions between paired objects in a human–object–object way. The innate interaction-affordance knowledge of the paired objects are learned from a labeled training dataset that contains a set of relative motions of the paired objects, human actions, and object labels. The learned knowledge is represented with a Bayesian Network, and the network can be used to improve the recognition reliability of both objects and human actions and to generate proper manipulation motion for a robot if a pair of objects is recognized. This paper also presents an image-based visual servoing approach that uses the learned motion features of the affordance in interaction as the control goals to control a robot to perform manipulation tasks.     

Anatomy of a mobilized lesson: Learning my way

With the mass adoption of mobile computing devices by the current school generation, significant opportunities have emerged for genuinely supporting differentiated and personalized learning experiences through mobile devices. In our school-based research work in introducing mobilized curricula to a class, we observe one compelling mobilized lesson that exploits the affordances of mobile learning to provide multiple learning pathways for elementary grade (primary) 2 students. Through the lesson, students move beyond classroom activities that merely mimic what the teacher says and does in the classroom, and yet they still learn in personally meaningful ways. In deconstructing the lesson, we provide an in-depth analysis of how the affordances of mobile computing enable personalized learning from four facets: (a) allowing multiple entry points and learning pathways, (b) supporting multi-modality, (c) enabling student improvisation in situ, and (d) supporting the sharing and creation of student artifacts on the move. A key property of mobile technology that enables these affordances lies with the small form factor and the lightweightness of these devices which make them non-obtrusive in the learning spaces of the student. This article makes a contribution on the design aspects of mobilized lessons, namely, what the affordances of mobile technologies can enable.     

Anatomy of a mobilized lesson: Learning my way

With the mass adoption of mobile computing devices by the current school generation, significant opportunities have emerged for genuinely supporting differentiated and personalized learning experiences through mobile devices. In our school-based research work in introducing mobilized curricula to a class, we observe one compelling mobilized lesson that exploits the affordances of mobile learning to provide multiple learning pathways for elementary grade (primary) 2 students. Through the lesson, students move beyond classroom activities that merely mimic what the teacher says and does in the classroom, and yet they still learn in personally meaningful ways. In deconstructing the lesson, we provide an in-depth analysis of how the affordances of mobile computing enable personalized learning from four facets: (a) allowing multiple entry points and learning pathways, (b) supporting multi-modality, (c) enabling student improvisation in situ, and (d) supporting the sharing and creation of student artifacts on the move. A key property of mobile technology that enables these affordances lies with the small form factor and the lightweightness of these devices which make them non-obtrusive in the learning spaces of the student. This article makes a contribution on the design aspects of mobilized lessons, namely, what the affordances of mobile technologies can enable.     

Learning cooperative grasping with the graph representation of a state-action space

In this paper we present a method for two robot manipulators to learn cooperative tasks. If a single robot is unable to grasp an object in a certain orientation, it can only continue with the help of other robots. The grasping can be realized by a sequence of cooperative operations that re-orient the object. Several sequences are needed to handle the different situations in which an object is not graspable for the robot. It is shown that a distributed learning method based on a Markov decision process is able to learn the sequences for the involved robots, a master robot that needs to grasp and a helping robot that supports him with the re-orientation. A novel state-action graph is used to store the reinforcement values of the learning process. Further an example of aggregate assembly shows the generality of this approach.     

Robot docking based on omnidirectional vision and reinforcement learning

We present a system for visual robotic docking using an omnidirectional camera coupled with the actor critic reinforcement learning algorithm. The system enables a PeopleBot robot to locate and approach a table so that it can pick an object from it using the pan-tilt camera mounted on the robot. We use a staged approach to solve this problem as there are distinct subtasks and different sensors used. Starting with random wandering of the robot until the table is located via a landmark, then a network trained via reinforcement allows the robot to turn to and approach the table. Once at the table the robot is to pick the object from it. We argue that our approach has a lot of potential allowing the learning of robot control for navigation and remove the need for internal maps of the environment. This is achieved by allowing the robot to learn couplings between motor actions and the position of a landmark.     

INTEGRATING PLANNING,EXECUTION, AND LEARNING TO IMPROVE PLAN EXECUTION

Algorithms for planning under uncertainty require accurate action models that explicitly capture the uncertainty of the environment. Unfortunately, obtaining these models is usually complex. In environments with uncertainty, actions may produce countless outcomes and hence, specifying them and their probability is a hard task. As a consequence, when implementing agents with planning capabilities, practitioners frequently opt for architectures that interleave classical planning and execution monitoring following a replanning when failure paradigm. Though this approach is more practical, it may produce fragile plans that need continuous replanning episodes or even worse, that result in execution dead‐ends. In this paper, we propose a new architecture to relieve these shortcomings. The architecture is based on the integration of a relational learning component and the traditional planning and execution monitoring components. The new component allows the architecture to learn probabilistic rules of the success of actions from the execution of plans and to automatically upgrade the planning model with these rules. The upgraded models can be used by any classical planner that handles metric functions or, alternatively, by any probabilistic planner. This architecture proposal is designed to integrate off‐the‐shelf interchangeable planning and learning components so it can profit from the last advances in both fields without modifying the architecture.     

Grounding semantic categories in behavioral interactions: Experiments with 100 objects

From an early stage in their development, human infants show a profound drive to explore the objects around them. Research in psychology has shown that this exploration is fundamental for learning the names of objects and object categories. To address this problem in robotics, this paper presents a behavior-grounded approach that enables a robot to recognize the semantic labels of objects using its own behavioral interaction with them. To test this method, our robot interacted with 100 different objects grouped according to 20 different object categories. The robot performed 10 different behaviors on them, while using three sensory modalities (vision, proprioception and audio) to detect any perceptual changes. The results show that the robot was able to use multiple sensorimotor contexts in order to recognize a large number of object categories. Furthermore, the category recognition model presented in this paper was able to identify sensorimotor contexts that can be used to detect specific categories. Most importantly, the robot’s model was able to reduce exploration time by half by dynamically selecting which exploratory behavior should be applied next when classifying a novel object.     

Deep reinforcement learning-based safe interaction for industrial human-robot collaboration using intrinsic reward function

Aiming at human-robot collaboration in manufacturing, the operator's safety is the primary issue during the manufacturing operations. This paper presents a deep reinforcement learning approach to realize the real-time collision-free motion planning of an industrial robot for human-robot collaboration. Firstly, the safe human-robot collaboration manufacturing problem is formulated into a Markov decision process, and the mathematical expression of the reward function design problem is given. The goal is that the robot can autonomously learn a policy to reduce the accumulated risk and assure the task completion time during human-robot collaboration. To transform our optimization object into a reward function to guide the robot to learn the expected behaviour, a reward function optimizing approach based on the deterministic policy gradient is proposed to learn a parameterized intrinsic reward function. The reward function for the agent to learn the policy is the sum of the intrinsic reward function and the extrinsic reward function. Then, a deep reinforcement learning algorithm intrinsic reward-deep deterministic policy gradient (IRDDPG), which is the combination of the DDPG algorithm and the reward function optimizing approach, is proposed to learn the expected collision avoidance policy. Finally, the proposed algorithm is tested in a simulation environment, and the results show that the industrial robot can learn the expected policy to achieve the safety assurance for industrial human-robot collaboration without missing the original target. Moreover, the reward function optimizing approach can help make up for the designed reward function and improve policy performance.     

Q-学习及其在智能机器人局部路径规划中的应用研究

强化学习一词来自于行为心理学,这门学科把行为学习看成反复试验的过程,从而把环境状态映射成相应的动作．在设计智能机器人过程中,如何来实现行为主义的思想、在与环境的交互中学习行为动作？ 文中把机器人在未知环境中为躲避障碍所采取的动作看作一种行为,采用强化学习方法来实现智能机器人避碰行为学习．Ｑ－学习算法是类似于动态规划的一种强化学习方法,文中在介绍了Ｑ－学习的基本算法之后,提出了具有竞争思想和自组织机制的Ｑ－学习神经网络学习算法;然后研究了该算法在智能机器人局部路径规划中的应用,在文中的最后给出了详细的仿真结果