Learning Dextrous Manipulation Skills for Multifingered Robot Hands Using the Evolution Strategy

We present a method for autonomous learning of dextrous manipulation skills with multifingered robot hands. We use heuristics derived from observations made on human hands to reduce the degrees of freedom of the task and make learning tractable. Our approach consists of learning and storing a few basic manipulation primitives for a few prototypical objects and then using an associative memory to obtain the required parameters for new objects and/or manipulations. The parameter space of the robot is searched using a modified version of the evolution strategy, which is robust to the noise normally present in real-world complex robotic tasks. Given the difficulty of modeling and simulating accurately the interactions of multiple fingers and an object, and to ensure that the learned skills are applicable in the real world, our system does not rely on simulation; all the experimentation is performed by a physical robot, in this case the 16-degree-of-freedom Utah/MIT hand. Experimental results show that accurate dextrous manipulation skills can be learned by the robot in a short period of time. We also show the application of the learned primitives to perform an assembly task and how the primitives generalize to objects that are different from those used during the learning phase.     

Learning Dextrous Manipulation Skills for Multifingered Robot Hands Using the Evolution Strategy

We present a method for autonomous learning of dextrous manipulation skills with multifingered robot hands. We use heuristics derived from observations made on human hands to reduce the degrees of freedom of the task and make learning tractable. Our approach consists of learning and storing a few basic manipulation primitives for a few prototypical objects and then using an associative memory to obtain the required parameters for new objects and/or manipulations. The parameter space of the robot is searched using a modified version of the evolution strategy, which is robust to the noise normally present in real-world complex robotic tasks. Given the difficulty of modeling and simulating accurately the interactions of multiple fingers and an object, and to ensure that the learned skills are applicable in the real world, our system does not rely on simulation; all the experimentation is performed by a physical robot, in this case the 16-degree-of-freedom Utah/MIT hand. E xperimental results show that accurate dextrous manipulation skills can be learned by the robot in a short period of time. We also show the application of the learned primitives to perform an assembly task and how the primitives generalize to objects that are different from those used during the learning phase.     

Efficient vision-based navigation

In this article, we present a novel approach to learning efficient navigation policies for mobile robots that use visual features for localization. As fast movements of a mobile robot typically introduce inherent motion blur in the acquired images, the uncertainty of the robot about its pose increases in such situations. As a result, it cannot be ensured anymore that a navigation task can be executed efficiently since the robot’s pose estimate might not correspond to its true location. We present a reinforcement learning approach to determine a navigation policy to reach the destination reliably and, at the same time, as fast as possible. Using our technique, the robot learns to trade off velocity against localization accuracy and implicitly takes the impact of motion blur on observations into account. We furthermore developed a method to compress the learned policy via a clustering approach. In this way, the size of the policy representation is significantly reduced, which is especially desirable in the context of memory-constrained systems. Extensive simulated and real-world experiments carried out with two different robots demonstrate that our learned policy significantly outperforms policies using a constant velocity and more advanced heuristics. We furthermore show that the policy is generally applicable to different indoor and outdoor scenarios with varying landmark densities as well as to navigation tasks of different complexity.     

Using dialog and human observations to dictate tasks to a learning robot assistant

Robot assistants need to interact with people in a natural way in order to be accepted into people’s day-to-day lives. We have been researching robot assistants with capabilities that include visually tracking humans in the environment, identifying the context in which humans carry out their activities, understanding spoken language (with a fixed vocabulary), participating in spoken dialogs to resolve ambiguities, and learning task procedures. In this paper, we describe a robot task learning algorithm in which the human explicitly and interactively instructs a series of steps to the robot through spoken language. The training algorithm fuses the robot’s perception of the human with the understood speech data, maps the spoken language to robotic actions, and follows the human to gather the action applicability state information. The robot represents the acquired task as a conditional procedure and engages the human in a spoken-language dialog to fill in information that the human may have omitted.     

Adaptation of robot’s perception of fuzzy linguistic information by evaluating vocal cues for controlling a robot manipulator

This article proposes a method for adapting a robot’s perception of fuzzy linguistic information by evaluating vocal cues. The robot’s perception of fuzzy linguistic information such as “very little” depends on the environmental arrangements and the user’s expectations. Therefore, the robot’s perception of the corresponding environment is modified by acquiring the user’s perception through vocal cues. Fuzzy linguistic information related to primitive movements is evaluated by a behavior evaluation network (BEN). A vocal cue evaluation system (VCES) is used to evaluate the vocal cues for modifying the BEN. The user’s satisfactory level for the robot’s movements and the user’s willingness to change the robot’s perception are identified based on a series of vocal cues to improve the adaptation process. A situation of cooperative rearrangement of the user’s working space is used to illustrate the proposed system by a PA-10 robot manipulator.     

Localization of a mobile robot based on an ultrasonic sensor using dynamic obstacles

In this article, we propose a localization scheme for a mobile robot based on the distance between the robot and moving objects. This method combines the distance data obtained from ultrasonic sensors in a mobile robot, and estimates the location of the mobile robot and the moving object. The movement of the object is detected by a combination of data and the object’s estimated position. Then, the mobile robot’s location is derived from the a priori known initial state. We use kinematic modeling that represents the movement of a robot and an object. A Kalman-filtering algorithm is used for addressing estimation error and measurement noise. Throughout the computer simulation experiments, the performance is verified. Finally, the results of experiments are presented and discussed. The proposed approach allows a mobile robot to seek its own position in a weakly structured environment. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

Bipedal walking gait generation based on the Sequential Method of Analytical Potential (SMAP)

This paper presents a method we have called the Sequential Method of Analytical Potential (SMAP). By taking account of a system’s actual capabilities via Dynamic Propulsion Potentials (DPP), this method aims to generate dynamic walking gaits for bipedal robots. The objective is to move the robot by acting directly on the actuator forces. The various accelerations governing the movements of the robot are controlled by direct, precise modification of its own dynamic effects, taking account of the robot’s intrinsic dynamics as well as the capabilities of the actuators moving the joints.     

Dynamic motion learning for multi-DOF flexible-joint robots using active–passive motor babbling through deep learning

This paper proposes a learning strategy for robots with flexible joints having multi-degrees of freedom in order to achieve dynamic motion tasks. In spite of there being several potential benefits of flexible-joint robots such as exploitation of intrinsic dynamics and passive adaptation to environmental changes with mechanical compliance, controlling such robots is challenging because of increased complexity of their dynamics. To achieve dynamic movements, we introduce a two-phase learning framework of the body dynamics of the robot using a recurrent neural network motivated by a deep learning strategy. The proposed methodology comprises a pre-training phase with motor babbling and a fine-tuning phase with additional learning of the target tasks. In the pre-training phase, we consider active and passive exploratory motions for efficient acquisition of body dynamics. In the fine-tuning phase, the learned body dynamics are adjusted for specific tasks. We demonstrate the effectiveness of the proposed methodology in achieving dynamic tasks involving constrained movement requiring interactions with the environment on a simulated robot model and an actual PR2 robot both of which have a compliantly actuated seven degree-of-freedom arm. The results illustrate a reduction in the required number of training iterations for task learning and generalization capabilities for untrained situations.     

Codevelopmental learning between human and humanoid robot using a dynamic neural-network model.

This paper examines characteristics of interactive learning between human tutors and a robot having a dynamic neural-network model, which is inspired by human parietal cortex functions. A humanoid robot, with a recurrent neural network that has a hierarchical structure, learns to manipulate objects. Robots learn tasks in repeated self-trials with the assistance of human interaction, which provides physical guidance until the tasks are mastered and learning is consolidated within the neural networks. Experimental results and the analyses showed the following: 1) codevelopmental shaping of task behaviors stems from interactions between the robot and a tutor; 2) dynamic structures for articulating and sequencing of behavior primitives are self-organized in the hierarchically organized network; and 3) such structures can afford both generalization and context dependency in generating skilled behaviors.     

Artificial potential functions based camera movements and visual behaviors in attentive robots

An attentive robot needs to exhibit a plethora of different visual behaviors including free viewing, detecting visual onsets, search, remaining fixated and tracking depending on the vision task at hand. The robot’s associated camera movements—ranging from saccades to smooth pursuit—direct its optical axis in a manner that is dependent on the current visual behavior. This paper proposes a closed-loop dynamical systems approach to the generation of camera movements based on a family of artificial potential functions. Each movement from the current fixation point to the next is associated with an artificial potential function that encodes saliency and possibly inhibition depending on the visual behavior that the robot is engaged in. The novelty of this approach is that since the nature of resulting motion can vary from being saccadic to smooth pursuit, the full repertoire of visual behaviors all become possible within the same framework. The robot can switch its visual behavior simply by changing the parameters of the constructed artificial potential functions appropriately. Furthermore, automated reflexive changes among the different visual behaviors can be achieved via a simple switching automaton. Experimental results with APES robot serve to show the performance properties of a robot engaged in each different visual behavior.     

A Real-Time Hybrid Architecture for Biped Humanoids with Active Vision Mechanisms

A real-time hybrid control architecture for biped humanoid robots is proposed. The architecture is modular and hierarchical. The main robot’s functionalities are organized in four parallel modules: perception, actuation, world-modeling, and hybrid control. Hybrid control is divided in three behavior-based hierarchical layers: the planning layer, the deliberative layer, and the reactive layer, which work in parallel and have very different response speeds and planning capabilities. The architecture allows: (1) the coordination of multiple robots and the execution of group behaviors without disturbing the robot’s reactivity and responsivity, which is very relevant for biped humanoid robots whose gait control requires real-time processing. (2) The straightforward management of the robot’s resources using resource multiplexers. (3) The integration of active vision mechanisms in the reactive layer under control of behavior-dependant value functions from the deliberative layer. This adds flexibility in the implementation of complex functionalities, such as the ones required for playing soccer in robot teams. The architecture is validated using simulated and real Nao humanoid robots. Passive and active behaviors are tested in simulated and real robot soccer setups. In addition, the ability to execute group behaviors in real- time is tested in international robot soccer competitions.     

Reinforcement learning to adjust parametrized motor primitives to new situations

Humans manage to adapt learned movements very quickly to new situations by generalizing learned behaviors from similar situations. In contrast, robots currently often need to re-learn the complete movement. In this paper, we propose a method that learns to generalize parametrized motor plans by adapting a small set of global parameters, called meta-parameters. We employ reinforcement learning to learn the required meta-parameters to deal with the current situation, described by states. We introduce an appropriate reinforcement learning algorithm based on a kernelized version of the reward-weighted regression. To show its feasibility, we evaluate this algorithm on a toy example and compare it to several previous approaches. Subsequently, we apply the approach to three robot tasks, i.e., the generalization of throwing movements in darts, of hitting movements in table tennis, and of throwing balls where the tasks are learned on several different real physical robots, i.e., a Barrett WAM, a BioRob, the JST-ICORP/SARCOS CBi and a Kuka KR?6.     

Task Transfer by a Developmental Robot

Scaffolding is a process of transferring learned skills to new and more complex tasks through arranged experience in open-ended development. In this paper, we propose a developmental learning architecture that enables a robot to transfer skills acquired in early learning settings to later more complex task settings. We show that a basic mechanism that enables this transfer is sequential priming combined with attention, which is also the driving mechanism for classical conditioning, secondary conditioning, and instrumental conditioning in animal learning. A major challenge of this work is that training and testing must be conducted in the same program operational mode through online, real-time interactions between the agent and the trainers. In contrast with former modeling studies, the proposed architecture does not require the programmer to know the tasks to be learned and the environment is uncontrolled. All possible perceptions and actions, including the actual number of classes, are not available until the programming is finished and the robot starts to learn in the real world. Thus, a predesigned task-specific symbolic representation is not suited for such an open-ended developmental process. Experimental results on a robot are reported in which the trainer shaped the behaviors of the agent interactively, continuously, and incrementally through verbal commands and other sensory signals so that the robot learns new and more complex sensorimotor tasks by transferring sensorimotor skills learned in earlier periods of open-ended development     

Human robot cooperation with compliance adaptation along the motion trajectory

In this paper we propose a novel approach for intuitive and natural physical human–robot interaction in cooperative tasks. Through initial learning by demonstration, robot behavior naturally evolves into a cooperative task, where the human co-worker is allowed to modify both the spatial course of motion as well as the speed of execution at any stage. The main feature of the proposed adaptation scheme is that the robot adjusts its stiffness in path operational space, defined with a Frenet–Serret frame. Furthermore, the required dynamic capabilities of the robot are obtained by decoupling the robot dynamics in operational space, which is attached to the desired trajectory. Speed-scaled dynamic motion primitives are applied for the underlying task representation. The combination allows a human co-worker in a cooperative task to be less precise in parts of the task that require high precision, as the precision aspect is learned and provided by the robot. The user can also freely change the speed and/or the trajectory by simply applying force to the robot. The proposed scheme was experimentally validated on three illustrative tasks. The first task demonstrates novel two-stage learning by demonstration, where the spatial part of the trajectory is demonstrated independently from the velocity part. The second task shows how parts of the trajectory can be rapidly and significantly changed in one execution. The final experiment shows two Kuka LWR-4 robots in a bi-manual setting cooperating with a human while carrying an object.     

Improving biped walk stability with complementary corrective demonstration

We contribute a method for improving the skill execution performance of a robot by complementing an existing algorithmic solution with corrective human demonstration. We apply the proposed method to the biped walking problem, which is a good example of a complex low level skill due to the complicated dynamics of the walk process in a high dimensional state and action space. We introduce an incremental learning approach to improve the Nao humanoid robot’s stability during walking. First, we identify, extract, and record a complete walk cycle from the motion of the robot as it executes a given walk algorithm as a black box. Second, we apply offline advice operators for improving the stability of the learned open-loop walk cycle. Finally, we present an algorithm to directly modify the recorded walk cycle using real time corrective human demonstration. The demonstrator delivers the corrective feedback using a commercially available wireless game controller without touching the robot. Through the proposed algorithm, the robot learns a closed-loop correction policy for the open-loop walk by mapping the corrective demonstrations to the sensory readings received while walking. Experiment results demonstrate a significant improvement in the walk stability.     

Active learning of inverse models with intrinsically motivated goal exploration in robots

We introduce the Self-Adaptive Goal Generation Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsically motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills/policies that solve a corresponding distribution of parameterized tasks/goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy parameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters.We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: (1) learning the inverse kinematics in a highly-redundant robotic arm, (2) learning omnidirectional locomotion with motor primitives in a quadruped robot, and (3) an arm learning to control a fishing rod with a flexible wire. We show that (1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; (2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; (3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot.     

User—robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy

This paper describes a hands-off socially assistive therapist robot designed to monitor, assist, encourage, and socially interact with post-stroke users engaged in rehabilitation exercises. We investigate the role of the robot’s personality in the hands-off therapy process, focusing on the relationship between the level of extroversion–introversion of the robot and the user. We also demonstrate a behavior adaptation system capable of adjusting its social interaction parameters (e.g., interaction distances/proxemics, speed, and vocal content) toward customized post-stroke rehabilitation therapy based on the user’s personality traits and task performance. Three validation experiment sets are described. The first maps the user’s extroversion–introversion personality dimension to a spectrum of robot therapy styles that range from challenging to nurturing. The second and the third experiments adjust the personality matching dynamically to adapt the robot’s therapy styles based on user personality and performance. The reported results provide first evidence for user preference for personality matching in the assistive domain and demonstrate how the socially assistive robot’s autonomous behavior adaptation to the user’s personality can result in improved human task performance. This work was supported by USC Women in Science and Engineering (WiSE) Program and the Okawa Foundation.     

From dynamic movement primitives to associative skill memories

In recent years, research on movement primitives has gained increasing popularity. The original goals of movement primitives are based on the desire to have a sufficiently rich and abstract representation for movement generation, which allows for efficient teaching, trial-and-error learning, and generalization of motor skills (Schaal 1999). Thus, motor skills in robots should be acquired in a natural dialog with humans, e.g., by imitation learning and shaping, while skill refinement and generalization should be accomplished autonomously by the robot. Such a scenario resembles the way we teach children and connects to the bigger question of how the human brain accomplishes skill learning. In this paper, we review how a particular computational approach to movement primitives, called dynamic movement primitives, can contribute to learning motor skills. We will address imitation learning, generalization, trial-and-error learning by reinforcement learning, movement recognition, and control based on movement primitives. But we also want to go beyond the standard goals of movement primitives. The stereotypical movement generation with movement primitives entails predicting of sensory events in the environment. Indeed, all the sensory events associated with a movement primitive form an associative skill memory that has the potential of forming a most powerful representation of a complete motor skill.