A reward allocation method for reinforcement learning in stabilizing control tasks

Reinforcement learning is an area of machine learning that does not require detailed teaching signals by a human, which is expected to be applied to real robots. In its application to real robots, the learning processes are required to be finished in a short learning period of time. A reinforcement learning method of model-free type has fast convergence speeds in the tasks such as Sutton’s maze problem that aims to reach the target state in a minimum time. However, these methods are difficult to learn task to keep a stable state as long as possible. In this study, we improve the reward allocation method for the stabilizing control tasks. In stabilizing control tasks, we use the Semi-Markov decision process as an environment model. The validity of our method is demonstrated through simulation for stabilizing control of an inverted pendulum.     

Skillful control under uncertainty via direct reinforcement learning

Complexity and uncertainty in modern robots and other autonomous systems make it difficult to design controllers for such systems that can achieve desired levels of precision and robustness. Therefore learning methods are being incorporated into controllers for such systems, thereby providing the adaptibility necessary to meet the performance demands of the task. We argue that for learning tasks arising frequently in control applications, the most useful methods in practice probably are those we call direct associative reinforcement learning methods. We describe direct reinforcement learning methods and also illustrate with an example the utility of these methods for learning skilled robot control under uncertainty.     

Adaptation technique for integrating genetic programming and reinforcement learning for real robots

We propose an integrated technique of genetic programming (GP) and reinforcement learning (RL) to enable a real robot to adapt its actions to a real environment. Our technique does not require a precise simulator because learning is achieved through the real robot. In addition, our technique makes it possible for real robots to learn effective actions. Based on this proposed technique, we acquire common programs, using GP, which are applicable to various types of robots. Through this acquired program, we execute RL in a real robot. With our method, the robot can adapt to its own operational characteristics and learn effective actions. In this paper, we show experimental results from two different robots: a four-legged robot "AIBO" and a humanoid robot "HOAP-1." We present results showing that both effectively solved the box-moving task; the end result demonstrates that our proposed technique performs better than the traditional Q-learning method.     

基于再励学习的多移动机器人协调避障路径规划方法

随着多移动机器人协调系统的应用向未知环境发展,一些依赖于环境模型的路径规划方法不再适用。而利用再励学习与环境直接交互,不需要先验知识和样本数据的特点,该文将再励学习应用于多机器人协调系统中,提出了基于再励学习的避障路径规划方法,并将再励函数设计为基于行为分解的无模型非均匀结构。计算机仿真实验结果表明该方法有效,并有较好的鲁棒性,新的再励函数结构使得学习速度得以提高。     

Learning to Acquire Whole-Body Humanoid Center of Mass Movements to Achieve Dynamic Tasks

This paper presents a novel approach for acquiring dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass (CoM). In our approach, we combine both a model-based CoM controller and a model-free reinforcement learning (RL) method to acquire dynamic whole-body movements in humanoid robots. (i) To cope with high dimensionality, we use a model-based CoM controller as a basic controller that derives joint angular velocities from the desired CoM velocity. The balancing issue can also be considered in the controller. (ii) The RL method is used to acquire a controller that generates the desired CoM velocity based on the current state. To demonstrate the effectiveness of our approach, we apply it to a ball-punching task on a simulated humanoid robot model. The acquired whole-body punching movement was also demonstrated on Fujitsu's Hoap-2 humanoid robot.     

面向机器人系统的虚实迁移强化学习综述

近年来,基于环境交互的强化学习方法在机器人相关应用领域取得巨大成功,为机器人行为控制策略优化提供一个现实可行的解决方案.但在真实世界中收集交互样本存在高成本以及低效率等问题,因此仿真环境被广泛应用于机器人强化学习训练过程中.通过在虚拟仿真环境中以较低成本获取大量训练样本进行策略训练,并将学习策略迁移至真实环境,能有效缓解真实机器人训练中存在的安全性、可靠性以及实时性等问题.然而,由于仿真环境与真实环境存在差异,仿真环境中训练得到的策略直接迁移到真实机器人往往难以获得理想的性能表现.针对这一问题,虚实迁移强化学习方法被提出用以缩小环境差异,进而实现有效的策略迁移.按照迁移强化学习过程中信息的流动方向和智能化方法作用的不同对象,提出一个虚实迁移强化学习系统的流程框架,并基于此框架将现有相关工作分为3大类:基于真实环境的模型优化方法、基于仿真环境的知识迁移方法、基于虚实环境的策略迭代提升方法,并对每一分类中的代表技术与关联工作进行阐述.最后,讨论虚实迁移强化学习研究领域面临的机遇和挑战.     

A reinforcement learning approach to fail-safe design for multiple space robots—cooperation mechanism without communication and negotiation schemes

This paper explores a fail-safe design for multiple space robots, which enables robots to complete given tasks even when they can no longer be controlled due to a communication accident or negotiation problem. As the first step towards this goal, we propose new reinforcement learning methods that help robots avoid deadlock situations in addition to improving the degree of task completion without communications via ground stations or negotiations with other robots. Through intensive simulations on a truss construction task, we found that our reinforcement learning methods have great potential to contribute towards fail-safe design for multiple space robots in the above case. Furthermore, the simulations revealed the following detailed implications: (i) the first several planned behaviors must not be reinforced with negative rewards even in deadlock situations in order to derive cooperation among multiple robots, (ii) a certain amount of positive rewards added into negative rewards in deadlock situations contributes to reducing the computational cost of finding behavior plans for task completion, and (iii) an appropriate balance between positive and negative rewards in deadlock situations is indispensable for finding good behavior plans at a small computational cost.     

Reinforcement learning to adjust parametrized motor primitives to new situations

Humans manage to adapt learned movements very quickly to new situations by generalizing learned behaviors from similar situations. In contrast, robots currently often need to re-learn the complete movement. In this paper, we propose a method that learns to generalize parametrized motor plans by adapting a small set of global parameters, called meta-parameters. We employ reinforcement learning to learn the required meta-parameters to deal with the current situation, described by states. We introduce an appropriate reinforcement learning algorithm based on a kernelized version of the reward-weighted regression. To show its feasibility, we evaluate this algorithm on a toy example and compare it to several previous approaches. Subsequently, we apply the approach to three robot tasks, i.e., the generalization of throwing movements in darts, of hitting movements in table tennis, and of throwing balls where the tasks are learned on several different real physical robots, i.e., a Barrett WAM, a BioRob, the JST-ICORP/SARCOS CBi and a Kuka KR?6.     

基于改进深度强化学习的动态移动机器人协同计算卸载

移动边缘计算是解决机器人大计算量任务需求的一种方法。传统算法基于智能算法或凸优化方法,迭代时间长。深度强化学习通过一次前向传递即可求解,但只针对固定数量机器人进行求解。通过对深度强化学习分析研究,在深度强化学习神经网络中输入层前进行输入规整,在输出层后添加卷积层,使得网络能够自适应满足动态移动机器人数量的卸载需求。最后通过仿真实验验证,与自适应遗传算法和强化学习进行对比,验证了所提出算法的有效性及可行性。     

Efficient model learning methods for actor-critic control

We propose two new actor-critic algorithms for reinforcement learning. Both algorithms use local linear regression (LLR) to learn approximations of the functions involved. A crucial feature of the algorithms is that they also learn a process model, and this, in combination with LLR, provides an efficient policy update for faster learning. The first algorithm uses a novel model-based update rule for the actor parameters. The second algorithm does not use an explicit actor but learns a reference model which represents a desired behavior, from which desired control actions can be calculated using the inverse of the learned process model. The two novel methods and a standard actor-critic algorithm are applied to the pendulum swing-up problem, in which the novel methods achieve faster learning than the standard algorithm.     

未知环境中移动机器人柔性行为决策的调节发育学习

未知环境中移动机器人柔性的行为决策是完成各种任务的前提.目前的机器人行为决策方法在面对动态变化的环境时柔性较差,机器人难以获得持续稳定的学习能力.本文作者曾尝试通过集成小脑监督学习和基底神经节的强化学习来实现移动机器人动态环境下的柔性行为决策,但所提算法适应动态环境的能力有限.在前期工作基础上,本文设计了更有生物学意义的好奇度指标代替原来的警觉度指标,通过模拟蓝斑活动在基音模式和阶段模式之间的动态切换,实现移动机器人环境探索–利用的动态自适应调节.同时,设计随外部环境变化的自适应调节因子,实现移动机器人动态环境中基于小脑监督学习和基底神经节强化学习的柔性行为决策,使机器人可以获得持续稳定的学习能力.动态环境和实际环境中的实验结果验证了本文所提算法的有效性.     

TEXPLORE: real-time sample-efficient reinforcement learning for robots

The use of robots in society could be expanded by using reinforcement learning (RL) to allow robots to learn and adapt to new situations online. RL is a paradigm for learning sequential decision making tasks, usually formulated as a Markov Decision Process (MDP). For an RL algorithm to be practical for robotic control tasks, it must learn in very few samples, while continually taking actions in real-time. In addition, the algorithm must learn efficiently in the face of noise, sensor/actuator delays, and continuous state features. In this article, we present texplore, the first algorithm to address all of these challenges together. texplore is a model-based RL method that learns a random forest model of the domain which generalizes dynamics to unseen states. The agent explores states that are promising for the final policy, while ignoring states that do not appear promising. With sample-based planning and a novel parallel architecture, texplore can select actions continually in real-time whenever necessary. We empirically evaluate the importance of each component of texplore in isolation and then demonstrate the complete algorithm learning to control the velocity of an autonomous vehicle in real-time.     

Swarm robots reinforcement learning convergence accuracy-based learning classifier systems with gradient descent (XCS-GD)

This paper presented a novel approach accuracy-based learning classifier system with gradient descent (XCS-GD) to research on swarm robots reinforcement learning convergence. XCS-GD combines covering operator and genetic algorithm. XCS-GD is responsible for adjusting precision and reducing search space according to some reward obtained from the environment, XCS-GD’s innovation discovery component is responsible for discovering new better reinforcement learning rules. The experiment and simulation showed that XCS-GD approach can achieve convergence very quickly in swarm robots reinforcement learning.     

一种新的多智能体强化学习算法及其在多机器人协作任务中的应用

在多机器人系统中，评价一个机器人行为的好坏常常依赖于其它机器人的行为，此 时必须采用组合动作以实现多机器人的协作，但采用组合动作的强化学习算法由于学习空间 异常庞大而收敛得极慢．本文提出的新方法通过预测各机器人执行动作的概率来降低学习空 间的维数，并应用于多机器人协作任务之中．实验结果表明，基于预测的加速强化学习算法 可以比原始算法更快地获得多机器人的协作策略．     

Cooperative behavior control of robot group using stress antibody allotment reward

Lately, development in robotics for utilizing in both industry and home is in much progress. In this research, a group of robots is made to handle relatively complicated tasks. Cooperative action among robots is one of the research areas in robotics that is progressing remarkably well. Reinforcement learning is known as a common approach in robotics for deploying acquisition of action under dynamic environment. However, until recently, reinforcement learning is only applied to one agent problem. In multi-agent environment where plural robots exist, it was difficult to differentiate between learning of achievement of task and learning of performing cooperative action. This paper introduces a method of implementing reinforcement learning to induce cooperation among a group of robots where its task is to transport luggage of various weights to a destination. The general Q-learning method is used as a learning algorithm. Also, the switching of learning mode is proposed for reduction of learning time and learning area. Finally, grid world simulation is carried out to evaluate the proposed methods.     

Policy search for motor primitives in robotics

Many motor skills in humanoid robotics can be learned using parametrized motor primitives. While successful applications to date have been achieved with imitation learning, most of the interesting motor learning problems are high-dimensional reinforcement learning problems. These problems are often beyond the reach of current reinforcement learning methods. In this paper, we study parametrized policy search methods and apply these to benchmark problems of motor primitive learning in robotics. We show that many well-known parametrized policy search methods can be derived from a general, common framework. This framework yields both policy gradient methods and expectation-maximization (EM) inspired algorithms. We introduce a novel EM-inspired algorithm for policy learning that is particularly well-suited for dynamical system motor primitives. We compare this algorithm, both in simulation and on a real robot, to several well-known parametrized policy search methods such as episodic REINFORCE, ??Vanilla?? Policy Gradients with optimal baselines, episodic Natural Actor Critic, and episodic Reward-Weighted Regression. We show that the proposed method out-performs them on an empirical benchmark of learning dynamical system motor primitives both in simulation and on a real robot. We apply it in the context of motor learning and show that it can learn a complex Ball-in-a-Cup task on a real Barrett WAM? robot arm.     

强化学习理论在机器人应用中的几个关键问题探讨

文章在简单概述强化学习理论的基础上,对强化学习在实际机器人应用中经常遇到的连续状态-动作空间、信度分配、探索和利用的平衡、不完整信息等关键性问题进行了讨论,给出了一些常用的解决方法,以期为相关的研究和应用提供一个参考。     

Reinforcement learning of goal-directed obstacle-avoiding reaction strategies in an autonomous mobile robot

In this paper we argue for building reactive autonomous mobile robots through reinforcement connectionist learning. Nevertheless, basic reinforcement learning is a slow process. This paper describes an architecture which deals with complex— high-dimensional and/or continuous—situation and action spaces effectively. This architecture is based on two main ideas. The first is to organize the reactive component into a set of modules in such a way that, roughly, each one of them codifies the prototypical action for a given cluster of situations. The second idea is to use a particular kind of planning for figuring out what part of the action space deserves attention for each cluster of situations. Salient features of the planning process are that it is grounded and that it is invoked only when the reactive component does not generalize correctly its previous experience to the new situation. We also report our experience in solving a basic task that most autonomous mobile robots must face, namely path finding.     

Rapid, safe, and incremental learning of navigation strategies

In this paper we propose a reinforcement connectionist learning architecture that allows an autonomous robot to acquire efficient navigation strategies in a few trials. Besides rapid learning, the architecture has three further appealing features. First, the robot improves its performance incrementally as it interacts with an initially unknown environment, and it ends up learning to avoid collisions even in those situations in which its sensors cannot detect the obstacles. This is a definite advantage over nonlearning reactive robots. Second, since it learns from basic reflexes, the robot is operational from the very beginning and the learning process is safe. Third, the robot exhibits high tolerance to noisy sensory data and good generalization abilities. All these features make this learning robot's architecture very well suited to real-world applications. We report experimental results obtained with a real mobile robot in an indoor environment that demonstrate the appropriateness of our approach to real autonomous robot control.     

Autonomous driving at the handling limit using residual reinforcement learning

While driving a vehicle safely at its handling limit is essential in autonomous vehicles in Level 5 autonomy, it is a very challenging task for current conventional methods. Therefore, this study proposes a novel controller of trajectory planning and motion control for autonomous driving through manifold corners at the handling limit to improve the speed and shorten the lap time of the vehicle. The proposed controller innovatively combines the advantages of conventional model-based control algorithm, model-free reinforcement learning algorithm, and prior expert knowledge, to improve the training efficiency for autonomous driving in extreme conditions. The reward shaping of this algorithm refers to the procedure and experience of race training of professional drivers in real time. After training on track maps that exhibit different levels of difficulty, the proposed controller implemented a superior strategy compared to the original reference trajectory, and can to other tougher maps based on the basic driving knowledge learned from the simpler map, which verifies its superiority and extensibility. We believe this technology can be further applied to daily life to expand the application scenarios and maneuvering envelopes of autonomous vehicles.