Policy gradient learning for quadruped soccer robots

In real-world robotic applications, many factors, both at low level (e.g., vision, motion control and behaviors) and at high level (e.g., plans and strategies) determine the quality of the robot performance. Consequently, fine tuning of the parameters, in the implementation of the basic functionalities, as well as in the strategic decisions, is a key issue in robot software development. In recent years, machine learning techniques have been successfully used to find optimal parameters for typical robotic functionalities. However, one major drawback of learning techniques is time consumption: in practical applications, methods designed for physical robots must be effective with small amounts of data. In this paper, we present a method for concurrent learning of best strategy and optimal parameters using policy gradient reinforcement learning algorithm. The results of our experimental work in a simulated environment and on a real robot show a very high convergence rate.     

Robo-Erectus: a low-cost autonomous humanoid soccer robot

An exoskeleton robot can replace the wearer's motion function by operating the human's body. The purpose of this study is to propose a power assist method of walking, standing up and going up stairs based on autonomous motion of the exoskeleton robot suit, HAL (Hybrid assistive Limb), and verify the effectiveness of this method by experiment. In order to realize power assist of tasks (walking, standing up and going up stairs) autonomically, we used the Phase Sequence control which generates a task by transiting some simple basic motions called Phases. A task was divided into some Phases on the basis of the task performed by a normal person. The joint moving modes were categorized into active, passive and free modes according to the characteristic of the muscle force conditions. The autonomous motions which HAL generates in each Phase were designed corresponding to one of the categorized modes. The power assist experiments were performed by using the autonomous motion with a focus on the active mode. The experimental results showed that the wearer's muscle activation levels in each Phase were significantly reduced. With this, we confirmed the effectiveness of the proposed assist method.     

Reinforcement learning for robot soccer

Batch reinforcement learning methods provide a powerful framework for learning efficiently and effectively in autonomous robots. The paper reviews some recent work of the authors aiming at the successful application of reinforcement learning in a challenging and complex domain. It discusses several variants of the general batch learning framework, particularly tailored to the use of multilayer perceptrons to approximate value functions over continuous state spaces. The batch learning framework is successfully used to learn crucial skills in our soccer-playing robots participating in the RoboCup competitions. This is demonstrated on three different case studies.

一种改进的类人足球机器人彩色目标识别算法

针对类人足球机器人视觉需求,提出一种结合区域生长和基于形状判别的阈值自适应更新的彩色目标识别算法。该算法在HSI空间基于S分量把图像分为高饱和区域和低饱和区域,在高饱和区域基于H分量采用区域生长算法识别目标;通过目标形状判别自适应更新阈值,并用新阈值更新区域生长中原来的阈值,以稳定准确地识别彩色目标。在类人足球机器人系统中的成功应用表明,该算法能在不同光照条件下稳定地识别出彩色目标,对光照环境有良好的适应性和鲁棒性,具有良好的识别效果。     

Real-world reinforcement learning for autonomous humanoid robot docking

Reinforcement learning (RL) is a biologically supported learning paradigm, which allows an agent to learn through experience acquired by interaction with its environment. Its potential to learn complex action sequences has been proven for a variety of problems, such as navigation tasks. However, the interactive randomized exploration of the state space, common in reinforcement learning, makes it difficult to be used in real-world scenarios. In this work we describe a novel real-world reinforcement learning method. It uses a supervised reinforcement learning approach combined with Gaussian distributed state activation. We successfully tested this method in two real scenarios of humanoid robot navigation: first, backward movements for docking at a charging station and second, forward movements to prepare grasping. Our approach reduces the required learning steps by more than an order of magnitude, and it is robust and easy to be integrated into conventional RL techniques.     

Action-based sensor space segmentation for soccer robot learning

Robot learning, such as reinforcement learning, generally needs a well-defined state space in order to converge. However, building such a state space is one of the main issues of robot learning because of the interdependence between state and action spaces, which resembles the well-known "chicken and egg" problem. This article proposes a method of action-based state space construction for vision-based mobile robots. Basic ideas to cope with the interdependence are that we define a state as a cluster of input vectors from which the robot can reach the goal state or the state already obtained by a sequence of one kind of action primitive regardless of its length, and that this sequence is defined as one action. To realize these ideas, we need many data (experiences) of the robot and we must cluster the input vectors as hyper ellipsoids so that the whole state space is segmented into a state transition map in terms of action from which the optimal action sequence is obtained. To show the validity of the method, we apply it to a soccer robot that tries to shoot a ball into a goal. The simulation and real experiments are shown.     

类人足球机器人决策系统的设计

类人机器人足球比赛是机器人足球比赛的最高赛事.类人足球机器人的决策系统是基于独立视觉的自主决策系统,很大程度上决定着比赛的胜败.介绍了自主研发的类人足球机器人决策系统的架构及实现方法,并在此基础上运用有限状态机理论,对单个机器人的自主进攻策略进行了详细分析和研究,真实环境中的实验及比赛结果证明了其有效性.该决策系统的设计及研究工作对基于自主决策的多智能体协作以及服务性机器人决策系统的研究都具有重要的价值.     

基于ARM的类人足球机器人视觉识别系统的设计

针对类人足球机器人在复杂赛场环境中对目标色块识别能力的不足,设计一种基于ARM的类人足球机器人视觉识别系统。软硬件设计上,以搭载Linux系统的三星S3C6410作为主控图像处理芯片,用飞思卡尔MC9S12XS128单片机作为舵机控制器,以罗技C270摄像头作为视觉传感器。分割识别算法上,采用YUV颜色空间下的阈值法与区域生长的混合分割算法对色块信息进行快速分割和识别。实验表明,该设计可准确识别出较复杂环境中的目标色块并定位出中心坐标。     

Hardware design and gait generation of humanoid soccer robot Stepper-3D

This paper presents the hardware design and gait generation of humanoid soccer robot Stepper-3D. Virtual Slope Walking, inspired by Passive Dynamic Walking, is introduced for gait generation. In Virtual Slope Walking, by actively extending the stance leg and shortening the swing leg, the robot walks on level ground as it walks down a virtual slope. In practical, Virtual Slope Walking is generated by connecting three key frames in the sagittal plane with sinusoids. Aiming for improving the walking stability, the parallel double crank mechanism are adopted in the leg structure. Experimental results show that Stepper-3D achieves a fast forward walking speed of 0.5 m/s and accomplishes omnidirectional walking. Stepper-3D performed fast and stable walking in the RoboCup 2008 Humanoid competitions.     

Application of reinforcement learning in robot soccer

The robot soccer game has been proposed as a benchmark problem for the artificial intelligence and robotic researches. Decision-making system is the most important part of the robot soccer system. As the environment is dynamic and complex, one of the reinforcement learning (RL) method named FNN-RL is employed in learning the decision-making strategy. The FNN-RL system consists of the fuzzy neural network (FNN) and RL. RL is used for structure identification and parameters tuning of FNN. On the other hand, the curse of dimensionality problem of RL can be solved by the function approximation characteristics of FNN. Furthermore, the residual algorithm is used to calculate the gradient of the FNN-RL method in order to guarantee the convergence and rapidity of learning. The complex decision-making task is divided into multiple learning subtasks that include dynamic role assignment, action selection, and action implementation. They constitute a hierarchical learning system. We apply the proposed FNN-RL method to the soccer agents who attempt to learn each subtask at the various layers. The effectiveness of the proposed method is demonstrated by the simulation and the real experiments.     

Sensor information analysis for a humanoid robot

For a humanoid robot to safely walk in unknown environments, various sensors are used to identify the surface condition and recognize any obstacles. The humanoid robot is not fixed on the surface and the base/orientation of the kinematics change while it is walking. Therefore, if the foot contact changes from the estimated due to the unknown surface condition, the kinematics results are not correct. The robot may not be able to perform the motion commands based on the incorrect surface condition. Some robots have built-in range sensors but it’s difficult to accurately model the surface from the sensor readings because the movement of the robot should be considered and the robot localization should have zero error for correct interpretation of the sensor readings. In this paper, three infrared range sensors are used in order to perceive the floor state. Covariance analysis is incorporated to consider the uncertainties. The accelerometer and gyro sensor are also used in order to detect the moment a foot hits the surface. This information provides correction to the motion planner and robot kinematics when the environment is not modeled correctly.     

Development of a humanoid robot for underwater use

In this research, we have developed a swimming robot with a fluttering kick with two legs, which can swim freely both on the surface of water and under water. We have established a control method for all the different types of motion of this robot, e.g., swimming in a straight line, turning, diving, or rising up in the water. Furthermore, by optimizing the three-dimensional action of this underwater robot, we can expect an improvement in its performance for complex work.     

Evolving behavior sequences for a humanoid entertainment robot

One of the most important issues in developing an entertainment robot is human-robot interaction, in which the robot is expected to learn new behaviors specified by the user. In this article we present an imitation-based mechanism to support robot learning, and use evolutionary computing to learn new behavior sequences. We also propose several advanced techniques at the task level and the computational level to evolve complex sequences. To evaluate our approach, we use it to evolve different behaviors for a humanoid robot. The results show the promise of our approach.     

Designing a sociable humanoid robot for interdisciplinary research

This paper presents the humanoid robot BARTHOC and the smaller, but system-equal twin, BARTHOC Junior. Both robots have been developed to study human–robot interaction. The main focus of BARTHOC's design was to realize the expression and behavior of the robot to be as human-like as possible. This allows us to apply the platform to manifold research and demonstration areas. With its human-like look and mimic possibilities it differs from other platforms like ASIMO or QRIO and enables experiments even close to Mori's 'uncanny valley'. The paper describes details of the mechanical and electrical design of BARTHOC together with its PC control interface and an overview of the interaction architecture. Its humanoid appearance allows limited imitation of human behavior. The basic interaction software running on BARTHOC has been completely ported from a mobile robot except for some functionalities that could not be used due to hardware differences such as the lack of mobility. Based on these components, the robot's human-like appearance will enable us to study embodied interaction and to explore theories of human intelligence.     

Self-modeling in humanoid soccer robots

In this paper we discuss the applicability, potential benefits, open problems and expected contributions that an emerging set of self-modeling techniques might bring on the development of humanoid soccer robots. The idea is that robots might continuously generate, validate and adjust physical models of their sensorimotor interaction with the world. These models are exploited for adapting behavior in simulation, enhancing the learning skills of a robot with the regular transference of controllers developed in simulation to reality. Moreover, these simulations can be used to aid the execution of complex sensorimotor tasks, speed up adaptation and enhance task planning. We present experiments on the generation of behaviors for humanoid soccer robots using the Back-to-Reality algorithm. General motivations are presented, alternative algorithms are discussed and, most importantly, directions of research are proposed.     

A multi-agent reinforcement learning approach to robot soccer

In this paper, a multi-agent reinforcement learning method based on action prediction of other agent is proposed. In a multi-agent system, action selection of the learning agent is unavoidably impacted by other agents’ actions. Therefore, joint-state and joint-action are involved in the multi-agent reinforcement learning system. A novel agent action prediction method based on the probabilistic neural network (PNN) is proposed. PNN is used to predict the actions of other agents. Furthermore, the sharing policy mechanism is used to exchange the learning policy of multiple agents, the aim of which is to speed up the learning. Finally, the application of presented method to robot soccer is studied. Through learning, robot players can master the mapping policy from the state information to the action space. Moreover, multiple robots coordination and cooperation are well realized.     

Codevelopmental learning between human and humanoid robot using a dynamic neural-network model.

This paper examines characteristics of interactive learning between human tutors and a robot having a dynamic neural-network model, which is inspired by human parietal cortex functions. A humanoid robot, with a recurrent neural network that has a hierarchical structure, learns to manipulate objects. Robots learn tasks in repeated self-trials with the assistance of human interaction, which provides physical guidance until the tasks are mastered and learning is consolidated within the neural networks. Experimental results and the analyses showed the following: 1) codevelopmental shaping of task behaviors stems from interactions between the robot and a tutor; 2) dynamic structures for articulating and sequencing of behavior primitives are self-organized in the hierarchically organized network; and 3) such structures can afford both generalization and context dependency in generating skilled behaviors.     

Centroidal dynamics of a humanoid robot

The center of mass (CoM) of a humanoid robot occupies a special place in its dynamics. As the location of its effective total mass, and consequently, the point of resultant action of gravity, the CoM is also the point where the robot’s aggregate linear momentum and angular momentum are naturally defined. The overarching purpose of this paper is to refocus our attention to centroidal dynamics: the dynamics of a humanoid robot projected at its CoM. In this paper we specifically study the properties, structure and computation schemes for the centroidal momentum matrix (CMM), which projects the generalized velocities of a humanoid robot to its spatial centroidal momentum. Through a transformation diagram we graphically show the relationship between this matrix and the well-known joint-space inertia matrix. We also introduce the new concept of “average spatial velocity” of the humanoid that encompasses both linear and angular components and results in a novel decomposition of the kinetic energy. Further, we develop a very efficient $O(N)$ O ( N ) algorithm, expressed in a compact form using spatial notation, for computing the CMM, centroidal momentum, centroidal inertia, and average spatial velocity. Finally, as a practical use of centroidal dynamics we show that a momentum-based balance controller that directly employs the CMM can significantly reduce unnecessary trunk bending during balance maintenance against external disturbance.     

基于体感的仿人机器人步态学习与控制

针对现有理想化步态动力学模型规划方法复杂、人为指定参数过多、计算量大的问题,提出一种基于体感数据学习人体步态的仿人机器人步态生成方法。首先,用体感设备收集人体骨骼信息,基于最小二乘拟合方法建立人体关节局部坐标系;其次,搭建人体与机器人映射的运动学模型,根据两者间主要关节映射关系,生成机器人关节转角轨迹,实现机器人对人类行走姿态的学习;然后,基于零力矩点(ZMP)稳定性原则,对机器人脚踝关节转角采用梯度下降算法进行优化控制;最后,在步态稳定性分析上,提出使用安全系数来评价机器人行走稳定程度的方法。实验结果表明,步行过程中安全系数保持在0~0.85,期望为0.4825,ZMP接近于稳定区域中心,机器人实现了仿人姿态的稳定行走,证明了该方法的有效性。