Automatic neural-based pattern classification of motion behaviors in autonomous robots

This paper addresses the problem of providing autonomous robots with a system that allows them to classify the motion behavior patterns of groups of robots present in their surroundings. It is a first step in the development of a cognitive model that can detect and understand the events occurring in the environment that are not due to the robot's own actions. The recognition of motion patterns must be achieved from the input data acquired by the robot through its camera during real time operation and, consequently, it can be addressed as a high dimensional dynamic pattern classification problem. Artificial Neural Networks (ANN) have been widely used in this type of classification problems, where a preprocessing stage is typically introduced in order to reduce dimensionality. In this stage, the processing window size and the dimensional transformation parameters must be selected according to specific domain knowledge, and they remain fixed during the ANN classification process. Such an approach is not applicable here as there is no prior information on the number of robots present or the dimensional reduction level required to describe the possible robot motion behaviors. Consequently, this work proposes a hybrid approach based on the application of a classification system called ANPAC (Automatic Neural-based Pattern Classifier) that uses a variable size ANN to perform the classification and an advisor module to adjust the preprocessing parameters and, consequently, the size of the ANN, depending on the learning results of the network. The components and operation of ANPAC are described in depth and illustrated using an example related to the recognition of behavior patterns in the motion of flocks.     

融合元学习和PPO算法的四足机器人运动技能学习方法

具备学习能力是高等动物智能的典型表现特征, 为探明四足动物运动技能学习机理, 本文对四足机器人步 态学习任务进行研究, 复现了四足动物的节律步态学习过程. 近年来, 近端策略优化(PPO)算法作为深度强化学习 的典型代表, 普遍被用于四足机器人步态学习任务, 实验效果较好且仅需较少的超参数. 然而, 在多维输入输出场 景下, 其容易收敛到局部最优点, 表现为四足机器人学习到步态节律信号杂乱且重心震荡严重. 为解决上述问题, 在元学习启发下, 基于元学习具有刻画学习过程高维抽象表征优势, 本文提出了一种融合元学习和PPO思想的元近 端策略优化(MPPO)算法, 该算法可以让四足机器人进化学习到更优步态. 在PyBullet仿真平台上的仿真实验结果表 明, 本文提出的算法可以使四足机器人学会行走运动技能, 且与柔性行动者评价器(SAC)和PPO算法的对比实验显 示, 本文提出的MPPO算法具有步态节律信号更规律、行走速度更快等优势.     

Robotic grasping and manipulation through human visuomotor learning

A major goal of robotics research is to develop techniques that allow non-experts to teach robots dexterous skills. In this paper, we report our progress on the development of a framework which exploits human sensorimotor learning capability to address this aim. The idea is to place the human operator in the robot control loop where he/she can intuitively control the robot, and by practice, learn to perform the target task with the robot. Subsequently, by analyzing the robot control obtained by the human, it is possible to design a controller that allows the robot to autonomously perform the task. First, we introduce this framework with the ball-swapping task where a robot hand has to swap the position of the balls without dropping them, and present new analyses investigating the intrinsic dimension of the ball-swapping skill obtained through this framework. Then, we present new experiments toward obtaining an autonomous grasp controller on an anthropomorphic robot. In the experiments, the operator directly controls the (simulated) robot using visual feedback to achieve robust grasping with the robot. The data collected is then analyzed for inferring the grasping strategy discovered by the human operator. Finally, a method to generalize grasping actions using the collected data is presented, which allows the robot to autonomously generate grasping actions for different orientations of the target object.     

Simultaneous people tracking and motion pattern learning

The field of Human Robot Interaction (HRI) encompasses many difficult challenges as robots need a better understanding of human actions. Human detection and tracking play a major role in such scenarios. One of the main challenges is to track them with long term occlusions due to agile nature of human navigation. However, in general humans do not make random movements. They tend to follow common motion patterns depending on their intentions and environmental/physical constraints. Therefore, knowledge of such common motion patterns could allow a robotic device to robustly track people even with long term occlusions. On the other hand, once a robust tracking is achieved, they can be used to enhance common motion pattern models allowing robots to adapt to new motion patterns that could appear in the environment. Therefore, this paper proposes to learn human motion patterns based on Sampled Hidden Markov Model (SHMM) and simultaneously track people using a particle filter tracker. The proposed simultaneous people tracking and human motion pattern learning has not only improved the tracking robustness compared to more conservative approaches, it has also proven robustness to prolonged occlusions and maintaining identity. Furthermore, the integration of people tracking and on-line SHMM learning have led to improved learning performance. These claims are supported by real world experiments carried out on a robot with suite of sensors including a laser range finder.     

Accountable system design architecture for embodied AI: a focus on physical human support robots

Although the development of robot-based support systems for elderly people has become more popular, it is difficult for humans to understand the actions, plans, and behavior of autonomous robots and the reasons behind them, particularly when the robots include learning algorithms. Learning-based autonomous systems which are called AI are treated as an inherently untrustworthy ‘black box,’ because machine learning or deep learning algorithms are difficult for humans to understand. Robot systems such as assistive robots, which work closely with humans, however, should be trusted. Systems should therefore achieve accountability for all stakeholders. However, most research in this field has focused on particular systems and situations, and no general design architecture exists. In this study, we propose a new design method, focused on accountability and transparency, for learning-based robot systems. Describing the entire system is a necessary first step, and transcribing the described system for each stakeholder based on several principles is effective for achieving accountability. The method improves transparency for systems, including learning algorithms. A standing assistive robot is used as an example of the entire system to clarify which system parts require greater transparency. This study adopted the Systems Modeling Language (SysML) to describe the system and the described system is used for the information representation. Information should be represented considering the relationships between stakeholders, information, and the system interface. Because of their complexity, it is difficult for humans to understand the complete set of information available in robot systems. Systems should therefore present only the information required, depending on the situation. The stakeholder–interface relationship is also important because it is more beneficial for professionals to view information relevant to their specialized field, which would be difficult for others to understand. By contrast, the interface should be intuitive for general users. Visualization and sound are very useful means of transmitting information, with advantages and disadvantages for different circumstances. These relationships are important for achieving accountability. Finally, we show an example of implementation with a developed support system. It is confirmed that accountable systems can be designed based on the proposed design architecture.     

Energy-optimal gait analysis of quadruped robots

It is important for walking robots such as quadruped robots to have an efficient gait. Since animals and insects are the basic models for most walking robots, their walking patterns are good examples. In this study, the walking energy consumption of a quadruped robot is analyzed and compared with natural animal gaits. Genetic algorithms have been applied to obtain the energy-optimal gait when the quadruped robot is walking with a set velocity. In this method, an individual in a population represents the walking pattern of the quadruped robot. The gait (individual) which consumes the least energy is considered to be the best gait (individual) in this study. The energy-optimal gait is analyzed at several walking velocities, since the amount of walking energy consumption changes if the walking velocity of the robot is changed. The results of this study can be used to decide what type of gait should be generated for a quadruped robot as its walking velocity changes. This work was presented, in part, at the Sixth International Symposium on Artificial Life and Robotics, Tokyo, Japan, January 15–17, 2001.     

Adaptation technique for integrating genetic programming and reinforcement learning for real robots

We propose an integrated technique of genetic programming (GP) and reinforcement learning (RL) to enable a real robot to adapt its actions to a real environment. Our technique does not require a precise simulator because learning is achieved through the real robot. In addition, our technique makes it possible for real robots to learn effective actions. Based on this proposed technique, we acquire common programs, using GP, which are applicable to various types of robots. Through this acquired program, we execute RL in a real robot. With our method, the robot can adapt to its own operational characteristics and learn effective actions. In this paper, we show experimental results from two different robots: a four-legged robot "AIBO" and a humanoid robot "HOAP-1." We present results showing that both effectively solved the box-moving task; the end result demonstrates that our proposed technique performs better than the traditional Q-learning method.     

仿生机器人运动步态控制:强化学习方法综述

仿生机器人是一类典型的多关节非线性欠驱动系统，其步态控制是一个非常具有挑战性的问题。对于该问题，传统的控制和规划方法需要针对具体的运动任务进行专门设计，需要耗费大量时间和精力，而且所设计出来的控制器往往没有通用性。基于数据驱动的强化学习方法能对不同的任务进行自主学习，且对不同的机器人和运动任务具有良好的通用性。因此，近年来这种基于强化学习的方法在仿生机器人运动步态控制方面获得了不少应用。针对这方面的研究，本文从问题形式化、策略表示方法和策略学习方法3个方面对现有的研究情况进行了分析和总结，总结了强化学习应用于仿生机器人步态控制中尚待解决的问题，并指出了后续的发展方向。     

Generating human-like soccer primitives from human data

Recently, interest in analysis and generation of human and human-like motion has increased in various areas. In robotics, in order to operate a humanoid robot, it is necessary to generate motions that have strictly dynamic consistency. Furthermore, human-like motion for robots will bring advantages such as energy optimization.This paper presents a mechanism to generate two human-like motions, walking and kicking, for a biped robot using a simple model based on observation and analysis of human motion. Our ultimate goal is to establish a design principle of a controller in order to achieve natural human-like motions. The approach presented here rests on the principle that in most biological motor learning scenarios some form of optimization with respect to a physical criterion is taking place. In a similar way, the equations of motion for the humanoid robot systems are formulated in such a way that the resulting optimization problems can be solved reliably and efficiently.The simulation results show that faster and more accurate searching can be achieved to generate an efficient human-like gait. Comparison is made with methods that do not include observation of human gait. The gait has been successfully used to control Robo-Erectus, a soccer-playing humanoid robot, which is one of the foremost leading soccer-playing humanoid robots in the RoboCup Humanoid League.     

Gait optimization of step climbing for a hexapod robot

RHex-style hexapod robot is a type of legged robot which can perform multiple moving gaits according to different applications, due to its simple structure and strong mobility. However, traversing high obstacles has always been a big challenge for legged robots. In this paper, gait optimization of a hexapod robot is proposed for climbing steps at different heights, which even enables the robot to climb the step 3.9 times of the leg length. First, a previous step-climbing gait is optimized by adjusting body inclination when placing front legs on top of the step, which enables RHex with different sizes to perform the rising stage of the gait. Second, to improve the climbing heights, a novel quasi-static climbing gait is proposed by using the reversed claw-shape legs to reach the higher step. The nondeformable legs are used to raise the center of mass (COM) of the body by lifting the front and rear legs alternately so that the front legs can reach the top of the step, then the front and middle legs are lifted alternately to maneuver COM up onto the step. The simulations and dynamic analysis of climbing steps are utilized to verify the feasibility of the improved gait. Finally, the step-climbing experiments at different heights are performed with the optimized gaits to compare with the existing gaits. The results of simulations and experiments show the superiority of the proposed gaits due to climbing higher steps.     

Teaching robots to cooperate with humans in dynamic manipulation tasks based on multi-modal human-in-the-loop approach

We propose an approach to efficiently teach robots how to perform dynamic manipulation tasks in cooperation with a human partner. The approach utilises human sensorimotor learning ability where the human tutor controls the robot through a multi-modal interface to make it perform the desired task. During the tutoring, the robot simultaneously learns the action policy of the tutor and through time gains full autonomy. We demonstrate our approach by an experiment where we taught a robot how to perform a wood sawing task with a human partner using a two-person cross-cut saw. The challenge of this experiment is that it requires precise coordination of the robot’s motion and compliance according to the partner’s actions. To transfer the sawing skill from the tutor to the robot we used Locally Weighted Regression for trajectory generalisation, and adaptive oscillators for adaptation of the robot to the partner’s motion.     

Multisensor-Based Human Detection and Tracking for Mobile Service Robots

One of fundamental issues for service robots is human-robot interaction. In order to perform such a task and provide the desired services, these robots need to detect and track people in the surroundings. In this paper, we propose a solution for human tracking with a mobile robot that implements multisensor data fusion techniques. The system utilizes a new algorithm for laser-based leg detection using the onboard laser range finder (LRF). The approach is based on the recognition of typical leg patterns extracted from laser scans, which are shown to also be very discriminative in cluttered environments. These patterns can be used to localize both static and walking persons, even when the robot moves. Furthermore, faces are detected using the robot's camera, and the information is fused to the legs' position using a sequential implementation of unscented Kalman filter. The proposed solution is feasible for service robots with a similar device configuration and has been successfully implemented on two different mobile platforms. Several experiments illustrate the effectiveness of our approach, showing that robust human tracking can be performed within complex indoor environments.     

Interactive Teaching for Vision-Based Mobile Robots: A Sensory-Motor Approach

For the last decade, we have been developing a vision-based architecture for mobile robot navigation. Using our bio-inspired model of navigation, robots can perform sensory-motor tasks in real time in unknown indoor as well as outdoor environments. We address here the problem of autonomous incremental learning of a sensory-motor task, demonstrated by an operator guiding a robot. The proposed system allows for semisupervision of task learning and is able to adapt the environmental partitioning to the complexity of the desired behavior. A real dialogue based on actions emerges from the interactive teaching. The interaction leads the robot to autonomously build a precise sensory-motor dynamics that approximates the behavior of the teacher. The usability of the system is highlighted by experiments on real robots, in both indoor and outdoor environments. Accuracy measures are also proposed in order to evaluate the learned behavior as compared to the expected behavioral attractor. These measures, used first in a real experiment and then in a simulated experiment, demonstrate how a real interaction between the teacher and the robot influences the learning process.     

Improving human robot collaboration through Force/Torque based learning for object manipulation

Human–Robot Collaboration (HRC) is a term used to describe tasks in which robots and humans work together to achieve a goal. Unlike traditional industrial robots, collaborative robots need to be adaptive; able to alter their approach to better suit the situation and the needs of the human partner. As traditional programming techniques can struggle with the complexity required, an emerging approach is to learn a skill by observing human demonstration and imitating the motions; commonly known as Learning from Demonstration (LfD). In this work, we present a LfD methodology that combines an ensemble machine learning algorithm (i.e. Random Forest (RF)) with stochastic regression, using haptic information captured from human demonstration. The capabilities of the proposed method are evaluated using two collaborative tasks; co-manipulation of an object (where the human provides the guidance but the robot handles the objects weight) and collaborative assembly of simple interlocking parts. The proposed method is shown to be capable of imitation learning; interpreting human actions and producing equivalent robot motion across a diverse range of initial and final conditions. After verifying that ensemble machine learning can be utilised for real robotics problems, we propose a further extension utilising Weighted Random Forest (WRF) that attaches weights to each tree based on its performance. It is then shown that the WRF approach outperforms RF in HRC tasks.     

Emergent synthesis of motion patterns for locomotion robots

Emergence of stable gaits in locomotion robots is studied in this paper. A classifier system, implementing an instance-based reinforcement-learning scheme, is used for the sensory-motor control of an eight-legged mobile robot and for the synthesis of the robot gaits. The robot does not have a priori knowledge of the environment and its own internal model. It is only assumed that the robot can acquire stable gaits by learning how to reach a goal area. During the learning process the control system is self-organized by reinforcement signals. Reaching the goal area defines a global reward. Forward motion gets a local reward, while stepping back and falling down get a local punishment. As learning progresses, the number of the action rules in the classifier systems is stabilized to a certain level, corresponding to the acquired gait patterns. Feasibility of the proposed self-organized system is tested under simulation and experiment. A minimal simulation model that does not require sophisticated computational schemes is constructed and used in simulations. The simulation data, evolved on the minimal model of the robot, is downloaded to the control system of the real robot. Overall, of 10 simulation data seven are successful in running the real robot.     

A Paradigm for Dynamic Coordination of Multiple Robots

In this paper, we present a paradigm for coordinating multiple robots in the execution of cooperative tasks. The basic idea in the paper is to assign to each robot in the team, a role that determines its actions during the cooperation. The robots dynamically assume and exchange roles in a synchronized manner in order to perform the task successfully, adapting to unexpected events in the environment. We model this mechanism using a hybrid systems framework and apply it in different cooperative tasks: cooperative manipulation and cooperative search and transportation. Simulations and real experiments demonstrating the effectiveness of the proposed paradigm are presented.     

A Study on Acquiring Underlying Behavioral Criteria for Manipulator Motion by Focusing on Learning Efficiency

Conventional humanoid robotic behaviors are directly programmed depending on the programmer's personal experience. With this method, the behaviors usually appear unnatural. It is believed that a humanoid robot can acquire new adaptive behaviors from a human, if the robot has the criteria underlying such behaviors. The aim of this paper is to establish a method of acquiring human behavioral criteria. The advantage of acquiring behavioral criteria is that the humanoid robots can then autonomously produce behaviors for similar tasks with the same behavioral criteria but without transforming data obtained from morphologically different humans every time for every task. In this paper, a manipulator robot learns a model behavior, and another robot is created to perform the model behavior instead of being performed by a person. The model robot is presented some behavioral criteria, but the learning manipulator robot does not know them and tries to infer them. In addition, because of the difference between human and robot bodies, the body sizes of the learning robot and the model robot are also made different. The method of obtaining behavioral criteria is realized by comparing the efficiencies with which the learning robot learns the model behaviors. Results from the simulation have demonstrated that the proposed method is effective for obtaining behavioral criteria. The proposed method, the details regarding the simulation, and the results are presented in this paper.     

Robots learn to dance through interaction with humans

In this paper, we investigated an approach for robots to learn to adapt dance actions to human’s preferences through interaction and feedback. Human’s preferences were extracted by analysing the common action patterns with positive or negative feedback from the human during robot dancing. By using a buffering technique to store the dance actions before a feedback, each individual’s preferences can be extracted even when a reward is received late. The extracted preferred dance actions from different people were then combined to generate improved dance sequences, i.e. performing more of what was preferred and less of that was not preferred. Together with Softmax action-selection method, the Sarsa reinforcement learning algorithm was used as the underlining learning algorithm and to effectively control the trade-off between exploitation of the learnt dance skills and exploration of new dance actions. The results showed that the robot learnt, using interactive reinforcement learning, the preferences of human partners, and the dance improved with the extracted preferences from more human partners.     

Power Consumption Optimization for a Hexapod Walking Robot

Power consumption is one of the main operational restrictions on autonomous walking robots. In this paper, an energy efficiency analysis is performed for a hexapod walking robot to reduce these energy costs. To meet the power-saving demands of legged robots, the torque distribution algorithm required to minimize the system’s energy costs was established with an energy-consumption model formulated. In contrast to the force distribution method, where the objective function is related to the tip-point force components, the torque distribution scheme is based on minimization of the mechanical energy cost and heat loss power. The simulation results show that this scheme could reduce the system energy costs with use of the appropriate walking velocities and duty factors for the robot. The paper also discusses the effects of the gait patterns and the mechanical structure on the system energy costs. For this purpose, the prescribed periodic walking gait of the robot is described in terms of several parameters, including the duty factor, the stride length, the body height, and the foot trajectory lateral offset. The numerical results indicate some analogies between the characteristics of the simulated walking robot and those of animals in nature. The optimized parameters derived here are intended for robot platform development applications.