Model Learning with Local Gaussian Process Regression

Precise models of robot inverse dynamics allow the design of significantly more accurate, energy-efficient and compliant robot control. However, in some cases the accuracy of rigid-body models does not suffice for sound control performance due to unmodeled nonlinearities arising from hydraulic cable dynamics, complex friction or actuator dynamics. In such cases, estimating the inverse dynamics model from measured data poses an interesting alternative. Nonparametric regression methods, such as Gaussian process regression (GPR) or locally weighted projection regression (LWPR), are not as restrictive as parametric models and, thus, offer a more flexible framework for approximating unknown nonlinearities. In this paper, we propose a local approximation to the standard GPR, called local GPR (LGP), for real-time model online learning by combining the strengths of both regression methods, i.e., the high accuracy of GPR and the fast speed of LWPR. The approach is shown to have competitive learning performance for high-dimensional data while being sufficiently fast for real-time learning. The effectiveness of LGP is exhibited by a comparison with the state-of-the-art regression techniques, such as GPR, LWPR and ν-support vector regression. The applicability of the proposed LGP method is demonstrated by real-time online learning of the inverse dynamics model for robot model-based control on a Barrett WAM robot arm.     

Robot Control Optimization Using Reinforcement Learning

Conventional robot control schemes are basically model-based methods. However, exact modeling of robot dynamics poses considerable problems and faces various uncertainties in task execution. This paper proposes a reinforcement learning control approach for overcoming such drawbacks. An artificial neural network (ANN) serves as the learning structure, and an applied stochastic real-valued (SRV) unit as the learning method. Initially, force tracking control of a two-link robot arm is simulated to verify the control design. The simulation results confirm that even without information related to the robot dynamic model and environment states, operation rules for simultaneous controlling force and velocity are achievable by repetitive exploration. Hitherto, however, an acceptable performance has demanded many learning iterations and the learning speed proved too slow for practical applications. The approach herein, therefore, improves the tracking performance by combining a conventional controller with a reinforcement learning strategy. Experimental results demonstrate improved trajectory tracking performance of a two-link direct-drive robot manipulator using the proposed method.     

Learning gas distribution models using sparse Gaussian process mixtures

In this paper, we consider the problem of learning two-dimensional spatial models of gas distributions. To build models of gas distributions that can be used to accurately predict the gas concentration at query locations is a challenging task due to the chaotic nature of gas dispersal. We formulate this task as a regression problem. To deal with the specific properties of gas distributions, we propose a sparse Gaussian process mixture model, which allows us to accurately represent the smooth background signal and the areas with patches of high concentrations. We furthermore integrate the sparsification of the training data into an EM procedure that we apply for learning the mixture components and the gating function. Our approach has been implemented and tested using datasets recorded with a real mobile robot equipped with an electronic nose. The experiments demonstrate that our technique is well-suited for predicting gas concentrations at new query locations and that it outperforms alternative and previously proposed methods in robotics.     

Learning model trees from evolving data streams

The problem of real-time extraction of meaningful patterns from time-changing data streams is of increasing importance for the machine learning and data mining communities. Regression in time-changing data streams is a relatively unexplored topic, despite the apparent applications. This paper proposes an efficient and incremental stream mining algorithm which is able to learn regression and model trees from possibly unbounded, high-speed and time-changing data streams. The algorithm is evaluated extensively in a variety of settings involving artificial and real data. To the best of our knowledge there is no other general purpose algorithm for incremental learning regression/model trees able to perform explicit change detection and informed adaptation. The algorithm performs online and in real-time, observes each example only once at the speed of arrival, and maintains at any-time a ready-to-use model tree. The tree leaves contain linear models induced online from the examples assigned to them, a process with low complexity. The algorithm has mechanisms for drift detection and model adaptation, which enable it to maintain accurate and updated regression models at any time. The drift detection mechanism exploits the structure of the tree in the process of local change detection. As a response to local drift, the algorithm is able to update the tree structure only locally. This approach improves the any-time performance and greatly reduces the costs of adaptation.     

Multi-level control architecture for Bionic Handling Assistant robot augmented by learning from demonstration for apple-picking

The control of soft continuum robots is challenging owing to their mechanical elasticity and complex dynamics. An additional challenge emerges when we want to apply Learning from Demonstration (LfD) and need to collect necessary demonstrations due to the inherent control difficulty. In this paper, we provide a multi-level architecture from low-level control to high-level motion planning for the Bionic Handling Assistant (BHA) robot. We deploy learning across all levels to enable the application of LfD for a real-world manipulation task. To record the demonstrations, an actively compliant controller is used. A variant of dynamical systems' application that are able to encode both position and orientation then maps the recorded 6D end-effector pose data into a virtual attractor space. A recent LfD method encodes the pose attractors within the same model for point-to-point motion planning. In the proposed architecture, hybrid models that combine an analytical approach and machine learning techniques are used to overcome the inherent slow dynamics and model imprecision of the BHA. The performance and generalization capability of the proposed multi-level approach are evaluated in simulation and with the real BHA robot in an apple-picking scenario which requires high accuracy to control the pose of the robot's end-effector.     

面向机器人系统的虚实迁移强化学习综述

近年来,基于环境交互的强化学习方法在机器人相关应用领域取得巨大成功,为机器人行为控制策略优化提供一个现实可行的解决方案.但在真实世界中收集交互样本存在高成本以及低效率等问题,因此仿真环境被广泛应用于机器人强化学习训练过程中.通过在虚拟仿真环境中以较低成本获取大量训练样本进行策略训练,并将学习策略迁移至真实环境,能有效缓解真实机器人训练中存在的安全性、可靠性以及实时性等问题.然而,由于仿真环境与真实环境存在差异,仿真环境中训练得到的策略直接迁移到真实机器人往往难以获得理想的性能表现.针对这一问题,虚实迁移强化学习方法被提出用以缩小环境差异,进而实现有效的策略迁移.按照迁移强化学习过程中信息的流动方向和智能化方法作用的不同对象,提出一个虚实迁移强化学习系统的流程框架,并基于此框架将现有相关工作分为3大类:基于真实环境的模型优化方法、基于仿真环境的知识迁移方法、基于虚实环境的策略迭代提升方法,并对每一分类中的代表技术与关联工作进行阐述.最后,讨论虚实迁移强化学习研究领域面临的机遇和挑战.     

Indirect robot model learning for tracking control

Learning task-space tracking control on redundant robot manipulators is an important but difficult problem. A main difficulty is the non-uniqueness of the solution: a task-space trajectory has multiple joint-space trajectories associated, therefore averaging over non-convex solution space needs to be done if treated as a regression problem. A second class of difficulties arise for those robots when the physical model is either too complex or even not available. In this situation machine learning methods may be a suitable alternative to classical approaches. We propose a learning framework for tracking control that is applicable for underactuated or non-rigid robots where an analytical physical model of the robot is unavailable. The proposed framework builds on the insight that tracking problems are well defined in the joint task- and joint-space coordinates and consequently predictions can be obtained via local optimization. Physical experiments show that state-of-the art accuracy can be achieved in both online and offline tracking control learning. Furthermore, we show that the presented method is capable of controlling underactuated robot architectures as well.     

Neural net robot controller with guaranteed tracking performance

A neural net (NN) controller for a general serial-link robot arm is developed. The NN has two layers so that linearity in the parameters holds, but the "net functional reconstruction error" and robot disturbance input are taken as nonzero. The structure of the NN controller is derived using a filtered error/passivity approach, leading to new NN passivity properties. Online weight tuning algorithms including a correction term to backpropagation, plus an added robustifying signal, guarantee tracking as well as bounded NN weights. The NN controller structure has an outer tracking loop so that the NN weights are conveniently initialized at zero, with learning occurring online in real-time. It is shown that standard backpropagation, when used for real-time closed-loop control, can yield unbounded NN weights if (1) the net cannot exactly reconstruct a certain required control function or (2) there are bounded unknown disturbances in the robot dynamics. The role of persistency of excitation is explored.     

Balance Control of a Biped Robot on a Rotating Platform Based on Efficient Reinforcement Learning

In this work, we combined the model based reinforcement learning (MBRL) and model free reinforcement learning (MFRL) to stabilize a biped robot (NAO robot) on a rotating platform, where the angular velocity of the platform is unknown for the proposed learning algorithm and treated as the external disturbance. Nonparametric Gaussian processes normally require a large number of training data points to deal with the discontinuity of the estimated model. Although some improved method such as probabilistic inference for learning control (PILCO) does not require an explicit global model as the actions are obtained by directly searching the policy space, the overfitting and lack of model complexity may still result in a large deviation between the prediction and the real system. Besides, none of these approaches consider the data error and measurement noise during the training process and test process, respectively. We propose a hierarchical Gaussian processes (GP) models, containing two layers of independent GPs, where the physically continuous probability transition model of the robot is obtained. Due to the physically continuous estimation, the algorithm overcomes the overfitting problem with a guaranteed model complexity, and the number of training data is also reduced. The policy for any given initial state is generated automatically by minimizing the expected cost according to the predefined cost function and the obtained probability distribution of the state. Furthermore, a novel Q(λ) based MFRL method scheme is employed to improve the policy. Simulation results show that the proposed RL algorithm is able to balance NAO robot on a rotating platform, and it is capable of adapting to the platform with varying angular velocity.      

Denoising and recognition using hidden Markov models with observation distributions modeled by hidden Markov trees

Hidden Markov models have been found very useful for a wide range of applications in machine learning and pattern recognition. The wavelet transform has emerged as a new tool for signal and image analysis. Learning models for wavelet coefficients have been mainly based on fixed-length sequences, but real applications often require to model variable-length, very long or real-time sequences. In this paper, we propose a new learning architecture for sequences analyzed on short-term basis, but not assuming stationarity within each frame. Long-term dependencies will be modeled with a hidden Markov model which, in each internal state, will deal with the local dynamics in the wavelet domain, using a hidden Markov tree. The training algorithms for all the parameters in the composite model are developed using the expectation-maximization framework. This novel learning architecture could be useful for a wide range of applications. We detail two experiments with artificial and real data: model-based denoising and speech recognition. Denoising results indicate that the proposed model and learning algorithm are more effective than previous approaches based on isolated hidden Markov trees. In the case of the ‘Doppler’ benchmark sequence, with 1024 samples and additive white noise, the new method reduced the mean squared error from 1.0 to 0.0842. The proposed methods for feature extraction, modeling and learning, increased the phoneme recognition rates in 28.13%, with better convergence than models based on Gaussian mixtures.     

基于深度学习的移动机器人目标跟踪系统

针对当前智能移动机器人在跟踪过程中常因目标发生外观形态上的变化而丢失跟踪目标的问题,利用Caffe深度学习框架和ROS机器人操作系统作为开发平台,设计一个高准确度及高实时性的移动机器人目标跟踪系统并进行了研究.使用对于目标形变、视角、轻微遮挡及光照变化具有鲁棒性的基于孪生卷积神经网络的GOTURN目标跟踪算法,通过ROS系统为桥梁使离线训练的跟踪模型实时应用于TurtleBot移动机器人上,并开展了详细的测试.实验结果表明,该目标跟踪系统不仅设计方案可行,实现了移动机器人在各种复杂场景下有效地跟踪目标,还具有成本低、性能高和易扩展等特点.     

内部动机驱动的机器人未知环境在线自主学习

针对目前智能移动机器人在未知环境中学习遇到的如学习主动性、实时性差,无法在线积累学习的知识和经验等问题,受心理学中内部动机的启发,提出一种内部动机驱动的移动机器人未知环境在线自主学习方法,在一定程度上弥补目前该领域存在的一些问题。该方法通过在移动机器人Q学习的框架下,将奖励机制用基于心理学启发的内部动机取代,提高其对于未知环境的学习主动性,同时,采用增量自组织神经网络代替经典Q学习中的查找表,实现输入输出空间的映射,使得机器人能够在线增量地学习未知环境。实验结果表明,通过内部动机驱动的方法,移动机器人对于未知环境的学习主动性得到了提高,智能程度有了明显改进。     

在线鲁棒最小二乘支持向量机回归建模

鉴于工业过程的时变特性以及现场采集的数据通常具有非线性特性且包含离群点,利用最小二乘支持向量机回归（least squares support vector regression,LSSVR）建模易受离群点的影响.针对这一问题,结合鲁棒学习算法（robust learning algorithm,RLA）,本文提出了一种在线鲁棒最小二乘支持向量机回归建模方法.该方法首先利用LSSVR模型对过程输出进行预测,与真实输出相比较得到预测误差;然后利用RLA方法训练LSSVR模型的权值,建立鲁棒LSSVR模型;最后应用增量学习方法在线更新鲁棒LSSVR模型,从而得到在线鲁棒LSSVR模型.仿真研究验证了所提方法的有效性.     

A nonparametric Bayesian approach toward robot learning by demonstration

In the past years, many authors have considered application of machine learning methodologies to effect robot learning by demonstration. Gaussian mixture regression (GMR) is one of the most successful methodologies used for this purpose. A major limitation of GMR models concerns automatic selection of the proper number of model states, i.e., the number of model component densities. Existing methods, including likelihood- or entropy-based criteria, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori. Under this motivation, to resolve the aforementioned issues of GMR-based methods for robot learning by demonstration, in this paper we introduce a nonparametric Bayesian formulation for the GMR model, the Dirichlet process GMR model. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally investigate its efficacy as a robot learning by demonstration methodology, considering a number of demanding robot learning by demonstration scenarios.     

Humanoids skill learning based on real-time human motion imitation using Kinect

In this paper, a novel framework which enables humanoid robots to learn new skills from demonstration is proposed. The proposed framework makes use of real-time human motion imitation module as a demonstration interface for providing the desired motion to the learning module in an efficient and user-friendly way. This interface overcomes many problems of the currently used interfaces like direct motion recording, kinesthetic teaching, and immersive teleoperation. This method gives the human demonstrator the ability to control almost all body parts of the humanoid robot in real time (including hand shape and orientation which are essential to perform object grasping). The humanoid robot is controlled remotely and without using any sophisticated haptic devices, where it depends only on an inexpensive Kinect sensor and two additional force sensors. To the best of our knowledge, this is the first time for Kinect sensor to be used in estimating hand shape and orientation for object grasping within the field of real-time human motion imitation. Then, the observed motions are projected onto a latent space using Gaussian process latent variable model to extract the relevant features. These relevant features are then used to train regression models through the variational heteroscedastic Gaussian process regression algorithm which is proved to be a very accurate and very fast regression algorithm. Our proposed framework is validated using different activities concerned with both human upper and lower body parts and object grasping also.     

A multi-level control architecture for the bionic handling assistant

The bionic handling assistant is one of the largest soft continuum robots and very special in being a pneumatically operated platform that is able to bend, stretch, and grasp in all directions. It nevertheless shares many challenges with smaller continuum and other soft robots such as parallel actuation, complex movement dynamics, slow pneumatic actuation, non-stationary behavior, and a lack of analytic models. To master the control of this challenging robot, we argue for a tight integration of standard analytic tools, simulation, control, and state-of-the-art machine learning into an overall architecture that can serve as blueprint for control design also beyond the BHA. To this aim, we show how to integrate specific modes of operation and different levels of control in a synergistic manner, which is enabled by using modern paradigms of software architecture and middleware. We thereby achieve an architecture with unique overall control abilities for a soft continuum robot that allow for flexible experimentation toward compliant user-interaction, grasping, and online learning of internal models.     

Learning control of mobile robots using a multiprocessor system

A real-time multiprocessor system is proposed for the solution of the tracking problem of mobile robots operating in a real context with environmental disturbances and parameter uncertainties. The proposed control scheme utilizes multiple models of the robot for its identification in an adaptive and learning control framework. Radial Basis Function Networks (RBFNs) are considered for the multiple models in order to exploit the net non-linear approximation capabilities for modeling the kinematic behavior of the vehicle and for reducing unmodeled contributions to tracking errors. The training of the nets and the tests of the achieved control performance have been done in a real experimental setup. The proposed control architecture improves the robot tracking performance achieving fast and accurate control actions in presence of large and time-varying uncertainties in dynamical environments. The experimental results are satisfactory in terms of tracking errors and computational efforts.     

Active balance of humanoids with foot positioning compensation and non-parametric adaptation

To maintain human-like active balance for a humanoid robot, this paper proposes a novel adaptive non-parametric foot positioning compensation approach that can modify predefined step position and step duration online with sensor feedback. A constrained inverted pendulum model taking into account of supporting area to CoM acceleration is used to generate offline training samples with constrained nonlinear optimization programming. To speed up real-time computation and make online model adjustable, a non-parametric regression model based on extended Gaussian Process model is applied for online foot positioning compensation. In addition, a real-time and sample-efficient local adaptation algorithm is proposed for the non-parametric model to enable online adaptation of foot positioning compensation on a humanoid system. Simulation and experiments on a full-body humanoid robot validate the effectiveness of the proposed method.     

Auditory learning: a developmental method

Motivated by the human autonomous development process from infancy to adulthood, we have built a robot that develops its cognitive and behavioral skills through real-time interactions with the environment. We call such a robot a developmental robot. In this paper, we present the theory and the architecture to implement a developmental robot and discuss the related techniques that address an array of challenging technical issues. As an application, experimental results on a real robot, self-organizing, autonomous, incremental learner (SAIL), are presented with emphasis on its audition perception and audition-related action generation. In particular, the SAIL robot conducts the auditory learning from unsegmented and unlabeled speech streams without any prior knowledge about the auditory signals, such as the designated language or the phoneme models. Neither available before learning starts are the actions that the robot is expected to perform. SAIL learns the auditory commands and the desired actions from physical contacts with the environment including the trainers.     

Incremental Tensor Subspace Learning and Its Applications to Foreground Segmentation and Tracking

Appearance modeling is very important for background modeling and object tracking. Subspace learning-based algorithms have been used to model the appearances of objects or scenes. Current vector subspace-based algorithms cannot effectively represent spatial correlations between pixel values. Current tensor subspace-based algorithms construct an offline representation of image ensembles, and current online tensor subspace learning algorithms cannot be applied to background modeling and object tracking. In this paper, we propose an online tensor subspace learning algorithm which models appearance changes by incrementally learning a tensor subspace representation through adaptively updating the sample mean and an eigenbasis for each unfolding matrix of the tensor. The proposed incremental tensor subspace learning algorithm is applied to foreground segmentation and object tracking for grayscale and color image sequences. The new background models capture the intrinsic spatiotemporal characteristics of scenes. The new tracking algorithm captures the appearance characteristics of an object during tracking and uses a particle filter to estimate the optimal object state. Experimental evaluations against state-of-the-art algorithms demonstrate the promise and effectiveness of the proposed incremental tensor subspace learning algorithm, and its applications to foreground segmentation and object tracking.