强化学习中基于定性模型的知识传递方法

本文提出一种基于定性模糊网络的强化学习知识传递方法。该方法通过建立系统的定性模型,并用定性模糊网络抽取基于定性动作的次优策略的共同特征获得与系统参数无关知识。这些知识能有效描述参数值不同的系统所具有的共同控制规律,加快在新参数值的系统中强化学习的收敛速度。     

融合认知行为模型的深度强化学习框架及算法

面对高维连续状态空间或稀疏奖励等复杂任务时,仅依靠深度强化学习算法从零学习最优策略十分困难,如何将已有知识表示为人与学习型智能体之间相互可理解的形式,并有效地加速策略收敛仍是一个难题.对此,提出一种融合认知行为模型的深度强化学习框架,将领域内先验知识建模为基于信念-愿望-意图(belief- desire-intention, BDI)的认知行为模型,用于引导智能体策略学习.基于此框架,分别提出融合认知行为模型的深度Q学习算法和近端策略优化算法,并定量化设计认知行为模型对智能体策略更新的引导方式.最后,通过典型gym环境和空战机动决策对抗环境,验证所提出算法可以高效利用认知行为模型加速策略学习,有效缓解状态空间巨大和环境奖励稀疏的影响.     

Efficient Exploration for Multi-Agent Reinforcement Learning via Transferable Successor Features

In multi-agent reinforcement learning (MARL), the behaviors of each agent can influence the learning of others, and the agents have to search in an exponentially enlarged joint-action space. Hence, it is challenging for the multi-agent teams to explore in the environment. Agents may achieve suboptimal policies and fail to solve some complex tasks. To improve the exploring efficiency as well as the performance of MARL tasks, in this paper, we propose a new approach by transferring the knowledge across tasks. Differently from the traditional MARL algorithms, we first assume that the reward functions can be computed by linear combinations of a shared feature function and a set of task-specific weights. Then, we define a set of basic MARL tasks in the source domain and pre-train them as the basic knowledge for further use. Finally, once the weights for target tasks are available, it will be easier to get a well-performed policy to explore in the target domain. Hence, the learning process of agents for target tasks is speeded up by taking full use of the basic knowledge that was learned previously. We evaluate the proposed algorithm on two challenging MARL tasks: cooperative box-pushing and non-monotonic predator-prey. The experiment results have demonstrated the improved performance compared with state-of-the-art MARL algorithms.      

A feature selection method for a sample-based stochastic policy

Stochastic policy gradient methods have been applied to a variety of robot control tasks such as robot’s acquisition of motor skills because they have an advantage in learning in high-dimensional and continuous feature spaces by combining some heuristics like motor primitives. However, when we apply one of them to a real-world task, it is difficult to represent the task well by designing the policy function and the feature space due to the lack of enough prior knowledge about the task. In this research, we propose a method to extract a preferred feature space autonomously to achieve a task using a stochastic policy gradient method for a sample-based policy. We apply our method to a control of linear dynamical system and the computer simulation result shows that a desirable controller is obtained and that the performance of the controller is improved by the feature selection.     

基于自动编码器的中文词汇特征无监督学习

大规模未标注语料中蕴含了丰富的词汇信息,有助于提高中文分词词性标注模型效果。该文从未标注语料中抽取词汇的分布信息,表示为高维向量,进一步使用自动编码器神经网络,无监督地学习对高维向量的编码算法,最终得到可直接用于分词词性标注模型的低维特征表示。在宾州中文树库5.0数据集上的实验表明,所得到的词汇特征对分词词性标注模型效果有较大帮助,在词性标注上优于主成分分析与k均值聚类结合的无监督特征学习方法。     

An informational perspective on skill transfer in human-machine systems

Differentiation of perceptual invariants is proposed as a theoretical approach to explain skill transfer for control at the human-machine interface. I propose that sensitivity to perceptual invariants is enhanced during learning and that this sensitivity forms the basis for transfer of skill from one task to another. The hypothesis implies that detection and discrimination of critical features, patterns, and dimension of difference are important for learning and for transfer. This account goes beyond other similarity conceptions of transfer. To the extent that those conceptions are specific, they cannot account for effects in which performance is better following training on tasks that are less rather than more similar to the criterion task. In essence, this is a theory about the central role of low-dimensional informational patterns for control of behavior within a high-dimensional environment, and about the adjustment of an actor's sensitivity to changes in those low-dimensional patterns.     

基于局部结构学习的非线性属性选择算法

针对大多数高维数据之间不仅有相似性,而且还有非线性关系等特点,提出一种基于局部结构学习的非线性属性选择算法。该算法首先通过核函数把数据映射到高维空间,在高维空间中表示出数据属性之间的非线性关系;然后在低维空间中通过局部结构学习来充分挖掘属性之间的相似性,同时通过低秩约束来排除噪声的干扰;最后通过稀疏正则化因子来进行属性选择。其通过核函数映射来找出数据属性之间的非线性关系,运用局部结构学习来找出数据属性之间的相似性,是一种嵌入了局部结构学习的非线性属性选择算法。实验结果表明,该算法相比其他的对比算法,有更好的效果。     

结构交互驱动的机器人深度强化学习控制方法

针对深度强化学习在高维机器人行为控制中训练效率低下和策略不可解释等问题,提出一种基于结构交互驱动的机器人深度强化学习方法(structure-motivated interactive deep reinforcement learning, SMILE).首先,利用结构分解方法将高维的单机器人控制问题转化为低维的多关节控制器协同学习问题,从而缓解连续运动控制的维度灾难难题;其次,通过两种协同图模型(ATTENTION和PODT)动态推理控制器之间的关联关系,实现机器人内部关节的信息交互和协同学习;最后,为了平衡ATTENTION和PODT协同图模型的计算复杂度和信息冗余度,进一步提出两种协同图模型更新方法 APDODT和PATTENTION,实现控制器之间长期关联关系和短期关联关系的动态自适应调整.实验结果表明,基于结构驱动的机器人强化学习方法能显著提升机器人控制策略学习效率.此外,基于协同图模型的关系推理及协同机制,可为最终学习策略提供更为直观和有效的解释.     

点云配准中多维度信息融合的特征挖掘方法

数据挖掘是使用人工智能等方法在大型数据集中提取隐含潜在信息的过程,为从大量信息中获取有价值的知识提供了有效途径.在使用深度学习解决点云配准任务的过程中,数据挖掘也无处不在.全局特征提取和刚体变换估计是无对应点云配准的2个关键阶段,挖掘隐藏在2个阶段中的丰富信息是点云配准的重要任务之一.然而,最近提出的方法在提取全局特征时容易忽略低维局部特征,导致大量点云信息的丢失,使得后续刚体变换估计阶段求解变换参数时精度无法达到预期.首先,提出了一种基于多维度信息融合的特征挖掘网络,充分挖掘点云中的高维全局和低维局部信息,有效弥补了点云配准的全局特征提取阶段局部特征的缺失.其次,在刚体变换估计阶段使用了对偶四元数估计姿态,其可以在一个公共框架内同时表示旋转和平移,为姿态估计提供紧凑和精确的表示.最后,在ModelNet40数据集上进行的大量实验表明：与现有前沿的无对应点云配准方法相比,提出的方法可以获得更高的精度,同时对噪声具有较强的鲁棒性.     

Robust reinforcement learning

This letter proposes a new reinforcement learning (RL) paradigm that explicitly takes into account input disturbance as well as modeling errors. The use of environmental models in RL is quite popular for both offline learning using simulations and for online action planning. However, the difference between the model and the real environment can lead to unpredictable, and often unwanted, results. Based on the theory of H(infinity) control, we consider a differential game in which a "disturbing" agent tries to make the worst possible disturbance while a "control" agent tries to make the best control input. The problem is formulated as finding a min-max solution of a value function that takes into account the amount of the reward and the norm of the disturbance. We derive online learning algorithms for estimating the value function and for calculating the worst disturbance and the best control in reference to the value function. We tested the paradigm, which we call robust reinforcement learning (RRL), on the control task of an inverted pendulum. In the linear domain, the policy and the value function learned by online algorithms coincided with those derived analytically by the linear H(infinity) control theory. For a fully nonlinear swing-up task, RRL achieved robust performance with changes in the pendulum weight and friction, while a standard reinforcement learning algorithm could not deal with these changes. We also applied RRL to the cart-pole swing-up task, and a robust swing-up policy was acquired.     

Active learning of inverse models with intrinsically motivated goal exploration in robots

We introduce the Self-Adaptive Goal Generation Robust Intelligent Adaptive Curiosity (SAGG-RIAC) architecture as an intrinsically motivated goal exploration mechanism which allows active learning of inverse models in high-dimensional redundant robots. This allows a robot to efficiently and actively learn distributions of parameterized motor skills/policies that solve a corresponding distribution of parameterized tasks/goals. The architecture makes the robot sample actively novel parameterized tasks in the task space, based on a measure of competence progress, each of which triggers low-level goal-directed learning of the motor policy parameters that allow to solve it. For both learning and generalization, the system leverages regression techniques which allow to infer the motor policy parameters corresponding to a given novel parameterized task, and based on the previously learnt correspondences between policy and task parameters.We present experiments with high-dimensional continuous sensorimotor spaces in three different robotic setups: (1) learning the inverse kinematics in a highly-redundant robotic arm, (2) learning omnidirectional locomotion with motor primitives in a quadruped robot, and (3) an arm learning to control a fishing rod with a flexible wire. We show that (1) exploration in the task space can be a lot faster than exploration in the actuator space for learning inverse models in redundant robots; (2) selecting goals maximizing competence progress creates developmental trajectories driving the robot to progressively focus on tasks of increasing complexity and is statistically significantly more efficient than selecting tasks randomly, as well as more efficient than different standard active motor babbling methods; (3) this architecture allows the robot to actively discover which parts of its task space it can learn to reach and which part it cannot.     

L_(2,1)范数正则化的不相关判别分析及其在人脸识别中的应用

对高维数据降维并选取有效特征对分类起着关键作用。针对人脸识别中存在的高维和小样本问题,从特征选取和子空间学习入手,提出了一种L_(2,1)范数正则化的不相关判别分析算法。该算法首先对训练样本矩阵进行奇异值分解;然后通过一系列变换,将原非线性的Fisher鉴别准则函数转化为线性模型;最后加入L_(2,1)范数惩罚项进行求解,得到一组最佳鉴别矢量。将训练样本和测试样本投影到该低维子空间中,利用最近欧氏距离分类器进行分类。由于加入了L_(2,1)范数惩罚项,该算法能使特征选取和子空间学习同时进行,有效改善识别性能。在ORL、YaleB及PIE人脸库上的实验结果表明,算法在有效降维的同时能进一步提高鉴别能力。     

高维大数据分析的无监督异常检测方法

高维数据的无监督异常检测是机器学习的重要挑战之一.虽然先前基于单一深度自动编码器和密度估计的方法已经取得了显著的进展,但是其仅通过一个深度自编码器来生成低维表示,这表明没有足够的信息来执行后续的密度估计任务.为了解决上述问题,文中提出了一种混合自动编码器高斯混合模型(Mixed Auto-encoding Gaussi...     

Data-efficient learning of robotic clothing assistance using Bayesian Gaussian process latent variable model

ABSTRACT

Motor-skill learning for complex robotic tasks is a challenging problem due to the high task variability. Robotic clothing assistance is one such challenging problem that can greatly improve the quality-of-life for the elderly and disabled. In this study, we propose a data-efficient representation to encode task-specific motor-skills of the robot using Bayesian nonparametric latent variable models. The effectivity of the proposed motor-skill representation is demonstrated in two ways: (1) through a real-time controller that can be used as a tool for learning from demonstration to impart novel skills to the robot and (2) by demonstrating that policy search reinforcement learning in such a task-specific latent space outperforms learning in the high-dimensional joint configuration space of the robot. We implement our proposed framework in a practical setting with a dual-arm robot performing clothing assistance tasks.     

Divide, conquer and coordinate: globally coordinated switching linear dynamical system

The goal of this work is to learn a parsimonious and informative representation for high-dimensional time series. Conceptually, this comprises two distinct yet tightly coupled tasks: learning a low-dimensional manifold and modeling the dynamical process. These two tasks have a complementary relationship as the temporal constraints provide valuable neighborhood information for dimensionality reduction and, conversely, the low-dimensional space allows dynamics to be learned efficiently. Solving these two tasks simultaneously allows important information to be exchanged mutually. If nonlinear models are required to capture the rich complexity of time series, then the learning problem becomes harder as the nonlinearities in both tasks are coupled. A divide, conquer, and coordinate method is proposed. The solution approximates the nonlinear manifold and dynamics using simple piecewise linear models. The interactions and coordinations among the linear models are captured in a graphical model. The model structure setup and parameter learning are done using a variational Bayesian approach, which enables automatic Bayesian model structure selection, hence solving the problem of overfitting. By exploiting the model structure, efficient inference and learning algorithms are obtained without oversimplifying the model of the underlying dynamical process. Evaluation of the proposed framework with competing approaches is conducted in three sets of experiments: dimensionality reduction and reconstruction using synthetic time series, video synthesis using a dynamic texture database, and human motion synthesis, classification, and tracking on a benchmark data set. In all experiments, the proposed approach provides superior performance.     

任务间共享和特有结构分解的多任务TSK模糊系统建模

现有的多任务Takagi-Sugeno-Kang (TSK) 模糊建模方法更注重利用任务间的相关性信息,而忽略了单个任务的特殊性。针对此问题,本文提出了一种考虑所有任务之间的共享结构和特有结构的TSK模糊系统多任务建模新方法。该方法将后件参数分解为共享参数和特有参数两个分量,既充分利用了任务间共享信息,又有效地保留了单个任务的特性。最后,本文利用增广拉格朗日乘子法(ALM)求解该最优化问题。实验结果表明,该方法比现有的模型获得了更好的表现。     

Deep imitation learning for 3D navigation tasks

Deep learning techniques have shown success in learning from raw high-dimensional data in various applications. While deep reinforcement learning is recently gaining popularity as a method to train intelligent agents, utilizing deep learning in imitation learning has been scarcely explored. Imitation learning can be an efficient method to teach intelligent agents by providing a set of demonstrations to learn from. However, generalizing to situations that are not represented in the demonstrations can be challenging, especially in 3D environments. In this paper, we propose a deep imitation learning method to learn navigation tasks from demonstrations in a 3D environment. The supervised policy is refined using active learning in order to generalize to unseen situations. This approach is compared to two popular deep reinforcement learning techniques: deep-Q-networks and Asynchronous actor-critic (A3C). The proposed method as well as the reinforcement learning methods employ deep convolutional neural networks and learn directly from raw visual input. Methods for combining learning from demonstrations and experience are also investigated. This combination aims to join the generalization ability of learning by experience with the efficiency of learning by imitation. The proposed methods are evaluated on 4 navigation tasks in a 3D simulated environment. Navigation tasks are a typical problem that is relevant to many real applications. They pose the challenge of requiring demonstrations of long trajectories to reach the target and only providing delayed rewards (usually terminal) to the agent. The experiments show that the proposed method can successfully learn navigation tasks from raw visual input while learning from experience methods fail to learn an effective policy. Moreover, it is shown that active learning can significantly improve the performance of the initially learned policy using a small number of active samples.

     

Privacy Preserving Demand Side Management Method via Multi-Agent Reinforcement Learning

The smart grid utilizes the demand side management technology to motivate energy users towards cutting demand during peak power consumption periods, which greatly improves the operation efficiency of the power grid. However, as the number of energy users participating in the smart grid continues to increase, the demand side management strategy of individual agent is greatly affected by the dynamic strategies of other agents. In addition, the existing demand side management methods, which need to obtain users’ power consumption information, seriously threaten the users’ privacy. To address the dynamic issue in the multi-microgrid demand side management model, a novel multi-agent reinforcement learning method based on centralized training and decentralized execution paradigm is presented to mitigate the damage of training performance caused by the instability of training experience. In order to protect users’ privacy, we design a neural network with fixed parameters as the encryptor to transform the users’ energy consumption information from low-dimensional to high-dimensional and theoretically prove that the proposed encryptor-based privacy preserving method will not affect the convergence property of the reinforcement learning algorithm. We verify the effectiveness of the proposed demand side management scheme with the real-world energy consumption data of Xi’an, Shaanxi, China. Simulation results show that the proposed method can effectively improve users’ satisfaction while reducing the bill payment compared with traditional reinforcement learning (RL) methods (i.e., deep Q learning (DQN), deep deterministic policy gradient (DDPG), QMIX and multi-agent deep deterministic policy gradient (MADDPG)). The results also demonstrate that the proposed privacy protection scheme can effectively protect users’ privacy while ensuring the performance of the algorithm.

     

新的连续自编码网络流形学习研究

发现高维观测数据空间的低维流形结构,是流形学习的主要目标。在前人利用神经网络进行非线性降维的基础上,提出一种新的连续自编码（Continuous Autoencoder,C-Autoencoder）网络,该方法特别采用CRBM（Continuous Restricted Boltzmann Machine）的网络结构,通过训练具有多个中间层的双向深层神经网络可将高维连续数据转换成低维嵌套并继而重构高维连续数据。特别地,这种连续自编码网络可以提供高维连续数据空间和低维嵌套结构的双向映射,不仅有效解决了大多数非线性降维方法所不具备的逆向映射问题,而且特别适用于高维连续数据的降维和重构。将C-Autoencoder用于人工连续数据的实验表明,C-Autoencoder不仅能发现嵌入在高维连续数据中的非线性流形结构,也能有效地从低维嵌套中恢复原始高维连续数据。     

ROS2多线程执行器上DAG任务的优先级分配方法

随着机器人操作系统（robot operating system, ROS）的日益普及,系统也变得更加复杂,这类系统的计算平台正逐渐转变为多核心平台. 在ROS中,任务执行的顺序取决于底层任务调度策略和分配给任务的优先级,而最大限度地缩短所有任务的执行时间是并行系统任务调度的一个重要目标. 受强化学习在解决各种组合优化问题的最新研究成果的启发,在考虑ROS2多线程执行器的调度机制和执行约束的前提下,提出了一种基于强化学习的任务优先级分配方法,该方法提取了基于有向无环图形式表示的任务集的时间和结构特征,通过策略梯度和蒙特卡洛树搜索（Monte Carlo tree search, MCTS）方法有效地学习ROS2调度策略并给出合理的优先级设置方案,最终达到最小化并行任务的最大完工时间的目的. 通过模拟平台环境下随机生成的任务图以评估所提方法,结果表明所提方法明显优于基准方法. 作为一种离线分析方法,所提方法可以很容易地扩展到复杂的ROS中,在可接受的时间内找到接近最优的解决方案.