Off-Policy Reinforcement Learning with Gaussian Processes

Abstract—An off-policy Bayesian nonparameteric approximate reinforcement learning framework, termed as GPQ, that employs a Gaussian processes (GP) model of the value (Q) function is presented in both the batch and online settings. Sufficient conditions on GP hyperparameter selection are established to guarantee convergence of off-policy GPQ in the batch setting, and theoretical and practical extensions are provided for the online case. Empirical results demonstrate GPQ has competitive learning speed in addition to its convergence guarantees and its ability to automatically choose its own bases locations.      

Multitask Learning

Multitask Learning is an approach to inductive transfer that improves generalization by using the domain information contained in the training signals of related tasks as an inductive bias. It does this by learning tasks in parallel while using a shared representation; what is learned for each task can help other tasks be learned better. This paper reviews prior work on MTL, presents new evidence that MTL in backprop nets discovers task relatedness without the need of supervisory signals, and presents new results for MTL with k-nearest neighbor and kernel regression. In this paper we demonstrate multitask learning in three domains. We explain how multitask learning works, and show that there are many opportunities for multitask learning in real domains. We present an algorithm and results for multitask learning with case-based methods like k-nearest neighbor and kernel regression, and sketch an algorithm for multitask learning in decision trees. Because multitask learning works, can be applied to many different kinds of domains, and can be used with different learning algorithms, we conjecture there will be many opportunities for its use on real-world problems.     

Semisupervised Multitask Learning

Context plays an important role when performing classification, and in this paper we examine context from two perspectives. First, the classification of items within a single task is placed within the context of distinct concurrent or previous classification tasks (multiple distinct data collections). This is referred to as multi-task learning (MTL), and is implemented here in a statistical manner, using a simplified form of the Dirichlet process. In addition, when performing many classification tasks one has simultaneous access to all unlabeled data that must be classified, and therefore there is an opportunity to place the classification of any one feature vector within the context of all unlabeled feature vectors; this is referred to as semi-supervised learning. In this paper we integrate MTL and semi-supervised learning into a single framework, thereby exploiting two forms of contextual information. Example results are presented on a "toy" example, to demonstrate the concept, and the algorithm is also applied to three real data sets.     

一种可并行的贝叶斯集合在线学习算法

无论是Boosting还是Bagging算法，在使用连续样本集进行分类器集合学习时，均需缓存大量数据，这对大容量样本集的应用不可行。本文提出一种基于贝叶斯集合的在线学习算法BEPOL，在保持Boosting算法加权采样思想的前提下，只需对样本集进行一次扫描，就可实现对贝叶斯集合的在线更新学习。算法针对串行训练时间长、成员相关性差的缺点，采用了并行学习的思想，通过将各贝叶斯分量映射到并行计算结构上，提高集合学习的效率。通过UCI数据集的实验表明，算法BEPOL具有与批量学习算法相近的分类性能和更小的时间开销，这使得算法对某些具有时间和空间限制的应用，如大型数据集或连续型数据集应用尤其有效。     

How Online Learning Approaches Ornstein Uhlenbeck Processes

We show that under reasonable conditions, online learning near a local minimum is similar to a multivariate Ornstein Uhlenbeck process. This implies that the parameter state oscillates randomly around the minimum point, with a Gaussian limiting distribution. We also develop a simple hypothesis test that detects Ornstein Uhlenbeck properties without storing the history of the learning process.     

贝叶斯网络参数的在线学习算法及应用

以EM算法为基础，在给定贝叶斯网络结构情况下。研究分析了Voting EM算法并利用该算法对防洪决策贝叶斯网络进行在线参数学习，将该算法与EM算法的学习结果进行了比较分析，结果表明Voting EM算法不但能够进行在线参数学习，而且也具有较高的学习精度．     

一种大规模离散空间中的高斯强化学习方法

针对大规模离散空间中强化学习的"维数灾"问题,即状态空间的大小随着特征的增加而发生指教级的增长,提出了一种基于高斯过程的强化学习方法.在本方法中,高斯过程模型有表示函数分布的能力,使用该模型之后,可以得到的不只是一个所需的估计值,而是关于该值的一个分布.实验结果表明,结合了高斯过程的强化学习方法在各方面性能,如收敛速度以及最终实验效果等都有所提高.使用高斯方法的回归模型可以在一定程度上解决大规模离散空间上的"维数灾"问题.     

基于深度表示学习和高斯过程迁移学习的情感分析方法

情感分析是自然语言处理领域的重要研究问题。现有方法往往难以克服样本偏置与领域依赖问题,严重制约了情感分析的发展和应用。为此,该文提出了一种基于深度表示学习和高斯过程知识迁移学习的情感分析方法。该方法首先利用深度神经网络获得文本样本的分布式表示,而后基于深度高斯过程,从辅助数据中迁移与测试集数据分布相符的高质量样例扩充训练数据集用于分类器训练,以此提高文本情感分类系统性能。在COAE2014文本情感分类数据集上进行的实验结果显示,该文提出的方法可以有效提高文本情感分类性能,同时可以有效缓解训练数据的样本偏置以及领域依赖问题的影响。     

Upper and Lower Bounds on the Learning Curve for Gaussian Processes

In this paper we introduce and illustrate non-trivial upper and lower bounds on the learning curves for one-dimensional Guassian Processes. The analysis is carried out emphasising the effects induced on the bounds by the smoothness of the random process described by the Modified Bessel and the Squared Exponential covariance functions. We present an explanation of the early, linearly-decreasing behavior of the learning curves and the bounds as well as a study of the asymptotic behavior of the curves. The effects of the noise level and the lengthscale on the tightness of the bounds are also discussed.     

贝叶斯网络结构在线学习算法研究及应用

现代大型机电设备的日趋复杂化和自动化导致设备故障现象和机理之间具有很大的不确定性,因此对故障诊断技术提出了更高的要求。针对汽车发动机的工作原理及其故障知识结构特征,基于贝叶斯网络理论,以机器学习中的增量学习为基础提出和研究了在线式贝叶斯网络结构学习方法,并利用该方法对汽车发动机故障结构网络进行在线学习。最后通过实验分析验证了在线式贝叶斯网络故障诊断方法比起传统的贝叶斯网络方法以及专家系统方法,该方法在汽车发动机故障诊断结果中具有更高的准确性和可靠性。     

时变过程在线辨识的即时递推核学习方法研究

为了及时跟踪非线性化工过程的时变特性, 提出即时递推核学习 (Kernel learning, KL)的在线辨识方法. 针对待预测的新样本点, 采用即时学习 (Just-in-time kernel learning, JITL)策略, 通过构造累积相似度因子, 选择与其相似的样本集建立核学习辨识模型. 为避免传统即时学习对每个待预测点都重新建模的繁琐, 利用两个临近时刻相似样本集的异同点, 采用递推方法有效添加新样本, 并删减旧模型的样本, 以快速建立新即时模型. 通过一时变连续搅拌釜式反应过程的在线辨识, 表明了所提出方法在保证计算效率的同时, 较传统递推核学习方法提高了辨识的准确程度, 能更好地辨识时变过程.     

基于Laplace逼近Gaussian过程的指节图像中层偏移测度特征学习

在人机协调装配中,为了准确描述手部位姿,需要精确的指节图像特征提取与识 别。为了丰富手部信息,提出了基于 Laplace 逼近 Gaussian 过程的多分类算法,以实现基于手 部图像的指节识别。在类别信息无关联的假设基础上,将中层偏移测度特征的学习转化为对随 机量的学习;然后通过分析二值多分类高斯场上的后验计算,给出了基于 Laplace 逼近 Gaussian 过程的多分类高斯过程学习算法;通过构造中层随机信息的正定核函数,给出了基于 Laplace 的多分类高斯过程预测算法。最后,利用中层数据的分布学习与预测算法进行了指节图像训练 学习和固定阈值的图像识别。识别结果显示,该方法具有一定的指节识别能力。     

基于迁移学习的多任务分配算法

对于多任务分配问题，传统的方法针对每一个任务独立地寻找一个最优分配方案，没有考虑任务间的关联以及历史经验对新任务分配的影响，因而复杂度较高。研究了多智能体系统中的多任务分配问题，通过迁移学习来加速任务分配以及子任务的完成。在分配目标任务时，通过计算当前任务和历史任务的相似度找到最适合的源任务，再将源任务的分配模式迁移到目标任务中，并在完成子任务的过程中使用迁移学习，从而提高效率，节约时间。最后，通过“格子世界”的实验证明了该算法在运行时间和平均带折扣回报方面都优于基于Q学习的任务分配算法。     

高斯渐进贝叶斯滤波器

渐进贝叶斯方法将先验分布到后验分布的演化描述为一阶动态系统,通过在伪时间上连续地引入观测信息实现后验状态估计.该方法的一般形式解,即动态系统的时间导数,是难以得到的.本文提出一种高斯型渐进贝叶斯滤波器.首先在线性高斯条件下推导了时间导数的解析解;然后证明了在该条件下,由该解析解确定的一阶动态系统与常量状态估计的Kalman-Bucy滤波器是一致的,且由此导出的高斯渐进贝叶斯滤波器与卡尔曼滤波器是一致的.最后利用一阶Taylor展开推导了滤波器在非线性高斯条件下的近似解表达式,并采用Monte Carlo方法给出了具体实现方法.通过若干仿真算例表明,新滤波器具有较高的精度,且在一定精度条件下的时间复杂度低于一般粒子滤波器.     

Tweet Retweet Prediction Based on Deep Multitask Learning

Neural Processing Letters - Today, as social networks play an increasingly important role, people are more likely to use them to discuss hot topics. Thus, reposting behavior plays a crucial role in...     

面向合同信息抽取的动态多任务学习方法

对于合同文本中要素和条款两类信息的准确提取, 可以有效提升合同的审查效率, 为贸易各方提供便利化服务. 然而当前的合同信息抽取方法一般训练单任务模型对要素和条款分别进行抽取, 并没有深挖合同文本的特征, 忽略了不同任务间的关联性. 因此, 采用深度神经网络结构对要素抽取和条款抽取两个任务间的相关性进行研究, 并提出多任务学习方法. 所提方法首先将上述两种任务进行融合, 构建一种应用于合同信息抽取的基本多任务学习模型; 然后对其进行优化, 利用Attention机制进一步挖掘其相关性, 形成基于Attention机制的动态多任务学习模型; 最后针对篇章级合同文本中复杂的语义环境, 在前两者的基础上提出一种融合词汇知识的动态多任务学习模型. 实验结果表明, 所提方法可以充分捕捉任务间的共享特征, 不仅取得了比单任务模型更好的信息抽取结果, 而且能够有效解决合同文本中要素与条款间实体嵌套的问题, 实现合同要素与条款的信息联合抽取. 此外, 为了验证该方法的鲁棒性, 在多个领域的公开数据集上进行实验, 结果表明该方法的效果均优于基线方法.     

基于多任务学习的邮件过滤系统的研究

随着电子邮件的广泛使用,如何有效地避免和防范垃圾邮件的侵扰已成为一个亟待解决的问题.受机器学习在邮件过滤中研究和应用的启发,利用多任务学习(multitask learning)的特性,将判断一个用户的邮件是否为垃圾邮件看作一个任务(task),利用多任务学习中任务相关性假设,提出一种基于多任务学习的邮件过滤系统.实验表明,该系统对中英文邮件语料都是可靠和有效的,尤其对于同一邮件列表(mail list)中的用户的邮件.     

Gaussian Processes for Object Categorization

Discriminative methods for visual object category recognition are typically non-probabilistic, predicting class labels but not directly providing an estimate of uncertainty. Gaussian Processes (GPs) provide a framework for deriving regression techniques with explicit uncertainty models; we show here how Gaussian Processes with covariance functions defined based on a Pyramid Match Kernel (PMK) can be used for probabilistic object category recognition. Our probabilistic formulation provides a principled way to learn hyperparameters, which we utilize to learn an optimal combination of multiple covariance functions. It also offers confidence estimates at test points, and naturally allows for an active learning paradigm in which points are optimally selected for interactive labeling. We show that with an appropriate combination of kernels a significant boost in classification performance is possible. Further, our experiments indicate the utility of active learning with probabilistic predictive models, especially when the amount of training data labels that may be sought for a category is ultimately very small.     

Online Gaussian Process State-space Model: Learning and Planning for Partially Observable Dynamical Systems

International Journal of Control, Automation and Systems - This paper proposes an online learning method of Gaussian process state-space model (GP-SSM). GP-SSM is a probabilistic representation...     

面向轨道分割与侵入物检测的多任务学习

轨道车智能防护会涉及轨道车侵入物检测与行驶区域分割任务,在深度学习领域已有针对各任务的算法,却无法很好满足多任务情形时的需求.该算法使用轻量级卷积神经网络(CNN)作为编码器提取特征图,随之将特征图送到两个基于one-stage检测网络的解码器中,进而完成各自的任务.不同级别和尺度的语义特征在编码器输出的特征图中被融合,良好地完成像素级语义预测,在检测和分割效果上有明显提升.采用本算法的设备将掌握对新目标的识别检测判断与追踪,为提升轨道车行驶安全做出保障.