连续空间增量最近邻时域差分学习

针对连续空间强化学习问题,提出一种基于局部加权学习的增量最近邻时域差分(TD)学习框架。通过增量方式在线选取部分已观测状态构建实例词典,采用新观测状态的范围最近邻实例逼近其值函数与策略,并结合TD算法对词典中各实例的值函数和资格迹迭代更新。就框架各主要组成部分给出多种设计方案,并对其收敛性进行理论分析。对24种方案组合进行仿真验证的实验结果表明, SNDN组合具有较好的学习性能和计算效率。     

Incremental Multi-Step Q-Learning

This paper presents a novel incremental algorithm that combines Q-learning, a well-known dynamic-programming based reinforcement learning method, with the TD() return estimation process, which is typically used in actor-critic learning, another well-known dynamic-programming based reinforcement learning method. The parameter is used to distribute credit throughout sequences of actions, leading to faster learning and also helping to alleviate the non-Markovian effect of coarse state-space quantization. The resulting algorithm, Q()-learning, thus combines some of the best features of the Q-learning and actor-critic learning paradigms. The behavior of this algorithm has been demonstrated through computer simulations.     

Comments on “Co-Evolution in the Successful Learning of Backgammon Strategy”

The results obtained by Pollack and Blair substantially underperform my 1992 TD Learning results. This is shown by directly benchmarking the 1992 TD nets against Pubeval. A plausible hypothesis for this underperformance is that, unlike TD learning, the hillclimbing algorithm fails to capture nonlinear structure inherent in the problem, and despite the presence of hidden units, only obtains a linear approximation to the optimal policy for backgammon. Two lines of evidence supporting this hypothesis are discussed, the first coming from the structure of the Pubeval benchmark program, and the second coming from experiments replicating the Pollack and Blair results.     

基础素描与结构素描的差异性研究

基础素描与结构素描都是艺术基础教学的重要组成部分,本文研究了基础素描和结构素描的学习和表现方法,并从观察方法与构图安排、方法步骤、表现形式和手段、空间观念和细节表现这四个方面分析了基础素描和结构素描之间的差异性。     

平均报酬模型的多步强化学习算法

讨论模型未知的平均报酬强化学习算法。通过结合即时差分学习与Ｒ学习算法,将折扣问题中的一些方法推广到了平均准则问题中,提出了两类算法：Ｒ（λ）学习。现有的Ｒ学习可视为Ｒ（λ）学习和ＴＴＤ（λ）学习当λ＝０时的一个特例。仿真结果表明,λ取中间值的Ｒ（λ）和ＴＴＤ（λ）学习比现有的方法在可靠性与收敛速度上均有提高。     

Technical Update: Least-Squares Temporal Difference Learning

TD./ is a popular family of algorithms for approximate policy evaluation in large MDPs. TD./ works by incrementally updating the value function after each observed transition. It has two major drawbacks: it may make inefficient use of data, and it requires the user to manually tune a stepsize schedule for good performance. For the case of linear value function approximations and = 0, the Least-Squares TD (LSTD) algorithm of Bradtke and Barto (1996, Machine learning, 22:1–3, 33–57) eliminates all stepsize parameters and improves data efficiency.This paper updates Bradtke and Barto's work in three significant ways. First, it presents a simpler derivation of the LSTD algorithm. Second, it generalizes from = 0 to arbitrary values of ; at the extreme of = 1, the resulting new algorithm is shown to be a practical, incremental formulation of supervised linear regression. Third, it presents a novel and intuitive interpretation of LSTD as a model-based reinforcement learning technique.     

Co-Evolution in the Successful Learning of Backgammon Strategy

Following Tesauro's work on TD-Gammon, we used a 4,000 parameter feedforward neural network to develop a competitive backgammon evaluation function. Play proceeds by a roll of the dice, application of the network to all legal moves, and selection of the position with the highest evaluation. However, no backpropagation, reinforcement or temporal difference learning methods were employed. Instead we apply simple hillclimbing in a relative fitness environment. We start with an initial champion of all zero weights and proceed simply by playing the current champion network against a slightly mutated challenger and changing weights if the challenger wins. Surprisingly, this worked rather well. We investigate how the peculiar dynamics of this domain enabled a previously discarded weak method to succeed, by preventing suboptimal equilibria in a meta-game of self-learning.     

基于递推最小二乘法的多步时序差分学习算法

强化学习是一种重要的机器学习方法。为了提高强化学习过程的收敛速度和减少学习过程值函数估计的误差,提出了基于递推最小二乘法的多步时序差分学习算法（RLS-TD（λ））。证明了在满足一定条件下,该算法的权值将以概率1收敛到唯一解,并且得出和证明了值函数估计值的误差应满足的关系式。迷宫实验表明,与RLS-TD（0）算法相比,该算法能加快学习过程的收敛,与传统的TD（λ）算法相比,该算法减少了值函数估计误差,从而提高了精度。     

Learning to Play Chess Using Temporal Differences

In this paper we present TDLEAF(), a variation on the TD() algorithm that enables it to be used in conjunction with game-tree search. We present some experiments in which our chess program KnightCap used TDLEAF() to learn its evaluation function while playing on Internet chess servers. The main success we report is that KnightCap improved from a 1650 rating to a 2150 rating in just 308 games and 3 days of play. As a reference, a rating of 1650 corresponds to about level B human play (on a scale from E (1000) to A (1800)), while 2150 is human master level. We discuss some of the reasons for this success, principle among them being the use of on-line, rather than self-play. We also investigate whether TDLEAF() can yield better results in the domain of backgammon, where TD() has previously yielded striking success.     

中国象棋人机对弈的自学习方法研究

机器博弈被认为是人工智能领域最具挑战性的研究方向之一。中国象棋计算机博弈的难度绝不亚于国际象棋，但是涉足学者太少，具有自学习能力的就更少了。介绍了中国象棋人机对弈原理，给出了近年来几类典型的评估函数学习方法及其原理，通过比较得出了最适合中国象棋使用的学习方法。分析了这些方法尚存在的问题，并提出了未来研究方向。     

一种基于递归最小二乘法的强化学习算法及其应用研究

文章推导了递归最小二乘瞬时差分法,较通常的瞬时差分法有样本使用效率高,收敛速度快,计算量少等特点。并将基于递归最小二乘的强化学习应用于船舶航向控制,克服了通常智能算法的学习需要一定数量样本数据的缺陷,对控制器的参数进行在线学习与调整,可以在一定程度上解决船舶运动中的不确定性问题,仿真结果表明,在有各种分浪流干扰的条件下,船舶航向的控制仍能取得令人满意的效果,说明该算法是有效可行的。     

中国象棋人机对弈的自学习方法研究

机器博弈被认为是人工智能领域最具挑战性的研究方向之一。中国象棋计算机博弈的难度绝不亚于国际象棋,但是涉足学者太少,具有自学习能力的就更少了。介绍了中国象棋人机对弈原理,给出了近年来几类典型的评估函数自学习方法及其原理,通过比较得出了最适合中国象棋使用的学习方法。分析了这些方法尚存在的问题,并提出了未来的研究方向。     

Reinforcement Learning with Replacing Eligibility Traces

The eligibility trace is one of the basic mechanisms used in reinforcement learning to handle delayed reward. In this paper we introduce a new kind of eligibility trace, the replacing trace, analyze it theoretically, and show that it results in faster, more reliable learning than the conventional trace. Both kinds of trace assign credit to prior events according to how recently they occurred, but only the conventional trace gives greater credit to repeated events. Our analysis is for conventional and replace-trace versions of the offline TD(1) algorithm applied to undiscounted absorbing Markov chains. First, we show that these methods converge under repeated presentations of the training set to the same predictions as two well known Monte Carlo methods. We then analyze the relative efficiency of the two Monte Carlo methods. We show that the method corresponding to conventional TD is biased, whereas the method corresponding to replace-trace TD is unbiased. In addition, we show that the method corresponding to replacing traces is closely related to the maximum likelihood solution for these tasks, and that its mean squared error is always lower in the long run. Computational results confirm these analyses and show that they are applicable more generally. In particular, we show that replacing traces significantly improve performance and reduce parameter sensitivity on the "Mountain-Car" task, a full reinforcement-learning problem with a continuous state space, when using a feature-based function approximator.     

神经元动态规划综述

神经元动态规划是近年发展起来的一种优化方法．它采用计算机仿真和函数近似,简化对状态空间的搜索,可以有效克服“维数危机” ,有广阔的应用前景．本文对神经元动态规划作一综述,希望能对相关研究有所帮助．     

Computational Intelligence and Games： Challenges and Opportunities

The last few decades have seen a phenomenal increase in the quality, diversity and pervasiveness of computer games. The worldwide computer games market is estimated to be worth around USD 21bn annually, and is predicted to continue to grow rapidly. This paper reviews some of the recent developments in applying computational intelligence （CI） methods to games, points out some of the potential pitfalls, and suggests some fruitful directions for future research.     

混沌系统的混合遗传神经网络控制

在小扰动控制技术基础上,将暂态误差预测方法和遗传算法结合起来,提出了一种混合遗传神经网络控制非线性混沌系统的新方法(简称HyGANN).通过增强学习训练,HyGANN可产生控制混沌状态的小扰动时间序列信号,Henon映射的计算机仿真结果表明,它不仅有效镇定混沌周期1,2等低周期轨道,还可成功将高周期混轨道变成期望周期行为.     

基于替代传导径迹增强式学习的自主式微直升机控制

随着微电子机械系统（MEMS）的迅猛发展,自主式微直升机的研究也已成为这一领域内的研究热点之一．由于微直升机尺寸的限制,不能安装功能很强的传感器和处理器,难以获得完全的环境信息,所以传统的基于模型的控制方法不适用于环境是动态的自主微直升机控制．基于行为的控制方法采用累次逼近的方法,不需要环境的精确模型,因此系统的稳定性较好．本文采用基于替代传导径迹的增强式学习,结合即时差分方法,提高其学习效率,仿真实验验证了该学习算法的有效性．最后,本文介绍了微直升机控制中存在的一些问题和我们以后的改进方向．     

权重向量的一种新求法

在确定不同元素 (方案 )重要性权重的问题中 ,为避免直接确定重要性比值的困难 ,提出了从先定性判定差异 ,再对差异量化处理的新思路。文中给出了量化的步骤和方法。