连续状态自适应离散化基于K-均值聚类的强化学习方法

使用聚类算法对连续状态空间进行自适应离散化．得到了基于K-均值聚类的强化学习方法．该方法的学习过程分为两部分：对连续状态空间进行自适应离散化的状态空间学习,使用K-均值聚类算法;寻找最优策略的策略学习．使用替代合适迹Sarsa学习算法．对连续状态的强化学习基准问题进行仿真实验,结果表明该方法能实现对连续状态空间的自适应离散化,并最终学习到最优策略．与基于CMAC网络的强化学习方法进行比较．结果表明该方法具有节省存储空间和缩短计算时间的优点．     

基于LSTM神经网络的管道缺陷模式识别方法研究

针对复杂环境下,管道振动信号特征微弱难以提取的问题,提出一种基于长短时记忆网络(LSTM)深度学习神经网络的管道缺陷模式识别方法;首先利用改进型自适应噪声的完全集合经验模态分解(ICEEMDAN)对采集的原始信号进行分解得到若干个固有模态函数(IMF)分量,随后根据信息熵理论计算IMF分量的近似熵作为管道典型状态的特征值构造特征向量集合,然后构造LSTM深度学习神经网络训练模型并调节深度神经网络在训练过程中的相关参数进行网络的结构优化,最后将特征向量输入到LSTM神经网络模型进行训练和识别;结果表明:针对管道振动信号特征微弱难以提取的问题,该方法对管道缺陷模式识别的准确率达到了95％,在消除管道振动信号的背景噪声、挖掘特征信息和保证识别准确性方面优势明显.     

gBoost: a mathematical programming approach to graph classification and regression

Graph mining methods enumerate frequently appearing subgraph patterns, which can be used as features for subsequent classification or regression. However, frequent patterns are not necessarily informative for the given learning problem. We propose a mathematical programming boosting method (gBoost) that progressively collects informative patterns. Compared to AdaBoost, gBoost can build the prediction rule with fewer iterations. To apply the boosting method to graph data, a branch-and-bound pattern search algorithm is developed based on the DFS code tree. The constructed search space is reused in later iterations to minimize the computation time. Our method can learn more efficiently than the simpler method based on frequent substructure mining, because the output labels are used as an extra information source for pruning the search space. Furthermore, by engineering the mathematical program, a wide range of machine learning problems can be solved without modifying the pattern search algorithm.     

基于频繁项挖掘的贝叶斯网络结构学习算法BNSL-FIM

贝叶斯网络能够表示不确定知识并进行推理计算表达,但由于实际样本数据存在噪声和大小限制以及网络空间搜索的复杂性,贝叶斯网络结构学习始终会存在一定的误差。为了提高贝叶斯网络结构学习的准确度,提出了以最大频繁项集和关联规则分析结果为先验知识的贝叶斯网络结构学习算法BNSL-FIM 。首先从数据中挖掘出最大频繁项集并对该项集进行结构学习,之后使用关联规则分析结果对其进行校正,从而确定基于频繁项挖掘和关联规则分析的先验知识。然后提出一种融合先验知识的BDeu评分算法进行贝叶斯网络结构学习。最后在6个公开标准的数据集上开展了实验,并对比引入先验/不引入先验的结构与原始网络结构的汉明距离,结果表明所提算法与未引入先验的BDeu评分算法相比显著提高了贝叶斯网络结构学习的准确度。     

面向资源分配问题的Q-CF多智能体强化学习

多智能体强化学习算法在用于复杂的分布式系统时存在着状态空间大、学习效率低等问题.针对网络环境中的资源分配问题对多智能体强化学习算法进行了研究,将Q-学习算法和链式反馈(chain feedback,CF)学习算法相结合,提出了Q-CF多智能体强化学习算法,利用一种称为信息链式反馈的机制实现了多智能体之间的高效协同.仿真...     

基于卷积计算的多层脉冲神经网络的监督学习

针对脉冲神经元基于精确定时的多脉冲编码信息的特点,提出了一种基于卷积计算的多层脉冲神经网络监督学习的新算法。该算法应用核函数的卷积计算将离散的脉冲序列转换为连续函数,在多层前馈脉冲神经网络结构中,使用梯度下降的方法得到基于核函数卷积表示的学习规则,并用来调整神经元连接的突触权值。在实验部分,首先验证了该算法学习脉冲序列的效果,然后应用该算法对Iris数据集进行分类。结果显示,该算法能够实现脉冲序列复杂时空模式的学习,对非线性模式分类问题具有较高的分类正确率。     

强化学习在足球机器人基本动作学习中的应用

主要研究了强化学习算法及其在机器人足球比赛技术动作学习问题中的应用．强化学习的状态空间 和动作空间过大或变量连续,往往导致学习的速度过慢甚至难于收敛．针对这一问题,提出了基于T-S 模型模糊 神经网络的强化学习方法,能够有效地实现强化学习状态空间到动作空间的映射．此外,使用提出的强化学习方 法设计了足球机器人的技术动作,研究了在不需要专家知识和环境模型情况下机器人的行为学习问题．最后,通 过实验证明了所研究方法的有效性,其能够满足机器人足球比赛的需要．     

Reinforcement learning of a continuous motor sequence with hidden states

Reinforcement learning is the scheme for unsupervised learning in which robots are expected to acquire behavior skills through self-explorations based on reward signals. There are some difficulties, however, in applying conventional reinforcement learning algorithms to motion control tasks of a robot because most algorithms are concerned with discrete state space and based on the assumption of complete observability of the state. Real-world environments often have partial observablility; therefore, robots have to estimate the unobservable hidden states. This paper proposes a method to solve these two problems by combining the reinforcement learning algorithm and a learning algorithm for a continuous time recurrent neural network (CTRNN). The CTRNN can learn spatio-temporal structures in a continuous time and space domain, and can preserve the contextual flow by a self-organizing appropriate internal memory structure. This enables the robot to deal with the hidden state problem. We carried out an experiment on the pendulum swing-up task without rotational speed information. As a result, this task is accomplished in several hundred trials using the proposed algorithm. In addition, it is shown that the information about the rotational speed of the pendulum, which is considered as a hidden state, is estimated and encoded on the activation of a context neuron.     

基于一维卷积循环神经网络的深度强化学习算法

针对现有深度强化学习算法在状态空间维度大的环境中难以收敛的问题,提出了在时间维度上提取特征的基于一维卷积循环网络的强化学习算法;首先在深度Q网络(DQN,deep Q network)的基础上构建一个深度强化学习系统;然后在深度循环Q网络(DRQN,deep recurrent Q network)的神经网络结构基础上加入了一层一维卷积层,用于在长短时记忆(LSTM,long short-term memory)层之前提取时间维度上的特征;最后在与时序相关的环境下对该新型强化学习算法进行训练和测试;实验结果表明这一改动可以提高智能体的决策水平,并使得深度强化学习算法在非图像输入的时序相关环境中有更好的表现。     

A hybrid neural network classifier combining ordered fuzzy ARTMAP and the dynamic decay adjustment algorithm

This paper presents a novel conflict-resolving neural network classifier that combines the ordering algorithm, fuzzy ARTMAP (FAM), and the dynamic decay adjustment (DDA) algorithm, into a unified framework. The hybrid classifier, known as Ordered FAMDDA, applies the DDA algorithm to overcome the limitations of FAM and ordered FAM in achieving a good generalization/performance. Prior to network learning, the ordering algorithm is first used to identify a fixed order of training patterns. The main aim is to reduce and/or avoid the formation of overlapping prototypes of different classes in FAM during learning. However, the effectiveness of the ordering algorithm in resolving overlapping prototypes of different classes is compromised when dealing with complex datasets. Ordered FAMDDA not only is able to determine a fixed order of training patterns for yielding good generalization, but also is able to reduce/resolve overlapping regions of different classes in the feature space for minimizing misclassification during the network learning phase. To illustrate the effectiveness of Ordered FAMDDA, a total of ten benchmark datasets are experimented. The results are analyzed and compared with those from FAM and Ordered FAM. The outcomes demonstrate that Ordered FAMDDA, in general, outperforms FAM and Ordered FAM in tackling pattern classification problems.     

An Improved Prototype Pattern Selection Algorithm

［1］Wang Hai Long,Qi Feihu. The Learning Algorithm Based on Superposition of Information. Journal of Infrared Millim. Waves,2000,19(6) ［2］H. Haken. Synergetic Computers and Cognition, Vol. 50 of Springer Series Synergetics. Springer,Berlin, 1991 ［3］Wagne     

一种基于编码关联的快速多模式匹配算法

多模式匹配算法经常使用有限自动状态机来实现多个模式串的并行匹配。针对基于自动状态机的多模式匹配算法在应用于中文编码时存在的存储空间膨胀问题,使用中文字符的拆分编码构造自动状态机,以优化算法自动状态机的存储空间,并利用中文编码的编码关联性,设计了一种基于编码关联跳转的失效跳转表,使用启发式跳跃规则提升匹配算法的时间性能。最后通过实验证明,中文编码环境下,相比于其它使用自动状态机的多模式匹配算法,改良算法拥有更小的空间消耗与更快的运行速度。     

基于径向基神经网络的多步Sarsa控制算法

针对具有连续状态空间的无模型非线性系统,提出一种基于径向基(radial basis function, RBF)神经网络的多步强化学习控制算法.首先,将神经网络引入强化学习系统,利用RBF神经网络的函数逼近功能近似表示状态-动作值函数,解决连续状态空间表达问题;然后,结合资格迹机制形成多步Sarsa算法,通过记录经历过的状态提高系统的学习效率;最后,采用温度参数衰减的方式改进softmax策略,优化动作的选择概率,达到平衡探索和利用关系的目的. MountainCar任务的仿真实验表明:所提出算法经过少量训练能够有效实现无模型情况下的连续非线性系统控制;与单步算法相比,该算法完成任务所用的平均收敛步数更少,效果更稳定,表明非线性值函数近似与多步算法结合在控制任务中同样可以具有良好的性能.     

基于深度强化学习求解作业车间机器与 AGV联合调度问题

针对作业车间中自动引导运输车(automated guided vehicle, AGV)与机器联合调度问题,以完工时间最小化为目标,提出一种基于卷积神经网络和深度强化学习的集成算法框架.首先,对含AGV的作业车间调度析取图进行分析,将问题转化为一个序列决策问题,并将其表述为马尔可夫决策过程.接着,针对问题的求解特点,设计一种基于析取图的空间状态与5个直接状态特征;在动作空间的设置上,设计包含工序选择和AGV指派的二维动作空间;根据作业车间中加工时间与有效运输时间为定值这一特点,构造奖励函数来引导智能体进行学习.最后,设计针对二维动作空间的2D-PPO算法进行训练和学习,以快速响应AGV与机器的联合调度决策.通过实例验证,基于2D-PPO算法的调度算法具有较好的学习性能和可扩展性效果.     

一种基于联合神经网络的连续空间行动者评论家学习方法

在复杂的连续空间应用场景中,经典的离散空间强化学习方法已难以满足实际需要,而已有的连续空间强化学习方法主要采用线性拟合方法逼近状态值函数和动作选择函数,存在精度不高的问题。提出一种基于联合神经网络非线性行动者评论家方法(actor-critic approach based on union neural network, UNN-AC)。该方法将动作选择函数和评论值函数表示为统一的联合神经网络模型,利用联合神经网络非线性拟合状态值函数和动作选择概率。与已有的线性拟合方法相比,非线性UNN-AC提高了对评论值函数和动作选择函数的拟合精度。实验结果表明,UNN-AC算法能够有效求解连续空间中近似最优策略问题。与经典的连续动作空间算法相比,该算法具有收敛速度快和稳定性高的优点。     

基于启发式奖赏塑形方法的智能化攻击路径发现

渗透测试作为一种评估网络系统安全性能的重要手段, 是以攻击者的角度模拟真实的网络攻击, 找出网络系统中的脆弱点。而自动化渗透测试则是利用各种智能化方法实现渗透测试过程的自动化, 从而大幅降低渗透测试的成本。攻击路径发现作为自动化渗透测试中的关键技术, 如何快速有效地在网络系统中实现智能化攻击路径发现, 一直受到学术界的广泛关注。现有的自动化渗透测试方法主要基于强化学习框架实现智能化攻击路径发现, 但还存在奖赏稀疏、学习效率低等问题, 导致算法收敛速度慢, 攻击路径发现难以满足渗透测试的高时效性需求。为此, 提出一种基于势能的启发式奖赏塑形函数的分层强化学习算法(HRL-HRSF), 该算法首先利用渗透测试的特性, 根据网络攻击的先验知识提出了一种基于深度横向渗透的启发式方法, 并利用该启发式方法设计出基于势能的启发式奖赏塑形函数, 以此为智能体前期探索提供正向反馈, 有效缓解了奖赏稀疏的问题;然后将该塑形函数与分层强化学习算法相结合, 不仅能够有效减少环境状态空间与动作空间大小, 还能大幅度提高智能体在攻击路径发现过程中的奖赏反馈, 加快智能体的学习效率。实验结果表明, HRL-HRSF 相较于没有奖赏塑形的分层强化学习算法、DQN 及其改进算法更加快速有效, 并且随着网络规模和主机漏洞数目的增大, HRL-HRSF 均能保持更好地学习效率, 拥有良好的鲁棒性和泛化性。     

无项头表的FP-Growth算法

针对FP-Growth算法中频繁模式树的遍历低效问题,提出了一种无项头表的频繁模式增长算法。该算法利用递归回溯的方式遍历频繁模式树以求取条件模式基,解决了对同一树路径多次重复遍历的问题。从理论分析和实际挖掘能力两方面,将新算法与FP-Growth算法进行了对比。结果表明,新算法有效减少了条件模式基的搜索开销,使频繁模式挖掘的效率提高了2~5倍,在时间和空间性能上均优于FP-Growth算法。将该算法应用于通信告警关联规则挖掘,较快地挖掘出了关联规则结果,且正确规则的覆盖率达到了83.3%。     

Design and analysis of a nonequilibrium cross-coupled network witha detectable similarity measure

In this paper, a nonequilibrium network which works as a dynamical associative memory is designed. The design is based on a new similarity measure between any stored pattern and a state of the network. Although conventional similarity measures, such as Hamming distance, direction cosine, and so on, are not detectable in a cross-coupled network, the similarity measure proposed in this paper is. The new similarity measure is employed in our design. The network should include the following properties in its output pattern sequence, so that the dynamics of cross-coupled network may be designed: 1) Stored patterns are frequently associated in the dynamical association. 2) The dynamical association is very robust against variation of distributed parameters. Property 1) is achieved by introducing the next two operation modes with inverse N-shaped function into the dynamics of the proposed network, 1) When the state of the network is close enough to a stored pattern at a time step, the state is forced to evolve at the next time step, 2) The state of the network converges to a stored one while it is not close to any stored patterns. By considering these two operation modes, the frequency of associating stored patterns is increased. The authors emphasize the property 2) which is very important for a silicon implementation of the proposed network. In the silicon implementation, parameters of the network must be represented by transistors, resistors, capacitors, and other electric components which exhibit variation in their characteristics. Thus the second property guarantees the easy silicon implementation of the nonequilibrium network proposed in this paper.     

Artificial neural networks for feature extraction and multivariatedata projection

Classical feature extraction and data projection methods have been well studied in the pattern recognition and exploratory data analysis literature. We propose a number of networks and learning algorithms which provide new or alternative tools for feature extraction and data projection. These networks include a network (SAMANN) for J.W. Sammon's (1969) nonlinear projection, a linear discriminant analysis (LDA) network, a nonlinear discriminant analysis (NDA) network, and a network for nonlinear projection (NP-SOM) based on Kohonen's self-organizing map. A common attribute of these networks is that they all employ adaptive learning algorithms which makes them suitable in some environments where the distribution of patterns in feature space changes with respect to time. The availability of these networks also facilitates hardware implementation of well-known classical feature extraction and projection approaches. Moreover, the SAMANN network offers the generalization ability of projecting new data, which is not present in the original Sammon's projection algorithm; the NDA method and NP-SOM network provide new powerful approaches for visualizing high dimensional data. We evaluate five representative neural networks for feature extraction and data projection based on a visual judgement of the two-dimensional projection maps and three quantitative criteria on eight data sets with various properties.