一种基于递归神经网络的自适应控制方法研究

本文针对快速、多变量、强非线性的复杂系统的控制问题,在强化学习方式的基础上,提出一种新的自适应控制方法.该方法在没有先验知识的条件下,基于递归神经网络并结合强化学习的自调节能力,通过自身神经网络的在线学习,有效控制不稳定的非线性系统.本文以一级倒立摆系统为实验对象,仿真实验结果表明所提出的控制方法具有非常好的控制效果和稳定精度,抗干扰能力强.     

基于内部回归神经网络的强化学习

在强化学习和动态规划算法的基础上,提出一种不需要预测模型的改进的强化学习算法：该学习算法在模型未知和没有先验经验的条件下,能通过自身神经网络的在线学习,有效控制不稳定的非线性系统：该学习算法的网络结构采用内部回归神经网络,这样可以增强网络本身处理动态信息的能力,使其更适合复杂系统的稳定控制。应用到一级倒立摆物理实体的控制．实验结果表明了该学习算法在性能上优于其他同类强化学习算法．     

针对纯滞后系统神经网络Dahlin控制的研究

针对传统的基于Dahlin算法的控制器在大惯性、纯滞后、时变性、非线性对象的控制效果不佳,甚至发生不稳定现象的弱点,提出了以CMAC神经网络与Dahlia算法相结合的控制方法.以CMAC神经网络作为一个前馈控制器,实现时滞系统的自适应稳定控制.仿真实验表明,这种复合控制方法保留了Dahlin算法与CMAC神经网络的各自特长,同时具备学习速度快、适应能力强的优点,具有良好的稳定性和控制效果.     

状态变量合成三级倒立摆模糊神经网络控制

为解决模糊神经网络在控制多变量系统时的规则组合爆炸问题,提出了用状态变量合成模糊神经网络控制三级倒立摆的方法。该方法既能解决具有快速、强非线性、绝对不稳定系统的控制问题,又能适用对状态变量可按性质和类型分类的多变量系统的控制,大大减少了模糊神经网络控制器的规则数,有利于利用专家的控制经验。实验结果证实了该方法的控制效果好,鲁棒性强。     

二级倒立摆的状态变量合成模糊神经网络控制

为解决模糊神经网络在控制多变量系统时的规则组合爆炸问题，提出用状态变量合成模糊神经网络控制二级倒立摆。该方法既能解决具有快速、强非线性、绝对不稳定系统的控制问题，又能对状态变量可摈性质和类型分类的多变量系统，大大减少模糊神经网络控制器的规则数，有利于利用专家的控制经验。实验结果证实了该方法的控制效果好，鲁棒性强。     

基于强化学习规则的两轮机器人自平衡控制

两轮机器人是一个典型的不稳定,非线性,强耦合的自平衡系统,在两轮机器人系统模型未知和没有先验经验的条件下,将强化学习算法和模糊神经网络有效结合,保证了函数逼近的快速性和收敛性,成功地实现两轮机器人的自学习平衡控制,并解决了两轮机器人连续状态空间和动作空间的强化学习问题;仿真和实验表明:该方法不仅在很短的时间内成功地完成对两轮机器人的平衡控制,而且在两轮机器人参数变化较大时,仍能维持两轮机器人的平衡。     

基于强化学习补偿器的水下航行器姿态控制

自主式无人水下航行器(AUV)的动力学模型是一个多输入多输出、欠驱动强耦合的非线性系统,同时AUV的工作环境复杂多变,因此,对其姿态进行高精度控制是一个挑战。针对上述问题,本文提出了一个基于强化学习补偿器的AUV姿态控制方法,通过对历史经验数据的学习,实现在野外航行过程中抵抗未建模的不确定扰动和逐步提升姿态控制性能。主要贡献：(1)融合经典控制器和强化学习补偿器,通过经典控制器保障强化学习补偿器在训练过程中的系统稳定,通过训练好的强化学习补偿器抵抗不确定扰动和提升最终性能;(2)改进了传统的二次型的强化学习奖励函数,提升了训练速度和最终控制性能;(3)通过仿真验证了在神经网络权值随机初始化的条件下,本文设计的强化补偿控制器经过训练后可以收敛到稳定一致的性能。     

一类动态递归神经网络的智能控制器

提出一种改进型动态递归神经网络的自适应控制方法，研究了动态递归网络的学习算法，分析了学习算法的收敛性，并推导了保证算法收敛的有效学习率范围，在此基础上提出了模糊推理自适应学习率方法。计算机仿真实验表明，本文控制方法对于未知、非线性被控对象的控制是有效的。     

基于Dahlin算法的CMAC控制器设计

针对传统的基于Dahlin算法的控制器在对变时滞系统进行控制时控制效果恶化,甚至发生不稳定现象的弱点,提出了以CMAC神经网络与Dahlin算法相结合的控制方法。以CMAC神经网络作为一个前馈控制器,通过对Dahlin控制器输出的学习,实现时滞系统的自适应稳定控制。仿真实验表明,这种复合控制方法保留了Dahlin算法与CMAC神经网络的各自特长,同时具备学习速度快,适应能力强的优点,具有良好的稳定性控制效果。     

采样非线性系统的动态神经网络稳定自适应控制

研究了一类采样数据非线性系统的动态神经网络稳定自适应控制方法.不同于静态神经网络自适应控制,动态神经网络自适应控制中神经网络用于逼近整个采样数据非线性系统,而不是动态系统中的非线性分量.系统的控制律由神经网络系统的动态逆、自适应补偿项和神经变结构鲁棒控制项组成.神经变结构控制用于保证系统的全局稳定性,并加速动态神经网络系统的适近速度.证明了动态神经网络自适应控制系统的稳定性,并得到了动态神经网络系统的学习算法.仿真研究表明,基于动态神经网络的非线性系统稳定自适应控制方法较基于静态神经网络的自适应方法具有更好的性能.     

基于径向基神经网络的多步Sarsa控制算法

针对具有连续状态空间的无模型非线性系统,提出一种基于径向基(radial basis function, RBF)神经网络的多步强化学习控制算法.首先,将神经网络引入强化学习系统,利用RBF神经网络的函数逼近功能近似表示状态-动作值函数,解决连续状态空间表达问题;然后,结合资格迹机制形成多步Sarsa算法,通过记录经历过的状态提高系统的学习效率;最后,采用温度参数衰减的方式改进softmax策略,优化动作的选择概率,达到平衡探索和利用关系的目的. MountainCar任务的仿真实验表明:所提出算法经过少量训练能够有效实现无模型情况下的连续非线性系统控制;与单步算法相比,该算法完成任务所用的平均收敛步数更少,效果更稳定,表明非线性值函数近似与多步算法结合在控制任务中同样可以具有良好的性能.     

Reinforcement learning to adaptive control of nonlinear systems

Based on the feedback linearization theory, this paper presents how a reinforcement learning scheme that is adopted to construct artificial neural networks (ANNs) can linearize a nonlinear system effectively. The proposed reinforcement linearization learning system (RLLS) consists of two sub-systems: The evaluation predictor (EP) is a long-term policy selector, and the other is a short-term action selector composed of linearizing control (LC) and reinforce predictor (RP) elements. In addition, a reference model plays the role of the environment, which provides the reinforcement signal to the linearizing process. The RLLS thus receives reinforcement signals to accomplish the linearizing behavior to control a nonlinear system such that it can behave similarly to the reference model. Eventually, the RLLS performs identification and linearization concurrently. Simulation results demonstrate that the proposed learning scheme, which is applied to linearizing a pendulum system, provides better control reliability and robustness than conventional ANN schemes. Furthermore, a PI controller is used to control the linearized plant where the affine system behaves like a linear system.     

Model-free learning control of neutralization processes using reinforcement learning

The pH process dynamic often exhibits severe nonlinear and time-varying behavior and therefore cannot be adequately controlled with a conventional PI control. This article discusses an alternative approach to pH process control using model-free learning control (MFLC), which is based on reinforcement learning algorithms. The MFLC control technique is proposed because this algorithm gives a general solution for acid–base systems, yet is simple enough to be implemented in existing control hardware without a model. Reinforcement learning is selected because it is a learning technique based on interaction with a dynamic system or process for which a goal-seeking control task must be performed. This “on-the-fly” learning is suitable for time varying or nonlinear processes for which the development of a model is too costly, time consuming or even not feasible. Results obtained in a laboratory plant show that MFLC gives good performance for pH process control. Also, control actions generated by MFLC are much smoother than conventional PID controller.     

基于强化学习的模型参考自适应控制

提出了一种基于强化学习的模型参考自适应控制方法,控制器采用自适应启发评价算法,它由两部分组成:自适应评价单元及联想搜索单元.由参考模型给出系统的性能指标,利用系统反馈的强化信号在线更新控制器的参数.仿真结果表明:基于强化学习的模型参考自适应控制方法可以实现对一类复杂的非线性系统的稳定控制和鲁棒控制,该控制方法不仅响应速度快,而且具有较高的学习速率,实时性较强.     

REINFORCEMENT LINEARIZATION CONTROL SYSTEM

The objective of the article is to provide an effective linearization control approach for a nonlinear system. Three reinforcement back propagation learning algorithms (RBPs), based on different step-ahead predictions, are proposed to build the affine linear model of a nonlinear system by means of a composed neural network structure. The approach is used to cancel the effect of nonlinearity of a plant. Reinforcement back propagations can compensate the nonlinearity of the system dynamics between the outputs of the reference model and the system responses. In other words, the role of the composed neural plant is to perform model matching for a linearized system. Based on the derivation of RBPs, a synthetic model, a reinforcement nonlinear control system (RNCS) is developed. This scheme excels the conventional approaches and RBPs. The proposed learning schemes are implemented to linearize a pendulum system. The simulation has been done to illustrate the performance of the proposed learning schemes.     

Multiple model-based reinforcement learning

We propose a modular reinforcement learning architecture for nonlinear, nonstationary control tasks, which we call multiple model-based reinforcement learning (MMRL). The basic idea is to decompose a complex task into multiple domains in space and time based on the predictability of the environmental dynamics. The system is composed of multiple modules, each of which consists of a state prediction model and a reinforcement learning controller. The "responsibility signal," which is given by the softmax function of the prediction errors, is used to weight the outputs of multiple modules, as well as to gate the learning of the prediction models and the reinforcement learning controllers. We formulate MMRL for both discrete-time, finite-state case and continuous-time, continuous-state case. The performance of MMRL was demonstrated for discrete case in a nonstationary hunting task in a grid world and for continuous case in a nonlinear, nonstationary control task of swinging up a pendulum with variable physical parameters.     

Consciousness-driven reinforcement learning: An online learning control framework

As a powerful tool for solving nonlinear complex system control problems, the model-free reinforcement learning hardly guarantees system stability in the early stage of learning, especially with high complicity learning components applied. In this paper, a reinforcement learning framework imitating many cognitive mechanisms of brain such as attention, competition, and integration is proposed to realize sample-efficient self-stabilized online learning control. Inspired by the generation of consciousness in human brain, multiple actors that work either competitively for best interaction results or cooperatively for more accurate modeling and predictions were applied. A deep reinforcement learning implementation for challenging control tasks and a real-time control implementation of the proposed framework are respectively given to demonstrate the high sample efficiency and the capability of maintaining system stability in the online learning process without requiring an initial admissible control.     

基于强化学习算法的智能飞控开发系统

无人机控制器的设计开发是一项复杂的系统工程, 传统的基于代码编程的开发方式存在开发难度大、周期长及错误率高等缺点. 同时, 强化学习智能飞控算法虽在仿真中取得很好的性能, 但在实际中仍缺乏一套完备的开发系统. 本文提出一套基于模型的智能飞控开发系统, 使用模块化编程及自动代码生成技术, 将强化学习算法应用于飞控的嵌入式开发与部署. 该系统可以实现强化学习算法的训练仿真、测试及硬件部署, 旨在提升以强化学习为代表的智能控制算法的部署速度, 同时降低智能飞行控制系统的开发难度.     

仿生机器人运动步态控制:强化学习方法综述

仿生机器人是一类典型的多关节非线性欠驱动系统，其步态控制是一个非常具有挑战性的问题。对于该问题，传统的控制和规划方法需要针对具体的运动任务进行专门设计，需要耗费大量时间和精力，而且所设计出来的控制器往往没有通用性。基于数据驱动的强化学习方法能对不同的任务进行自主学习，且对不同的机器人和运动任务具有良好的通用性。因此，近年来这种基于强化学习的方法在仿生机器人运动步态控制方面获得了不少应用。针对这方面的研究，本文从问题形式化、策略表示方法和策略学习方法3个方面对现有的研究情况进行了分析和总结，总结了强化学习应用于仿生机器人步态控制中尚待解决的问题，并指出了后续的发展方向。