基于强化学习的部分线性离散时间系统的最优输出调节

针对同时具有线性外部干扰与非线性不确定性下的离散时间部分线性系统的最优输出调节问题, 提出了仅利用在线数据的基于强化学习的数据驱动控制方法. 首先, 该问题可拆分为一个受约束的静态优化问题和一个动态规划问题, 第一个问题可以解出调节器方程的解. 第二个问题可以确定出控制器的最优反馈增益. 然后, 运用小增益定理证明了存在非线性不确定性离散时间部分线性系统的最优输出调节问题的稳定性. 针对传统的控制方法需要准确的系统模型参数用来解决这两个优化问题, 提出了一种数据驱动离线策略更新算法, 该算法仅使用在线数据找到动态规划问题的解. 然后, 基于动态规划问题的解, 利用在线数据为静态优化问题提供了最优解. 最后, 仿真结果验证了该方法的有效性.     

数据驱动的保证收敛速率最优输出调节

针对具有外部系统扰动的线性离散时间系统的输出调节问题, 提出了可保证收敛速率的数据驱动最优输出调节方法, 包括状态可在线测量系统的基于状态反馈的算法, 与状态不可在线测量系统的基于输出反馈的算法. 首先, 该问题被分解为输出调节方程求解问题与反馈控制律设计问题, 基于输出调节方程的解, 通过引入收敛速率参数, 建立了可保证收敛速率的最优控制问题, 通过求解该问题得到具有保证收敛速率的输出调节器. 之后, 利用强化学习的方法, 设计基于值迭代的数据驱动状态反馈控制器, 学习得到基于状态反馈的最优输出调节器. 对于状态无法在线测量的被控对象, 利用历史输入输出数据对状态进行重构, 并以此为基础设计基于值迭代的数据驱动输出反馈控制器. 仿真结果验证了所提方法的有效性.     

无线网络环境下数据驱动混合选别浓密过程双率控制方法

无线网络环境下赤铁矿混合选别浓密过程控制问题是以底流矿浆泵频率为内环输入,以底流矿浆流量为内环输出外环输入,以底流矿浆浓度为外环输出的非线性串级工业过程控制问题.其外环反馈回路存在丢包,且模型参数难以辨识,故本文利用工业运行过程的在线数据,设计不依赖模型参数的跟踪控制器.首先,利用浓密过程运行在工作点附近的特点进行线性化,对流量过程设计Q-学习控制器,保证流量过程能够跟踪给定的流量设定值;然后采用提升技术,得到统一时间尺度的以底流矿浆流量设定值为输入,以底流矿浆浓度为输出的被控对象;最后,考虑到在无线网络环境下浓度过程存在反馈丢包,当前的状态可能无法获得,故采用史密斯预估器的思想,利用历史的数据估计系统当前的状态,设计丢包Q-学习设定值控制器为流量过程提供最优设定值.通过仿真实验验证所提算法的有效性.     

一类带扰动线性开关系统的误差反馈输出调节问题

研究了带扰动线性开关系统的输出调节问题,在开关系统的一个凸组合系统输出调节问题可解的条件下,设计每个子系统的控制器及开关系统的切换规律,从而给出开关系统输出调节问题的控制策略。     

基于自适应动态规划和梯度下降法的自适应最优输出调节

当设计前馈-反馈控制器来实现连续时间线性系统的自适应最优输出调节时,往往需要依赖调节器方程的解,然而在系统动态不确定的前提下,获得这个方程的解一般非常困难.现有的自适应最优输出调节方法从选取调节方程解的基矩阵出发,通过在线数据学习每个基矩阵所对应的权重.但是,当被控对象或外部系统阶数过高时,这种方法的计算复杂度亦非常高.对此,提出一种新的无模型方法来实现闭环系统的自适应最优输出调节,调节器方程的解由梯度下降法逼近.需要指出的是,该方法通过在线数据而不是模型信息来获得梯度.仿真结果验证了所提出方法的有效性.     

基于强化学习的目标偏移策略跟踪方法

针对计算机视觉领域的目标跟踪问题,提出基于强化学习的目标框偏移决策跟踪方法。近年来目标跟踪主要研究趋势是基于孪生网络或相关滤波的跟踪方法。与现有跟踪方法不同,提出的方法通过将目标跟踪视为动作序贯决策问题,设计动作决策网络,使得该网络可以用于预测目标边框的偏移,连续移动目标框实现目标跟踪。动作决策网络基于强化学习方式训练,使得该决策网络能使用标注不全的数据集训练。实验表明,提出的跟踪方法在视觉目标跟踪(Visual object tracking, VOT)数据集上表现优异,能有效应对多种复杂场景,具备较高的跟踪精度。     

强化学习系统及其基于可靠度最优的学习算法

归纳了强化学习的主要理论方法，提出了一个区分主客观因素的强化学习系统描述，引入了任务域的概念，针对以往强化学习采用的期望最优准则描述任务域能力的不足，考虑了目标水平准则下的首达时间可靠度优准则模型，分别结合随机逼近理论和时间差分理论，提出了基于概率估计的Ｊ－学习和无需建增量Ｒ－学习。     

基于能量控制与近似输出调节的二级倒立摆起摆和跟踪控制

二级倒立摆是一个典型的欠驱动非线性系统,其控制问题具有一定的挑战性.为了解决时变参考信号下二级倒立摆的起摆和跟踪控制问题,本文提出了一种基于能量控制与近似输出调节方法的起摆和三阶控制器设计方案.首先,采用能量控制方法将第1级摆杆从下垂位置摆起到倒立位置附近;其次,采用滑模控制方法将第1级摆杆稳定在倒立位置,同时,采用等效小车与能量控制相结合的方法将第2级摆杆摆起到倒立位置附近;最后,采用基于近似输出调节理论的多项式近似方法设计三阶控制器实现二级倒立摆的位置跟踪控制.仿真和实验结果均验证了该控制方案的有效性.     

无线环境下基于丢包区分算法的流媒体拥塞控制

无线环境中存在的链路质量以及切换等的问题严重影响了拥塞控制机制的性能,该文针对此问题,考虑到实际无线链路的差错控制对上层的影响,提出了一种新的基于(M,K)统计测量方式的端到端丢包区分算法,并基于此设计了适合于无线环境的流媒体拥塞控制机制(WTFRC)。通过NS模拟验证,相对于其他端到端丢包区分算法,该文提出的算法的准确度更高,WTFRC可以在实际的无线环境中保持良好的性能。     

一类非线性输出反馈系统的自适应输出调节问题

本文讨论一类非线性系统的全局鲁棒输出调节问题.假定被控非线性系统的系统输入方向未知,且产生参考或扰动信号的外部系统含未知参数,这为控制律的设计带来了挑战.文章使用自适应控制方法和内模原理,解决了一类相对阶为1的非线性输出反馈系统的输出调节问题,并将结果应用于处理Lorenz系统的渐近跟踪问题.     

基于强化学习的一类具有输入约束非线性系统最优控制

针对部分系统存在输入约束和不可测状态的最优控制问题,本文将强化学习中基于执行–评价结构的近似最优算法与反步法相结合,提出了一种最优跟踪控制策略.首先,利用神经网络构造非线性观测器估计系统的不可测状态.然后,设计一种非二次型效用函数解决系统的输入约束问题.相比现有的最优方法,本文提出的最优跟踪控制方法不仅具有反步法在处理...     

Optimal linear estimation for systems with multiple packet dropouts

This paper is concerned with the optimal linear estimation problem for linear discrete-time stochastic systems with multiple packet dropouts. Based on a packet dropout model, the optimal linear estimators including filter, predictor and smoother are developed via an innovation analysis approach. The estimators are computed recursively in terms of the solution of a Riccati difference equation of dimension equal to the order of the system state plus that of the measurement output. The steady-state estimators are also investigated. A sufficient condition for the convergence of the optimal linear estimators is given. Simulation results show the effectiveness of the proposed optimal linear estimators.     

一类非线性动态系统基于强化学习的最优控制制

提出一类非线性不确定动态系统基于强化学习的最优控制方法. 该方法利用欧拉强化学习算法估计对象的未知非线性函数, 给出了强化学习中回报函数和策略函数迭代的在线学习规则. 通过采用向前欧拉差分迭代公式对学习过程中的时序误差进行离散化, 实现了对值函数的估计和控制策略的改进. 基于值函数的梯度值和时序误差指标值, 给出了该算法的步骤和误差估计定理. 小车爬山问题的仿真结果表明了所提出方法的有效性.

     

Global output regulation for strict‐feedback nonlinear systems with mismatched nonvanishing disturbances

This paper concerns the problem of global output regulation for a class of strict‐feedback nonlinear systems subject to mismatched nonvanishing disturbances. A composite control scheme is developed using a nonlinear disturbance observer‐based control approach. A novel idea is that the disturbance estimation is introduced into the design of virtual control laws in each step. Global stability analysis for the closed‐loop system is presented by the direct Lyapunov function method. It is shown that the system output asymptotically converges to zero in the presence of mismatched nonvanishing disturbances without the requirement of solving any partial differential equations involved with the traditional output regulation theory. An application design example of a single‐machine infinite‐bus system with static var compensator is presented with simulation results to demonstrate the effectiveness of the proposed method. Copyright © 2014 John Wiley & Sons, Ltd.     

Optimal mixed block withholding attacks based on reinforcement learning

The vulnerabilities in cryptographic currencies facilitate the adversarial attacks. Therefore, the attackers have incentives to increase their rewards by strategic behaviors. Block withholding attacks (BWH) are such behaviors that attackers withhold blocks in the target pools to subvert the blockchain ecosystem. Furthermore, BWH attacks may dwarf the countermeasures by combining with selfish mining attacks or other strategic behaviors, for example, fork after withholding (FAW) attacks and power adaptive withholding (PAW) attacks. That is, the attackers may be intelligent enough such that they can dynamically gear their behaviors to optimal attacking strategies. In this paper, we propose mixed-BWH attacks with respect to intelligent attackers, who leverage reinforcement learning to pin down optimal strategic behaviors to maximize their rewards. More specifically, the intelligent attackers strategically toggle among BWH, FAW, and PAW attacks. Their main target is to fine-tune the optimal behaviors, which incur maximal rewards. The attackers pinpoint the optimal attacking actions with reinforcement learning, which is formalized into a Markov decision process. The simulation results show that the rewards of the mixed strategy are much higher than that of honest strategy for the attackers. Therefore, the attackers have enough incentives to adopt the mixed strategy.     

Piecewise output feedback control for affine systems with disturbances based on linear temporal logic specifications

In the paper,we investigate the problem of finding a piecewise output feedback control law for an uncertain affine system such that the resulting closed-loop output satisfies a desired linear temporal logic (LTL) specification.A two-level hierarchical approach is proposed to solve the problem in a triangularized output space.In the lower level,we explore whether there exists a robust output feedback control law to make the output starting in a simplex either remains in it or leaves via a specific facet.In the higher level,for the triangularization,we construct the transition system according to the reachability relationship obtained in the lower level and search for feasible paths that meet the LTL specification.The control approach is then applied to solve a motion planning problem.     

基于即时学习的非线性系统优化控制

基于数据驱动机制的逆控制是一种非线性系统控制方法,关键问题在于局部逆控制模型的准确性,但尚无校验机制来保证其能否产生期望的输出．为此,提出一种k-VNN即时学习算法,提高了逆控制模型的建模精度．将该算法与性能指标优化策略相结合,在线修正逆控制模型顶估的系统控制量。可得到系统的一步最优控制量。实现非线性系统的跟踪控制,为提高控制系统的泛化能力,提出一种数据库数据更新策略．仿真结果表明了所提出方法的有效性．     

具有有限连续丢包网络控制系统的最优线性估值器

本文研究了观测数据和控制输入数据传输具有有限连续丢包的线性离散随机系统的最优估计问题.利用两个满足Bernoulli分布的随机变量来分别描述从传感器到估值器和从控制器到执行器之间的数据丢包现象.通过引入两组新的变量,将原系统转化为一个带有随机参数的系统.利用射影理论,提出了线性最小方差最优线性估值器,包括滤波器、预报器和平滑器.最后研究了稳态线性估值器,并给出了稳态存在的一个充分条件.仿真例子验证了算法的有效性.     

Optimal linear estimators for systems with multiple random measurement delays and packet dropouts

This article is concerned with the optimal linear estimation problem for linear discrete-time stochastic systems with possible multiple random measurement delays and packet dropouts, where the largest random delay is limited within a known bound and packet dropouts can be infinite. A new model is constructed to describe the phenomena of multiple random delays and packet dropouts by employing some random variables of Bernoulli distribution. By state augmentation, the system with random delays and packet dropouts is transferred to a system with random parameters. Based on the new model, the least mean square optimal linear estimators including filter, predictor and smoother are easily obtained via an innovation analysis approach. The estimators are recursively computed in terms of the solutions of a Riccati difference equation and a Lyapunov difference equation. A sufficient condition for the existence of the steady-state estimators is given. An example shows the effectiveness of the proposed algorithms.     

Event-triggered-based integral reinforcement learning output feedback optimal control for partially unknown constrained-input nonlinear systems

In this paper, an adaptive output feedback event-triggered optimal control algorithm is proposed for partially unknown constrained-input continuous-time nonlinear systems. First, a neural network observer is constructed to estimate unmeasurable state. Next, an event-triggered condition is established, and only when the event-triggered condition is violated will the event be triggered and the state be sampled. Then, an event-triggered-based synchronous integral reinforcement learning (ET-SIRL) control algorithm with critic-actor neural networks (NNs) architecture is proposed to solve the event-triggered Hamilton–Jacobi–Bellman equation under the established event-triggered condition. The critic and actor NNs are used to approximate cost function and optimal event-triggered optimal control law, respectively. Meanwhile, the event-triggered-based closed-loop system state and all the neural network weight estimation errors are uniformly ultimately bounded proved by Lyapunov stability theory, and there is no Zeno behavior. Finally, two numerical examples are presented to show the effectiveness of the proposed ET-SIRL control algorithm.