Reinforcement learning-based event-triggered optimal control for unknown nonlinear systems with input delay

The optimal control issue of discrete-time nonlinear unknown systems with time-delay control input is the focus of this work. In order to reduce communication costs, a reinforcement learning-based event-triggered controller is proposed. By applying the proposed control method, closed-loop system's asymptotic stability is demonstrated, and a maximum upper bound for the infinite-horizon performance index can be calculated beforehand. The event-triggered condition requires the next time state information. In an effort to forecast the next state and achieve optimal control, three neural networks (NNs) are introduced and used to approximate system state, value function, and optimal control. Additionally, a M NN is utilized to cope with the time-delay term of control input. Moreover, taking the estimation errors of NNs into account, the uniformly ultimately boundedness of state and NNs weight estimation errors can be guaranteed. Ultimately, the validity of proposed approach is illustrated by simulations.     

Distributed event-triggered optimal bipartite consensus control for multiagent systems with input delay via reinforcement learning method

In this article, an optimal bipartite consensus control (OBCC) scheme is proposed for heterogeneous multiagent systems (MASs) with input delay by reinforcement learning (RL) algorithm. A directed signed graph is established to construct MASs with competitive and cooperative relationships, and model reduction method is developed to tackle input delay problem. Then, based on the Hamilton–Jacobi–Bellman (HJB) equation, policy iteration method is utilized to design the bipartite consensus controller, which consists of value function and optimal controller. Further, a distributed event-triggered function is proposed to increase control efficiency, which only requires information from its own agent and neighboring agents. Based on the input-to-state stability (ISS) function and Lyapunov function, sufficient conditions for the stability of MASs can be derived. Apart from that, RL algorithm is employed to solve the event-triggered OBCC problem in MASs, where critic neural networks (NNs) and actor NNs estimate value function and control policy, respectively. Finally, simulation results are given to validate the feasibility and efficiency of the proposed algorithm.     

非线性零和微分对策的事件触发自适应动态规划算法

针对一类非线性零和微分对策问题,本文提出了一种事件触发自适应动态规划(event-triggered adaptive dynamic programming,ET--ADP)算法在线求解其鞍点.首先,提出一个新的自适应事件触发条件.然后,利用一个输入为采样数据的神经网络(评价网络)近似最优值函数,并设计了新型的神经网络权值更新律使得值函数、控制策略及扰动策略仅在事件触发时刻同步更新.进一步地,利用Lyapunov稳定性理论证明了所提出的算法能够在线获得非线性零和微分对策的鞍点且不会引起Zeno行为.所提出的ET--ADP算法仅在事件触发条件满足时才更新值函数、控制策略和扰动策略,因而可有效减少计算量和降低网络负荷.最后,两个仿真例子验证了所提出的ET--ADP算法的有效性.     

离散非线性零和博弈的事件驱动最优控制方案

在求解离散非线性零和博弈问题时,为了在有效降低网络通讯和控制器执行次数的同时保证良好的控制效果,本文提出了一种基于事件驱动机制的最优控制方案.首先,设计了一个采用新型事件驱动阈值的事件驱动条件,并根据贝尔曼最优性原理获得了最优控制对的表达式.为了求解该表达式中的最优值函数,提出了一种单网络值迭代算法.利用一个神经网络构建评价网.设计了新的评价网权值更新规则.通过在评价网、控制策略及扰动策略之间不断迭代,最终获得零和博弈问题的最优值函数和最优控制对.然后,利用Lyapunov稳定性理论证明了闭环系统的稳定性.最后,将该事件驱动最优控制方案应用到了两个仿真例子中,验证了所提方法的有效性.     

Event-Triggered Optimal Adaptive Control Algorithm for Continuous-Time Nonlinear Systems

This paper proposes a novel optimal adaptive eventtriggered control algorithm for nonlinear continuous-time systems. The goal is to reduce the controller updates, by sampling the state only when an event is triggered to maintain stability and optimality. The online algorithm is implemented based on an actor/critic neural network structure. A critic neural network is used to approximate the cost and an actor neural network is used to approximate the optimal event-triggered controller. Since in the algorithm proposed there are dynamics that exhibit continuous evolutions described by ordinary differential equations and instantaneous jumps or impulses, we will use an impulsive system approach. A Lyapunov stability proof ensures that the closed-loop system is asymptotically stable. Finally, we illustrate the effectiveness of the proposed solution compared to a timetriggered controller.      

Fixed-time synchronization of neural networks with time delay via quantized intermittent control

This paper presents an approach for fixed-time synchronization (FIXTS) of neural networks (NNs) by designing quantized intermittent controller. Under the intermittent controller, the synchronization between neural network systems with time delay can be realized. Based on intermittent strategy, FIXTs theory is proposed, and a sufficient condition is established to realize the FIXTS of the master–slave NNs. At the same time, the establishment time of FIXTS is estimated. Finally, the simulation of Gilli attractor to prove the validity of the proposed method.     

基于事件触发的离散 MIMO 系统自适应评判容错控制

本文针对具有执行器故障的一类离散非线性多输入多输出(Multi-input multi-output, MIMO)系统, 提出了一种基于事件触发的自适应评判容错控制方案. 该控制方案包括评价和执行网络. 在评价网络里, 为了缓解现有的非光滑二值效用函数可能引起的执行网络跳变问题, 利用高斯函数构建了一个光滑的效用函数, 并采用评价网络近似最优性能指标函数. 在执行网络里, 通过变量替换将系统状态的将来信息转化成关于系统当前状态的函数, 并结合事件触发机制设计了最优跟踪控制器. 该控制器引入了动态补偿项, 不仅能够抑制执行器故障对系统性能的影响, 而且能够改善系统的控制性能. 稳定性分析表明所有信号最终一致有界且跟踪误差收敛于原点的有界小邻域内. 数值系统和实际系统的仿真结果验证了该方案的有效性.     

Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design

This paper addresses the model-free nonlinear optimal control problem based on data by introducing the reinforcement learning (RL) technique. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton–Jacobi–Bellman (HJB) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, most practical systems are too complicated to establish an accurate mathematical model. To overcome these difficulties, we propose a data-based approximate policy iteration (API) method by using real system data rather than a system model. Firstly, a model-free policy iteration algorithm is derived and its convergence is proved. The implementation of the algorithm is based on the actor–critic structure, where actor and critic neural networks (NNs) are employed to approximate the control policy and cost function, respectively. To update the weights of actor and critic NNs, a least-square approach is developed based on the method of weighted residuals. The data-based API is an off-policy RL method, where the “exploration” is improved by arbitrarily sampling data on the state and input domain. Finally, we test the data-based API control design method on a simple nonlinear system, and further apply it to a rotational/translational actuator system. The simulation results demonstrate the effectiveness of the proposed method.     

具有指定性能和全状态约束的多智能体系统事件触发控制

针对一类非严格反馈的非线性多智能体系统一致性跟踪问题,在考虑全状态约束和指定性能的基础上提出了一种事件触发自适应控制算法.首先,通过设计性能函数,使跟踪误差在规定时间内收敛于指定范围.然后,在反步法中引入Barrier Lyapunov函数使所有状态满足约束条件,结合动态面技术解决传统反步法产生的"计算爆炸"问题,并利用径向基函数神经网络（Radial basis function neural networks,RBF NNs）处理系统中的未知非线性函数.最后基于Lyapunov稳定性理论证明系统中所有信号都是半全局一致最终有界的,跟踪误差收敛于原点的有界邻域内且满足指定性能.仿真结果验证了该控制算法的有效性.     

Event-triggered-based finite-time cooperative path following control for underactuated multiple marine surface vehicles

This paper develops an event-triggered-based finite-time cooperative path following (CPF) control scheme for underactuated marine surface vehicles (MSVs) with model parameter uncertainties and unknown ocean disturbances. First, a finite-time extended state observer (FTESO) is proposed, in which the FTESO can estimate the velocities and compound disturbances in finite time. Then, the finite-time LOS guidance law based on velocity estimation values is designed to obtain the desired surge velocity and the desired yaw rate. In order to realize the cooperative control of multiple paths in finite time, the cooperative control law for the path variable is designed. In addition, the relative threshold event-triggered control (ETC) mechanism is incorporated into the formation control algorithm, and an event-triggered-based finite-time CPF controller is designed, which not only effectively reduces the update frequency of controller and the mechanical loss of actuator but also improves the control performance of system. Furthermore, by using homogeneous method, Lyapunov theory, and finite-time stability theory, it is proved that under the proposed finite-time CPF control scheme, the formation errors can converge to a small neighborhood around origin in finite time. Finally, numerical simulation results illustrate the effectiveness of the proposed control scheme.     

基于自适应事件触发牵制控制的多时滞随机耦合神经网络簇同步

本文通过自适应事件触发牵制控制策略,研究了多时滞的随机耦合神经网络在均方意义下以指数速率进行簇同步的问题.在耦合神经网络中,同一簇中的节点只需与相应的孤立节点同步,而对于不同簇中节点之间的同步状态没有要求.首先,本文提出了一种事件触发牵制控制方法来解决耦合神经网络中节点数量众多、通讯复杂的问题.该方法不仅能减少耦合神经网络中控制器的数量,还可以减少控制信号的传输次数、减轻网络传输压力.然后根据M矩阵方法,建立了随机耦合神经网络均方指数稳定的充分条件.同时,利用自适应控制策略,给出了反馈增益的更新规律.最后,通过一个数值例子验证了所提出的自适应事件触发牵制控制策略的有效性和适用性.     

A Novel Distributed Optimal Adaptive Control Algorithm for Nonlinear Multi-Agent Differential Graphical Games

In this paper, an online optimal distributed learning algorithm is proposed to solve leader-synchronization problem of nonlinear multi-agent differential graphical games. Each player approximates its optimal control policy using a single-network approximate dynamic programming (ADP) where only one critic neural network (NN) is employed instead of typical actorcritic structure composed of two NNs. The proposed distributed weight tuning laws for critic NNs guarantee stability in the sense of uniform ultimate boundedness (UUB) and convergence of control policies to the Nash equilibrium. In this paper, by introducing novel distributed local operators in weight tuning laws, there is no more requirement for initial stabilizing control policies. Furthermore, the overall closed-loop system stability is guaranteed by Lyapunov stability analysis. Finally, Simulation results show the effectiveness of the proposed algorithm.      

ODE–热方程级联系统的事件触发控制

本文针对常微分方程(ODE)耦合偏微分方程(PDE)建模的分布式参数多智能体系统进行研究, 针对一致性同步问题, 提出了事件触发的网络化ODE–热方程级联系统多智能体一致性边界交互协议. 本文考虑的热方程左边界为Neumann边界条件, 并且与ODE系统耦合, 右边界为绝热边界条件. 假设网络化多智能体系统的连接方式为全联通有向拓扑图, 给出ODE–热方程级联系统的多智能体的一致性控制协议. 另外针对现有数字式控制器, 设计了事件触发的一致性控制协议, 并利用李雅普诺夫函数验证了在事件触发条件下ODE–热方程级联系统的稳定性. 最后给出了由5个ODE–热方程级联的多智能体系统的仿真结果, 验证了事件触发控制器的有效性.     

Fixed time event-triggered control for high-order nonlinear uncertain systems with time-varying full state constraints

The fixed time event-triggered control for high-order nonlinear uncertain systems with time-varying full state constraints is investigated in this paper. First, the event-triggered control (ETC) mechanism is introduced to reduce the data transmission in the communication channel. In consideration of the physical constraints and engineering requirements, time-varying barrier Lyapunov function (BLF) is deployed to make all the system states confined in the given time-varying constraints. Then, the radial basis function neural networks (RBF NNs) are used to approximate the unknown nonlinear terms. Further, the fixed time stability strategy is deployed to make the system achieve semiglobal practical fixed time stability (SPFTS) and the convergence time is independent of the initial conditions. Finally, the proposed control scheme is verified by two simulation examples.     

Simultaneous estimations of temperature-dependent thermal conductivity and heat capacity using a time efficient novel strategy based on MEGA–NN

The aim of the presented novel strategy is to find the best values of input parameters, while the objective functions are not explicitly known in terms of input parameters and their values only can be calculated by a time-consuming simulation. In this paper, a hybrid modified elitist genetic algorithm–neural network (MEGA–NN) strategy is proposed for such optimization problems. The good approximation performance of neural network (NN) and the effective and robust evolutionary searching ability of modified elitist genetic algorithm (MEGA) are applied in hybrid sense, where NNs are employed in predicting the objective value, and MEGA is adopted in searching optimal designs based on the predicted fitness values. The proposed strategy (MEGA–NN) is used to estimate the temperature-dependent thermal conductivity and heat capacity using inverse heat transfer method. In order to demonstrate the accuracy and time efficiency of the proposed strategy, the results are compared to those of pre-selected parameters and MEGA. Finally, the results show that proposed MEGA–NN could save a great deal of time depending on the case.     

Multiple model-based event-triggered adaptive control of a class of discrete-time nonlinear systems

In this study, the problem of event-triggered-based adaptive control (ETAC) for a class of discrete-time nonlinear systems with unknown parameters and nonlinear uncertainties is considered. Both neural network (NN) based and linear identifiers are used to approximate the unknown system dynamics. The feedback output signals are transmitted, and the parameters and the NN weights of the identifiers are tuned in an aperiodic manner at the event sample instants. A switching mechanism is provided to evaluate the approximate performance of each identifier and decide which estimated output is utilised for the event-triggered controller design, during any two events. The linear identifier with an auxiliary output and an improved adaptive law is introduced so that the nonlinear uncertainties are no longer assumed to be Lipschitz. The number of transmission times are significantly reduced by incorporating multiple model schemes into ETAC. The boundedness of both the parameters of identifiers and the system outputs is demonstrated though the Lyapunov approach. Simulation results demonstrate the effectiveness of the proposed method.     

Event-triggered optimal tracking control of multiplayer unknown nonlinear systems via adaptive critic designs

In this article, the event-triggered optimal tracking control problem for multiplayer unknown nonlinear systems is investigated by using adaptive critic designs. By constructing a neural network (NN)-based observer with input–output data, the system dynamics of multiplayer unknown nonlinear systems is obtained. Subsequently, the optimal tracking control problem is converted to an optimal regulation problem by establishing a tracking error system. Then, the optimal tracking control policy for each player is derived by solving coupled event-triggered Hamilton-Jacobi (HJ) equation via a critic NN. Meanwhile, a novel weight updating rule is designed by adopting concurrent learning method to relax the persistence of excitation (PE) condition. Moreover, an event-triggering condition is designed by using Lyapunov's direct method to guarantee the uniform ultimate boundedness (UUB) of the closed-loop multiplayer systems. Finally, the effectiveness of the developed method is verified by two different multiplayer nonlinear systems.     

Decentralised event-triggered impulsive synchronisation for semi-Markovian jump delayed neural networks with leakage delay and randomly occurring uncertainties

This study examines the problem of decentralised event-triggered impulsive synchronisation for the semi-Markovian jump neutral type neural networks with leakage delay and randomly occurring uncertainties. An improved globally asymptotic stability criterion is derived to guarantee impulsive synchronisation of the response systems with the drive systems. In order to reduce the network traffic and the resource of computation, we propose a new decentralised event-triggered scheme for the considered delayed NNs. In order to make full use of the sawtooth structure characteristic of the sampling input delay, a discontinuous Lyapunov functional is proposed. By establishing a suitable Lyapunov–Krasovskii functional (LKF) with triple integral terms and applying Writinger based integral method, auxiliary function based integral inequalities, reciprocal convex approach and improved inequality techniques, a delay dependent stability criterion is derived in terms of linear matrix inequalities (LMIs). Finally, numerical examples are given to illustrate the effectiveness of the proposed results.     

Distributed optimisation for resource allocation with event-triggered communication over general directed topology

This paper focuses on the resource allocation problem(RAP) with constraints under a fixed general directed topology by using the distributed sub-gradient algorithm with event-triggered scheme in multi-agent systems, where each agent owns a cost function and its state value is bounded. The distributed sub-gradient algorithm aims to minimise the total cost by a distributed manner while achieving an optimal solution. Unlike centralised methods, the triggering condition and algorithm for each agent are fully decentralised. At each instant of time, each agent updates its state by employing the states which are collected from itself and its neighbouring agents at their last triggering time. In order to illustrate the effectiveness of the proposed sub-gradient algorithm with event-triggered control law, one simulation example is presented before the conclusion.     

Adaptive critic learning for event-triggered safe control of nonlinear safety-critical systems

In this paper, an event-triggered safe control method based on adaptive critic learning (ACL) is proposed for a class of nonlinear safety-critical systems. First, a safe cost function is constructed by adding a control barrier function (CBF) to the traditional quadratic cost function; the optimization problem with safety constraints that is difficult to deal with by classical ACL methods is solved. Subsequently, the event-triggered scheme is introduced to reduce the amount of computation. Further, combining the properties of CBF with the ACL-based event-triggering mechanism, the event-triggered safe Hamilton–Jacobi–Bellman (HJB) equation is derived, and a single critic neural network (NN) framework is constructed to approximate the solution of the event-triggered safe HJB equation. In addition, the concurrent learning method is applied to the NN learning process, so that the persistence of excitation (PE) condition is not required. The weight approximation error of the NN and the states of the system are proven to be uniformly ultimately bounded (UUB) in the safe set with the Lyapunov theory. Finally, the availability of the presented method can be validated through the simulation.