Reinforcement learning-based event-triggered optimal control for unknown nonlinear systems with input delay

The optimal control issue of discrete-time nonlinear unknown systems with time-delay control input is the focus of this work. In order to reduce communication costs, a reinforcement learning-based event-triggered controller is proposed. By applying the proposed control method, closed-loop system's asymptotic stability is demonstrated, and a maximum upper bound for the infinite-horizon performance index can be calculated beforehand. The event-triggered condition requires the next time state information. In an effort to forecast the next state and achieve optimal control, three neural networks (NNs) are introduced and used to approximate system state, value function, and optimal control. Additionally, a M NN is utilized to cope with the time-delay term of control input. Moreover, taking the estimation errors of NNs into account, the uniformly ultimately boundedness of state and NNs weight estimation errors can be guaranteed. Ultimately, the validity of proposed approach is illustrated by simulations.     

通信延时和数据丢包下事件驱动的多智能体系统一致性研究

本文研究受网络通信延时和数据随机丢包的多智能系统一致性问题, 探索事件驱动的分布式协同控制策略. 首先针对两类普遍应用的事件触发器, 提出了一个可用于选择触发策略的触发频率比较方法. 然后提出了分布式协同控制律以保证系统的渐近一致性, 并给出了相应的时滞依赖Markov切换控制器设计新方法. 本文所提的控制策略不仅保证系统一致性目标, 而且能显著减少通信数据传输量并降低控制器计算负担. 最后,通过仿真算例验证了所提方法的有效性。     

Event-triggered-based integral reinforcement learning output feedback optimal control for partially unknown constrained-input nonlinear systems

In this paper, an adaptive output feedback event-triggered optimal control algorithm is proposed for partially unknown constrained-input continuous-time nonlinear systems. First, a neural network observer is constructed to estimate unmeasurable state. Next, an event-triggered condition is established, and only when the event-triggered condition is violated will the event be triggered and the state be sampled. Then, an event-triggered-based synchronous integral reinforcement learning (ET-SIRL) control algorithm with critic-actor neural networks (NNs) architecture is proposed to solve the event-triggered Hamilton–Jacobi–Bellman equation under the established event-triggered condition. The critic and actor NNs are used to approximate cost function and optimal event-triggered optimal control law, respectively. Meanwhile, the event-triggered-based closed-loop system state and all the neural network weight estimation errors are uniformly ultimately bounded proved by Lyapunov stability theory, and there is no Zeno behavior. Finally, two numerical examples are presented to show the effectiveness of the proposed ET-SIRL control algorithm.     

离散非线性零和博弈的事件驱动最优控制方案

在求解离散非线性零和博弈问题时,为了在有效降低网络通讯和控制器执行次数的同时保证良好的控制效果,本文提出了一种基于事件驱动机制的最优控制方案.首先,设计了一个采用新型事件驱动阈值的事件驱动条件,并根据贝尔曼最优性原理获得了最优控制对的表达式.为了求解该表达式中的最优值函数,提出了一种单网络值迭代算法.利用一个神经网络构建评价网.设计了新的评价网权值更新规则.通过在评价网、控制策略及扰动策略之间不断迭代,最终获得零和博弈问题的最优值函数和最优控制对.然后,利用Lyapunov稳定性理论证明了闭环系统的稳定性.最后,将该事件驱动最优控制方案应用到了两个仿真例子中,验证了所提方法的有效性.     

对抗网络下时滞非线性多智能体系统固定时间二分一致性

本文针对一类存在输入时延的非线性多智能体系统,研究了其在结构平衡的无向符号图下的固定时间二分一致性问题.首先,本文针对智能体间相互合作与相互竞争的关系,设计了一类存在输入时延的多智能体系统固定时间分布式一致性控制协议,使得系统状态在固定时间内收敛到数值相同但符号相反的两个值,且收敛时间上界与初始状态无关.随后,利用Lyapunov稳定性理论和代数图论给出了在存在输入时延的情况下多智能体系统实现固定时间二分一致性的充分条件和收敛时间的上界值,证明了控制算法的稳定性.最后,仿真实例验证了所提固定时间二分一致性算法和理论结果的有效性.     

Data-based approximate policy iteration for affine nonlinear continuous-time optimal control design

This paper addresses the model-free nonlinear optimal control problem based on data by introducing the reinforcement learning (RL) technique. It is known that the nonlinear optimal control problem relies on the solution of the Hamilton–Jacobi–Bellman (HJB) equation, which is a nonlinear partial differential equation that is generally impossible to be solved analytically. Even worse, most practical systems are too complicated to establish an accurate mathematical model. To overcome these difficulties, we propose a data-based approximate policy iteration (API) method by using real system data rather than a system model. Firstly, a model-free policy iteration algorithm is derived and its convergence is proved. The implementation of the algorithm is based on the actor–critic structure, where actor and critic neural networks (NNs) are employed to approximate the control policy and cost function, respectively. To update the weights of actor and critic NNs, a least-square approach is developed based on the method of weighted residuals. The data-based API is an off-policy RL method, where the “exploration” is improved by arbitrarily sampling data on the state and input domain. Finally, we test the data-based API control design method on a simple nonlinear system, and further apply it to a rotational/translational actuator system. The simulation results demonstrate the effectiveness of the proposed method.     

Semi-global Adaptive Bipartite Output Consensus of Multi-agent Systems Subject to Input Saturation and External Disturbance Under Switching Network

This paper investigates semi-global adaptive bipartite output consensus of continuous-time multi-agent systems (MASs) with input saturation and non-identical external disturbance under jointly connected switching network. An adaptive bipartite output consensus protocol of MASs is proposed by using low-gain feedback technology. It is turned out that semi-global adaptive bipartite consensus of MASs can be achieved under the protocol. Furthermore, the proposed control protocol can be applied for MASs under fixed network, and semi-global adaptive bipartite output consensus can be also achieved in this case. Finally, the simulations will verify the effectiveness of theoretical results.

     

Neural-based online finite-time optimal tracking control for wheeled mobile robotic system with inequality constraints

In this study, a finite-time online optimal controller was designed for a nonlinear wheeled mobile robotic system (WMRS) with inequality constraints, based on reinforcement learning (RL) neural networks. In addition, an extended cost function, obtained by introducing a penalty function to the original long-time cost function, was proposed to deal with the optimal control problem of the system with inequality constraints. A novel Hamilton-Jacobi-Bellman (HJB) equation containing the constraint conditions was defined to determine the optimal control input. Furthermore, two neural networks (NNs), a critic and an actor NN, were established to approximate the extended cost function and the optimal control input, respectively. The adaptation laws of the critic and actor NN were obtained with the gradient descent method. The semi-global practical finite-time stability (SGPFS) was proved using Lyapunov's stability theory. The tracking error converges to a small region near zero within the constraints in a finite period. Finally, the effectiveness of the proposed optimal controller was verified by a simulation based on a practical wheeled mobile robot model.     

Distributed fuzzy adaptive event-triggered finite-time consensus tracking control for uncertain nonlinear multi-agent systems with asymmetric output constraint

This article investigates the consensus problem for uncertain nonlinear multi-agent systems (MASs) with asymmetric output constraint. Different from BLF-based constraint consensus tracking control, a novel approach based on nonlinear state-dependent function is proposed to solve the asymmetric output constraint, which need not convert output constraint into tracking error bound. First-order sliding mode differentiator is incorporated into each step of backstepping control design to reduce computation burden. Further, in combination of proposed event-triggered mechanism based on time-varying threshold, a distributed fuzzy adaptive event-triggered finite-time consensus method is developed. It can ensure that the consensus tracking error tends to a small neighbor in a finite time and the asymmetric output constraint of each subsystem is not violated. Two simulations are given to demonstrate the effectiveness of control method.     

输入饱和下非线性系统的事件触发控制器设计

本文研究了具有输入饱和的非线性系统事件触发控制策略设计问题.首先,针对输入饱和下非线性系统,建立混杂系统模型.其次,当非线性函数满足Lipschitz条件下,给出闭环混杂系统局部一致渐近稳定性的稳定判据,并设计了事件触发饱和控制器.然后,当非线性函数满足扇区条件时,给出闭环混杂系统框架下满足局部一致渐近稳定性的LMI条件,并设计了事件触发饱和控制器.进一步地,在事件触发饱和控制器作用下,分析了非线性系统的半全局鲁棒镇定性.最后,结合两个仿真实例说明了所提出事件触发控制策略的有效性.     

Bipartite Byzantine-resilient event-triggered consensus control of heterogeneous multi-agent systems

This article studies the bipartite resilient event-triggered consensus control for a class of the heterogeneous multi-agent systems. Due to the external cyberattacks, some agents may become the Byzantine agents and will affect the behavior of the other agents. To improve the security of the multi-agent systems against the Byzantine agents, a novel bipartite event-triggered heterogeneous mean-subsequence-reduced algorithm is designed. First, to handle the heterogeneous multi-agent systems, a state transformation is carefully designed, to facilitate the design and analysis of the bipartite resilient consensus algorithm. Based upon the designed state transformation, the bipartite resilient control inputs are constructed, where the structural balance analysis shows that the resulting effective signed graph and the equivalent signed graph are both structurally balanced, if the signed graph of the multi-agent systems is structurally balanced. In addition, a dynamic event-triggered mechanism is proposed, where a set of dynamic factors are introduced into the event-triggered functions to prevent the usage of the global topology information. By virtue of the designed algorithm, it is guaranteed that the heterogeneous multi-agent systems can achieve the bipartite consensus in the presence of the Byzantine agents, and the communication burden among the agents can be reduced. The numerical simulations are conducted to verify the effectiveness of the proposed algorithm.     

带状态约束的事件触发积分强化学习控制

为克服全状态对称约束以及控制策略频繁更新的局限,同时使得无限时间的代价函数最优,针对一类具有部分动力学未知的仿射非线性连续系统,提出一种带状态约束的事件触发积分强化学习的控制器设计方法。该方法是一种基于数据的在线策略迭代方法。引入系统转换将带有全状态约束的系统转化为不含约束的系统。基于事件触发机制以及积分强化学习算法,通过交替执行系统转换、策略评估、策略改进,最终系统在满足全状态约束的情况下,代价函数以及控制策略将分别收敛于最优值,并能降低控制策略的更新频率。此外,通过构建李亚普诺夫函数对系统以及评论神经网络权重误差的稳定性进行严格的分析。单连杆机械臂的仿真实验也进一步说明算法的可行性。     

基于事件触发的离散多智能体二分一致性

针对一阶离散多智能体系统,研究了事件触发控制下的二分一致性问题.首先考虑智能体间通信拓扑结构为无向连通结构平衡图的情形,针对各智能体设计事件触发控制,包括仅依赖于自身及邻居智能体采样状态的控制输入,以及仅依赖自身状态的事件触发条件,实现了对通信资源的节约利用.基于图论、离散系统稳定性理论,证明系统能够实现二分一致性.同时,合理设置控制输入及事件触发条件中参数,保证系统不存在Zeno现象.之后,进一步分析设计了包含有向生成树的结构平衡图下,多智能体系统的事件触发控制.最后利用仿真实例验证了理论结果的有效性.     

含时延约束的多智能体系统二分一致性

针对多智能体系统中信息交互存在通信时延这一约束,在无向符号图拓扑结构下分别研究了含固定时延和时变时延的一阶多智能体系统二分一致性问题。通过设计相应的控制协议,使得该系统收敛到两个模值相同但符号不同的状态。在稳定性分析中,利用广义Nyquist准则的方法,得到含固定时延多智能体系统实现二分一致性的充分条件;对含时变时延系统构造包含三重积分项的Lyapunov函数,利用积分不等式和线性矩阵不等式理论,并结合自由矩阵的方法得到含时变时延多智能体系统实现二分一致性的充分条件。最后,数值仿真验证了所得结论的有效性和正确性。     

非线性零和微分对策的事件触发自适应动态规划算法

针对一类非线性零和微分对策问题,本文提出了一种事件触发自适应动态规划(event-triggered adaptive dynamic programming,ET--ADP)算法在线求解其鞍点.首先,提出一个新的自适应事件触发条件.然后,利用一个输入为采样数据的神经网络(评价网络)近似最优值函数,并设计了新型的神经网络权值更新律使得值函数、控制策略及扰动策略仅在事件触发时刻同步更新.进一步地,利用Lyapunov稳定性理论证明了所提出的算法能够在线获得非线性零和微分对策的鞍点且不会引起Zeno行为.所提出的ET--ADP算法仅在事件触发条件满足时才更新值函数、控制策略和扰动策略,因而可有效减少计算量和降低网络负荷.最后,两个仿真例子验证了所提出的ET--ADP算法的有效性.     

Dual-terminal event-triggered dynamic output feedback H∞ control for aero engine networked control systems with quantization

This study deals with the problem of dual-terminal event-triggered dynamic output feedback (DOF) control for aero engine networked control systems (NCSs) subject to network-induced delay, external disturbance, and quantization effects. First, we established a generalized mathematical model of the aero engine. Second, a dual-terminal dynamic event-triggered mechanism (DETM) was designed to reduce the utilization of the network bandwidth. In addition, a DOF controller with a compensation function was proposed to stabilize the system. By utilizing the Lyapunov-Krasovskii (LK) method, the stability criteria were determined. Accordingly, the design conditions of the DOF controller and DETMs were presented. Furthermore, based on a genetic algorithm (GA), a parameter tuning method was proposed to obtain the allowable delay upper bound with less conservatism. Finally, some examples were presented to show the effectiveness and superiority of the presented scheme.     

Event-triggered containment control for multi-agent systems under hybrid cyber attacks

This paper studies event-triggered containment control problem of multi-agent systems (MASs) under deception attacks and denial-of-service (DoS) attacks. First, to save limited network resources, an event-triggered mechanism is proposed for MASs under hybrid cyber attacks. Different from the existing event-triggered mechanisms, the negative influences of deception attacks and DoS attacks are considered in the proposed triggering function. The communication frequencies between agents are reduced. Then, based on the proposed event-triggered mechanism, a corresponding control protocol is proposed to ensure that the followers will converge to the convex hull formed by the leaders under deception attacks and DoS attacks. Compared with the previous researches about containment control, in addition to hybrid cyber attacks being considered, the nonlinear functions related to the states of the agents are applied to describe the deception attack signals in the MAS. By orthogonal transformation of deception attack signals, the containment control problem under deception attacks and DoS attacks is reformulated as a stability problem. Then, the sufficient conditions on containment control can be obtained. Finally, a set of simulation example is used to verify the effectiveness of the proposed method.     

Connectivity-preserving consensus: An adaptive event-triggered strategy

The connectivity of communication graph is indispensable for consensus of multi-agent systems (MASs), which in many applications (e.g., wireless sensor) depends on the relative distances between agents. But, in the adaptive setting particularly with system nonlinearities and event-triggered communication, it is rather difficult to enforce the relative distances within the limited range for the connectivity. This paper focuses on developing an adaptive event-triggered control strategy with connectivity preservation in the context of nonidentical unknown control coefficients and heterogeneous nonlinearities coupling with parameter uncertainties. First, a group of potential functions are introduced acting as control barrier functions to constrain the relative distances between agents within the limited range for all time. Also, two dynamic gains are specialized for each agent to handle the system uncertainties, system nonlinearities and negative effect of the execution error. Then, an adaptive event-triggered protocol is designed for each agent such that the connectivity-preserving consensus of MASs is achieved and Zeno behavior is excluded. Moreover, an extended study is conducted on a leader-following scenario. Two simulation examples illustrate the effectiveness of the proposed event-triggered control strategy.     

Decentralised event-triggered impulsive synchronisation for semi-Markovian jump delayed neural networks with leakage delay and randomly occurring uncertainties

This study examines the problem of decentralised event-triggered impulsive synchronisation for the semi-Markovian jump neutral type neural networks with leakage delay and randomly occurring uncertainties. An improved globally asymptotic stability criterion is derived to guarantee impulsive synchronisation of the response systems with the drive systems. In order to reduce the network traffic and the resource of computation, we propose a new decentralised event-triggered scheme for the considered delayed NNs. In order to make full use of the sawtooth structure characteristic of the sampling input delay, a discontinuous Lyapunov functional is proposed. By establishing a suitable Lyapunov–Krasovskii functional (LKF) with triple integral terms and applying Writinger based integral method, auxiliary function based integral inequalities, reciprocal convex approach and improved inequality techniques, a delay dependent stability criterion is derived in terms of linear matrix inequalities (LMIs). Finally, numerical examples are given to illustrate the effectiveness of the proposed results.     

Dynamic event-triggered bipartite consensus for uncertain high-order nonlinear multi-agent systems

In this paper, the bipartite consensus problem is studied for a class of uncertain high-order nonlinear multi-agent systems. A signed digraph is presented to describe the collaborative and competitive interactions among agents. For each agent with lower triangular structure, a time-varying gain compensator is first designed by relative output information of neighboring agents. Subsequently, a distributed controller with dynamic event-triggered mechanism is proposed to drive the bipartite consensus error to zero. It is worth noting that an internal dynamic variable is introduced in triggering function, which plays an essential role in excluding the Zeno behavior and reducing energy consumption. Furthermore, the dynamic event-triggered control protocol is developed for upper triangular multi-agent systems to realize the bipartite consensus without Zeno behavior. Finally, simulation examples are provided to illustrate the effectiveness of the presented results.