Reinforcement learning-based event-triggered optimal control for unknown nonlinear systems with input delay

The optimal control issue of discrete-time nonlinear unknown systems with time-delay control input is the focus of this work. In order to reduce communication costs, a reinforcement learning-based event-triggered controller is proposed. By applying the proposed control method, closed-loop system's asymptotic stability is demonstrated, and a maximum upper bound for the infinite-horizon performance index can be calculated beforehand. The event-triggered condition requires the next time state information. In an effort to forecast the next state and achieve optimal control, three neural networks (NNs) are introduced and used to approximate system state, value function, and optimal control. Additionally, a M NN is utilized to cope with the time-delay term of control input. Moreover, taking the estimation errors of NNs into account, the uniformly ultimately boundedness of state and NNs weight estimation errors can be guaranteed. Ultimately, the validity of proposed approach is illustrated by simulations.     

Global output feedback control for nonlinear cascade systems with unknown output functions and unknown control directions

This paper investigates the output feedback control for the uncertain nonlinear system with the integral input‐to‐state stable (iISS) cascade subsystem, which allow not only the unknown control direction but also the unknown output function. The unknown output function only needs to have a generalized derivative (which may not be derivable), and the upper and lower bounds of the generalized derivative need not to be known. To deal with the challenge raised by the unknown output function and the unknown control direction, we choose a special Nussbaum function with a faster growth rate to ensure the integrability for the derivative of the selected Lyapunov function. Then, a dynamic output feedback controller is designed to drive the system states to the origin while keeping the boundedness for all other closed‐loop signals. Moreover, via some appropriate transformations, the proposed control scheme is extended to deal with more general uncertain nonlinear cascade systems with quantized input signals. Finally, two simulation examples are given to show the effectiveness of the control scheme.     

Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning

In this paper, a new formulation for the optimal tracking control problem (OTCP) of continuous-time nonlinear systems is presented. This formulation extends the integral reinforcement learning (IRL) technique, a method for solving optimal regulation problems, to learn the solution to the OTCP. Unlike existing solutions to the OTCP, the proposed method does not need to have or to identify knowledge of the system drift dynamics, and it also takes into account the input constraints a priori. An augmented system composed of the error system dynamics and the command generator dynamics is used to introduce a new nonquadratic discounted performance function for the OTCP. This encodes the input constrains into the optimization problem. A tracking Hamilton–Jacobi–Bellman (HJB) equation associated with this nonquadratic performance function is derived which gives the optimal control solution. An online IRL algorithm is presented to learn the solution to the tracking HJB equation without knowing the system drift dynamics. Convergence to a near-optimal control solution and stability of the whole system are shown under a persistence of excitation condition. Simulation examples are provided to show the effectiveness of the proposed method.     

Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems

In this paper, an integral reinforcement learning (IRL) algorithm on an actor–critic structure is developed to learn online the solution to the Hamilton–Jacobi–Bellman equation for partially-unknown constrained-input systems. The technique of experience replay is used to update the critic weights to solve an IRL Bellman equation. This means, unlike existing reinforcement learning algorithms, recorded past experiences are used concurrently with current data for adaptation of the critic weights. It is shown that using this technique, instead of the traditional persistence of excitation condition which is often difficult or impossible to verify online, an easy-to-check condition on the richness of the recorded data is sufficient to guarantee convergence to a near-optimal control law. Stability of the proposed feedback control law is shown and the effectiveness of the proposed method is illustrated with simulation examples.     

Robust optimal control of the multi-input systems with unknown disturbance based on adaptive integral reinforcement learning Q-function

Considering overshoot and chatter caused by the unknown interference, this article studies the adaptive robust optimal controls of continuous-time (CT) multi-input systems with an approximate dynamic programming (ADP) based Q-function scheme. An adaptive integral reinforcement learning (IRL) scheme is proposed to study the optimal solutions of Q-functions. First, multi-input value functions are presented, and Nash equilibrium is analyzed. A complex Hamilton–Jacobi–Issacs (HJI) equation is constructed with the multi-input system and the zero-sum-game-based value function. It is a challenging task to solve the HJI equation for nonlinear system. Thus, A transformation of the HJI equation is constructed as a Q-function. The neural network (NN) is applied to learn the solution of the transformed Q-functions based on the adaptive IRL scheme. Moreover, an error information is added to the Q-function for the issue of insufficient initial incentives to relax the persistent excitation (PE) condition. Simultaneously, an IRL signal of the critic networks is introduced to study the saddle-point intractable solution, such that the system drift and NN derivatives in the HJI equation are relaxed. The convergence of weight parameters is proved, and the closed-loop stability of the multi-system with the proposed IRL Q-function scheme is analyzed. Finally, a two-engine driven F-16 aircraft plant and a nonlinear system are presented to verify the effectiveness of the proposed adaptive IRL Q-function scheme.     

基于强化学习的一类具有输入约束非线性系统最优控制

针对部分系统存在输入约束和不可测状态的最优控制问题,本文将强化学习中基于执行–评价结构的近似最优算法与反步法相结合,提出了一种最优跟踪控制策略.首先,利用神经网络构造非线性观测器估计系统的不可测状态.然后,设计一种非二次型效用函数解决系统的输入约束问题.相比现有的最优方法,本文提出的最优跟踪控制方法不仅具有反步法在处理...     

Event-triggered adaptive tracking control for nonlinear systems with input saturation and unknown control directions

This article is concerned with event-triggered adaptive tracking control design of strict-feedback nonlinear systems, which are subject to input saturation and unknown control directions. In the design procedure, a smooth nonlinear function is employed to approximate the saturation function so that the controller can be designed under the framework of backstepping. The Nussbaum gain technique is employed to address the issue of the unknown control directions. A predetermined time convergent performance function and a nonlinear mapping technique are introduced to guarantee that the tracking error can converge in the predetermined time with a fast convergence rate and a high accuracy. Then the event-triggered adaptive prescribed performance tracking control strategy is proposed, which not only ensures the boundedness of all the closed-loop signals and the convergence of tracking error but also reduces the communication burden from the controller to the actuator. At last, the simulation study further tests the availability of the proposed control strategy.     

Dynamic event-triggered control for a class of p-normal nonlinear time-delay systems via output feedback

In this article, the event-triggered (ET) output feedback control problem is discussed for a class of $p$-normal nonlinear time-delay systems. Different from the related existing literature, the linear growth condition is relaxed to the homogeneous one and the growth rate is unknown. In the case of unknown time-varying delays and control coefficients, nonsmooth control law and some extra redundant terms will be encountered in the design of ET controller, which would bring substantial challenges to the achievement of global convergence. To cope with unknown growth rate and time-varying delays, two dynamic gains and an appropriate Lyapunov–Krasovskii functional (LKF) are introduced. Then, a new dynamic ET mechanism is given, in which the triggering threshold can be tuned dynamically. It is proved that all the signals of the closed-loop system (CLS) are bounded. Corresponding examples are given to indicate the validity of the developed theoretical results.     

Input-constrained optimal output synchronization of heterogeneous multiagent systems via observer-based model-free reinforcement learning

This paper presents a new formulation of input-constrained optimal output synchronization problem and proposes an observer-based distributed optimal control protocol for discrete-time heterogeneous multiagent systems with input constraints via model-free reinforcement learning. First, distributed adaptive observers are designed for all agents to estimate the leader's trajectory without requiring its dynamics knowledge. Subsequently, the optimal control input associated with the optimal value function is derived based on the solution to the tracking Hamilton-Jacobi-Bellman equation, which is always difficult to solve analytically. To this end, motivated by reinforcement learning technique, a model-free Q-learning policy iteration algorithm is proposed, and the actor-critic neural network structure is implemented to iteratively find the optimal tracking control input without knowing system dynamics. Moreover, inputs of all agents are constrained in the permitted bounds by inserting a nonquadratic function into the performance function, where input constraints are encoded into the optimization problem. Finally, a numerical simulation example is provided to illustrate the effectiveness of the proposed theoretical results.     

Quantized feedback control for nonlinear feedforward systems with unknown output functions and unknown control coefficients

This paper investigates the quantized feedback control for nonlinear feedforward systems with unknown output functions and unknown control coefficients. The unknown output function is Lipschitz continuous but may not be derivable, and the unknown control coefficients are assumed to be bounded. To deal with this challenging quantized control problem, a time‐varying low‐gain observer is designed and a delicate time‐varying scaling transformation is introduced, which can avoid using the derivative information of the output function. Then, based on the well‐known backstepping method and the sector bound approach, a time‐varying quantized feedback controller is designed using the quantized output, which can achieve the boundedness of the closed‐loop system states and the convergence of the original system states. Moreover, a guideline is provided for choosing the parameters of the input and output quantizers such that the closed‐loop system is stable. Finally, two simulation examples are given to show the effectiveness of the control scheme.     

Two-loop reinforcement learning algorithm for finite-horizon optimal control of continuous-time affine nonlinear systems

This article proposes three novel time-varying policy iteration algorithms for finite-horizon optimal control problem of continuous-time affine nonlinear systems. We first propose a model-based time-varying policy iteration algorithm. The method considers time-varying solutions to the Hamiltonian–Jacobi–Bellman equation for finite-horizon optimal control. Based on this algorithm, value function approximation is applied to the Bellman equation by establishing neural networks with time-varying weights. A novel update law for time-varying weights is put forward based on the idea of iterative learning control, which obtains optimal solutions more efficiently compared to previous works. Considering that system models may be unknown in real applications, we propose a partially model-free time-varying policy iteration algorithm that applies integral reinforcement learning to acquiring the time-varying value function. Moreover, analysis of convergence, stability, and optimality is provided for every algorithm. Finally, simulations for different cases are given to verify the convenience and effectiveness of the proposed algorithms.     

具有未知输出函数的非线性系统全局输出反馈控制

研究了一类具有未知输出函数的非线性系统全局输出反馈控制问题．由于输出函数未知,传统的观测器将无法实现．为解决这个问题,首先设计了一个与输出函数无关的状态补偿器,使得标称线性系统全局渐近稳定．然后,应用齐次控制方法通过适当选择增益参数,使得不确定非线性系统在有限时间内全局渐近稳定．数值算例表明该算法的有效性．     

Distributed event-triggered optimal bipartite consensus control for multiagent systems with input delay via reinforcement learning method

In this article, an optimal bipartite consensus control (OBCC) scheme is proposed for heterogeneous multiagent systems (MASs) with input delay by reinforcement learning (RL) algorithm. A directed signed graph is established to construct MASs with competitive and cooperative relationships, and model reduction method is developed to tackle input delay problem. Then, based on the Hamilton–Jacobi–Bellman (HJB) equation, policy iteration method is utilized to design the bipartite consensus controller, which consists of value function and optimal controller. Further, a distributed event-triggered function is proposed to increase control efficiency, which only requires information from its own agent and neighboring agents. Based on the input-to-state stability (ISS) function and Lyapunov function, sufficient conditions for the stability of MASs can be derived. Apart from that, RL algorithm is employed to solve the event-triggered OBCC problem in MASs, where critic neural networks (NNs) and actor NNs estimate value function and control policy, respectively. Finally, simulation results are given to validate the feasibility and efficiency of the proposed algorithm.     

Online adaptive algorithm for optimal control with integral reinforcement learning

In this paper, we introduce an online algorithm that uses integral reinforcement knowledge for learning the continuous‐time optimal control solution for nonlinear systems with infinite horizon costs and partial knowledge of the system dynamics. This algorithm is a data‐based approach to the solution of the Hamilton–Jacobi–Bellman equation, and it does not require explicit knowledge on the system's drift dynamics. A novel adaptive control algorithm is given that is based on policy iteration and implemented using an actor/critic structure having two adaptive approximator structures. Both actor and critic approximation networks are adapted simultaneously. A persistence of excitation condition is required to guarantee convergence of the critic to the actual optimal value function. Novel adaptive control tuning algorithms are given for both critic and actor networks, with extra terms in the actor tuning law being required to guarantee closed loop dynamical stability. The approximate convergence to the optimal controller is proven, and stability of the system is also guaranteed. Simulation examples support the theoretical result. Copyright © 2013 John Wiley & Sons, Ltd.     

Global adaptive stabilisation for nonlinear systems with unknown control directions and input disturbance

This paper addresses the global adaptive stabilisation via switching and learning strategies for a class of uncertain nonlinear systems. Remarkably, the systems in question simultaneously have unknown control directions, unknown input disturbance and unknown growth rate, which makes the problem in question challenging to solve and essentially different from those in the existing literature. To solve the problem, an adaptive scheme via switching and learning is proposed by skilfully integrating the techniques of backstepping design, adaptive learning and adaptive switching. One key point in the design scheme is the introduction of the learning mechanism, in order to compensate the unknown input disturbance, and the other one is the design of the switching mechanism, through tuning the design parameters online to deal with the unknown control directions, unknown bound and period of input disturbance and unknown growth rate. The designed controller guarantees that all the signals of the resulting closed-loop systems are bounded, and furthermore, the closed-loop system states globally converge to zero.     

Observer-based adaptive control for nonlinear input-delay systems with unknown control directions

This paper investigates the problem of adaptive control for strict-feedback nonlinear systems with input delay and unknown control directions. The Nussbaum function is utilised to deal with the unknown control directions and a novel compensation system is introduced to handle the time-varying input delay. By using neural network(NN) approximation and backstepping approaches, an adaptive NN controller is designed which can guarantee the semi-global boundedness of all the signals in the closed-loop system. Two simulation examples are also given to illustrate the effectiveness of the proposed method.     

Multigradient recursive reinforcement learning NN control for affine nonlinear systems with unmodeled dynamics

In this paper, an adaptive reinforcement learning approach is developed for a class of discrete‐time affine nonlinear systems with unmodeled dynamics. The multigradient recursive (MGR) algorithm is employed to solve the local optimal problem, which is inherent in gradient descent method. The MGR radial basis function neural network approximates the utility functions and unmodeled dynamics, which has a faster rate of convergence than that of the gradient descent method. A novel strategic utility function and cost function are defined for the affine systems. Finally, it concludes that all the signals in the closed‐loop system are semiglobal uniformly ultimately bounded through differential Lyapunov function method, and two simulation examples are presented to demonstrate the effectiveness of the proposed scheme.     

Event-triggered adaptive consensus control of nonlinear multi-agent systems with unknown control directions and actuator saturation

This article addresses the event-triggered adaptive consensus control of nonlinear multi-agent systems with unknown control direction and actuator saturation. A new robust adaptive control algorithm based on an event-triggered mechanism is designed. The smooth Lipschitz function approximates the saturated nonlinear function, while the Nussbaum function handles unknown control directions and residual terms. The event-triggered mechanism is designed to determine the time of communication, significantly reducing the communication burden. An additional estimator is utilized to deal with unknown parameters involved in neighbor dynamics and prevent information exchange to consistency errors between connected subsystems. The results show that all the signals of the closed-loop system are uniformly bounded, and the consensus tracking error converges to a bounded set. Meanwhile, Zeno's behavior is eliminated. Simulation results confirm the superiority of the proposed method.     

不确定非线性系统自适应动态事件触发输出反馈镇定

本文研究了一类不确定非线性系统的动态事件触发输出反馈镇定问题. 显著不同的是系统具有依赖于不可测状态的增长且增长率为输出的未知多项式. 尽管已有一些连续自适应控制器, 但需要巧妙融合非线性状态观测器、系统未知性的动态补偿以及非线性的抵御, 因此这些控制器具有一定的脆弱性, 不能平凡地拓展到不连续情形 (采样误差导致). 为此, 首先通过引入动态高增益和基于高增益的观测器来分别抵御未知增长率和重构系统不可测状态. 进而, 意识到静态事件触发机制的无效性, 通过引入动态事件触发机制, 成功设计出了事件触发输出反馈控制器, 确保了系统状态的全局有界性和收敛性. 数值仿真验证了所设计控制器的有效性.     

Global adaptive output feedback control for a class of stochastic nonlinear systems with unknown homogeneous growth rates

This article focuses on the adaptive output feedback stabilization for a class of stochastic nonlinear systems whose drift and diffusion terms satisfy homogeneous growth conditions. Since the homogeneous growth rates are unknown, two dynamic gains are coupled into the full-order homogeneous observer. By virtue of adding a power integrator technique and the homogeneity theory, two adaptive laws and a homogeneous output feedback controller are designed. Based on the celebrated nonnegative semimartingale convergence theorem and the general stochastic Barbˇlat's lemma, it is indicated that all the signals of the closed-loop system are bounded almost surely, and all the system states of the closed-loop system converge to origin almost surely. Finally, the effectiveness of the proposed control scheme is verified by means of both numerical and practical examples.