Online adaptive optimal control for continuous-time nonlinear systems with completely unknown dynamics

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.     

Observer-based Adaptive Optimal Control for Unknown Singularly Perturbed Nonlinear Systems With Input Constraints

This paper introduces an observer-based adaptive optimal control method for unknown singularly perturbed nonlinear systems with input constraints. First, a multi-time scales dynamic neural network (MTSDNN) observer with a novel updating law derived from a properly designed Lyapunov function is proposed to estimate the system states. Then, an adaptive learning rule driven by the critic NN weight error is presented for the critic NN, which is used to approximate the optimal cost function. Finally, the optimal control action is calculated by online solving the Hamilton-Jacobi-Bellman (HJB) equation associated with the MTSDNN observer and critic NN. The stability of the overall closed-loop system consisting of the MTSDNN observer, the critic NN and the optimal control action is proved. The proposed observer-based optimal control approach has an essential advantage that the system dynamics are not needed for implementation, and only the measured input/output data is needed. Moreover, the proposed optimal control design takes the input constraints into consideration and thus can overcome the restriction of actuator saturation. Simulation results are presented to confirm the validity of the investigated approach.      

Reinforcement learning neural-network-based controller for nonlinear discrete-time systems with input constraints.

A novel adaptive-critic-based neural network (NN) controller in discrete time is designed to deliver a desired tracking performance for a class of nonlinear systems in the presence of actuator constraints. The constraints of the actuator are treated in the controller design as the saturation nonlinearity. The adaptive critic NN controller architecture based on state feedback includes two NNs: the critic NN is used to approximate the "strategic" utility function, whereas the action NN is employed to minimize both the strategic utility function and the unknown nonlinear dynamic estimation errors. The critic and action NN weight updates are derived by minimizing certain quadratic performance indexes. Using the Lyapunov approach and with novel weight updates, the uniformly ultimate boundedness of the closed-loop tracking error and weight estimates is shown in the presence of NN approximation errors and bounded unknown disturbances. The proposed NN controller works in the presence of multiple nonlinearities, unlike other schemes that normally approximate one nonlinearity. Moreover, the adaptive critic NN controller does not require an explicit offline training phase, and the NN weights can be initialized at zero or random. Simulation results justify the theoretical analysis.     

Event-triggered optimal tracking control of multiplayer unknown nonlinear systems via adaptive critic designs

In this article, the event-triggered optimal tracking control problem for multiplayer unknown nonlinear systems is investigated by using adaptive critic designs. By constructing a neural network (NN)-based observer with input–output data, the system dynamics of multiplayer unknown nonlinear systems is obtained. Subsequently, the optimal tracking control problem is converted to an optimal regulation problem by establishing a tracking error system. Then, the optimal tracking control policy for each player is derived by solving coupled event-triggered Hamilton-Jacobi (HJ) equation via a critic NN. Meanwhile, a novel weight updating rule is designed by adopting concurrent learning method to relax the persistence of excitation (PE) condition. Moreover, an event-triggering condition is designed by using Lyapunov's direct method to guarantee the uniform ultimate boundedness (UUB) of the closed-loop multiplayer systems. Finally, the effectiveness of the developed method is verified by two different multiplayer nonlinear systems.     

基于事件触发的离散 MIMO 系统自适应评判容错控制

本文针对具有执行器故障的一类离散非线性多输入多输出(Multi-input multi-output, MIMO)系统, 提出了一种基于事件触发的自适应评判容错控制方案. 该控制方案包括评价和执行网络. 在评价网络里, 为了缓解现有的非光滑二值效用函数可能引起的执行网络跳变问题, 利用高斯函数构建了一个光滑的效用函数, 并采用评价网络近似最优性能指标函数. 在执行网络里, 通过变量替换将系统状态的将来信息转化成关于系统当前状态的函数, 并结合事件触发机制设计了最优跟踪控制器. 该控制器引入了动态补偿项, 不仅能够抑制执行器故障对系统性能的影响, 而且能够改善系统的控制性能. 稳定性分析表明所有信号最终一致有界且跟踪误差收敛于原点的有界小邻域内. 数值系统和实际系统的仿真结果验证了该方案的有效性.     

Online H∞ control for completely unknown nonlinear systems via an identifier–critic-based ADP structure

ABSTRACT

In this paper, we propose an identifier–critic-based approximate dynamic programming (ADP) structure to online solve H∞ control problem of nonlinear continuous-time systems without knowing precise system dynamics, where the actor neural network (NN) that has been widely used in the standard ADP learning structure is avoided. We first use an identifier NN to approximate the completely unknown nonlinear system dynamics and disturbances. Then, another critic NN is proposed to approximate the solution of the induced optimal equation. The H∞ control pair is obtained by using the proposed identifier–critic ADP structure. A recently developed adaptation algorithm is used to online directly estimate the unknown NN weights simultaneously, where the convergence to the optimal solution can be rigorously guaranteed, and the stability of the closed-loop system is analysed. Thus, this new ADP scheme can improve the computational efficiency of H∞ control implementation. Finally, simulation results confirm the effectiveness of the proposed methods.     

基于自适应动态规划的非线性鲁棒近似最优跟踪控制

为克服现有近似最优跟踪控制方法只能跟踪连续可微参考输入的局限,本文针对一类具有未知动态的连续时间非线性时不变仿射系统,提出了一种新的基于自适应动态规划的鲁棒近似最优跟踪控制方法.首先采用递归神经网络建立系统模型,然后建立评价神经网络对最优性能指标进行估计,从而得到最优性能指标偏导数的估计值,进而得到近似最优跟踪控制器,最后利用系统输出与参考输入之间的跟踪误差设计鲁棒项对神经网络建模误差进行补偿.分别针对两个非线性系统进行仿真实验,仿真结果表明了所提方法的有效性和优越性.     

Parallel Control for Optimal Tracking via Adaptive Dynamic Programming

This paper studies the problem of optimal parallel tracking control for continuous-time general nonlinear systems. Unlike existing optimal state feedback control, the control input of the optimal parallel control is introduced into the feedback system. However, due to the introduction of control input into the feedback system, the optimal state feedback control methods can not be applied directly. To address this problem, an augmented system and an augmented performance index function are proposed firstly. Thus, the general nonlinear system is transformed into an affine nonlinear system. The difference between the optimal parallel control and the optimal state feedback control is analyzed theoretically. It is proven that the optimal parallel control with the augmented performance index function can be seen as the suboptimal state feedback control with the traditional performance index function. Moreover, an adaptive dynamic programming (ADP) technique is utilized to implement the optimal parallel tracking control using a critic neural network (NN) to approximate the value function online. The stability analysis of the closed-loop system is performed using the Lyapunov theory, and the tracking error and NN weights errors are uniformly ultimately bounded (UUB). Also, the optimal parallel controller guarantees the continuity of the control input under the circumstance that there are finite jump discontinuities in the reference signals. Finally, the effectiveness of the developed optimal parallel control method is verified in two cases.      

动态规划最优控制在非线性系统中的应用

应用一种新的自适应动态最优化方法(ADP),在线实现对非线性连续系统的最优控制。首先应用汉密尔顿函数（Hamilton-Jacobi-Bellman, HJB）求解系统的最优控制,并应用神经网络BP算法对汉密尔顿函数中的性能指标进行估计,进而得到非线性连续系统的最优控制。同时引进一种新的自适应算法,基于参数误差,在线实现对系统进行动态最优求解,而且通过李亚普诺夫方法对参数收敛情况也进行详细的分析。最后,用仿真结果来验证所提出的方法的可行性。     

Predefined-time control for Euler–Lagrange systems with model uncertainties and actuator faults

A novel adaptive predefined-time tracking control algorithm is proposed for the Euler–Lagrange systems (ELSs) with model uncertainties and actuator faults. Compared with traditional finite-time and fixed-time studies, the system output tracking error under the proposed predefined-time controller converges to a small neighborhood of zero in finite time, whose upper bound is exactly a design parameter in the control algorithm. For the uncertain model, radial-based function neural network (RBFNN) is utilized to approximate the continuous uncertain dynamics. To deal with the actuator faults, an adaptive control law is involved in the fault-tolerant controller. In order to achieve the predefined-time bounded, a novel predefined-time sliding mode surface is designed. It is proved that the tracking error vector trajectory of closed-loop system is semi-globally uniformly ultimately predefined-time bounded, and the upper bounds of both the system settling time and the corresponding output tracking error can be adjusted with a simple parameter. Simulation examples finally demonstrate the effectiveness of the proposed control algorithm.     

Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

In this paper, an adaptive dynamic programming (ADP) strategy is investigated for discrete-time nonlinear systems with unknown nonlinear dynamics subject to input saturation. To save the communication resources between the controller and the actuators, stochastic communication protocols (SCPs) are adopted to schedule the control signal, and therefore the closed-loop system is essentially a protocol-induced switching system. A neural network (NN)-based identifier with a robust term is exploited for approximating the unknown nonlinear system, and a set of switch-based updating rules with an additional tunable parameter of NN weights are developed with the help of the gradient descent. By virtue of a novel Lyapunov function, a sufficient condition is proposed to achieve the stability of both system identification errors and the update dynamics of NN weights. Then, a value iterative ADP algorithm in an offline way is proposed to solve the optimal control of protocol-induced switching systems with saturation constraints, and the convergence is profoundly discussed in light of mathematical induction. Furthermore, an actor-critic NN scheme is developed to approximate the control law and the proposed performance index function in the framework of ADP, and the stability of the closed-loop system is analyzed in view of the Lyapunov theory. Finally, the numerical simulation results are presented to demonstrate the effectiveness of the proposed control scheme.      

Neural-based online finite-time optimal tracking control for wheeled mobile robotic system with inequality constraints

In this study, a finite-time online optimal controller was designed for a nonlinear wheeled mobile robotic system (WMRS) with inequality constraints, based on reinforcement learning (RL) neural networks. In addition, an extended cost function, obtained by introducing a penalty function to the original long-time cost function, was proposed to deal with the optimal control problem of the system with inequality constraints. A novel Hamilton-Jacobi-Bellman (HJB) equation containing the constraint conditions was defined to determine the optimal control input. Furthermore, two neural networks (NNs), a critic and an actor NN, were established to approximate the extended cost function and the optimal control input, respectively. The adaptation laws of the critic and actor NN were obtained with the gradient descent method. The semi-global practical finite-time stability (SGPFS) was proved using Lyapunov's stability theory. The tracking error converges to a small region near zero within the constraints in a finite period. Finally, the effectiveness of the proposed optimal controller was verified by a simulation based on a practical wheeled mobile robot model.     

Multiple model-based event-triggered adaptive control of a class of discrete-time nonlinear systems

In this study, the problem of event-triggered-based adaptive control (ETAC) for a class of discrete-time nonlinear systems with unknown parameters and nonlinear uncertainties is considered. Both neural network (NN) based and linear identifiers are used to approximate the unknown system dynamics. The feedback output signals are transmitted, and the parameters and the NN weights of the identifiers are tuned in an aperiodic manner at the event sample instants. A switching mechanism is provided to evaluate the approximate performance of each identifier and decide which estimated output is utilised for the event-triggered controller design, during any two events. The linear identifier with an auxiliary output and an improved adaptive law is introduced so that the nonlinear uncertainties are no longer assumed to be Lipschitz. The number of transmission times are significantly reduced by incorporating multiple model schemes into ETAC. The boundedness of both the parameters of identifiers and the system outputs is demonstrated though the Lyapunov approach. Simulation results demonstrate the effectiveness of the proposed method.     

Neural-network-observer-based optimal control for unknown nonlinear systems using adaptive dynamic programming

In this paper, an observer-based optimal control scheme is developed for unknown nonlinear systems using adaptive dynamic programming (ADP) algorithm. First, a neural-network (NN) observer is designed to estimate system states. Then, based on the observed states, a neuro-controller is constructed via ADP method to obtain the optimal control. In this design, two NN structures are used: a three-layer NN is used to construct the observer which can be applied to systems with higher degrees of nonlinearity and without a priori knowledge of system dynamics, and a critic NN is employed to approximate the value function. The optimal control law is computed using the critic NN and the observer NN. Uniform ultimate boundedness of the closed-loop system is guaranteed. The actor, critic, and observer structures are all implemented in real-time, continuously and simultaneously. Finally, simulation results are presented to demonstrate the effectiveness of the proposed control scheme.     

Model-free Adaptive Dynamic Programming Based Near-optimal Decentralized Tracking Control of Reconfigurable Manipulators

In this paper, a model-free near-optimal decentralized tracking control (DTC) scheme is developed for reconfigurable manipulators via adaptive dynamic programming algorithm. The proposed controller can be divided into two parts, namely local desired controller and local tracking error controller. In order to remove the normboundedness assumption of interconnections, desired states of coupled subsystems are employed to substitute their actual states. Using the local input/output data, the unknown subsystem dynamics of reconfigurable manipulators can be identified by constructing local neural network (NN) identifiers. With the help of the identified dynamics, the local desired control can be derived directly with corresponding desired states. Then, for tracking error subsystems, the local tracking error control is investigated by the approximate improved local cost function via local critic NN and the identified input gain matrix. To overcome the overall error caused by the substitution, identification and critic NN approximation, a robust compensation is added to construct the improved local cost function that reflects the overall error, regulation and control simultaneously. Therefore, the closed-loop tracking system can be guaranteed to be asymptotically stable via Lyapunov stability theorem. Two 2-degree of freedom reconfigurable manipulators with different configurations are employed to demonstrate the effectiveness of the proposed modelfree near-optimal DTC scheme.     

Event-Triggered Optimal Adaptive Control Algorithm for Continuous-Time Nonlinear Systems

This paper proposes a novel optimal adaptive eventtriggered control algorithm for nonlinear continuous-time systems. The goal is to reduce the controller updates, by sampling the state only when an event is triggered to maintain stability and optimality. The online algorithm is implemented based on an actor/critic neural network structure. A critic neural network is used to approximate the cost and an actor neural network is used to approximate the optimal event-triggered controller. Since in the algorithm proposed there are dynamics that exhibit continuous evolutions described by ordinary differential equations and instantaneous jumps or impulses, we will use an impulsive system approach. A Lyapunov stability proof ensures that the closed-loop system is asymptotically stable. Finally, we illustrate the effectiveness of the proposed solution compared to a timetriggered controller.      

Real-time torque control using discrete-time recurrent high-order neural networks

This paper presents a discrete-time direct current (DC) motor torque tracking controller, based on a recurrent high-order neural network to identify the plant model. In order to train the neural identifier, the extended Kalman filter (EKF) based training algorithm is used. The neural identifier is in series-parallel configuration that constitutes a well approximation method of the real plant by the neural identifier. Using the neural identifier structure that is in the nonlinear controllable form, the block control (BC) combined with sliding modes (SM) control techniques in discrete-time are applied. The BC technique is used to design a nonlinear sliding manifold such that the resulting sliding mode dynamics are described by a desired linear system. For the SM control technique, the equivalent control law is used in order to the plant output tracks a reference signal. For reducing the effect of unknown terms, it is proposed a specific desired dynamics for the sliding variables. The control problem is solved by the indirect approach, where an appropriate neural network (NN) identification model is selected; the NN parameters (synaptic weights) are adjusted according to a specific adaptive law (EKF), such that the response of the NN identifier approximates the response of the real plant for the same input. Then, based on the designed NN identifier a stabilizing or reference tracking controller is proposed (BC combined with SM). The proposed neural identifier and control applicability are illustrated by torque trajectory tracking for a DC motor with separate winding excitation via real-time implementation.     

Neural-network adaptive controller for nonlinear systems and its application in pneumatic servo systems

In this paper, a novel control law is presented, which uses neural-network techniques to approximate the affine class nonlinear system having unknown or uncertain dynamics and noise disturbances. It adopts an adaptive control law to adjust the network parameters online and adds another control component according to H-infinity control theory to attenuate the disturbance. This control law is applied to the position tracking control of pneumatic servo systems. Simulation and experimental results show that the tracking precision and convergence speed is obviously superior to the results by using the basic BP-network controller and self-tuning adaptive controller.     

Asymptotic tracking by a reinforcement learning-based adaptive critic controller

Adaptive critic (AC) based controllers are typically discrete and/or yield a uniformly ultimately bounded stability result because of the presence of disturbances and unknown approximation errors. A continuous-time AC controller is developed that yields asymptotic tracking of a class of uncertain nonlinear systems with bounded disturbances. The proposed AC-based controller consists of two neural networks (NNs) – an action NN, also called the actor, which approximates the plant dynamics and generates appropriate control actions; and a critic NN, which evaluates the performance of the actor based on some performance index. The reinforcement signal from the critic is used to develop a composite weight tuning law for the action NN based on Lyapunov stability analysis. A recently developed robust feedback technique, robust integral of the sign of the error (RISE), is used in conjunction with the feedforward action neural network to yield a semiglobal asymptotic result. Experimental results are provided that illustrate the performance of the developed controller.     

不确定机器人的自适应神经网络控制与学习

针对具有未知动态的电驱动机器人,研究其自适应神经网络控制与学习问题.首先,设计了稳定的自适应神经网络控制器,径向基函数(RBF)神经网络被用来逼近电驱动机器人的未知闭环系统动态,并根据李雅普诺夫稳定性理论推导了神经网络权值更新律.在对回归轨迹实现跟踪控制的过程中,闭环系统内部信号的部分持续激励(PE)条件得到满足.随着PE条件的满足,设计的自适应神经网络控制器被证明在稳定的跟踪控制过程中实现了电驱动机器人未知闭环系统动态的准确逼近.接着,使用学过的知识设计了新颖的学习控制器,实现了闭环系统稳定、改进了控制性能.最后,通过数字仿真验证了所提控制方法的正确性和有效性.