Generalized Hamilton–Jacobi–Bellman Formulation -Based Neural Network Control of Affine Nonlinear Discrete-Time Systems

In this paper, we consider the use of nonlinear networks towards obtaining nearly optimal solutions to the control of nonlinear discrete-time (DT) systems. The method is based on least squares successive approximation solution of the generalized Hamilton-Jacobi-Bellman (GHJB) equation which appears in optimization problems. Successive approximation using the GHJB has not been applied for nonlinear DT systems. The proposed recursive method solves the GHJB equation in DT on a well-defined region of attraction. The definition of GHJB, pre-Hamiltonian function, HJB equation, and method of updating the control function for the affine nonlinear DT systems under small perturbation assumption are proposed. A neural network (NN) is used to approximate the GHJB solution. It is shown that the result is a closed-loop control based on an NN that has been tuned a priori in offline mode. Numerical examples show that, for the linear DT system, the updated control laws will converge to the optimal control, and for nonlinear DT systems, the updated control laws will converge to the suboptimal control.     

An Iterative Method for Optimal Feedback Control and Generalized HJB Equation

In this paper, a new iterative method is proposed to solve the generalized Hamilton-Jacobi-Bellman (GHJB) equation through successively approximate it. Firstly, the GHJB equation is converted to an algebraic equation with the vector norm, which is essentially a set of simultaneous nonlinear equations in the case of dynamic systems. Then, the proposed algorithm solves GHJB equation numerically for points near the origin by considering the linearization of the non-linear equations under a good initial control guess. Finally, the procedure is proved to converge to the optimal stabilizing solution with respect to the iteration variable. In addition, it is shown that the result is a closed-loop control based on this iterative approach. Illustrative examples show that the update control laws will converge to optimal control for nonlinear systems.      

Galerkin approximationwith Legendre polynomials for a continuous-time nonlinear optimal control problem

We investigate the use of an approximation method for obtaining near-optimal solutions to a kind of nonlinear continuous-time (CT) system. The approach derived from the Galerkin approximation is used to solve the generalized Hamilton-Jacobi-Bellman (GHJB) equations. The Galerkin approximation with Legendre polynomials (GALP) for GHJB equations has not been applied to nonlinear CT systems. The proposed GALP method solves the GHJB equations in CT systems on some well-defined region of attraction. The integrals that need to be computed are much fewer due to the orthogonal properties of Legendre polynomials, which is a significant advantage of this approach. The stabilization and convergence properties with regard to the iterative variable have been proved. Numerical examples show that the update control laws converge to the optimal control for nonlinear CT systems.     

Online adaptive optimal control for continuous-time nonlinear systems with completely unknown dynamics

An online adaptive optimal control is proposed for continuous-time nonlinear systems with completely unknown dynamics, which is achieved by developing a novel identifier-critic-based approximate dynamic programming algorithm with a dual neural network (NN) approximation structure. First, an adaptive NN identifier is designed to obviate the requirement of complete knowledge of system dynamics, and a critic NN is employed to approximate the optimal value function. Then, the optimal control law is computed based on the information from the identifier NN and the critic NN, so that the actor NN is not needed. In particular, a novel adaptive law design method with the parameter estimation error is proposed to online update the weights of both identifier NN and critic NN simultaneously, which converge to small neighbourhoods around their ideal values. The closed-loop system stability and the convergence to small vicinity around the optimal solution are all proved by means of the Lyapunov theory. The proposed adaptation algorithm is also improved to achieve finite-time convergence of the NN weights. Finally, simulation results are provided to exemplify the efficacy of the proposed methods.     

Direct adaptive control for a class of discrete‐time unknown nonaffine nonlinear systems using neural networks

In this paper, a direct adaptive state‐feedback control approach is developed for a class of nonlinear systems in discrete‐time (DT) domain. We study MIMO unknown nonaffine nonlinear DT systems and employ a two‐layer NN to design the controller. By using the presented method, the NN approximation is able to cancel the nonlinearity of the unknown DT plant. Meanwhile, pretraining is not required, and the weights of NNs used in adaptive control are directly updated online. Moreover, unlike standard NN adaptive controllers yielding uniform ultimate boundedness results, the tracking error is guaranteed to be uniformly asymptotically stable by utilizing Lyapunov's direct method. Two illustrative examples are provided to demonstrate the effectiveness and the applicability of the theoretical results. Copyright © 2014 John Wiley & Sons, Ltd.     

Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation

In this paper we study the convergence of the Galerkin approximation method applied to the generalized Hamilton-Jacobi-Bellman (GHJB) equation over a compact set containing the origin. The GHJB equation gives the cost of an arbitrary control law and can be used to improve the performance of this control. The GHJB equation can also be used to successively approximate the Hamilton-Jacobi-Bellman equation. We state sufficient conditions that guarantee that the Galerkin approximation converges to the solution of the GHJB equation and that the resulting approximate control is stabilizing on the same region as the initial control. The method is demonstrated on a simple nonlinear system and is compared to a result obtained by using exact feedback linearization in conjunction with the LQR design method.     

Online Adaptive Approximate Optimal Tracking Control with Simplified Dual Approximation Structure for Continuous-time Unknown Nonlinear Systems

This paper proposes an online adaptive approximate solution for the infinite-horizon optimal tracking control problem of continuous-time nonlinear systems with unknown dynamics. The requirement of the complete knowledge of system dynamics is avoided by employing an adaptive identifier in conjunction with a novel adaptive law, such that the estimated identifier weights converge to a small neighborhood of their ideal values. An adaptive steady-state controller is developed to maintain the desired tracking performance at the steady-state, and an adaptive optimal controller is designed to stabilize the tracking error dynamics in an optimal manner. For this purpose, a critic neural network (NN) is utilized to approximate the optimal value function of the Hamilton-Jacobi-Bellman (HJB) equation, which is used in the construction of the optimal controller. The learning of two NNs, i.e., the identifier NN and the critic NN, is continuous and simultaneous by means of a novel adaptive law design methodology based on the parameter estimation error. Stability of the whole system consisting of the identifier NN, the critic NN and the optimal tracking control is guaranteed using Lyapunov theory; convergence to a near-optimal control law is proved. Simulation results exemplify the effectiveness of the proposed method.      

Adaptive neural control with fast approximation for uncertain nonlinear systems: A novel composite learning approach

In existing adaptive neural control approaches, only when the regressor satisfies the persistent excitation (PE) or interval excitation (IE) conditions, the constant optimal weights of neural network (NN) can be identified, which can be used to establish uncertainties in nonlinear systems. This paper proposes a novel composite learning approach based on adaptive neural control. The focus of this approach is to make the NN approximate uncertainties in nonlinear systems quickly and accurately without identifying the constant optimal weights of the NN. Hence, the regressor does not need to satisfy the PE or IE conditions. In this paper, regressor filtering scheme is adopted to generate prediction error, and then the prediction error and tracking error simultaneously drive the update of NN weights. Under the framework of Lyapulov theory, the proposed composite learning approach can ensure that approximation error of the uncertainty and tracking error of the system states converge to an arbitrarily small neighborhood of zero exponentially. The simulation results verify the effectiveness and advantages of the proposed approach in terms of fast approximation.     

State observer design for nonlinear systems using neural network

In this paper, an observer design is proposed for nonlinear systems. The Hamilton–Jacobi–Bellman (HJB) equation based formulation has been developed. The HJB equation is formulated using a suitable non-quadratic term in the performance functional to tackle magnitude constraints on the observer gain. Utilizing Lyapunov's direct method, observer is proved to be optimal with respect to meaningful cost. In the present algorithm, neural network (NN) is used to approximate value function to find approximate solution of HJB equation using least squares method. With time-varying HJB solution, we proposed a dynamic optimal observer for the nonlinear system. Proposed algorithm has been applied on nonlinear systems with finite-time-horizon and infinite-time-horizon. Necessary theoretical and simulation results are presented to validate proposed algorithm.     

Optimized tracking control using reinforcement learning strategy for a class of nonlinear systems

This paper is to develop a simplified optimized tracking control using reinforcement learning (RL) strategy for a class of nonlinear systems. Since the nonlinear control gain function is considered in the system modeling, it is challenging to extend the existing RL-based optimal methods to the tracking control. The main reasons are that these methods' algorithm are very complex; meanwhile, they also require to meet some strict conditions. Different with these exiting RL-based optimal methods that derive the actor and critic training laws from the square of Bellman residual error, which is a complex function consisting of multiple nonlinear terms, the proposed optimized scheme derives the two RL training laws from negative gradient of a simple positive function, so that the algorithm can be significantly simplified. Moreover, the actor and critic in RL are constructed by employing neural network (NN) to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation. Finally, the feasibility of the proposed method is demonstrated in accordance with both Lyapunov stability theory and simulation example.     

Consensus-based distributed cooperative learning control for a group of discrete-time nonlinear multi-agent systems using neural networks

This paper focuses on the cooperative learning capability of radial basis function neural networks in adaptive neural controllers for a group of uncertain discrete-time nonlinear systems where system structures are identical but reference signals are different. By constructing an interconnection topology among learning laws of NN weights in order to share their learned knowledge on-line, a novel adaptive NN control scheme, called distributed cooperative learning control scheme, is proposed. It is guaranteed that if the interconnection topology is undirected and connected, all closed-loop signals are uniform ultimate bounded and tracking errors of all systems can converge to a small neighborhood around the origin. Moreover, it is proved that all estimated NN weights converge to a small neighborhood of their common optimal value along the union of all state trajectories, which means that the estimated NN weights reach consensus with a small consensus error. Thus, all learned NN models have the better generalization capability than ones obtained by the deterministic learning method. The learned knowledge is also adopted to control a class of uncertain systems with the same structure but different reference signals. Finally, a simulation example is provided to verify the effectiveness and advantages of the distributed cooperative learning control scheme developed in this paper.     

Near Optimal Output Feedback Control of Nonlinear Discrete-time Systems Based on Reinforcement Neural Network Learning

In this paper, the output feedback based finitehorizon near optimal regulation of nonlinear affine discretetime systems with unknown system dynamics is considered by using neural networks (NNs) to approximate Hamilton-Jacobi-Bellman (HJB) equation solution. First, a NN-based Luenberger observer is proposed to reconstruct both the system states and the control coefficient matrix. Next, reinforcement learning methodology with actor-critic structure is utilized to approximate the time-varying solution, referred to as the value function, of the HJB equation by using a NN. To properly satisfy the terminal constraint, a new error term is defined and incorporated in the NN update law so that the terminal constraint error is also minimized over time. The NN with constant weights and timedependent activation function is employed to approximate the time-varying value function which is subsequently utilized to generate the finite-horizon near optimal control policy due to NN reconstruction errors. The proposed scheme functions in a forward-in-time manner without offline training phase. Lyapunov analysis is used to investigate the stability of the overall closedloop system. Simulation results are given to show the effectiveness and feasibility of the proposed method.      

Online optimal control of nonlinear discrete-time systems using approximate dynamic programming

In this paper,the optimal control of a class of general affine nonlinear discrete-time(DT) systems is undertaken by solving the Hamilton Jacobi-Bellman(HJB) equation online and forward in time.The proposed approach,referred normally as adaptive or approximate dynamic programming(ADP),uses online approximators(OLAs) to solve the infinite horizon optimal regulation and tracking control problems for affine nonlinear DT systems in the presence of unknown internal dynamics.Both the regulation and tracking contro...     

非线性扩展结构大系统自适应神经网络跟踪控制

针对一类非线性关联大系统在结构扩展时的跟踪控制问题, 提出一种采用自适应神经网络的控制方法. 该方法要求在不改变原结构系统控制律的前提下设计新加入子系统的控制律和自适应律, 使扩展后所有子系统都具有很好的跟踪性能. 这里主要利用神经网络的逼近功能以及Backstepping 技术来设计自适应律和控制律, 通过Lyapunov 理论证明在该控制器的作用下闭环系统的所有信号均是有界的, 并可使系统准确跟踪. 仿真结果验证了所提出方法的有效性.

     

Adaptive critic learning for event-triggered safe control of nonlinear safety-critical systems

In this paper, an event-triggered safe control method based on adaptive critic learning (ACL) is proposed for a class of nonlinear safety-critical systems. First, a safe cost function is constructed by adding a control barrier function (CBF) to the traditional quadratic cost function; the optimization problem with safety constraints that is difficult to deal with by classical ACL methods is solved. Subsequently, the event-triggered scheme is introduced to reduce the amount of computation. Further, combining the properties of CBF with the ACL-based event-triggering mechanism, the event-triggered safe Hamilton–Jacobi–Bellman (HJB) equation is derived, and a single critic neural network (NN) framework is constructed to approximate the solution of the event-triggered safe HJB equation. In addition, the concurrent learning method is applied to the NN learning process, so that the persistence of excitation (PE) condition is not required. The weight approximation error of the NN and the states of the system are proven to be uniformly ultimately bounded (UUB) in the safe set with the Lyapunov theory. Finally, the availability of the presented method can be validated through the simulation.     

An online fault tolerant actor-critic neuro-control for a class of nonlinear systems using neural network HJB approach

In this paper, we propose an actor-critic neuro-control for a class of continuous-time nonlinear systems under nonlinear abrupt faults, which is combined with an adaptive fault diagnosis observer (AFDO). Together with its estimation laws, an AFDO scheme, which estimates the faults in real time, is designed based on Lyapunov analysis. Then, based on the designed AFDO, a fault tolerant actor- critic control scheme is proposed where the critic neural network (NN) is used to approximate the value function and the actor NN updates the fault tolerant policy based on the approximated value function in the critic NN. The weight update laws for critic NN and actor NN are designed using the gradient descent method. By Lyapunov analysis, we prove the uniform ultimately boundedness (UUB) of all the states, their estimation errors, and NN weights of the fault tolerant system under the unpredictable faults. Finally, we verify the effectiveness of the proposed method through numerical simulations.     

动态规划最优控制在非线性系统中的应用

应用一种新的自适应动态最优化方法(ADP),在线实现对非线性连续系统的最优控制。首先应用汉密尔顿函数（Hamilton-Jacobi-Bellman, HJB）求解系统的最优控制,并应用神经网络BP算法对汉密尔顿函数中的性能指标进行估计,进而得到非线性连续系统的最优控制。同时引进一种新的自适应算法,基于参数误差,在线实现对系统进行动态最优求解,而且通过李亚普诺夫方法对参数收敛情况也进行详细的分析。最后,用仿真结果来验证所提出的方法的可行性。     

Adaptive radial basis function neural network control with variable variance parameters

The paper presents a direct adaptive control architecture for a class of nonlinear dynamic systems, which are either ill defined or rather complex. The direct adaptive architecture employs radial basis function (RBF) neural network (NN) systems to reconstruct the ideal feedback linearization control. With the modified adaptation algorithm proposed herein, the on-line function approximation capability of the RBF NN a system is enhanced to remove the auxiliary control term and switching element in a conventional RBF-NN-based controller; simultaneously, the tracking performance is upgraded. Global asymptotic stability of the on-line algorithm is established in the Lyapunov sense to guarantee that the tracking error can converge to a small neighbourhood of the origin. Simulation validations for an inverted pendulum system are finally performed to verify the effectiveness of the proposed controller and the theoretical discussion.     

一类非线性奇异摄动系统的近似最优控制

针对一类非线性奇异摄动系统,基于自适应动态规划算法提出了一种新型的近似最优控制设计方法.该方法基于奇异摄动系统的快、慢Hamilton-Jacobi-Bellman(HJB)方程,从初始性能指标开始,通过神经网络的近似和控制律与性能指标的逐步更新迭代,最终收敛到最优的性能指标,而不用直接求解复杂的HJB方程.同时给出了...     

Optimal State Estimation and Fault Diagnosis for a Class of Nonlinear Systems

This study proposes a scheme for state estimation and,consequently,fault diagnosis in nonlinear systems.Initially,an optimal nonlinear observer is designed for nonlinear systems subject to an actuator or plant fault.By utilizing Lyapunov's direct method,the observer is proved to be optimal with respect to a performance function,including the magnitude of the observer gain and the convergence time.The observer gain is obtained by using approximation of Hamilton-Jacobi-Bellman(HJB)equation.The approximation is determined via an online trained neural network(NN).Next a class of affine nonlinear systems is considered which is subject to unknown disturbances in addition to fault signals.In this case,for each fault the original system is transformed to a new form in which the proposed optimal observer can be applied for state estimation and fault detection and isolation(FDI).Simulation results of a singlelink flexible joint robot(SLFJR)electric drive system show the effectiveness of the proposed methodology.