Generalized Hamilton–Jacobi–Bellman Formulation -Based Neural Network Control of Affine Nonlinear Discrete-Time Systems

In this paper, we consider the use of nonlinear networks towards obtaining nearly optimal solutions to the control of nonlinear discrete-time (DT) systems. The method is based on least squares successive approximation solution of the generalized Hamilton-Jacobi-Bellman (GHJB) equation which appears in optimization problems. Successive approximation using the GHJB has not been applied for nonlinear DT systems. The proposed recursive method solves the GHJB equation in DT on a well-defined region of attraction. The definition of GHJB, pre-Hamiltonian function, HJB equation, and method of updating the control function for the affine nonlinear DT systems under small perturbation assumption are proposed. A neural network (NN) is used to approximate the GHJB solution. It is shown that the result is a closed-loop control based on an NN that has been tuned a priori in offline mode. Numerical examples show that, for the linear DT system, the updated control laws will converge to the optimal control, and for nonlinear DT systems, the updated control laws will converge to the suboptimal control.     

Galerkin approximationwith Legendre polynomials for a continuous-time nonlinear optimal control problem

We investigate the use of an approximation method for obtaining near-optimal solutions to a kind of nonlinear continuous-time (CT) system. The approach derived from the Galerkin approximation is used to solve the generalized Hamilton-Jacobi-Bellman (GHJB) equations. The Galerkin approximation with Legendre polynomials (GALP) for GHJB equations has not been applied to nonlinear CT systems. The proposed GALP method solves the GHJB equations in CT systems on some well-defined region of attraction. The integrals that need to be computed are much fewer due to the orthogonal properties of Legendre polynomials, which is a significant advantage of this approach. The stabilization and convergence properties with regard to the iterative variable have been proved. Numerical examples show that the update control laws converge to the optimal control for nonlinear CT systems.     

Generalized hamilton-jacobi-bellman formulation -based neural network control of affine nonlinear discrete-time systems

In this paper, we consider the use of nonlinear networks towards obtaining nearly optimal solutions to the control of nonlinear discrete-time (DT) systems. The method is based on least squares successive approximation solution of the generalized Hamilton-Jacobi-Bellman (GHJB) equation which appears in optimization problems. Successive approximation using the GHJB has not been applied for nonlinear DT systems. The proposed recursive method solves the GHJB equation in DT on a well-defined region of attraction. The definition of GHJB, pre-Hamiltonian function, HJB equation, and method of updating the control function for the affine nonlinear DT systems under small perturbation assumption are proposed. A neural network (NN) is used to approximate the GHJB solution. It is shown that the result is a closed-loop control based on an NN that has been tuned a priori in offline mode. Numerical examples show that, for the linear DT system, the updated control laws will converge to the optimal control, and for nonlinear DT systems, the updated control laws will converge to the suboptimal control.     

Numerical methods for the solution of the degenerate nonlinear elliptic equations arising in optimal stochastic control theory

Three distinct but related results are obtained. First, an iterative method is derived for obtaining the solution of optimal control problems for Markov chains. The method usually converges much faster, and requires less computer storage space, than the methods of Howard or Eaton and Zadeh. Second, nonlinear finite difference equations, which "approximate" the nonlinear degenerate elliptic equation (2) arising out of the stochastic optimization problem (1), are found. The difference equations, and their solution, may have a meaning for the control problem even when it cannot be proved that (2) has a solution. The iterative methods for the iterative solution of these nonlinear systems are discussed and compared. Both converge to the solution, provided that the difference equations were derived using the method introduced in the paper; one, new to this paper, often much faster than the other (Theorem 2). In fact, the typical time required for the numerical solution is about the time required for a related linear problem. The method of obtaining the difference equations, and the proof of convergence of the associated iterative procedures, are illustrated by a detailed example. Finally, specific numerical results for a "minimum average time" type of optimization problem are presented and discussed.     

Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation

In this paper we study the convergence of the Galerkin approximation method applied to the generalized Hamilton-Jacobi-Bellman (GHJB) equation over a compact set containing the origin. The GHJB equation gives the cost of an arbitrary control law and can be used to improve the performance of this control. The GHJB equation can also be used to successively approximate the Hamilton-Jacobi-Bellman equation. We state sufficient conditions that guarantee that the Galerkin approximation converges to the solution of the GHJB equation and that the resulting approximate control is stabilizing on the same region as the initial control. The method is demonstrated on a simple nonlinear system and is compared to a result obtained by using exact feedback linearization in conjunction with the LQR design method.     

A hybrid algorithm for approximate optimal control of nonlinear Fredholm integral equations

In this paper, a novel hybrid method based on two approaches, evolutionary algorithms and an iterative scheme, for obtaining the approximate solution of optimal control governed by nonlinear Fredholm integral equations is presented. By converting the problem to a discretized form, it is considered as a quasi-assignment problem and then an iterative method is applied to find an approximate solution for the discretized form of the integral equation. An analysis for convergence of the proposed iterative method and its implementation for numerical examples are also given.     

Finite horizon optimal control of discrete-time nonlinear systems with unfixed initial state using adaptive dynamic programming

In this paper, we aim to solve the finite horizon optimal control problem for a class of discrete-time nonlinear systems with unfixed initial state using adaptive dynamic programming (ADP) approach. A new ε-optimal control algorithm based on the iterative ADP approach is proposed which makes the performance index function converge iteratively to the greatest lower bound of all performance indices within an error according to ε within finite time. The optimal number of control steps can also be obtained by the proposed ε-optimal control algorithm for the situation where the initial state of the system is unfixed. Neural networks are used to approximate the performance index function and compute the optimal control policy, respectively, for facilitating the implementation of the ε-optimal control algorithm. Finally, a simulation example is given to show the results of the proposed method.     

Stable iterative adaptive dynamic programming algorithm with approximation errors for discrete-time nonlinear systems

In this paper, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon optimal control problems for discrete-time nonlinear systems. When the iterative control law and iterative performance index function in each iteration cannot be accurately obtained, it is shown that the iterative controls can make the performance index function converge to within a finite error bound of the optimal performance index function. Stability properties are presented to show that the system can be stabilized under the iterative control law which makes the present iterative ADP algorithm feasible for implementation both on-line and off-line. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.     

Direct adaptive iterative learning control of nonlinear systems using an output-recurrent fuzzy neural network

In this paper, a direct adaptive iterative learning control (DAILC) based on a new output-recurrent fuzzy neural network (ORFNN) is presented for a class of repeatable nonlinear systems with unknown nonlinearities and variable initial resetting errors. In order to overcome the design difficulty due to initial state errors at the beginning of each iteration, a concept of time-varying boundary layer is employed to construct an error equation. The learning controller is then designed by using the given ORFNN to approximate an optimal equivalent controller. Some auxiliary control components are applied to eliminate approximation error and ensure learning convergence. Since the optimal ORFNN parameters for a best approximation are generally unavailable, an adaptive algorithm with projection mechanism is derived to update all the consequent, premise, and recurrent parameters during iteration processes. Only one network is required to design the ORFNN-based DAILC and the plant nonlinearities, especially the nonlinear input gain, are allowed to be totally unknown. Based on a Lyapunov-like analysis, we show that all adjustable parameters and internal signals remain bounded for all iterations. Furthermore, the norm of state tracking error vector will asymptotically converge to a tunable residual set as iteration goes to infinity. Finally, iterative learning control of two nonlinear systems, inverted pendulum system and Chua's chaotic circuit, are performed to verify the tracking performance of the proposed learning scheme.     

An Iterative Relaxation Approach to the Solution of the Hamilton-Jacobi-Bellman-Isaacs Equation in Nonlinear Optimal Control

In this paper, we propose an iterative relaxation method for solving the Hamilton-Jacobi-Bellman-Isaacs equation (HJBIE) arising in deterministic optimal control of affine nonlinear systems. Local convergence of the method is established under fairly mild assumptions, and examples are solved to demonstrate the effectiveness of the method. An extension of the approach to Lyapunov equations is also discussed. The preliminary results presented are promising, and it is hoped that the approach will ultimately develop into an efficient computational tool for solving the HJBIEs.      

Computationally efficient simultaneous policy update algorithm for nonlinear H∞ state feedback control with Galerkin's method

The main bottleneck for the application of H_∞ control theory on practical nonlinear systems is the need to solve the Hamilton–Jacobi–Isaacs (HJI) equation. The HJI equation is a nonlinear partial differential equation (PDE) that has proven to be impossible to solve analytically, even the approximate solution is still difficult to obtain. In this paper, we propose a simultaneous policy update algorithm (SPUA), in which the nonlinear HJI equation is solved by iteratively solving a sequence of Lyapunov function equations that are linear PDEs. By constructing a fixed point equation, the convergence of the SPUA is established rigorously by proving that it is essentially a Newton's iteration method for finding the fixed point. Subsequently, a computationally efficient SPUA (CESPUA) based on Galerkin's method, is developed to solve Lyapunov function equations in each iterative step of SPUA. The CESPUA is simple for implementation because only one iterative loop is included. Through the simulation studies on three examples, the results demonstrate that the proposed CESPUA is valid and efficient. Copyright © 2012 John Wiley & Sons, Ltd.     

Neuro-optimal tracking control for a class of discrete-time nonlinear systems via generalized value iteration adaptive dynamic programming approach

In this paper, a novel value iteration adaptive dynamic programming (ADP) algorithm, called “generalized value iteration ADP” algorithm, is developed to solve infinite horizon optimal tracking control problems for a class of discrete-time nonlinear systems. The developed generalized value iteration ADP algorithm permits an arbitrary positive semi-definite function to initialize it, which overcomes the disadvantage of traditional value iteration algorithms. Convergence property is developed to guarantee that the iterative performance index function will converge to the optimum. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, a simulation example is given to illustrate the performance of the developed algorithm.     

非线性离散系统的近似最优跟踪控制

研究非线性离散系统的最优跟踪控制问题. 通过在由最优控制问题所导致的非线性两点边值问题中引入灵敏度参数, 并对它进行Maclaurin级数展开, 将原最优跟踪控制问题转化为一族非齐次线性两点边值问题. 得到的最优跟踪控制由解析的前馈反馈项和级数形式的补偿项组成. 解析的前馈反馈项可以由求解一个Riccati差分方程和一个矩阵差分方程得到. 级数补偿项可以由一个求解伴随向量的迭代算法近似求得. 以连续槽式反应器为例进行仿真验证了该方法的有效性．     

An integrated optimal control algorithm for discrete-time nonlinear stochastic system

Consider a discrete-time nonlinear system with random disturbances appearing in the real plant and the output channel where the randomly perturbed output is measurable. An iterative procedure based on the linear quadratic Gaussian optimal control model is developed for solving the optimal control of this stochastic system. The optimal state estimate provided by Kalman filtering theory and the optimal control law obtained from the linear quadratic regulator problem are then integrated into the dynamic integrated system optimisation and parameter estimation algorithm. The iterative solutions of the optimal control problem for the model obtained converge to the solution of the original optimal control problem of the discrete-time nonlinear system, despite model-reality differences, when the convergence is achieved. An illustrative example is solved using the method proposed. The results obtained show the effectiveness of the algorithm proposed.     

退化高阶抛物型分布参数系统的迭代学习控制

研究一类高阶分布参数系统的迭代学习控制问题,该类系统由退化高阶抛物型偏微分方程构成.根据系统所满足的性质,基于P型学习算法构建得到迭代学习控制器.利用压缩映射原理,证明该算法能使得系统的输出跟踪误差于L~2空间内沿迭代轴方向收敛于零.最后,仿真算例验证了算法的有效性.     

Optimal existence conditions for second order periodic solutions of delay differential equations with upper and lower solutions in the reverse order

In this paper we show that the monotone iterative technique provides two monotone sequences that converge uniformly to extremal (periodic) solutions of second order delay differential equations without assuming properties of monotonicity in the nonlinear part. Moreover, we obtain optimal existence conditions with upper and lower solutions in the reverse order. Our results are new even for ordinary differential equations.     

基于无线传感器网络的控制系统采样频率优化算法

针对基于无线传感网的网络化控制系统，讨论了采样频率的优化问题．建立了以数字和模拟控制系统性能差距指数最低为目标，以无线节点的通信容量为约束条件的非线性优化模型，并以障碍函数法进行求解，提出了基于节点缓冲区信息的分布武迭代算法．该算法在传感节点的计算量小，易于实现．仿真表明该算法能有效收敛到系统的最优目标点，并能适应于系统的负载变化．     

Adaptive iterative learning control for switched nonlinear continuous-time systems

In this paper, an adaptive iterative learning control (ILC) method is proposed for switched nonlinear continuous-time systems with time-varying parametric uncertainties. First, an iterative learning controller is constructed with a state feedback term in the time domain and an adaptive learning term in the iteration domain. Then a switched nonlinear continuous-discrete two-dimensional (2D) system is built to describe the adaptive ILC system. Multiple 2D Lyapunov functions-based analysis ensures that the 2D system is exponentially stable, and the tracking error will converge to zero in the iteration domain. The design method of the iterative learning controller is obtained by solving a linear matrix inequality. Finally, the efficacy of the proposed controller is demonstrated by the simulation results.     

Energy control of distributed parameter systems via speed-gradient method: case study of string and sine-Gordon benchmark models

Energy control problems are analysed for infinite dimensional systems. Benchmark linear wave equation and nonlinear sine-Gordon equation are chosen for exposition. The relatively simple case of distributed yet uniform over the space control is considered. The speed-gradient method for energy control of Hamiltonian systems proposed by A. Fradkov in 1996, has already successfully been applied to numerous nonlinear and adaptive control problems is presently developed and justified for the above partial differential equations (PDEs). An infinite dimensional version of the Krasovskii–LaSalle principle is validated for the resulting closed-loop systems. By applying this principle, the closed-loop trajectories are shown to either approach the desired energy level set or converge to a system equilibrium. The numerical study of the underlying closed-loop systems reveals reasonably fast transient processes and the feasibility of a desired energy level if initialised with a lower energy level.     

Adaptive iterative learning controller with input learning technique for a class of uncertain MIMO nonlinear systems

In this paper, an adaptive iterative learning controller (AILC) with input learning technique is presented for uncertain multi-input multi-output (MIMO) nonlinear systems in the normal form. The proposed AILC learns the internal parameter of the state equation as well as the input gain parameter, and also estimates the desired input using an input learning rule to track the whole history of command trajectory. The features of the proposed control scheme can be briefly summarized as follows: 1) To the best of authors’ knowledge, the AILC with input learning is first developed for uncertain MIMO nonlinear systems in the normal form; 2) The convergence of learning input error is ensured; 3) The input learning rule is simple; therefore, it can be easily implemented in industrial applications. With the proposed AILC scheme, the tracking error and desired input error converge to zero as the repetition of the learning operation increases. Single-link and two-link manipulators are presented as simulation examples to confirm the feasibility and performance of the proposed AILC.