Application of stochastic optimal control to a hydrofoil boat problem

In this paper the solution of a stochastic optimal control problem described by linear equations of motion and a nonquadratic performance index is presented. The theory is then applied to the dynamics of a single-foil and a hydrofoil boat flying on rough water. The random disturbances caused by sea waves are represented as the response of an auxiliary system to a white noise input. The control objective is formulated as an integral performance index containing a quadratic acceleration term and a nonquadratic term of the submergence deviation of the foil from calm water submergence. The stochastic version of the maximum principle is used in the formulation of a feedback control law. The Riccati equations and the feedback gains associated with a nonquadratic performance index are non-linear functions of the state and auxiliary state variables. These equations are integrated forward with the state equations for the steady-state solution of the problem. The controller for a nonquadratic performance index contains computing elements which perform the integration of the Riccati equations to generate the instantaneous values of the feedback gains. The effect of a nonquadratic penalty on the submergence deviation and the effect of a nonquadratic control penalty on the response of the system are investigated. A comparison between an optimal nonlinear control law and a suboptimal linear control law is presented.     

Discrete time neural network synthesis using input and outputactivation functions

A new very fast algorithm for synthesis of a new structure of discrete-time neural networks (NN) is proposed. For this purpose the following concepts are employed: (i) combination of input and output activation functions, (ii) input time-varying signal distribution, (iii) time-discrete domain synthesis and (iv) one-step learning iteration approach. The problem of input-output mappings of time-varying vectors is solved. Simulation results based on the synthesis of a new structure of feedforward NN of an universal logical unit are presented. The proposed NN synthesis procedure is useful for applications to identification and control of nonlinear, very fast, dynamical systems. In this sense a feedforward NN for an adaptive nonlinear robot control is designed. Finally, a new algorithm for the direct inverse modeling of input/output nonquadratic systems is discussed.     

不对称约束多人非零和博弈的自适应评判控制

本文针对连续时间非线性系统的不对称约束多人非零和博弈问题, 建立了一种基于神经网络的自适应评判控制方法. 首先, 本文提出了一种新颖的非二次型函数来处理不对称约束问题, 并且推导出最优控制律和耦合Hamilton-Jacobi方程. 值得注意的是, 当系统状态为零时, 最优控制策略是不为零的, 这与以往不同. 然后, 通过构建单一评判网络来近似每个玩家的最优代价函数, 从而获得相关的近似最优控制策略. 同时, 在评判学习期间发展了一种新的权值更新规则. 此外, 通过利用Lyapunov理论证明了评判网络权值近似误差和闭环系统状态的稳定性. 最后, 仿真结果验证了本文所提方法的有效性     

Training trajectories by continuous recurrent multilayer networks

This paper addresses the problem of training trajectories by means of continuous recurrent neural networks whose feedforward parts are multilayer perceptrons. Such networks can approximate a general nonlinear dynamic system with arbitrary accuracy. The learning process is transformed into an optimal control framework where the weights are the controls to be determined. A training algorithm based upon a variational formulation of Pontryagin's maximum principle is proposed for such networks. Computer examples demonstrating the efficiency of the given approach are also presented.     

A neural-network-based iterative GDHP approach for solving a class of nonlinear optimal control problems with control constraints

In this paper, a novel neural-network-based iterative adaptive dynamic programming (ADP) algorithm is proposed. It aims at solving the optimal control problem of a class of nonlinear discrete-time systems with control constraints. By introducing a generalized nonquadratic functional, the iterative ADP algorithm through globalized dual heuristic programming technique is developed to design optimal controller with convergence analysis. Three neural networks are constructed as parametric structures to facilitate the implementation of the iterative algorithm. They are used for approximating at each iteration the cost function, the optimal control law, and the controlled nonlinear discrete-time system, respectively. A simulation example is also provided to verify the effectiveness of the control scheme in solving the constrained optimal control problem.     

一类控制受约束非线性系统的基于单网络贪婪迭代DHP算法的近似最优镇定

提出一种贪婪迭代DHP (Dual heuristic programming)算法, 解决了一类控制受约束非线性系统的近似最优镇定问题. 针对系统的控制约束, 首先引入一个非二次泛函把约束问题转换为无约束问题, 然后基于协状态函数提出一种贪婪迭代DHP算法以求解系统的HJB (Hamilton-Jacobi-Bellman)方程. 在算法的每个迭代步, 利用一个神经网络来近似系统的协状态函数, 而后根据协状态函数直接计算系统的最优控制策略, 从而消除了常规近似动态规划方法中的控制网络. 最后通过两个仿真例子证明了本文提出的最优控制方案的有效性和可行性.     

Bounded robust control of nonlinear systems using neural network�Cbased HJB solution

In this paper, a Hamilton–Jacobi–Bellman (HJB) equation–based optimal control algorithm for robust controller design is proposed for nonlinear systems. The HJB equation is formulated using a suitable nonquadratic term in the performance functional to tackle constraints on the control input. Utilizing the direct method of Lyapunov stability, the controller is shown to be optimal with respect to a cost functional, which includes penalty on the control effort and the maximum bound on system uncertainty. The bounded controller requires the knowledge of the upper bound of system uncertainty. In the proposed algorithm, neural network is used to approximate the solution of HJB equation using least squares method. Proposed algorithm has been applied on the nonlinear system with matched and unmatched type system uncertainties and uncertainties in the input matrix. Necessary theoretical and simulation results are presented to validate proposed algorithm.     

Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning

In this paper, a new formulation for the optimal tracking control problem (OTCP) of continuous-time nonlinear systems is presented. This formulation extends the integral reinforcement learning (IRL) technique, a method for solving optimal regulation problems, to learn the solution to the OTCP. Unlike existing solutions to the OTCP, the proposed method does not need to have or to identify knowledge of the system drift dynamics, and it also takes into account the input constraints a priori. An augmented system composed of the error system dynamics and the command generator dynamics is used to introduce a new nonquadratic discounted performance function for the OTCP. This encodes the input constrains into the optimization problem. A tracking Hamilton–Jacobi–Bellman (HJB) equation associated with this nonquadratic performance function is derived which gives the optimal control solution. An online IRL algorithm is presented to learn the solution to the tracking HJB equation without knowing the system drift dynamics. Convergence to a near-optimal control solution and stability of the whole system are shown under a persistence of excitation condition. Simulation examples are provided to show the effectiveness of the proposed method.     

Neural-Network-Based Near-Optimal Control for a Class of Discrete-Time Affine Nonlinear Systems With Control Constraints

In this paper, the near-optimal control problem for a class of nonlinear discrete-time systems with control constraints is solved by iterative adaptive dynamic programming algorithm. First, a novel nonquadratic performance functional is introduced to overcome the control constraints, and then an iterative adaptive dynamic programming algorithm is developed to solve the optimal feedback control problem of the original constrained system with convergence analysis. In the present control scheme, there are three neural networks used as parametric structures for facilitating the implementation of the iterative algorithm. Two examples are given to demonstrate the convergence and feasibility of the proposed optimal control scheme.     

A CMAC neural-network-based algorithm for the kinematic control of a walking machine

In this paper a hierarchical, neural network control architecture of a walking machine is proposed. The neural network is based on the theory of the Cerebellum Model Articulation Controller (CMAC) which is a neuromuscular control system. Some preliminary studies of kinematic control and gait synthesis are presented to demonstrate the effectiveness of the CMAC neural network. After having been trained to learn the multivariable, nonlinear relationships of the leg kinematics and gaits, CMAC is utilized to perform feedforward kinematic control of a quadruped in straight-line walking and step climbing. Simulation examples are provided and discussed. This algorithm can be extended to control other highly nonlinear processes which are hierarchical in nature and cannot be modeled by mathematical equations.     

Partial‐state stabilization and optimal feedback control

In this paper, we develop a unified framework to address the problem of optimal nonlinear analysis and feedback control for partial stability and partial‐state stabilization. Partial asymptotic stability of the closed‐loop nonlinear system is guaranteed by means of a Lyapunov function that is positive definite and decrescent with respect to part of the system state, which can clearly be seen to be the solution to the steady‐state form of the Hamilton–Jacobi–Bellman equation and hence guaranteeing both partial stability and optimality. The overall framework provides the foundation for extending optimal linear‐quadratic controller synthesis to nonlinear nonquadratic optimal partial‐state stabilization. Connections to optimal linear and nonlinear regulation for linear and nonlinear time‐varying systems with quadratic and nonlinear nonquadratic cost functionals are also provided. Finally, we also develop optimal feedback controllers for affine nonlinear systems using an inverse optimality framework tailored to the partial‐state stabilization problem and use this result to address polynomial and multilinear forms in the performance criterion. Copyright © 2015 John Wiley & Sons, Ltd.     

Neural networks for identification of nonlinear systems underrandom piecewise polynomial disturbances

The problem of identification of a nonlinear dynamic system is considered. A two-layer neural network is used for the solution of the problem. Systems disturbed with unmeasurable noise are considered, although it is known that the disturbance is a random piecewise polynomial process. Absorption polynomials and nonquadratic loss functions are used to reduce the effect of this disturbance on the estimates of the optimal memory of the neural-network model.     

A neural network-based approximation method for discrete-timenonlinear servomechanism problem

A feedback controller that solves the discrete-time nonlinear servomechanism problem relies on the solution of a set of nonlinear functional equations known as the discrete regulator equations. The exact solution of the discrete regulator equations is usually unavailable due to the nonlinearity of the system. The paper proposes to approximately solve the discrete regulator equations using a feedforward neural network. This approach leads to an effective way to practically solve the discrete nonlinear servomechanism problem. The approach has been illustrated using the well-known inverted pendulum on a cart system. The simulation shows that the control law designed by the proposed approach performs much better than the conventional linear control law.     

Dynamic Neural-Network-Based Model-Predictive Control of an Industrial Baker's Yeast Drying Process

This paper presents dynamic neural-network-based model-predictive control (MPC) structure for a baker's yeast drying process. Mathematical model consists of two partial nonlinear differential equations that are obtained from heat and mass balances inside dried granules. The drying curves that are obtained from granule-based model were used as training data for neural network (NN) models. The target is to predict the moisture content and product activity, which are very important parameters in drying process, for different horizon values. Genetic-based search algorithm determines the optimal drying profile by solving optimization problem in MPC. As a result of the performance evaluation of the proposed control structure, which is compared with the model based on nonlinear partial differential equation (PDE) and with feedforward neural network (FFN) models, it is particularly satisfactory for the drying process of a baker's yeast.      

Neural networks for nonlinear state estimation

Estimating the state of a nonlinear stochastic system (observed through a nonlinear noisy measurement channel) has been the goal of considerable research to solve both filtering and control problems. In this paper, an original approach to the solution of the optimal state estimation problem by means of neural networks is proposed, which consists in constraining the state estimator to take on the structure of a multilayer feedforward network. Both non-recursive and recursive estimation schemes are considered, which enable one to reduce the original functional problem to a nonlinear programming one. As this reduction entails approximations for the optimal estimation strategy, quantitative results on the accuracy of such approximations are reported. Simulation results confirm the effectiveness of the proposed method.     

A neural network approach to freeway network traffic control

The paper investigates the application of a feedforward neural network approach to freeway network control via variable direction recommendations at bifurcation locations. A nonlinear control problem is formulated and solved first by use of computationally expensive nonlinear optimization techniques. A feedforward neural network is then trained by optimally adjusting its weights so as to reproduce the optimal control law for a limited number of traffic scenarios. Generalisation properties of the neural network are investigated and a discussion of advantages and disadvantages compared with alternative control approaches is provided.     

Synthesis of optimal control using neural network with mixed structure

This paper proposes firstly to use a neural network with a mixed structure for learning the system dynamics of a nonlinear plant, which consists of multilayer and recurrent structure. Since a neural network with a mixed structure can learn time series, it can learn the dynamics of a plant without knowing the plant order. Secondly, a novel method of synthesizing the optimal control is developed using the proposed neural network. Procedures are as follows: (1) Let a neural network with a mixed structure learn the unknown dynamics of a nonlinear plant with arbitrary order, (2) after the learning is completed, the network is expanded into an equivalent feedforward multilayer network, (3) it is shown that the gradient of a criterion functional to be optimized can be easily obtained from this multilayer network, and then (4) the optimal control is generated by applying any of the existing non-linear programming algorithm based on this gradient information. The proposed method is successfully applied to the optimal control synthesis problem of a nonlinear coupled vibratory plant with a linear quadratic criterion functional.     

Global optimal feedback control for general nonlinear systems with nonquadratic performance criteria

Optimal control of general nonlinear nonaffine controlled systems with nonquadratic performance criteria (that permit state- and control-dependent time-varying weighting parameters), is solved classically using a sequence of linear- quadratic and time-varying problems. The proposed method introduces an “approximating sequence of Riccati equations” (ASRE) to explicitly construct nonlinear time-varying optimal state-feedback controllers for such nonlinear systems. Under very mild conditions of local Lipschitz continuity, the sequences converge (globally) to nonlinear optimal stabilizing feedback controls. The computational simplicity and effectiveness of the ASRE algorithm is an appealing alternative to the tedious and laborious task of solving the Hamilton–Jacobi–Bellman partial differential equation. So the optimality of the ASRE control is studied by considering the original nonlinear-nonquadratic optimization problem and the corresponding necessary conditions for optimality, derived from Pontryagin's maximum principle. Global optimal stabilizing state-feedback control laws are then constructed. This is compared with the optimality of the ASRE control by considering a nonlinear fighter aircraft control system, which is nonaffine in the control. Numerical simulations are used to illustrate the application of the ASRE methodology, which demonstrate its superior performance and optimality.     

非线性离散系统的近似最优跟踪控制

研究非线性离散系统的最优跟踪控制问题. 通过在由最优控制问题所导致的非线性两点边值问题中引入灵敏度参数, 并对它进行Maclaurin级数展开, 将原最优跟踪控制问题转化为一族非齐次线性两点边值问题. 得到的最优跟踪控制由解析的前馈反馈项和级数形式的补偿项组成. 解析的前馈反馈项可以由求解一个Riccati差分方程和一个矩阵差分方程得到. 级数补偿项可以由一个求解伴随向量的迭代算法近似求得. 以连续槽式反应器为例进行仿真验证了该方法的有效性．     

Data‐driven optimal event‐triggered consensus control for unknown nonlinear multiagent systems with control constraints

This paper considers optimal consensus control problem for unknown nonlinear multiagent systems (MASs) subjected to control constraints by utilizing event‐triggered adaptive dynamic programming (ETADP) technique. To deal with the control constraints, we introduce nonquadratic energy consumption functions into performance indices and formulate the Hamilton‐Jacobi‐Bellman (HJB) equations. Then, based on the Bellman's optimality principle, constrained optimal consensus control policies are designed from the HJB equations. In order to implement the ETADP algorithm, the critic networks and action networks are developed to approximate the value functions and consensus control policies respectively based on the measurable system data. Under the event‐triggered control framework, the weights of the critic networks and action networks are only updated at the triggering instants which are decided by the designed adaptive triggered conditions. The Lyapunov method is used to prove that the local neighbor consensus errors and the weight estimation errors of the critic networks and action networks are ultimately bounded. Finally, a numerical example is provided to show the effectiveness of the proposed ETADP method.