Online optimal tracking control of continuous-time linear systems with unknown dynamics by using adaptive dynamic programming

In this paper, a novel theoretic formulation based on adaptive dynamic programming (ADP) is developed to solve online the optimal tracking problem of the continuous-time linear system with unknown dynamics. First, the original system dynamics and the reference trajectory dynamics are transformed into an augmented system. Then, under the same performance index with the original system dynamics, an augmented algebraic Riccati equation is derived. Furthermore, the solutions for the optimal control problem of the augmented system are proven to be equal to the standard solutions for the optimal tracking problem of the original system dynamics. Moreover, a new online algorithm based on the ADP technique is presented to solve the optimal tracking problem of the linear system with unknown system dynamics. Finally, simulation results are given to verify the effectiveness of the theoretic results.     

具有加性和乘性噪声的线性离散时间随机系统的无模型最优跟踪控制

本文研究一类同时受加性和乘性噪声影响的离散时间随机系统的最优跟踪控制问题.通过构造由原始系统和参考轨迹组成的增广系统,将随机线性二次跟踪控制(SLQT)的成本函数转化为与增广状态相关的二次型函数,由此推导出用于求解SLQT的贝尔曼方程和增广随机代数黎卡提方程(SARE),而后进一步针对系统和参考轨迹动力学信息完全未知的情形,提出一种Q-学习算法来在线求解增广SARE,证明了该算法的收敛性,并采用批处理最小二乘法(BLS)解决该在线无模型控制算法的实现问题.通过对单相电压源UPS逆变器的仿真,验证了所提出控制方案的有效性.     

An ADDHP-based Q-learning algorithm for optimal tracking control of linear discrete-time systems with unknown dynamics

This paper investigates a novel Q-learning algorithm based on action dependent dual heuristic programming (ADDHP) to solve the infinite-time domain linear quadratic tracker (LQT) for unknown linear discrete-time systems. The proposed method is conducted based on only system data without requiring the knowledge of the system matrices. After the reference system is determined, an augmented system composed of the original system and the reference system is constructed, and it is proved that the value function of LQT is quadratic concerning the state of the augmented system. Using the quadratic value function, the augmented algebraic Riccati equation (ARE) is derived to solve the LQT. Due to the difficulty of directly solving the augmented ARE, a Q-learning algorithm based on ADDHP structure is used to solve this problem. With unknown system matrices, a model neural network is developed to reconstruct system dynamics incorporating stability analysis. The estimated system matrices are employed to the proposed algorithm to calculate the optimal control by policy iteration. Moreover, the convergence of the algorithm is proved. Two simulation examples are used to validate the performance of the method, where all results have demonstrated the effectiveness of the proposed ADDHP-based Q-learning method without a priori knowledge of system matrices for LQT.     

Adaptive composite suboptimal control for linear singularly perturbed systems with unknown slow dynamics

In this article, using singular perturbation theory and adaptive dynamic programming (ADP) approach, an adaptive composite suboptimal control method is proposed for linear singularly perturbed systems (SPSs) with unknown slow dynamics. First, the system is decomposed into fast‐ and slow‐subsystems and the original optimal control problem is reduced to two subproblems in different time‐scales. Afterward, the fast subproblem is solved based on the known model of the fast‐subsystem and a fast optimal control law is designed by solving the algebraic Riccati equation corresponding to the fast‐subsystem. Then, the slow subproblem is reformulated by introducing a system transformation for the slow‐subsystem. An online learning algorithm is proposed to design a slow optimal control law by using the information of the original system state in the framework of ADP. As a result, the obtained fast and slow optimal control laws constitute the adaptive composite suboptimal control law for the original SPSs. Furthermore, convergence of the learning algorithm, suboptimality of the adaptive composite suboptimal control law and stability of the whole closed‐loop system are analyzed by singular perturbation theory. Finally, a numerical example is given to show the feasibility and effectiveness of the proposed methods.     

Stochastic Optimal Design for Unknown Linear Discrete‐Time System Zero‐Sum Games in Input‐Output form Under Communication Constraints

In this paper, stochastic optimal strategy for unknown linear discrete‐time system quadratic zero‐sum games in input‐output form with communication imperfections such as network‐induced delays and packet losses, otherwise referred to as networked control system (NCS) zero‐sum games, relating to the H_∞ optimal control problem is solved in a forward‐in‐time manner. First, the linear discrete‐time zero sum state space representation is transformed into a linear NCS in the state space form after incorporating random delays and packet losses and then into the input‐output form. Subsequently, the stochastic optimal approach, referred to as adaptive dynamic programming (ADP), is introduced which estimates the cost or value function to solve the infinite horizon optimal regulation of unknown linear NCS quadratic zero‐sum games in the presence of communication imperfections. The optimal control and worst case disturbance inputs are derived based on the estimated value function in the absence of state measurements. An update law for tuning the unknown parameters of the value function estimator is derived and Lyapunov theory is used to show that all signals are asymptotically stable (AS) and that the estimated control and disturbance signals converge to optimal control and worst case disturbances, respectively. Simulation results are included to verify the theoretical claims.     

An integrated optimal control algorithm for discrete-time nonlinear stochastic system

Consider a discrete-time nonlinear system with random disturbances appearing in the real plant and the output channel where the randomly perturbed output is measurable. An iterative procedure based on the linear quadratic Gaussian optimal control model is developed for solving the optimal control of this stochastic system. The optimal state estimate provided by Kalman filtering theory and the optimal control law obtained from the linear quadratic regulator problem are then integrated into the dynamic integrated system optimisation and parameter estimation algorithm. The iterative solutions of the optimal control problem for the model obtained converge to the solution of the original optimal control problem of the discrete-time nonlinear system, despite model-reality differences, when the convergence is achieved. An illustrative example is solved using the method proposed. The results obtained show the effectiveness of the algorithm proposed.     

Suboptimal control of linear stochastic multivariable systems with unknown parameters

An algorithm is proposed for suboptimal control of linear multivariable systems with unknown parameters and output noise covariances. This algorithm is based on the idea of explicitly separating the functions of identification, estimation and control. The parameters and states of the system are estimated in a bootstrap manner by the stochastic approximation method. A suboptimal controller is then obtained which utilizes a deterministic control gain derived using the estimates obtained from the parameter and state estimators. This suboptimal controller will approach the optimal strategy when the estimated system parameters approach their true values.     

基于数据ADP算法的一类带有执行器饱和的未知离散时间系统最优跟踪控制

针对一类带有执行器饱和的未知动态离散时间非线性系统, 提出了一种新的最优跟踪控制方案. 该方案基于迭代自适应动态规划算法, 为了实现最优控制, 首先建立了未知系统动态的数据辨识器. 通过引入M网络, 获得了稳态控制的精确表达式. 为了消除执行器饱和的影响, 提出了一个非二次的性能指标函数. 然后提出了一种迭代自适应动态规划算法获得最优跟踪控制的解, 并给出了收敛性分析. 为了实现最优控制方案, 神经网络被用来构建数据辨识器、计算性能指标函数、近似最优控制策略和求解稳态控制. 仿真结果验证了本文所提出的最优跟踪控制方法的有效性.     

Neural-Network-Based Control for Discrete-Time Nonlinear Systems with Input Saturation Under Stochastic Communication Protocol

In this paper, an adaptive dynamic programming (ADP) strategy is investigated for discrete-time nonlinear systems with unknown nonlinear dynamics subject to input saturation. To save the communication resources between the controller and the actuators, stochastic communication protocols (SCPs) are adopted to schedule the control signal, and therefore the closed-loop system is essentially a protocol-induced switching system. A neural network (NN)-based identifier with a robust term is exploited for approximating the unknown nonlinear system, and a set of switch-based updating rules with an additional tunable parameter of NN weights are developed with the help of the gradient descent. By virtue of a novel Lyapunov function, a sufficient condition is proposed to achieve the stability of both system identification errors and the update dynamics of NN weights. Then, a value iterative ADP algorithm in an offline way is proposed to solve the optimal control of protocol-induced switching systems with saturation constraints, and the convergence is profoundly discussed in light of mathematical induction. Furthermore, an actor-critic NN scheme is developed to approximate the control law and the proposed performance index function in the framework of ADP, and the stability of the closed-loop system is analyzed in view of the Lyapunov theory. Finally, the numerical simulation results are presented to demonstrate the effectiveness of the proposed control scheme.      

A Wiener-type recurrent neural network and its control strategy for nonlinear dynamic applications

This paper presents a Wiener-type recurrent neural network with a systematic identification algorithm and a control strategy for the identification and control of unknown dynamic nonlinear systems. The proposed Wiener-type recurrent network resembles the conventional Wiener model that consists of a dynamic linear subsystem cascaded with a static nonlinear subsystem. The novelties of our network include: (1) the two subsystems are integrated into a single network whose output is expressed by a nonlinear transformation of a linear state-space equation; (2) the characteristics of the trained network can be analyzed by its associated state-space equation using the well-developed theory of linear systems; and (3) the size of the network structure is determined by the number of state variables (or the system order) of the unknown systems to be identified. To effectively identify a given unknown system from its input–output data, we have developed a systematic identification algorithm that consists of an order determination procedure, a parameterization procedure, and an online learning procedure. The false nearest neighbors algorithm was adopted to acquire a minimal embedding dimension from the input–output data as the system order, and then the eigensystem realization algorithm (ERA) was used to initialize a best-fit state-space representation according to the acquired system order. To improve the overall identification performance, we have derived an online parameter learning algorithm based on an ordered derivatives and momentum terms. Subsequently, a simple feedback linear controller was designed to control the unknown dynamic nonlinear systems without much complexity. Computer simulations and comparisons with some existing recurrent networks have conducted to confirm the effectiveness and superiority of the proposed Wiener-type network, identification algorithm and control strategy.     

Neural-network-based approach to finite-time optimal control for a class of unknown nonlinear systems

This paper proposes a novel finite-time optimal control method based on input–output data for unknown nonlinear systems using adaptive dynamic programming (ADP) algorithm. In this method, the single-hidden layer feed-forward network (SLFN) with extreme learning machine (ELM) is used to construct the data-based identifier of the unknown system dynamics. Based on the data-based identifier, the finite-time optimal control method is established by ADP algorithm. Two other SLFNs with ELM are used in ADP method to facilitate the implementation of the iterative algorithm, which aim to approximate the performance index function and the optimal control law at each iteration, respectively. A simulation example is provided to demonstrate the effectiveness of the proposed control scheme.     

Preview control for a class of linear stochastic systems with multiplicative noise

ABSTRACT

In this paper, the preview control problem for a class of linear continuous time stochastic systems with multiplicative noise is studied based on the augmented error system method. First, a deterministic assistant system is introduced, and the original system is translated to the assistant system. Then, the integrator is employed to ensure the output of the closed-loop system tracking the reference signal accurately. Second, the augmented error system, which includes integrator vector, control vector and reference signal, is constructed based on the system after translation. As a result, the tracking problem is transformed into the optimal control problem of the augmented error system, and the optimal control input is obtained by the dynamic programming method. This control input is regarded as the preview controller of the original system. For a linear stochastic system with multiplicative noise, the difficulty being unable to construct an augmented error system by the derivation method is solved in this paper. And, the existence and uniqueness solution of the Riccati equation corresponding to the stochastic augmented error system is discussed. The numerical simulations show that the preview controller designed in this paper is very effective.     

Stochastic optimal control of unknown linear networked control system in the presence of random delays and packet losses

In this paper, the stochastic optimal control of linear networked control system (NCS) with uncertain system dynamics and in the presence of network imperfections such as random delays and packet losses is derived. The proposed stochastic optimal control method uses an adaptive estimator (AE) and ideas from Q-learning to solve the infinite horizon optimal regulation of unknown NCS with time-varying system matrices. Next, a stochastic suboptimal control scheme which uses AE and Q-learning is introduced for the regulation of unknown linear time-invariant NCS that is derived using certainty equivalence property. Update laws for online tuning the unknown parameters of the AE to obtain the Q-function are derived. Lyapunov theory is used to show that all signals are asymptotically stable (AS) and that the estimated control signals converge to optimal or suboptimal control inputs. Simulation results are included to show the effectiveness of the proposed schemes. The result is an optimal control scheme that operates forward-in-time manner for unknown linear systems in contrast with standard Riccati equation-based schemes which function backward-in-time.     

Optimal preview control for a linear continuous-time stochastic control system in finite-time horizon

This paper discusses the design of the optimal preview controller for a linear continuous-time stochastic control system in finite-time horizon, using the method of augmented error system. First, an assistant system is introduced for state shifting. Then, in order to overcome the difficulty of the state equation of the stochastic control system being unable to be differentiated because of Brownian motion, the integrator is introduced. Thus, the augmented error system which contains the integrator vector, control input, reference signal, error vector and state of the system is reconstructed. This leads to the tracking problem of the optimal preview control of the linear stochastic control system being transformed into the optimal output tracking problem of the augmented error system. With the method of dynamic programming in the theory of stochastic control, the optimal controller with previewable signals of the augmented error system being equal to the controller of the original system is obtained. Finally, numerical simulations show the effectiveness of the controller.     

Feedback linearisation control of an induction machine augmented by single-hidden layer neural networks

We consider adaptive output feedback control methodology of highly uncertain nonlinear systems with both parametric uncertainties and unmodelled dynamics. The approach is also applicable to systems of unknown, but bounded dimension. However, the relative degree of the regulated output is assumed to be known. This new control strategy is proposed to address the tracking problem of an induction motor based on a modified field-oriented control method. The obtained controller is then augmented by an online neural network that serves as an approximator for the neglected dynamics and modelling errors. The network weight adaptation rule is derived from the Lyapunov stability analysis, that guarantees boundedness of all the error signals of the closed-loop system. Computer simulations of an output feedback controlled induction machine, augmented via single-hidden-layer neural networks, demonstrate the practical potential of the proposed control algorithm.     

Multi-step Predictive Control of a PDF-shaping Problem

A predictive control strategy is proposed for the shaping of the output probability density function (PDF) of linear stochastic systems. The B-spline neural network is used to set up the output PDF model and therefore converts the PDF-shaping into the control of B-spline weights vector. The Diophantine equation is then introduced to formulate the predictive PDF model, based on which a moving-horizon control algorithm is developed so as to realize the predictive PDF tracking performance.     

Integrated fault estimation and fault‐tolerant control for stochastic systems with Brownian motions

This paper presents an integrated robust fault estimation and fault‐tolerant control technique for stochastic systems subjected to Brownian parameter perturbations. The augmented system approach, unknown input observer method, and optimization technique are integrated to achieve robust simultaneous estimates of the system states and the means of faults concerned. Meanwhile, a robust fault‐tolerant control strategy is developed by using actuator and sensor signal compensation techniques. Stochastic linear time‐invariant systems, stochastic systems with Lipschitz nonlinear constraint, and stochastic systems with quadratic inner‐bounded nonlinear constraint are respectively investigated, and the corresponding fault‐tolerant control algorithms are addressed. Finally, the effectiveness of the proposed fault‐tolerant control techniques is demonstrated via the drivetrain system of a 4.8 MW benchmark wind turbine, a 3‐tank system, and a numerical nonlinear model.     

输出概率密度函数形状的多步预测控制

A predictive control strategy is proposed for the shaping of the output probability density function (PDF) of linear stochastic systems. The B-spline neural network is used to set up the output PDF model and therefore converts the PDF-shaping into the control of B-spline weights vector. The Diophantine equation is then introduced to formulate the predictive PDF model, based on which a moving-horizon control algorithm is developed so as to realize the predictive PDF tracking performance.     

基于滚动优化的对偶控制策略

考虑具有未知参数的随机系统的最优控制问题.提出了一种新的基于滚动优化的对偶控制算法.在动态规划泛函方程中,用Kalman滤波对系统的状态进行估计;用线性化方法对阶段损失函数中的后验概率进行近似,然后,用滚动优化策略对控制与学习之间的耦合关系进行解耦.从而获得了原不可解泛函方程的解析递推表达式和一个易于实施的控制律的解析解.用一个例子说明了控制律的性能,仿真结果表明:该控制律具有良好的对偶性质,在学习和控制之间实现了较好的平衡.     

Combined identification of the input-output and noise dynamics of a closed-loop controlled linear system

The open-loop input—output dynamics and the noise dynamics of a feedback controlled linear system perturbed by coloured noise admitting a Markov representation are identified in state variable form using a two-stage algorithm. This system is equivalent to an augmented system driven by white noise.

First the input—output dynamics are identified through a stochastic approximation algorithm using superimposed white noise. Subtracting the model output from the system output yields correlated residuals which are then used to identify the noise dynamics using stochastic realization theory. An innovations representation is obtained that is equivalent to the above defined augmented system. The two stages are combined by a judicious coordinate transformation. The method can be applied on an operating feedback controlled process, regardless of the structure of the unknown suboptimal regulator.