Adaptive control of linear stochastic systems

An adaptive control algorithm for linear systems with unknown constant parameters and quadratic performance criterion has been obtained. The control is nonlinear in the estimate of the state of the plant and is given as the weighted integral of the model conditional optimal controls with the a-posteriori probabilities as weights. The control scheme is separated into a bank of model-conditional deterministic control gains, and a corresponding bank of known nonlinear functions of the model conditional, causal, mean-square state-vector estimate. The separation here can be viewed as a decomposition of the control into a bank of model conditional optimal non-adaptive linear controls, one for each admissible value of the unknown parameter, and the bank of a-posteriori model probabilities which incorporate the learning nature of the adaptive control. The computational requirements are reduced by a great extent for the special case when the uncertainity is only in the measurement matrix.     

Infinite horizon linear quadratic optimal control for discrete‐time stochastic systems

This paper is concerned with the infinite horizon linear quadratic optimal control for discrete‐time stochastic systems with both state and control‐dependent noise. Under assumptions of stabilization and exact observability, it is shown that the optimal control law and optimal value exist, and the properties of the associated discrete generalized algebraic Riccati equation (GARE) are also discussed. Copyright © 2008 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Robust optimal control for minimax stochastic linear quadratic problem

The robust maximum principle applied to the minimax linear quadratic problem is derived for stochastic differential equations containing a control-dependent diffusion term. The parametric families of the first and second order adjoint stochastic processes are obtained to construct the corresponding Hamiltonian formalism. The Hamiltonian function used for the construction of the robust optimal control is shown to be equal to the sum of the standard stochastic Hamiltonians corresponding to each value of the uncertain parameter from a given finite set. The cost function is considered on a finite horizon (contains the mathematical expectation of both an integral and a terminal term) and on an infinite one (a time-averaged losses function). These problems belong to the class of minimax stochastic optimization problems. It is shown that the construction of the minimax optimal controller can be reduced to an optimization problem on a finitedimensional simplex and consists in the analysis of the dependence of Riccati equation solution on the weight parameters to be found.     

Adaptive control of continuous-time linear stochastic systems

An adaptive control problem for linear, continuous-time stochastic systems is described and solved in this paper. A solution of the adaptive control problem means that the family of maximum likelihood estimators is shown to be strongly consistent and the average costs are shown to converge to the optimal average costs. The unknown parameters in the model appear affinely in the drift term of the stochastic differential equation. The assumptions that are made for the solution are natural and verifiable. A recursive equation is given for the maximum likelihood estimates. This research was partially supported by NSF Grants ECS-8403286-A01 and ECS-8718026.     

On linear quadratic optimal control of linear time-varying singular systems

Linear time-varying singular systems are treated in this paper. We focus on systems with constant-rank E matrices. It is shown that the existence of state feedback for impulse elimination is both sufficient and necessary for the existence of linear-quadratic optimal control. Also optimal control exists if and only if the corresponding fast subsystem is impulse-controllable. The results obtained are extensions of the existing time-invariant theory.     

On constrained infinite-time linear quadratic optimal control with stochastic disturbances

In this work, we study the infinite-time linear quadratic optimal control problem for systems with stochastic disturbances and constrained inputs. A number of stochastic problem formulations under the full state information (FSI) structure are considered with a particular focus on the subject of feedback structure and its impact on certainty equivalence. In particular, we clarify results concerning the open-loop hard constrained, closed-loop statistically constrained, and closed-loop hard constrained cases. Extension to the infinite-time framework provides a vehicle for interpreting these controllers and indicates that the last of the three is of most interest to regulation type applications. Additionally, the partial state information problem is considered, and conditions are given for which a separated configuration consisting of the optimal estimator cascaded with the FSI optimal controller remains optimal.     

基于策略迭代的连续时间系统的随机线性二次最优控制

针对模型参数部分未知的随机线性连续时间系统, 通过策略迭代算法求解无限时间随机线性二次(LQ) 最优控制问题. 求解随机LQ最优控制问题等价于求随机代数Riccati 方程(SARE) 的解. 首先利用伊藤公式将随机微分方程转化为确定性方程, 通过策略迭代算法给出SARE 的解序列; 然后证明SARE 的解序列收敛到SARE 的解, 而且在迭代过程中系统是均方可镇定的; 最后通过仿真例子表明策略迭代算法的可行性.

     

Value iteration algorithm for continuous-time linear quadratic stochastic optimal control problems

In this study, we investigate a continuous-time infinite-horizon linear quadratic stochastic optimal control problem with multiplicative noise in control and state variables. Using the techniques of stochastic stability, exact observability, and stochastic approximation, a value iteration algorithm is developed to solve the corresponding generalized algebraic Riccati equation. Unlike the existing policy iteration algorithm, this algorithm does not rely on an initial stabilizing control. Further,...     

Adaptive optimal control for continuous-time linear systems based on policy iteration

In this paper we propose a new scheme based on adaptive critics for finding online the state feedback, infinite horizon, optimal control solution of linear continuous-time systems using only partial knowledge regarding the system dynamics. In other words, the algorithm solves online an algebraic Riccati equation without knowing the internal dynamics model of the system. Being based on a policy iteration technique, the algorithm alternates between the policy evaluation and policy update steps until an update of the control policy will no longer improve the system performance. The result is a direct adaptive control algorithm which converges to the optimal control solution without using an explicit, a priori obtained, model of the system internal dynamics. The effectiveness of the algorithm is shown while finding the optimal-load-frequency controller for a power system.     

Piecewise linear quadratic optimal control

The use of piecewise quadratic cost functions is extended from stability analysis of piecewise linear systems to performance analysis and optimal control. Lower bounds on the optimal control cost are obtained by semidefinite programming based on the Bellman inequality. This also gives an approximation to the optimal control law. An upper bound to the optimal cost is obtained by another convex optimization problem using the given control law. A compact matrix notation is introduced to support the calculations and it is proved that the framework of piecewise linear systems can be used to analyze smooth nonlinear dynamics with arbitrary accuracy     

Neural-network-based stochastic linear quadratic optimal tracking control scheme for unknown discrete-time systems using adaptive dynamic programming

In this paper, a stochastic linear quadratic optimal tracking scheme is proposed for unknown linear discrete-time (DT) systems based on adaptive dynamic programming (ADP) algorithm. First, an augmented system composed of the original system and the command generator is constructed and then an augmented stochastic algebraic equation is derived based on the augmented system. Next, to obtain the optimal control strategy, the stochastic case is converted into the deterministic one by system transformation, and then an ADP algorithm is proposed with convergence analysis. For the purpose of realizing the ADP algorithm, three back propagation neural networks including model network, critic network and action network are devised to guarantee unknown system model, optimal value function and optimal control strategy, respectively. Finally, the obtained optimal control strategy is applied to the original stochastic system, and two simulations are provided to demonstrate the effectiveness of the proposed algorithm.     

Infinite horizon indefinite stochastic linear quadratic control for discrete-time systems

This paper discusses discrete-time stochastic linear quadratic (LQ) problem in the infinite horizon with state and control dependent noise, where the weighting matrices in the cost function are assumed to be indefinite. The problem gives rise to a generalized algebraic Riccati equation (GARE) that involves equality and inequality constraints. The well-posedness of the indefinite LQ problem is shown to be equivalent to the feasibility of a linear matrix inequality (LMI). Moreover, the existence of a stabilizing solution to the GARE is equivalent to the attainability of the LQ problem. All the optimal controls are obtained in terms of the solution to the GARE. Finally, we give an LMI -based approach to solve the GARE via a semidefinite programming.     

Solvability of indefinite stochastic Riccati equations and linear quadratic optimal control problems

A new approach to study the indefinite stochastic linear quadratic (LQ) optimal control problems, which we called the “equivalent cost functional method”, is introduced by Yu (2013) in the setup of Hamiltonian system. On the other hand, another important issue along this research direction, is the possible state feedback representation of optimal control and the solvability of associated indefinite stochastic Riccati equations. As the response, this paper continues to develop the equivalent cost functional method by extending it to the Riccati equation setup. Our analysis is featured by its introduction of some equivalent cost functionals which enable us to have the bridge between the indefinite and positive-definite stochastic LQ problems. With such bridge, some solvability relation between the indefinite and positive-definite Riccati equations is further characterized. It is remarkable the solvability of the former is rather complicated than the latter, hence our relation provides some alternative but useful viewpoint. Consequently, the corresponding indefinite linear quadratic problem is discussed for which the unique optimal control is derived in terms of state feedback via the solution of the Riccati equation. In addition, some example is studied using our theoretical results.     

严格反馈随机非线性系统的自适应逆最优控制

讨论了自适应逆最优控制问题可解定理，基于Ito微分规则和Backstepping方法，给出了具有标准Wiener噪声扰动和未知定常参数的严格反馈随机非线性系统的全局依概率渐近稳定和自适应逆最优控制策略的设计方法，该方法可同时获得控制律和自适应律。仿真结果表明了控制算法的有效性。     

Adaptive stochastic control for a class of linear systems

The problem considered in this paper deals with the control of linear discrete-time stochastic systems with unknown (possibly time-varying and random) gain parameters. The philosophy of control is based on the use of an open-loop feedback optimal (OLFO) control using a quadratic index of performance. It is shown that the OLFO system consists of 1) an identifier that estimates the system state variables and gain parameters and 2) a controller described by an "adaptive" gain and correction term. Several qualitative properties and asymptotic properties of the OLFO adaptive system are discussed. Simulation results dealing with the control of stable and unstable third-order plants are presented. The key quantitative result is the precise variation of the control system adaptive gains as a function of the future expected uncertainty of the parameters; thus, in this problem the ordinary "separation theorem" does not hold.     

Adaptive control with recursive identification for stochastic linear systems

This paper presents a stochastic adaptive control algorithm which is shown to possess the following properties when applied to a possibly unstable, inverse stable, linear stochastic system with unknown parameters, whenever that system satisfies a certain positive real condition on its (moving average) noise dynamics. 1) The adaptive control part of the algorithm stabilizes and asymptotically optimizes the behavior of the system in the sense that the (limit of the) sample mean-square variation of the-output around a given demand level equals that of a minimum variance control strategy implemented with known parameters. This optimal behavior is subject to an offset μ²where μ²is the variance of a dither signal added to the control action in order to produce a "continually disturbed control." Formu^{2} > 0, it is shown that the input-output process satisfies a persistent excitation property, and hence, subject to a simple identifiability condition, the next property holds. 2) The observed input and output of the controlled system may be taken as inputs to an approximate maximum likelihood algorithm (AML) which generates strongly consistent estimates of the system's parameters. Results are presented for the scalar and multivariable cases.     

Adaptive continuous-time linear quadratic Gaussian control

The adaptive linear quadratic Gaussian control problem, where the linear transformation of the state A and the linear transformation of the control B are unknown, is solved assuming only that (A, B) is controllable and (A, Q₁^1/2) is observable, where Q ₁ determines the quadratic form for the state in the integrand of the cost functional. A weighted least squares algorithm is modified by using a random regularization to ensure that the family of estimated models is uniformly controllable and observable. A diminishing excitation is used with the adaptive control to ensure that the family of estimates is strongly consistent. A lagged certainty equivalence control using this family of estimates is shown to be self-optimizing for an ergodic, quadratic cost functional     

Singular optimal control for stochastic linear quadratic singular system using ant colony programming

In this article, singular optimal control for stochastic linear singular system with quadratic performance is obtained using ant colony programming (ACP). To obtain the optimal control, the solution of matrix Riccati differential equation is computed by solving differential algebraic equation using a novel and nontraditional ACP approach. The obtained solution in this method is equivalent or very close to the exact solution of the problem. Accuracy of the solution computed by the ACP approach to the problem is qualitatively better. The solution of this novel method is compared with the traditional Runge Kutta method. An illustrative numerical example is presented for the proposed method.