Infinite horizon linear quadratic differential games for discrete-time stochastic systems

This paper deals with the infinite horizon linear quadratic(LQ)differential games for discrete-time stochastic systems with both state and control dependent noise.The Popov-Belevitch-Hautus(PBH)criteria for exact observability and exact detectability of discrete-time stochastic systems are presented.By means of them,we give the optimal strategies (Nash equilibrium strategies)and the optimal cost values for infinite horizon stochastic differential games.It indicates that the infinite horizon LQ stochastic differential games are associated with four coupled matrix-valued equations.Furthermore, an iterative algorithm is proposed to solve the four coupled equations.Finally,an example is given to demonstrate our results.     

Suboptimal Strategies of Linear Quadratic Closed-loop Differential Games: An BMI Approach

The suboptimal control program via memoryless state feedback strategies for LQ differential games with multiple players is studied in this paper. Sufficient conditions for the existence of the suboptimal strategies for LQ differential games are presented. It is shown that the suboptimal strategies of LQ differential games are associated with a coupled algebraic Riccati inequality. Furthermore, the problem of designing suboptimal strategies is considered. A non-convex optimization problem with BMI constrains is formulated to design the suboptimal strategies which minimizes the performance indices of the closed-loop LQ differential games and can be solved by using LMI Toolbox of MATLAB, An example is given to illustrate the proposed results.     

线性二次闭环微分对策次优对策的双线性矩阵不等式方法

The suboptimal control program via memoryless state feedback strategies for LQ differential games with multiple players is studied in this paper. Sufficient conditions for the existence of the suboptimal strategies for LQ differential games are presented. It is shown that the suboptimal strategies of LQ differential games are associated with a coupled algebraic Riccati inequality. Furthermore, the problem of designing suboptimal strategies is considered. A non-convex optimization problem with BMI constrains is formulated to design the suboptimal strategies which minimizes the performance indices of the closed-loop LQ differential games and can be solved by using LMI Toolbox of MATLAB. An example is given to illustrate the proposed results.     

基于LQ微分对策的最优规避策略与决策算法

针对信息化条件下飞行器通过"智能"机动方式来规避拦截这一问题,基于LQ微分对策方法对最优规避参数的决策问题进行了研究.首先,将微分对策引入LQ最优控制算法,建立了飞行器规避拦截问题的数学模型;其次,提出了将Hamil-ton-Yaceobi伴随向量与共轭法相结合的共轭决策方法,简化了微分对策的求解,并得到了最优规避参数的解析解;给出了规避策略中不同参数对规避效果的不同灵敏度,并给出了最优规避参数的决策算法;最后,针对不同的对抗情形,进行了数字仿真,检验了本文微分对策模型的合理性和算法的有效性.     

Online policy iteration algorithm for optimal control of linear hyperbolic PDE systems

In this paper, the linear quadratic (LQ) optimal control problem is considered for a class of linear distributed parameter systems described by first-order hyperbolic partial differential equations (PDEs). Reinforcement learning (RL) technique is introduced for adaptive optimal control design from the design-then-reduce (DTR) framework. Initially, a policy iteration (PI) algorithm is proposed, which learns the solution of the space-dependent Riccati differential equation (SDRDE) online without requiring the internal system dynamics of the PDE system. To prove its convergence, the PI algorithm is shown to be equivalent to an iterative procedure of a sequence of space-dependent Lyapunov differential equations (SDLDEs). Then, the convergence is established by showing that the solutions of SDLDEs are a monotone non-increasing sequence that converges to the solution of the SDRDE. For implementation purpose, an online least-square method is developed for the approximation of the solutions of the SDLDEs. Finally, the proposed design method is applied to the distributed control of a steam-jacketed tubular heat exchanger to illustrate its effectiveness.     

Linear quadratic suboptimal control with static output feedback

This paper considers the design of a stabilizing static output feedback gain which keeps linear quadratic (LQ) performance index less than a specified number (we call this an ‘LQ suboptimal controller’). Existence of such a controller is shown to be equivalent to the existence of a positive-definite matrix P such that P satisfies two linear matrix inequalities (LMIs) while P⁻¹ satisfies another LMI. All LQ suboptimal controllers are explicitly parametrized by the freedom in the choice of the positive-definite matrix P satisfying the LMIs, and an arbitrary positive scalar and an arbitrary matrix of fixed dimension with a norm bound. A modified version of the min/max algorithm is given to find a positive-definite solution P to the LMIs.     

Saturated linear quadratic regulation of uncertain linear systems: Stability region estimation and controller design

This paper considers the problems of estimating the stability region (domain of attraction) and controller design for uncertain linear continuous-time systems with input saturation when linear quadratic (LQ) optimal controller is used. By exploiting the structure of the LQ controller and the property of saturation functions, it is established that the estimation of stability region can be obtained by solving linear matrix inequality (LMI) problems. Moreover, an iterative LMI (ILMI) algorithm is presented to design an LQ controller such that the largest estimated stability region can be obtained. Two examples are given to compare our results with existing ones.     

A computational method for simultaneous LQ optimal control designvia piecewise constant output feedback

The paper is concerned with simultaneous linear-quadratic (LQ) optimal control design for a set of LTI systems via piecewise constant output feedback. First, the discrete-time simultaneous LQ optimal control design problem is reduced to solving a set of coupled matrix inequalities and an iterative LMI algorithm is presented to compute the feedback gain. Then, simultaneous stabilization and simultaneous LQ optimal control design of a set of LTI continuous-time systems are considered via periodic piecewise constant feedback gain. It is shown that the design of a periodic piecewise constant feedback gain simultaneously minimizing a set of given continuous-time performance indexes can be reduced to that of a constant feedback gain minimizing a set of equivalent discrete-time performance indexes. Explicit formulas for computing the equivalent discrete-time systems and performance indexes are derived. Examples are used to demonstrate the effectiveness of the proposed method.     

基于数据自适应评判的离散2-D系统零和博弈最优控制

提出了基于一种迭代自适应评判设计(ACD)算法解决一类离散时间Roesser型2-D系统的二人零和对策问题. 文章主要思想是采用自适应评判技术迭代的获得最优控制对使得性能指标函数达到零和对策的鞍点. 所提出的ACD可以通过输入输出数据进行实现而不需要系统的模型. 为了实现迭代ACD算法, 神经网络分别用来近似性能指标函数和计算最优控制率. 最后最优控制策略将应用到空气干燥过程控制中以证明其有效性.     

一类定量微分对策理论中最优策略的算法及其收敛性

本文利用不动点原理讨论了一类定量微分对策理论中最优策略的计算方法问题.首先构 造出了一种迭代方法,然后利用不动点原理分析了该迭代法的收敛性.本文给出的方法还可 用于一类Nash微分对策的Nash策略的分散计算方法.     

多组对策系统非劣Nash策略的最优均衡解算法

多组对策系统中求解组与组之间的非劣Nash策略至关重要.如何针对一般问题解析求出非劣Nash策略还没有有效的方法.本文阐述了一种利用组与组之间的非劣反应集构造求解非劣Nash策略的迭代算法.为此首先引进多组对策系统组内部合作对策的最优均衡值和最优均衡解的概念,然后通过证明最优均衡解是组内部隐含某一权重向量的合作对策的非劣解,得到求解合作对策的单目标规划问题.进一步说明在组内部该问题的解不仅是非劣解而且对所有局中人都优于不合作时的Nash平衡策略.最后给出了验证该算法有效性的一个实际例子.     

Multiple models,multiplicative noise and linear quadratic control—algorithmic aspects

Numerical solution of certain linear quadratic (LQ) control problems for robust design is addressed. An iterative method is suggested and analysed for solving a minimax multiple model LQ control problem. A convergent iterative method is studied for finding the constant feedback gains of a given linear controller so as to solve a spectral radius functional minimization problem for multiple plants. Furthermore, a convergent iterative method is proposed for solving the maximum entropy parametric LQ control problem with multiplicative noise.     

Optimal control of coupled parabolic–hyperbolic non-autonomous PDEs: infinite-dimensional state-space approach

This paper deals with the design of an optimal state-feedback linear-quadratic (LQ) controller for a system of coupled parabolic–hypebolic non-autonomous partial differential equations (PDEs). The infinite-dimensional state space representation and the corresponding operator Riccati differential equation are used to solve the control problem. Dynamical properties of the coupled system of interest are analysed to guarantee the existence and uniqueness of the solution of the LQ-optimal control problem and also to guarantee the exponential stability of the closed-loop system. Thanks to the eigenvalues and eigenfunctions of the parabolic operator and also the fact that the hyperbolic-associated operator Riccati differential equation can be converted to a scalar Riccati PDE, an algorithm to solve the LQ control problem has been presented. The results are applied to a non-isothermal packed-bed catalytic reactor. The LQ optimal controller designed in the early portion of the paper is implemented for the original non-linear model. Numerical simulations are performed to show the controller performances.     

LQ控制区段混合能矩阵的微分方程及其应用

本文根据计算结构力学与线性二次控制的对应关系,提出了连续时间有限区段的混合能 分块子矩阵Q2,G2及Φ2.推导出适用于LQ控制非定常课题的二区段连接的凝聚消元公式及 这些子矩阵的微分方程,可用级数展开求解这些方程.当△t很小时,这些分块子矩阵的高次 近似可以大大加速里卡提代数方程算法的收敛性.     

线性离散奇异系统的不定二次最优控制问题: Krein空间方法

The finite time horizon indefinite linear quadratic(LQ) optimal control problem for singular linear discrete time-varying systems is discussed. Indefinite LQ optimal control problem for singular systems can be transformed to that for standard state-space systems under a reasonable assumption. It is shown that the indefinite LQ optimal control problem is dual to that of projection for backward stochastic systems. Thus, the optimal LQ controller can be obtained by computing the gain matrices of Kalman filter. Necessary and sufficient conditions guaranteeing a unique solution for the indefinite LQ problem are given. An explicit solution for the problem is obtained in terms of the solution of Riccati difference equations.     

An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games

In this paper, a new iterative adaptive dynamic programming (ADP) method is proposed to solve a class of continuous-time nonlinear two-person zero-sum differential games. The idea is to use the ADP technique to obtain the optimal control pair iteratively which makes the performance index function reach the saddle point of the zero-sum differential games. If the saddle point does not exist, the mixed optimal control pair is obtained to make the performance index function reach the mixed optimum. Stability analysis of the nonlinear systems is presented and the convergence property of the performance index function is also proved. Two simulation examples are given to illustrate the performance of the proposed method.     

An equivalence result in linear-quadratic theory

We consider the zero-endpoint infinite-horizon LQ problem. We show that the existence of an optimal policy in the class of feedback controls is a sufficient condition for the existence of a stabilizing solution to the algebraic Riccati equation. This result is shown without assuming positive definiteness of the state weighting matrix. The feedback formulation of the optimization problem is natural in the context of differential games and we provide a characterization of feedback Nash equilibria both in a deterministic and stochastic context.     

Multi-agent graphical games with input constraints: an online learning solution

This paper studies an online iterative algorithm for solving discrete-time multi-agent dynamic graphical games with input constraints. In order to obtain the optimal strategy of each agent, it is necessary to solve a set of coupled Hamilton-Jacobi-Bellman (HJB) equations. It is very difficult to solve HJB equations by the traditional method. The relevant game problem will become more complex if the control input of each agent in the dynamic graphical game is constrained. In this paper, an online iterative algorithm is proposed to find the online solution to dynamic graphical game without the need for drift dynamics of agents. Actually, this algorithm is to find the optimal solution of Bellman equations online. This solution employs a distributed policy iteration process, using only the local information available to each agent. It can be proved that under certain conditions, when each agent updates its own strategy simultaneously, the whole multi-agent system will reach Nash equilibrium. In the process of algorithm implementation, for each agent, two layers of neural networks are used to fit the value function and control strategy, respectively. Finally, a simulation example is given to show the effectiveness of our method.     

Some new results on optimal and suboptimal regulators of the LQproblem with output feedback

The LQ (linear quadratic) regulator problem with output feedback is considered. In the problem studies an output feedback control is determined such that the cost is J*, the optimal cost for the LQ problem with state feedback, or βj* with some constant β>1. A set of sufficient conditions is given for determining such optimal and suboptimal output feedback control laws     

基于策略迭代的连续时间系统的随机线性二次最优控制

针对模型参数部分未知的随机线性连续时间系统, 通过策略迭代算法求解无限时间随机线性二次(LQ) 最优控制问题. 求解随机LQ最优控制问题等价于求随机代数Riccati 方程(SARE) 的解. 首先利用伊藤公式将随机微分方程转化为确定性方程, 通过策略迭代算法给出SARE 的解序列; 然后证明SARE 的解序列收敛到SARE 的解, 而且在迭代过程中系统是均方可镇定的; 最后通过仿真例子表明策略迭代算法的可行性.