Feedback control on Nash equilibrium for discrete-time stochastic systems with Markovian jumps: Finite-horizon case

In this paper, we consider the feedback control on nonzero-sum linear quadratic (LQ) differential games in finite horizon for discrete-time stochastic systems with Markovian jump parameters and multiplicative noise. Four-coupled generalized difference Riccati equations (GDREs) are obtained, which are essential to find the optimal Nash equilibrium strategies and the optimal cost values of the LQ differential games. Furthermore, an iterative algorithm is given to solve the four-coupled GDREs. Finally, a suboptimal solution of the LQ differential games is proposed based on a convex optimization approach and a simplification of the suboptimal solution is given. Simulation examples are presented to illustrate the effectiveness of the iterative algorithm and the suboptimal solution.     

Infinite time horizon nonzero-sum linear quadratic stochastic differential games with state and control-dependent noise

This paper discusses the infinite time horizon nonzero-sum linear quadratic （LQ） differential games of stochastic systems governed by Itoe＇s equation with state and control-dependent noise. First, the nonzero-sum LQ differential games are formulated by applying the results of stochastic LQ problems. Second, under the assumption of mean-square stabilizability of stochastic systems, necessary and sufficient conditions for the existence of the Nash strategy are presented by means of four coupled stochastic algebraic Riccati equations. Moreover, in order to demonstrate the usefulness of the obtained results, the stochastic H-two/H-infinity control with state, control and external disturbance-dependent noise is discussed as an immediate application.     

线性Markov 切换系统的随机Nash 微分博弈及混合H2/H∞ 控制

研究线性Markov切换系统的随机Nash微分博弈问题。首先借助线性Markov切换系统随机最优控制的相关结果,得到了有限时域和无线时域Nash均衡解的存在条件等价于其相应微分(代数) Riccati方程存在解,并给出了最优解的显式形式;然后应用相应的微分博弈结果分析线性Markov切换系统的混合H2/H∞控制问题;最后通过数值算例验证了所提出方法的可行性。     

Guaranteeing cost strategies for linear quadratic differential games under uncertain dynamics

This paper deals with the design of closed loop strategies for a class of two players zero-sum linear quadratic differential games, where each player does not know exactly the state equation and model it through a system subject to norm-bounded uncertainties. The finite horizon and the infinite horizon problems are both solved: it turns out that the optimal strategies, guaranteeing to each player a given level of performance, require, to be evaluated, the solution of two scaled differential (algebraic in the infinite horizon case) Riccati equations. A numerical example illustrates an application of the proposed technique.     

Infinite horizon indefinite stochastic linear quadratic control for discrete-time systems

This paper discusses discrete-time stochastic linear quadratic (LQ) problem in the infinite horizon with state and control dependent noise, where the weighting matrices in the cost function are assumed to be indefinite. The problem gives rise to a generalized algebraic Riccati equation (GARE) that involves equality and inequality constraints. The well-posedness of the indefinite LQ problem is shown to be equivalent to the feasibility of a linear matrix inequality (LMI). Moreover, the existence of a stabilizing solution to the GARE is equivalent to the attainability of the LQ problem. All the optimal controls are obtained in terms of the solution to the GARE. Finally, we give an LMI -based approach to solve the GARE via a semidefinite programming.     

线性离散奇异系统的不定二次最优控制问题: Krein空间方法

The finite time horizon indefinite linear quadratic(LQ) optimal control problem for singular linear discrete time-varying systems is discussed. Indefinite LQ optimal control problem for singular systems can be transformed to that for standard state-space systems under a reasonable assumption. It is shown that the indefinite LQ optimal control problem is dual to that of projection for backward stochastic systems. Thus, the optimal LQ controller can be obtained by computing the gain matrices of Kalman filter. Necessary and sufficient conditions guaranteeing a unique solution for the indefinite LQ problem are given. An explicit solution for the problem is obtained in terms of the solution of Riccati difference equations.     

离散时间不定号随机线性二次型最优控制:无限时区情形

应用渐近分析方法讨论了无限时区离散时间不定号随机线性二次型最优控制问题. 所进行的研究是建立在这一问题有限时区情形结果和系统均方能镇定假设基础之上的. 广义代数Riccati方程(GARE)解的一些性质也得到了考虑. 最后提供了两个例子来说明所推出的结果是有效的.     

Suboptimal Strategies of Linear Quadratic Closed-loop Differential Games: An BMI Approach

The suboptimal control program via memoryless state feedback strategies for LQ differential games with multiple players is studied in this paper. Sufficient conditions for the existence of the suboptimal strategies for LQ differential games are presented. It is shown that the suboptimal strategies of LQ differential games are associated with a coupled algebraic Riccati inequality. Furthermore, the problem of designing suboptimal strategies is considered. A non-convex optimization problem with BMI constrains is formulated to design the suboptimal strategies which minimizes the performance indices of the closed-loop LQ differential games and can be solved by using LMI Toolbox of MATLAB, An example is given to illustrate the proposed results.     

线性二次闭环微分对策次优对策的双线性矩阵不等式方法

The suboptimal control program via memoryless state feedback strategies for LQ differential games with multiple players is studied in this paper. Sufficient conditions for the existence of the suboptimal strategies for LQ differential games are presented. It is shown that the suboptimal strategies of LQ differential games are associated with a coupled algebraic Riccati inequality. Furthermore, the problem of designing suboptimal strategies is considered. A non-convex optimization problem with BMI constrains is formulated to design the suboptimal strategies which minimizes the performance indices of the closed-loop LQ differential games and can be solved by using LMI Toolbox of MATLAB. An example is given to illustrate the proposed results.     

Infinite horizon backward stochastic differential equation and exponential convergence index assignment of stochastic control systems

This paper studies exponential convergence index assignment of stochastic control systems from the viewpoint of backward stochastic differential equation. Like deterministic control systems, it is shown that the exact controllability of an open-loop stochastic system is equivalent to the possibility of assigning an arbitrary exponential convergence index to the solution of the closed-loop stochastic system, formed by means of suitable linear feedback of the states. As an application, a sufficient and necessary condition for the existence and uniqueness of the solution of a class of infinite horizon forward-backward stochastic differential equations is provided.     

CONTROL OF INTERCONNECTED JUMPING SYSTEMS: AN H∞ APPROACH

This paper investigates, by using an approach, the problems of stochastic stability and control for a class of interconnected systems with Markovian jumping parameters. Both cases of finite‐ and infinite‐horizon are studied. It is shown that the problems under consideration can be solved if a set of coupled differential or algebraic Riccati equations are solvable.     

Optimal control of discrete-time linear fractional-order systems with multiplicative noise

A finite horizon linear quadratic (LQ) optimal control problem is studied for a class of discrete-time linear fractional systems (LFSs) affected by multiplicative, independent random perturbations. Based on the dynamic programming technique, two methods are proposed for solving this problem. The first one seems to be new and uses a linear, expanded-state model of the LFS. The LQ optimal control problem reduces to a similar one for stochastic linear systems and the solution is obtained by solving Riccati equations. The second method appeals to the principle of optimality and provides an algorithm for the computation of the optimal control and cost by using directly the fractional system. As expected, in both cases, the optimal control is a linear function in the state and can be computed by a computer program. A numerical example and comparative simulations of the optimal trajectory prove the effectiveness of the two methods. Some other simulations are obtained for different values of the fractional order.     

乘性随机离散系统的最优控制

基于对系统随机不确定因素的分析,文中定义了一种新型随机离散系统--乘性随机 离散系统,并研究该类系统的线性二次型(LQ)最优控制问题.首先给出了该类系统的有限时间 和无限时间LQ最优控制律,并着重分析、证明了无限时间LQ最优控制问题的Riccati方程的 正定矩阵解的存在性及相应数值求解算法与收敛性,以及闭环系统的稳定性等问题.仿真结果 表明了该方法的有效性.     

Global optimal fuzzy tracker design based on local concept approach

In this paper, we propose a global optimal fuzzy tracking controller, implemented by fuzzily blending the individual local fuzzy tracking laws, for continuous and discrete-time fuzzy systems with the aim of solving, respectively, the continuous and discrete-time quadratic tracking problems with moving or model-following targets under finite or infinite horizon (time). The differential or recursive Riccati equations, and more, the differential or difference equations in tracing the variation of the target, are derived. Moreover, in the case of time-invariant fuzzy tracking systems, we show that the optimal tracking controller can be obtained by just solving algebraic Riccati equations and algebraic matrix equations. Grounding on this, several fascinating characteristics of the resultant closed-loop continuous or discrete time-invariant fuzzy tracking systems can be elicited easily. The stability of both closed-loop fuzzy tracking systems can be ensured by the designed optimal fuzzy tracking controllers. The optimal closed-loop fuzzy tracking systems cannot only be guaranteed to be exponentially stable, but also be stabilized to any desired degree. Moreover, the resulting closed-loop fuzzy tracking systems possess infinite gain margin; that is, their stability is guaranteed no matter how large the feedback gain becomes. Two examples are given to illustrate the performance of the proposed optimal fuzzy tracker design schemes and to demonstrate the proved stability properties     

Partially observed non-linear risk-sensitive optimal stopping control for non-linear discrete-time systems

In this paper we introduce and solve the partially observed optimal stopping non-linear risk-sensitive stochastic control problem for discrete-time non-linear systems. The presented results are closely related to previous results for finite horizon partially observed risk-sensitive stochastic control problem. An information state approach is used and a new (three-way) separation principle established that leads to a forward dynamic programming equation and a backward dynamic programming inequality equation (both infinite dimensional). A verification theorem is given that establishes the optimal control and optimal stopping time. The risk-neutral optimal stopping stochastic control problem is also discussed.     

On a general optimal algorithm for multirate output feedbackcontrollers for linear stochastic periodic systems

A modified optimal algorithm for multirate output feedback controllers of linear stochastic periodic systems is developed. By combining the discrete-time linear quadratic regulation (LQR) control problem and the discrete-time stochastic linear quadratic regulation (SLQR) control problem to obtain an extended linear quadratic regulation (ELQR) control problem, one derives a general optimal algorithm to balance the advantages of the optimal transient response of the LQR control problem and the optimal steady-state regulation of the SLQR control problem. In general, the solution of this algorithm is obtained by solving a set of coupled matrix equations. Special cases for which the coupled matrix equations can be reduced to a discrete-time algebraic Riccati equation are discussed. A reducable case is the optimal algorithm derived by H.M. Al-Rahmani and G.F. Franklin (1990), where the system has complete state information and the discrete-time quadratic performance index is transformed from a continuous-time one     

A computational method for simultaneous LQ optimal control designvia piecewise constant output feedback

The paper is concerned with simultaneous linear-quadratic (LQ) optimal control design for a set of LTI systems via piecewise constant output feedback. First, the discrete-time simultaneous LQ optimal control design problem is reduced to solving a set of coupled matrix inequalities and an iterative LMI algorithm is presented to compute the feedback gain. Then, simultaneous stabilization and simultaneous LQ optimal control design of a set of LTI continuous-time systems are considered via periodic piecewise constant feedback gain. It is shown that the design of a periodic piecewise constant feedback gain simultaneously minimizing a set of given continuous-time performance indexes can be reduced to that of a constant feedback gain minimizing a set of equivalent discrete-time performance indexes. Explicit formulas for computing the equivalent discrete-time systems and performance indexes are derived. Examples are used to demonstrate the effectiveness of the proposed method.     

Multi-player non-zero-sum games: Online adaptive learning solution of coupled Hamilton–Jacobi equations

In this paper we present an online adaptive control algorithm based on policy iteration reinforcement learning techniques to solve the continuous-time (CT) multi player non-zero-sum (NZS) game with infinite horizon for linear and nonlinear systems. NZS games allow for players to have a cooperative team component and an individual selfish component of strategy. The adaptive algorithm learns online the solution of coupled Riccati equations and coupled Hamilton–Jacobi equations for linear and nonlinear systems respectively. This adaptive control method finds in real-time approximations of the optimal value and the NZS Nash-equilibrium, while also guaranteeing closed-loop stability. The optimal-adaptive algorithm is implemented as a separate actor/critic parametric network approximator structure for every player, and involves simultaneous continuous-time adaptation of the actor/critic networks. A persistence of excitation condition is shown to guarantee convergence of every critic to the actual optimal value function for that player. A detailed mathematical analysis is done for 2-player NZS games. Novel tuning algorithms are given for the actor/critic networks. The convergence to the Nash equilibrium is proven and stability of the system is also guaranteed. This provides optimal adaptive control solutions for both non-zero-sum games and their special case, the zero-sum games. Simulation examples show the effectiveness of the new algorithm.     

Discrete-time LQ-optimal control problems for infinite Markov jumpparameter systems

Optimal control problems for discrete-time linear systems subject to Markovian jumps in the parameters are considered for the case in which the Markov chain takes values in a countably infinite set. Two situations are considered: the noiseless case and the case in which an additive noise is appended to the model. The solution for these problems relies, in part, on the study of a countably infinite set of coupled algebraic Riccati equations (ICARE). Conditions for existence and uniqueness of a positive semidefinite solution to the ICARE are obtained via the extended concepts of stochastic stabilizability (SS) and stochastic detectability (SD), which turn out to be equivalent to the spectral radius of certain infinite dimensional linear operators in a Banach space being less than one. For the long-run average cost, SS and SD guarantee existence and uniqueness of a stationary measure and consequently existence of an optimal stationary control policy. Furthermore, an extension of a Lyapunov equation result is derived for the countably infinite Markov state-space case     

Minimax games for stochastic systems subject to relative entropy uncertainty: applications to SDEs on Hilbert spaces

In this paper, we consider minimax games for stochastic uncertain systems with the pay-off being a nonlinear functional of the uncertain measure where the uncertainty is measured in terms of relative entropy between the uncertain and the nominal measure. The maximizing player is the uncertain measure, while the minimizer is the control which induces a nominal measure. Existence and uniqueness of minimax solutions are derived on suitable spaces of measures. Several examples are presented illustrating the results. Subsequently, the results are also applied to controlled stochastic differential equations on Hilbert spaces. Based on infinite dimensional extension of Girsanov’s measure transformation, martingale solutions are used in establishing existence and uniqueness of minimax strategies. Moreover, some basic properties of the relative entropy of measures on infinite dimensional spaces are presented and then applied to uncertain systems described by a stochastic differential inclusion on Hilbert space. An explicit expression for the worst case measure representing the maximizing player (adversary) is found.