Stochastic differential games and inverse optimal control and stopper policies

In this paper, we consider a two-player stochastic differential game problem over an infinite time horizon where the players invoke controller and stopper strategies on a nonlinear stochastic differential game problem driven by Brownian motion. The optimal strategies for the two players are given explicitly by exploiting connections between stochastic Lyapunov stability theory and stochastic Hamilton–Jacobi–Isaacs theory. In particular, we show that asymptotic stability in probability of the differential game problem is guaranteed by means of a Lyapunov function which can clearly be seen to be the solution to the steady-state form of the stochastic Hamilton–Jacobi–Isaacs equation, and hence, guaranteeing both stochastic stability and optimality of the closed-loop control and stopper policies. In addition, we develop optimal feedback controller and stopper policies for affine nonlinear systems using an inverse optimality framework tailored to the stochastic differential game problem. These results are then used to provide extensions of the linear feedback controller and stopper policies obtained in the literature to nonlinear feedback controllers and stoppers that minimise and maximise general polynomial and multilinear performance criteria.     

Existence of perfect equilibria in a class of multigenerational stochastic games of capital accumulation

In this paper we introduce a model of multigenerational stochastic games of capital accumulation where each generation consists of m different players. The main objective is to prove the existence of a perfect stationary equilibrium in an infinite horizon game. A suitable change in the terminology used in this paper provides (in the case of perfect altruism between generations) a new Nash equilibrium theorem for standard stochastic games with uncountable state space.     

The problem of optimal approach in locally Euclidean spaces

A differential game of optimal approach with simple motions when players move in locally Euclidean spaces is studied. The game-end moment is fixed, and the game payment is a distance between the pursuer and the evader at the game-end moment. The value of game is obtained in the explicit form for any initial positions of players. Moreover, the differential game of optimal approach for the denumerable number of pursuers and one evader in the Euclidean space is solved. All pursuers are controlled by one parameter.     

Stochastic nonlinear minimax dynamic games with noisy measurements

This note is concerned with nonlinear stochastic minimax dynamic games which are subject to noisy measurements. The minimizing players are control inputs while the maximizing players are square-integrable stochastic processes. The minimax dynamic game is formulated using an information state, which depends on the paths of the observed processes. The information state satisfies a partial differential equation of the Hamilton-Jacobi-Bellman (HJB) type. The HJB equation is employed to characterize the dissipation properties of the system, to derive a separation theorem between the design of the estimator and the controller, and to introduce a certainty-equivalence principle along the lines of Whittle. Finally, the separation theorem and the certainty-equi. valence principle are applied to solve the linear-quadratic-Gaussian minimax game. The results of this note generalize the L/sup 2/-gain of deterministic systems to stochastic analogs; they are related to the controller design of stochastic systems which employ risk-sensitive performance criteria, and to the controller design of deterministic systems which employ minimax performance criteria.     

Differential games of <Emphasis Type="Italic">N</Emphasis> players in stochastic systems with controlled diffusion terms

Differential Games of N players defined by stochastic systems with controlled diffusion terms are considered. Necessary conditions of equilibrium strategy are obtained. These conditions are specified for the linear quadratic differential game of N players in terms of differential Riccati equation for program and positional equilibrium situations.     

On a class of linear stochastic differential games

The solution for a class of stochastic pursuit-evasion differential games between two linear dynamic systems is given. This class includes the classical interception game in Euclidean space. The performance index which is optimized is quadratic, and one of the two players has imperfect (noisy) knowledge of the states of the two systems. The "certainty-equivalence principle' or, equivalently, the technique of separating the estimator and the controller which characterizes the standard stochastic control problem is shown to be applicable to this class of differential games.     

Feasible solution to discrete-time linear quadratic stochastic Stackelberg difference game

This paper investigates the discrete-time linear quadratic (LQ) stochastic Stackelberg game, which has not been thoroughly addressed in previous literature. Firstly, we derive the maximum principle for the stochastic Stackelberg difference game using the variational method, and obtain the necessary and sufficient solvability conditions. However, due to the coupling between the two players and the presence of stochastic noise, obtaining explicit optimal leader and follower's strategies becomes challenging. Therefore, we present a feasible suboptimal control strategy instead. As a result, we derive a feasible suboptimal control strategy. To achieve this, we assume a linear homogeneous relationship to decouple the group of stochastic game forward-backward stochastic differential equations (SG-FBSDEs), which serves as a compromise for obtaining the optimal solution. With this approach, we derive a feasible solution to the stochastic Stackelberg difference game based on the solution to symmetric Riccati equations.     

Reduction of stochastic parity to stochastic mean-payoff games

A stochastic graph game is played by two players on a game graph with probabilistic transitions. We consider stochastic graph games with ω-regular winning conditions specified as parity objectives, and mean-payoff (or limit-average) objectives. These games lie in NP ∩ coNP. We present a polynomial-time Turing reduction of stochastic parity games to stochastic mean-payoff games.     

On an approach to constructing a characteristic function in cooperative differential games

We propose a novel approach to constructing characteristic functions in cooperative differential games. A characteristic function of a coalition S is computed in two stages: first, optimal control strategies maximizing the total payoff of the players are found, and next, these strategies are used by the players from the coalition S, while the other players, those from N S, use strategies minimizing the total payoff of the players from S. The characteristic function obtained in this way is superadditive. In addition, it possesses a number of other useful properties. As an example, we compute values of a characteristic function for a specific differential game of pollution control.     

A Group Pursuit Game

A differential game of pursuit of an evader by m dynamic pursuers under simple motion is studied. The time of game completion is fixed. Pursuers’ controls obey integral constraints, whereas the evader control obeys either an integral constraint or a geometric constraint. A differential game with cost defined by the distance between the evader and his nearest pursuer at the game completion instant is studied. Optimal strategies for players are constructed and the game cost is determined.__________Translated from Avtomatika i Telemekhanika, No. 8, 2005, pp. 24–35.Original Russian Text Copyright © 2005 by Ibragimov.     

On the Existence and Uniqueness of Equilibrium Situations in Markovian Games with Discounting

Markovian (stochastic) two-person games with discounting are considered. It is proved that if the set of states of such a game and the set of decisions of the players are finite, then the game has values and both players have optimal stationary strategies. The proof, which is based on the principle of contracting mappings, is constructive and leads to a recurrent algorithm of finding solutions of the game. The question of uniqueness of an equilibrium situation in the game considered is also discussed. In addition to the Markovian game with discounting in the context of the principle of contracting mappings, its subgame, namely, a Markovian decision process with discounting is also studied.     

Stabilization of stochastic nonlinear systems driven by noise ofunknown covariance

This paper poses and solves a new problem of stochastic (nonlinear) disturbance attenuation where the task is to make the system solution bounded by a monotone function of the supremum of the covariance of the noise. This is a natural stochastic counterpart of the problem of input-to-state stabilization in the sense of Sontag (1989). Our development starts with a set of new global stochastic Lyapunov theorems. For an exemplary class of stochastic strict-feedback systems with vanishing nonlinearities, where the equilibrium is preserved in the presence of noise, we develop an adaptive stabilization scheme (based on tuning functions) that requires no a priori knowledge of a bound on the covariance. Next, we introduce a control Lyapunov function formula for stochastic disturbance attenuation. Finally, we address optimality and solve a differential game problem with the control and the noise covariance as opposing players; for strict-feedback systems the resulting Isaacs equation has a closed-form solution     

多目标动态优化中Pareto随机合作博弈研究综述

随着经济全球化的不断深入,“合作共赢”的发展战略越来越被人们接受,进而合作博弈也被合理地应用到多个领域.与静态合作博弈相比,动态博弈的约束条件为动态方程,其具有优化行为、多个玩家共同存在、决策结果的持久性以及对环境变化的鲁棒性等特点.由于动态系统总是受到某些随机波动的干扰,将这些内部随机波动和外部随机扰动考虑到系统模型中更为实际.随机动态合作博弈同时考虑策略行为、动态演化与随机因素之间的相互作用,其可能是最复杂的决策形式之一.鉴于此,对多目标动态优化中随机合作博弈的进展进行综述:首先,回顾多目标合作博弈的研究背景,给出Pareto最优性的定义和基本性质;其次,综述确定性的合作博弈;再次,分别论述随机合作博弈和平均场随机合作博弈;最后,提出随机合作博弈几个未来研究方向.     

Evolving an expert checkers playing program without using humanexpertise

An evolutionary algorithm has taught itself how to play the game of checkers without using features that would normally require human expertise. Using only the raw positions of pieces on the board and the piece differential, the evolutionary program optimized artificial neural networks to evaluate alternative positions in the game. Over the course of several hundred generations, the program taught itself to play at a level that is competitive with human experts (one level below human masters). This was verified by playing the best evolved neural network against 165 human players on an Internet gaming zone. The neural network's performance earned a rating that was better than 99.61% of all registered players at the Website. Control experiments between the best evolved neural network and a program that relies on material advantage indicate the superiority of the neural network both at equal levels of look ahead and CPU time. The results suggest that the principles of Darwinian evolution may he usefully applied to solving problems that have not yet been solved by human expertise     

Fuzzy differential games for nonlinear stochastic systems:suboptimal approach

A fuzzy differential game theory is proposed to solve the n-person (or n-player) nonlinear differential noncooperative game and cooperative game (team) problems, which are not easily tackled by the conventional methods. In the paper, both noncooperative and cooperative quadratic differential games are considered. First, the nonlinear stochastic system is approximated by a fuzzy model. Based on the fuzzy model, a fuzzy controller is proposed to deal with the noncooperative differential game in the sense of Nash equilibrium strategies or with the cooperative game in the sense of Pareto-optimal strategies. Using a suboptimal approach, the outcomes of the fuzzy differential games for both the noncooperative and the cooperative cases are parameterized in terms of an eigenvalue problem. Since the state variables are usually unavailable, a suboptimal fuzzy observer is also proposed in this study to estimate the states for these differential game problems. Finally, simulation examples are given to illustrate the design procedures and to indicate the performance of the proposed methods     

基于鲁棒控制的期权套期保值策略

在标的资产价格服从带有随机方差几何布朗运动的非完全市场假设条件下,应用随机微分对策方法,研究与标的资产有关的欧式期权的动态套期保值策略问题。建立了最优动态套期保值策略的随机微分对策数学模型,给出了基于鲁棒控制的均方复制误差最小的自融资动态套期保值策略。当方差为时间的确定性函数时,最优动态套期保值策略与用Black-Scholes套期比表示的delta套期保值策略是一致的。     

A BSDE approach to a risk-based optimal investment of an insurer

We discuss a backward stochastic differential equation, (BSDE), approach to a risk-based, optimal investment problem of an insurer. A simplified continuous-time economy with two investment vehicles, namely, a fixed interest security and a share, is considered. The insurer’s risk process is modeled by a diffusion approximation to a compound Poisson risk process. The goal of the insurer is to select an optimal portfolio so as to minimize the risk described by a convex risk measure of his/her terminal wealth. The optimal investment problem is then formulated as a zero-sum stochastic differential game between the insurer and the market. The BSDE approach is used to solve the game problem. It leads to a simple and natural approach for the existence and uniqueness of an optimal strategy of the game problem without Markov assumptions. Closed-form solutions to the optimal strategies of the insurer and the market are obtained in some particular cases.     

A Stochastic Game Model for Evaluating the Impacts of Security Attacks Against Cyber-Physical Systems

A quantitative security evaluation in the domain of cyber-physical systems (CPS), which operate under intentional disturbances, is an important open problem. In this paper, we propose a stochastic game model for quantifying the security of CPS. The proposed model divides the security modeling process of these systems into two phases: (1) intrusion process modeling and (2) disruption process modeling. In each phase, the game theory paradigm predicts the behaviors of the attackers and the system. By viewing the security states of the system as the elements of a stochastic game, Nash equilibriums and best-response strategies for the players are computed. After parameterization, the proposed model is analytically solved to compute some quantitative security measures of CPS. Furthermore, the impact of some attack factors and defensive countermeasures on the system availability and mean time-to-shutdown is investigated. Finally, the proposed model is applied to a boiling water power plant as an illustrative example.     

Strongly Subgame-Consistent Core in Stochastic Games

This paper investigates stochastic games on finite tree graphs. A given n-player normal-form game is defined at each node of a tree. Transition to a next node of the tree is random and depends on the strategy profile realized in a current game. We construct a cooperative solution of the game by maximizing the total expected payoff of the players. The core is used as the solution concept of the cooperative game. We introduce the definition of a strongly subgame-consistent (strongly time-consistent) core. Finally, we suggest a method for designing a cooperative distribution procedure of an imputation from the core that guarantees its strong subgame consistency.     

Stochastic minimax optimal control strategy for uncertain quasi-Hamiltonian systems using stochastic maximum principle

A stochastic minimax optimal control strategy for uncertain quasi-Hamiltonian systems is proposed based on the stochastic averaging method, stochastic maximum principle and stochastic differential game theory. First, the partially completed averaged Itô stochastic differential equations are derived from a given system by using the stochastic averaging method for quasi-Hamiltonian systems with uncertain parameters. Then, the stochastic Hamiltonian system for minimax optimal control with a given performance index is established based on the stochastic maximum principle. The worst disturbances are determined by minimizing the Hamiltonian function, and the worst-case optimal controls are obtained by maximizing the minimal Hamiltonian function. The differential equation for adjoint process as a function of system energy is derived from the adjoint equation by using the Itô differential rule. Finally, two examples of controlled uncertain quasi-Hamiltonian systems are worked out to illustrate the application and effectiveness of the proposed control strategy.