Connections between stochastic control and dynamic games

We consider duality relations between risk-sensitive stochastic control problems and dynamic games. They are derived from two basic duality results, the first involving free energy and relative entropy and resulting from a Legendre-type transformation, the second involving power functions. Our approach allows us to treat, in essentially the same way, continuous- and discrete-time problems, with complete and partial state observation, and leads to a very natural formal justification of the structure of the cost functional of the dual. It also allows us to obtain the solution of a stochastic game problem by solving a risk-sensitive control problem.     

Risk-sensitive filtering and smoothing for hidden Markov models

In this paper, we address the problem of risk-sensitive filtering and smoothing for discrete-time Hidden Markov Models (HMM) with finite-discrete states. The objective of risk-sensitive filtering is to minimise the expectation of the exponential of the squared estimation error weighted by a risk-sensitive parameter. We use the so-called Reference Probability Method in solving this problem. We achieve finite-dimensional linear recursions in the information state, and thereby the state estimate that minimises the risk-sensitive cost index. Also, fixed-interval smoothing results are derived. We show that L₂ or risk-neutral filtering for HMMs can be extracted as a limiting case of the risk-sensitive filtering problem when the risk-sensitive parameter approaches zero.     

A new risk-sensitive maximum principle

In this paper, a new maximum principle for the risk-sensitive control problem is established. One important feature of this result is that it applies to systems in which the diffusion term may depend on the control. Such control dependence gives rise to interesting phenomena not observed in the usual setting where control independence of the diffusion term is assumed. In particular, there is an additional second order adjoint equation and additional terms in the maximum condition that involve this second order process as well as the risk-sensitive parameter. Moreover, contrary to a conventional maximum principle, the first-order adjoint equation involved in our maximum principle is a nonlinear equation. An advantage of considering this new type of adjoint equation is that the risk-sensitive maximum principle derived is similar in form to its risk-neutral counterpart. The approach is based on the logarithmic transformation and the relationship between the adjoint variables and the value function. As an example, a linear-quadratic risk-sensitive problem is solved using the maximum principle derived.     

Nash equilibria in risk-sensitive dynamic games

Dynamic games in which each player has an exponential cost criterion are referred to as risk-sensitive dynamic games. In this note, Nash equilibria are considered for such games. Feedback risk-sensitive Nash equilibrium solutions are derived for two-person discrete time linear-quadratic nonzero-sum games, both under complete state observation and shared partial observation     

A note on risk-sensitive control of invariant models

According to Assaf, a dynamic programming problem is called invariant if its transition mechanism depends only on the chosen action. This paper studies properties of risk-sensitive invariant problems with a general state space. The main result establishes the optimality equation for the risk-sensitive average cost criterion without any restrictions on the risk factor. Moreover, a practical algorithm is provided for solving the optimality equation in case of a finite action space.     

Mixed risk-neutral/minimax control of discrete-time, finite-stateMarkov decision processes

Addresses the control design problem for discrete-time, finite-state Markov decision processes, when both risk-neutral and minimax objectives are of interest. We introduce the mixed risk-neutral/minimax objective and utilize results from risk-neutral and minimax control to derive an information state process and dynamic programming equations for the value function. We synthesize optimal control laws both on the finite and the infinite horizons. We study the effectiveness of both the mixed risk-neutral/minimax family and the risk-sensitive family of controllers as tools to tradeoff risk-neutral and minimax objectives. We conclude that the mixed risk-neutral/minimax family is more effective, at the cost of increased controller complexity     

Partially observed non-linear risk-sensitive optimal stopping control for non-linear discrete-time systems

In this paper we introduce and solve the partially observed optimal stopping non-linear risk-sensitive stochastic control problem for discrete-time non-linear systems. The presented results are closely related to previous results for finite horizon partially observed risk-sensitive stochastic control problem. An information state approach is used and a new (three-way) separation principle established that leads to a forward dynamic programming equation and a backward dynamic programming inequality equation (both infinite dimensional). A verification theorem is given that establishes the optimal control and optimal stopping time. The risk-neutral optimal stopping stochastic control problem is also discussed.     

Average cost criterion induced by the regular utility function for continuous-time Markov decision processes

In this paper we study the average cost criterion induced by the regular utility function (U-average cost criterion) for continuous-time Markov decision processes. This criterion is a generalization of the risk-sensitive average cost and expected average cost criteria. We first introduce an auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function under the slight conditions. Then we show that the pair of the optimal value functions of the risk-sensitive average cost criterion and the risk-sensitive first passage criterion is a solution to the optimality equation of the risk-sensitive average cost criterion allowing the risk-sensitivity parameter to take any nonzero value. Moreover, we have that the optimal value function of the risk-sensitive average cost criterion is continuous with respect to the risk-sensitivity parameter. Finally, we give the connections between the U-average cost criterion and the average cost criteria induced by the identity function and the exponential utility function, and prove the existence of a U-average optimal deterministic stationary policy in the class of all randomized Markov policies.     

The vanishing discount approach in Markov chains with risk-sensitive criteria

In this paper stochastic dynamic systems are studied, modeled by a countable state space Markov cost/reward chain, satisfying a Lyapunov-type stability condition. For an infinite planning horizon, risk-sensitive (exponential) discounted and average cost criteria are considered. The main contribution is the development of a vanishing discount approach to relate the discounted criterion problem with the average criterion one, as the discount factor increases to one, i.e., no discounting. In comparison to the well-established risk-neutral case, our results are novel and reveal several fundamental and surprising differences. Other contributions made include the use of convex analytic arguments to obtain appropriately convergent sequences and a verification theorem for the case of unbounded solutions to the average cost Poisson equation arising in the risk-sensitive case. Also of importance is the fact that our developments are very much self-contained and employ only basic probabilistic and analysis principles.     

Risk-sensitive optimal control of hidden Markov models: structuralresults

The authors consider a risk-sensitive optimal control problem for (finite state and action spaces) hidden Markov models (HMM). They present results of an investigation on the nature and structure of risk-sensitive controllers for HMM. Several general structural results are presented, as well as a particular case study of a popular benchmark problem. For the latter, they obtain structural results for the optimal risk-sensitive controller and compare it to that of the risk-neutral controller. Furthermore, they show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation as the risk factor goes to zero. They also study the infinite and general risk aversion cases     

Risk-sensitive control and dynamic games for partially observeddiscrete-time nonlinear systems

Solves a finite-horizon partially observed risk-sensitive stochastic optimal control problem for discrete-time nonlinear systems and obtains small noise and small risk limits. The small noise limit is interpreted as a deterministic partially observed dynamic game, and new insights into the optimal solution of such game problems are obtained. Both the risk-sensitive stochastic control problem and the deterministic dynamic game problem are solved using information states, dynamic programming, and associated separated policies. A certainty equivalence principle is also discussed. The authors' results have implications for the nonlinear robust stabilization problem. The small risk limit is a standard partially observed risk-neutral stochastic optimal control problem     

Quantum Risk-Sensitive Estimation and Robustness

This paper studies a quantum risk-sensitive estimation problem and investigates robustness properties of the filter. This is a direct extension to the quantum case of analogous classical results. All investigations are based on a discrete approximation model of the quantum system under consideration. This allows us to study the problem in a simple mathematical setting. We close the paper with some examples that demonstrate the robustness of the risk-sensitive estimator.     

Risk-sensitive control of Markov jump linear systems: Caveats and difficulties

In this technical note, we revisit the risk-sensitive optimal control problem for Markov jump linear systems (MJLSs). We first demonstrate the inherent difficulty in solving the risk-sensitive optimal control problem even if the system is linear and the cost function is quadratic. This is due to the nonlinear nature of the coupled set of Hamilton-Jacobi-Bellman (HJB) equations, stemming from the presence of the jump process. It thus follows that the standard quadratic form of the value function with a set of coupled Riccati differential equations cannot be a candidate solution to the coupled HJB equations. We subsequently show that there is no equivalence relationship between the problems of risk-sensitive control and H ^∞ control of MJLSs, which are shown to be equivalent in the absence of any jumps. Finally, we show that there does not exist a large deviation limit as well as a risk-neutral limit of the risk-sensitive optimal control problem due to the presence of a nonlinear coupling term in the HJB equations.

     

扩展风险性滤波算法的修正方法

针对非线性系统扩展风险性滤波(ERSF)估计精度较低的缺点,提出一种基于二阶Taylor展开对其进行高阶修正的算法.该算法利用高阶项对一阶扩展卡尔曼滤波(EKF)的状态估计向量及协方差阵进行适当修正,并由新息滤波方法得到修正的扩展风险性滤波(MERSF).高度非线性的仿真研究表明,所提出的算法在计算量增加不多的情况下,滤波精度有明显的提高.     

Finite Horizon Minimax Optimal Control of Stochastic Partially Observed Time Varying Uncertain Systems

We consider a linear-quadratic problem of minimax optimal control for stochastic uncertain control systems with output measurement. The uncertainty in the system satisfies a stochastic integral quadratic constraint. To convert the constrained optimization problem into an unconstrained one, a special S-procedure is applied. The resulting unconstrained game-type optimization problem is then converted into a risk-sensitive stochastic control problem with an exponential-of-integral cost functional. This is achieved via a certain duality relation between stochastic dynamic games and risk-sensitive stochastic control. The solution of the risk-sensitive stochastic control problem in terms of a pair of differential matrix Riccati equations is then used to establish a minimax optimal control law for the original uncertain system with uncertainty subject to the stochastic integral quadratic constraint. Date received: May 13, 1997. Date revised: March 18, 1998.     

Finite-dimensional risk-sensitive filters and smoothers fordiscrete-time nonlinear systems

Finite-dimensional optimal risk-sensitive filters and smoothers are obtained for discrete-time nonlinear systems by adjusting the standard exponential of a quadratic risk-sensitive cost index to one involving the plant nonlinearity. It is seen that these filters and smoothers are the same as those for a fictitious linear plant with the exponential of squared estimation error as the corresponding risk-sensitive cost index. Such finite-dimensional filters do not exist for nonlinear systems in the case of minimum variance filtering and control     

Stochastic nonlinear minimax dynamic games with noisy measurements

This note is concerned with nonlinear stochastic minimax dynamic games which are subject to noisy measurements. The minimizing players are control inputs while the maximizing players are square-integrable stochastic processes. The minimax dynamic game is formulated using an information state, which depends on the paths of the observed processes. The information state satisfies a partial differential equation of the Hamilton-Jacobi-Bellman (HJB) type. The HJB equation is employed to characterize the dissipation properties of the system, to derive a separation theorem between the design of the estimator and the controller, and to introduce a certainty-equivalence principle along the lines of Whittle. Finally, the separation theorem and the certainty-equi. valence principle are applied to solve the linear-quadratic-Gaussian minimax game. The results of this note generalize the L/sup 2/-gain of deterministic systems to stochastic analogs; they are related to the controller design of stochastic systems which employ risk-sensitive performance criteria, and to the controller design of deterministic systems which employ minimax performance criteria.     

NONLINEAR DISCRETE-TIME RISK-SENSITIVE OPTIMAL CONTROL

This paper is devoted to the study of the connections among risk-sensitive stochastic optimal control, dynamic game optimal control, risk-neutral stochastic optimal control and deterministic optimal control in a nonlinear, discrete-t ime context with complete state information. The analysis worked out sheds light on the profound links among these control strategies, which remain hidden in the linear context. In particular, it is shown that, under suitable parameterizations, risk-sensi tive control can be regarded as a control methodology which combines features of both stochastic risk-neutral control and deterministic dynamic game control.     

Risk-sensitive filtering for discrete-time systems with time-varying delay

In this paper, the risk-sensitive filtering problem with time-varying delay is investigated. The problem is transformed into Krein space as an equivalent optimisation problem. The observations with time-varying delays are restructured as ones with multiple constant delays by defining a binary variable model with respect to the arrival process of observations, containing the same state information as the original. Finally, the reorganised innovation analysis approach in Krein space allows the solution to the proposed risk-sensitive filtering in terms of the solutions to Riccati and matrix difference equations.