Robustness inequality for Markov control processes with unbounded costs

The paper discusses the robustness of discrete-time Markov control processes whose transition probabilities are known up to certain degree of accuracy. Upper bounds of increase of a discounted cost are derived when using an optimal control policy of the approximating process in order to control the original one. Bounds are given in terms of weighted total variation distance between transition probabilities. They hold for processes on Borel spaces with unbounded one-stage costs functions.     

Adaptive policies for discrete-time stochastic control systems with unknown disturbance distribution

We introduce adaptive policies for discrete-time, infinite horizon, stochastic control systems x₁₊₁ = F(x₁, a₁, ξ₁, T =0, 1, …, with discounted reward criterion, where the disturbance process ξ₁ is a sequence of i.i.d. random elements with unknown distribution. These policies are shown to be asymptotically optimal and for each of them we obtain (almost surely) uniform approximations of the optimal reward function.     

H 2 optimal control for a wide class of discrete-time linear stochastic systems

In this article, the problem of H ₂-control of a discrete-time linear system subject to Markovian jumping and independent random perturbations is considered. Different H ₂ performance criteria (often called H ₂-norms) are introduced and characterised via solutions of some suitable linear equations on certain spaces of symmetric matrices. Some aspects specific to the discrete-time framework are revealed. The problem of optimisation of H ₂-norms is solved under the assumption that full state vector is available for measurements. One shows that among all stabilising controllers of higher dimension, the best performance is achieved by a zero-order controller. The corresponding feedback gain of the optimal controller is constructed based on the stabilising solution of a system of discrete-time generalised Riccati equations.     

Quadratic control of stochastic hybrid systems with renewal transitions

We study the quadratic control of a class of stochastic hybrid systems with linear continuous dynamics for which the lengths of time that the system stays in each mode are independent random variables with given probability distribution functions. We derive a condition for finding the optimal feedback policy that minimizes a discounted infinite horizon cost. We show that the optimal cost is the solution to a set of differential equations with unknown boundary conditions. Furthermore, we provide a recursive algorithm for computing the optimal cost and the optimal feedback policy. The applicability of our result is illustrated through a numerical example, motivated by stochastic gene regulation in biology.     

随机马尔可夫跳变系统的变结构控制设计

采用滑动扇区方法,研究了不确定随机马尔可夫跳变系统的变结构控制设计问题。首先给出随机马尔可夫跳变系统滑动扇区的定义,然后基于线性矩阵不等式技术,提出一种滑动扇区及变结构控制律设计方法。经过理论证明该控制律能够确保随机马尔可夫跳变不确定系统二次稳定,并有效地抑制抖振。最后数值仿真算例验证了控制方案的有效性。     

Infinite horizon optimal control of linear discrete time systems with stochastic parameters

The infinite horizon optimal control problem is considered in the general case of linear discrete time systems and quadratic criteria, both with stochastic parameters which are independent with respect to time. A stronger stabilizability property and a weaker observability property than usual for deterministic systems are introduced. It is shown that the infinite horizon problem has a solution if the system has the first property. If in addition the problem has the second property the solution is unique and the control system is stable in the mean square sense. A simple necessary and sufficient condition, explicit in the system matrices, is given for the system to have the stronger stabilizability property. This condition also holds for deterministic systems to be stabilizable in the usual sense. The stronger stabilizability and weaker observability properties coincide with the usual ones if the parameters are deterministic.     

The average cost optimality equation for Markov control processes on Borel spaces

This paper deals with discrete-time Markov control processes with Borel state space, allowing unbounded costs and noncompact control sets. For these models, the existence of average optimal stationary policies has been recently established under very general assumptions, using an optimality inequality. Here we give a condition, which is a strengtened version of a variant of the ‘vanishing discount factor’ approach, for the optimality equation to hold.     

Uniformly optimal control of linear time-varying systems

An optimal control problem is formulated in the context of linear, discrete-time, time-varying systems. The cost is the supremum, over all exogenous inputs in a weighted ball, of the sum of the weighted energies of the plant's input and output. The controller is required to be causal and to achieve internal stability. Existence of an optical controller is proved and a formula for the minimum cost is derived.     

Properties of optimal stochastic control systems with dead-time

Structural, stability and sensitivity properties of optimal stochastic control systems for dead-time, stable minimum phase as well as non-minimum phase processes are presented. The processes are described by rational transfer functions plus dead-times and the disturbances by rational spectral densities. It is shown that although the frequency domain design techniques guarantee asymptotically stable systems for given process and disturbance models, many of the designs might be practically unstable. Necessary and sufficient conditions that must be imposed on the design to assure practically stable optimal systems are derived. The uncertainties in the parameters and in the structure of the process model are measured by means of an ignorance function. Sufficient conditions in terms of the ignorance function, which guarantee stable design and by means of which the bounds of the uncertainties for a given design may be estimated, are stated. Conditions under which the optimal designs possess attractive relative stability properties, namely gain and phase margins of at least 2 and 60°, respectively, are stated, too. It is further shown that any optimal controller, for the type of processes discussed in this paper, may be separated into a primary controller and into a dead-time compensator where the latter is completely independent of the cost and the disturbance properties. Such a decomposition gives excellent insight into the role of the cost and the disturbance in the design. When low order process and disturbance models are used, the conventional PI and PID control laws coupled with the dead-time compensator emerge.     

Optimal control of discrete-time bilinear systems with applications to switched linear stochastic systems

This paper aims at characterizing the most destabilizing switching law for discrete-time switched systems governed by a set of bounded linear operators. The switched system is embedded in a special class of discrete-time bilinear control systems. This allows us to apply the variational approach to the bilinear control system associated with a Mayer-type optimal control problem, and a second-order necessary optimality condition is derived. Optimal equivalence between the bilinear system and the switched system is analyzed, which shows that any optimal control law can be equivalently expressed as a switching law. This specific switching law is most unstable for the switched system, and thus can be used to determine stability under arbitrary switching. Based on the second-order moment of the state, the proposed approach is applied to analyze uniform mean-square stability of discrete-time switched linear stochastic systems. Numerical simulations are presented to verify the usefulness of the theoretic results.     

Approximations for optimal stopping of a piecewise-deterministic process

This paper deals with approximation techniques for the optimal stopping of a piecewise-deterministic Markov process (P.D.P.). Such processes consist of a mixture of deterministic motion and random jumps. In the first part of the paper (Section 3) we study the optimal stopping problem with lower semianalytic gain function; our main result is the construction of ε-optimal stopping times. In the second part (Section 4) we consider a P.D.P. satisfying some smoothness conditions, and forN integer we construct a discretized P.D.P. which retains the main characteristics of the original process. By iterations of the single jump operator from ℝ ^N to ℝ ^N , each iteration consisting ofN one-dimensional minimizations, we can calculate the payoff function of the discretized process. We demonstrate the convergence of the payoff functions, and for the case when the state space is compact we construct ε-optimal stopping times for the original problem using the payoff function of the discretized problem. A numerical example is presented.     

Discrete-time optimal control for stochastic nonlinear polynomial systems

This paper presents a solution to the discrete-time optimal control problem for stochastic nonlinear polynomial systems over linear observations and a quadratic criterion. The solution is obtained in two steps: the optimal control algorithm is developed for nonlinear polynomial systems by considering complete information when generating a control law. Then, the state estimate equations for discrete-time stochastic nonlinear polynomial system over linear observations are employed. The closed-form solution is finally obtained substituting the state estimates into the obtained control law. The designed optimal control algorithm can be applied to both distributed and lumped systems. To show effectiveness of the proposed controller, an illustrative example is presented for a second degree polynomial system. The obtained results are compared to the optimal control for the linearized system.     

Average cost optimal policies for Markov control processes with Borel state space and unbounded costs

We show the existence of average cost optimal stationary policies for Markov control processes with Borel state space and unbounded costs per stage, under a set of assumptions recently introduced by L.I. Sennott (1989) for control processes with countable state space and finite control sets.     

Preview control for a class of linear stochastic systems with multiplicative noise

ABSTRACT

In this paper, the preview control problem for a class of linear continuous time stochastic systems with multiplicative noise is studied based on the augmented error system method. First, a deterministic assistant system is introduced, and the original system is translated to the assistant system. Then, the integrator is employed to ensure the output of the closed-loop system tracking the reference signal accurately. Second, the augmented error system, which includes integrator vector, control vector and reference signal, is constructed based on the system after translation. As a result, the tracking problem is transformed into the optimal control problem of the augmented error system, and the optimal control input is obtained by the dynamic programming method. This control input is regarded as the preview controller of the original system. For a linear stochastic system with multiplicative noise, the difficulty being unable to construct an augmented error system by the derivation method is solved in this paper. And, the existence and uniqueness solution of the Riccati equation corresponding to the stochastic augmented error system is discussed. The numerical simulations show that the preview controller designed in this paper is very effective.     

Near optimal control for a class of stochastic hybrid systems

To address a computationally intractable optimal control problem for a class of stochastic hybrid systems, this paper proposes a near optimal state feedback control scheme, which is constructed by using a statistical prediction method based on approximate numerical solution that samples over the entire state space. A numerical example illustrates the potential of the approach.     

Adaptive control of a class of nonlinear time-varying systems with multiple models

The adaptive control of nonlinear systems that are linear in the unknown but time-varying parameters are treated in this paper. Since satisfactory transient performance is an important factor, multiple models are required as these parameters change abruptly in the parameter space. In this paper we consider both the multiple models with switching and tuning methodology as well as multiple models with second level adaptation for this class of systems. We demonstrate that the latter approach is better than the former.     

Optimal control of switching time in switched stochastic systems with multi-switching times and different costs

In this paper, we solve an optimal control problem for a class of time-invariant switched stochastic systems with multi-switching times, where the objective is to minimise a cost functional with different costs defined on the states. In particular, we focus on problems in which a pre-specified sequence of active subsystems is given and the switching times are the only control variables. Based on the calculus of variation, we derive the gradient of the cost functional with respect to the switching times on an especially simple form, which can be directly used in gradient descent algorithms to locate the optimal switching instants. Finally, a numerical example is given, highlighting the validity of the proposed methodology.     

Properties of linear switching time-varying discrete-time systems with applications

We study two discrete-time, linear switching time-varying (LSTV) structures, each of which consists of a periodic switch connected to several linear time-invariant (LTI) systems. Such structures can be used to represent any linear periodically time-varying (LPTV) systems. We give basic properties associated with the LSTV structures in terms of their LTI building blocks, and then apply the results to solve a general approximation problem: How to optimally approximate an LPTV system with period p by an LPTV system with period ? The optimality is measured using norms. The study is extended to general multirate periodic systems.