Linear convex stochastic control problems over an infinite horizon

A stochastic control problem over an infinite horizon which involves a linear system and a convex cost functional is analyzed. We prove the convergence of the dynamic programming algorithm associated with the problem, and we show the existence of a stationary Borel measurable optimal control law. The approach used illustrates how results on infinite time reachability [1] can be used for the analysis of dynamic programming algorithms over an infinite horizon subject to state constraints.     

Linear–quadratic switching control with switching cost

We study in this paper the linear–quadratic (LQ) optimal control problem of discrete-time switched systems with a constant switching cost for both finite and infinite time horizons. We reduce these problems into an auxiliary problem, which is an LQ optimal switching control problem with a cardinality constraint on the total number of switchings. Based on the solution structure derived from the dynamic programming (DP) procedure, we develop a lower bounding scheme by exploiting the monotonicity of the Riccati difference equation. Integrating such a lower bounding scheme into a branch and bound (BnB) framework, we offer an efficient numerical solution scheme for the LQ switching control problem with switching cost.     

Fast algorithm for a constrained infinite horizon LQ problem

This paper presents an efficient algorithmic solution to the infinite horizon linear quadratic optimal control problem for a discrete-time SISO plant subject to bound constraints on a scalar variable. The solution to the corresponding quadratic programming problem is based on the active set method and on dynamic programming. It is shown that the optimal solution can be updated after inclusion or removal of an active constraint by a simple procedure requiring in the order of kn operations, n being the system order and k the time at which the constraint is included or removed.     

Controllability, stabilizability, and continuous-time Markovianjump linear quadratic control

Consideration is given to the control of continuous-time linear systems that possess randomly jumping parameters which can be described by finite-state Markov processes. The relationship between appropriately defined controllability, stabilizability properties, and the solution of the infinite time jump linear quadratic (JLQ) optimal control problems is also examined. Although the solution of the continuous-time Markov JLQ problem with finite or infinite time horizons is known, only sufficient conditions for the existence of finite cost, constant, stabilizing controls for the infinite time problem appear in the literature. In this paper necessary and sufficient conditions are established. These conditions are based on new definitions of controllability, observability, stabilizability, and detectability that are appropriate for continuous-time Markovian jump linear systems. These definitions play the same role for the JLQ problem as the deterministic properties do for the linear quadratic regulator (LQR) problem     

Mixed Constrained Infinite Horizon Linear Quadratic Optimal Control

For a given initial state, a constrained infinite horizon linear quadratic optimal control problem can be reduced to a finite dimensional problem [12]. To find a conservative estimate of the size of the reduced problem, the existing algorithms require the on‐line solutions of quadratic programs [10] or a linear program [2]. In this paper, we first show based on the Lyapunov theorem that the closed‐loop system with a mixed constrained infinite horizon linear quadratic optimal control is exponentially stable on proper sets. Then the exponentially converging envelop of the closed‐loop trajectory that can be computed off‐line is employed to obtain a finite dimensional quadratic program equivalent to the mixed constrained infinite horizon linear quadratic optimal control problem without any on‐line optimization. The example considered in [2] showed that the proposed algorithm identifies less conservative size estimate of the reduced problem with much less computation.     

Conditions for optimality of Naïve quantized finite horizon control

This paper presents properties of a control law which quantizes the unconstrained solution to a unitary horizon quadratic programme. This naïve quantized control law underlies many popular algorithms, such as ΣΔ-converters and decision feedback equalities, and is easily shown to be globally optimal for horizon one. However, the question arises as to whether it is also globally optimal for horizons greater than one, i.e. whether it solves a finite horizon quadratic programme, where decision variables are restricted to belonging to a quantized set. By using dynamic programming, we develop sufficient conditions for this to hold. The present analysis is restricted to first order plants. However, this case already raises a number of highly non-trivial issues. The results can be applied to arbitrary horizons and quantized sets, which may contain a finite or an infinite (though countable) number of elements.     

A rigorous solution of the infinite time interval LQ problem with constant state tracking

A linear quadratic constant state tracking problem is considered over an infinite time horizon. It is shown that the solution cannot be obtained as a limit from a finite time horizon problem, as in general the limiting problem is ill-posed. To obtain a rigorous solution, the problem is split in two natural well-posed subproblems. One optimal control problem addresses the transient and the other optimal control problem concerns the steady state behavior. It is shown that the transient problem and the steady state problem are solved by the same control law.     

乘性随机离散系统的最优控制

基于对系统随机不确定因素的分析,文中定义了一种新型随机离散系统--乘性随机离散系统,并研究该类系统的线性二次型(LQ)最优控制问题.首先给出了该类系统的有限时间和无限时间LQ最优控制律,并着重分析、证明了无限时间LQ最优控制问题的Riccati方程的正定矩阵解的存在性及相应数值求解算法与收敛性,以及闭环系统的稳定性等问题.仿真结果表明了该方法的有效性.     

Sparse Regression Ensembles in Infinite and Finite Hypothesis Spaces

We examine methods for constructing regression ensembles based on a linear program (LP). The ensemble regression function consists of linear combinations of base hypotheses generated by some boosting-type base learning algorithm. Unlike the classification case, for regression the set of possible hypotheses producible by the base learning algorithm may be infinite. We explicitly tackle the issue of how to define and solve ensemble regression when the hypothesis space is infinite. Our approach is based on a semi-infinite linear program that has an infinite number of constraints and a finite number of variables. We show that the regression problem is well posed for infinite hypothesis spaces in both the primal and dual spaces. Most importantly, we prove there exists an optimal solution to the infinite hypothesis space problem consisting of a finite number of hypothesis. We propose two algorithms for solving the infinite and finite hypothesis problems. One uses a column generation simplex-type algorithm and the other adopts an exponential barrier approach. Furthermore, we give sufficient conditions for the base learning algorithm and the hypothesis set to be used for infinite regression ensembles. Computational results show that these methods are extremely promising.     

Differential dynamic programming methods for solving bang-bang control problems

Differential dynamic programming is a technique, based on dynamic programming rather than the calculus of variations, for determining the optimal control function of a nonlinear system. Unlike conventional dynamic programming where the optimal cost function is considered globally, differential dynamic programming applies the principle of optimality in the neighborhood of a nominal, possibly nonoptimal, trajectory. This allows the coefficients of a linear or quadratic expansion of the cost function to be computed in reverse time along the trajectory: these coefficients may then be used to yield a new improved trajectory (i.e., the algorithms are of the "successive sweep" type). A class of nonlinear control problems, linear in the control variables, is studied using differential dynamic programming. It is shown that for the free-end-point problem, the first partial derivatives of the optimal cost function are continuous throughout the state space, and the second partial derivatives experience jumps at switch points of the control function. A control problem that has an aualytic solution is used to illustrate these points. The fixed-end-point problem is converted into an equivalent free-end-point problem by adjoining the end-point constraints to the cost functional using Lagrange multipliers: a useful interpretation for Pontryagin's adjoint variables for this type of problem emerges from this treatment. The above results are used to devise new second- and first-order algorithms for determining the optimal bang-bang control by successively improving a nominal guessed control function. The usefulness of the proposed algorithms is illustrated by the computation of a number of control problem examples.     

Partially observed non-linear risk-sensitive optimal stopping control for non-linear discrete-time systems

In this paper we introduce and solve the partially observed optimal stopping non-linear risk-sensitive stochastic control problem for discrete-time non-linear systems. The presented results are closely related to previous results for finite horizon partially observed risk-sensitive stochastic control problem. An information state approach is used and a new (three-way) separation principle established that leads to a forward dynamic programming equation and a backward dynamic programming inequality equation (both infinite dimensional). A verification theorem is given that establishes the optimal control and optimal stopping time. The risk-neutral optimal stopping stochastic control problem is also discussed.     

Linear optimal stochastic control using instantaneous output feedback‡

The problem of determining the linear feedback control of the instantaneous system output which minimizes a quadratic performance measure for a linear system with state and control-dependent noise is solved in this paper. Both the finite and infinite terminal time versions of this problem are treated. For the latter case, a sufficient condition for the existence of an optimal control is obtained. For the finite terminal time problem, it is shown that a two-point boundary value problem must be solved to realize the optimal control. For the infinite terminal time case, two non-linear matrix equations must be solved to realize the optimal control. Some discussion on the numerical methods used by the author to solve these equations is included in the paper.     

Convex dynamic programming for hybrid systems

A classical linear programming approach to optimization of flow or transportation in a discrete graph is extended to hybrid systems. The problem is finite dimensional if the state space is discrete and finite, but becomes infinite dimensional for a continuous or hybrid state space. It is shown how strict lower bounds on the optimal loss function can be computed by gridding the continuous state space and restricting the linear program to a finite-dimensional subspace. Upper bounds can be obtained by evaluation of the corresponding control laws.     

Optimal stochastic control of linear systems with state- and control-dependent disturbances

A review of the solution of the linear regulator problem for linear systems with state- and control-dependent disturbances is presented. Both the finite and infinite terminal time cases are treated. The solution to the complete state feedback case is given in detail and that for the output feedback case is noted. The general conclusion is that control-dependent noise calls for conservative control (small gains) while state-dependent noise calls for vigorous control (large gains). Of course it is the degree of this behavior that is important and this is given explicitly by the algorithms in this paper.     

事件驱动的网络化系统最优控制

针对随机事件驱动的网络化控制系统, 研究其中的有限时域和无限时域内最优控制器的设计问题. 首先, 根据执行器介质访问机制将网络化控制系统建模为具有多个状态的马尔科夫跳变系统; 然后, 基于动态规划和马尔科夫跳变线性系统理论设计满足二次型性能指标的最优控制序列, 通过求解耦合黎卡提方程的镇定解, 给出最优控制律的计算方法, 使得网络化控制系统均方指数稳定; 最后, 通过仿真实验表明了所提出方法的有效性.

     

Monotone Smoothing Splines using General Linear Systems

In this paper, a method is proposed to solve the problem of monotone smoothing splines using general linear systems. This problem, also called monotone control theoretic splines, has been solved only when the curve generator is modeled by the second‐order integrator, but not for other cases. The difficulty in the problem is that the monotonicity constraint should be satisfied over an interval which has the cardinality of the continuum. To solve this problem, we first formulate the problem as a semi‐infinite quadratic programming problem, and then we adopt a discretization technique to obtain a finite‐dimensional quadratic programming problem. It is shown that the solution of the finite‐dimensional problem always satisfies the infinite‐dimensional monotonicity constraint. It is also proved that the approximated solution converges to the exact solution as the discretization grid‐size tends to zero. An example is presented to show the effectiveness of the proposed method.     

Maximal versus strong solution to algebraic Riccati equations arising in infinite Markov jump linear systems

We deal with a perturbed algebraic Riccati equation in an infinite dimensional Banach space which appears, for instance, in the optimal control problem for infinite Markov jump linear systems (from now on iMJLS). Infinite or finite here has to do with the state space of the Markov chain being infinite countable or finite (see, e.g., [M.D. Fragoso, J. Baczynski, Optimal control for continuous time LQ—problems with infinite Markov jump parameters, SIAM J. Control Optim. 40(1) (2001) 270–297]). By using a certain concept of stochastic stability (a sort of L₂-stability), we have proved in [J. Baczynski, M.D. Fragoso, Maximal solution to algebraic Riccati equations linked to infinite Markov jump linear systems, Internal Report LNCC, no. 6, 2006] existence (and uniqueness) of maximal solution for this class of equations. As it is noticed in this paper, unlike the finite case (including the linear case), we cannot guarantee anymore that maximal solution is a strong solution in this setting. Via a discussion on the main mathematical hindrance behind this issue, we devise some mild conditions for this implication to hold. Specifically, our main result here is that, under stochastic stability, along with a condition related with convergence in the infinite dimensional scenario, and another one related to spectrum—weaker than spectral continuity—we ensure the maximal solution to be also a strong solution. These conditions hold trivially in the finite case, allowing us to recover the result of strong solution of [C.E. de Souza, M.D. Fragoso, On the existence of maximal solution for generalized algebraic Riccati equations arising in stochastic control, Systems Control Lett. 14 (1990) 233–239] set for MJLS. The issue of whether the convergence condition is restrictive or not is brought to light and, together with some counterexamples, unveil further differences between the finite and the infinite countable case.     

线性时序逻辑约束下的滚动时域控制路径规划

针对有限确定性系统中的路径规划问题,本文提出了一种线性时序逻辑约束下的在线实时求解滚动时域控制的新方法。该方法将滚动时域控制方法和满足线性时序逻辑公式的策略相结合,控制目标是在满足高级别任务规范的同时,使收集的累积回报值最大化。其中,在有限时域内的每个时间步长上局部优化回报值,并应用当前时刻计算获得的最优控制序列。通过执行适当的约束,保证控制器产生的无限轨迹满足期望的时序逻辑公式。而且,由于地势影响因子的引入,所建议的方案更接近于真实情况。仿真实验结果验证了文中提出方法的可行性和有效性。     

On the linear regulator with cross terms in the performance index

The linear regulator theory which is perhaps the most used result of modern control theory is known to apply even when the performance index contains cross terms in control and state. Bryson and Ho (1960) present this result in terms of an equivalent problem. Conditions which guarantee the existence of a solution to the optimal gain equation for the finite and infinite time eases are well known for the equivalent problem. This note presents these conditions in terms of the original problem.