On a discrete-time stochastic learning control algorithm

In an earlier paper by the author (2001), the learning gain for a D-type learning algorithm, is derived based on minimizing the trace of the input error covariance matrix for linear time-varying systems. It is shown that, if the product of the input/output coupling matrices is full-column rank, then the input error covariance matrix converges uniformly to zero in the presence of uncorrelated random disturbances, whereas, the state error covariance matrix converges uniformly to zero in the presence of measurement noise. However, in general, the proposed algorithm requires knowledge of the state matrix. In this note, it is shown that equivalent results can be achieved without the knowledge of the state matrix. Furthermore, the convergence rate of the input error covariance matrix is shown to be inversely proportional to the number of learning iterations     

An integrated optimal control algorithm for discrete-time nonlinear stochastic system

Consider a discrete-time nonlinear system with random disturbances appearing in the real plant and the output channel where the randomly perturbed output is measurable. An iterative procedure based on the linear quadratic Gaussian optimal control model is developed for solving the optimal control of this stochastic system. The optimal state estimate provided by Kalman filtering theory and the optimal control law obtained from the linear quadratic regulator problem are then integrated into the dynamic integrated system optimisation and parameter estimation algorithm. The iterative solutions of the optimal control problem for the model obtained converge to the solution of the original optimal control problem of the discrete-time nonlinear system, despite model-reality differences, when the convergence is achieved. An illustrative example is solved using the method proposed. The results obtained show the effectiveness of the algorithm proposed.     

A discrete-time iterative learning algorithm for linear time-varying systems

In this paper, an iterative learning algorithm (ILC) is presented for a MIMO linear time-varying system. We consider the convergence of the algorithm. A necessary and sufficient condition for the existence of convergent algorithm is stated. Then, we prove that the same condition is sufficient for the robustness of the proposed learning algorithm against state disturbance, output measurement noise, and reinitialization error. Finally, a simulation example is given to illustrate the results.     

A general Martingale approach to discrete-time stochastic control and estimation

A general method of constructing system models for the solution of discrete-time stochastic control and estimation problems is presented. The method involves the application of modern martingale theory and entails the judicious choice of certain sigma-algebras and martingales. General estimation equations are derived for observations taking values in a countable space, and previously obtained estimation equations are exhibited as special cases. Finally, an example of the application of these methods to a stochastic control problem is analyzed.     

Minimax control for discrete-time time-varying stochastic systems

This paper gives a self-contained presentation of minimax control for discrete-time time-varying stochastic systems under finite- and infinite-horizon expected total cost performance criteria. Suitable conditions for the existence of minimax strategies are proposed. Also, we prove that the values of the finite-horizon problem converge to the values of the infinite-horizon problems. Moreover, for finite-horizon problems an algorithm of calculation of minimax strategies is developed and tested by using time-varying stochastic systems.     

Adaptive dual control of discrete-time distributed-parameter stochastic systems

The problem of adaptive dual control of discrete-time distributed-parameter stochastic systems is examined. It is shown that there exists an important difference between feedback and closed-loop policies of control for this type of system as for the lumped parameter case. This difference is based on the adaptivity feature of the control. Namely, when the control policy affects both the state and its uncertainty (dual effect) it possesses the so-called feature of active adaptivity and can only be a characteristic of a closed-loop policy, whereas a feedback policy can only be passively adaptive. These results can be used to develop a control algorithm for non-linear problems for which the realization of optimal control laws involves control strategies with both learning and control features.     

Deterministic and stochastic control of discrete-time bilinear systems

Optimal control of a class of time invariant single-input, discrete bilinear systems is investigated in this paper. Both deterministic and stochastic problems are considered.

In the deterministic problem, for the initial state in a certain set ∑₀, the solution is the same as the solution to the associated linear system. The optimal path may be a regular path or a singular path.

The stochastic control problem is considered with perfect state observation, and additive and multiplicative noise in the state equation. It is demonstrated that the presence of noise simplifies the analysis compared to that in the determinstic case.     

A new algorithm for suboptimal stochastic control

An apparently new stochastic control algorithm, calledM-measurement-optimal feedback control, is described for discrete-time systems. This scheme incorporatesMfuture measurements into the control computations: whenMis zero,it reduces to the well-known open-loop-optimal feedback control; whenMis the actual number of measurements remaining in the problem, it becomes the truly optimal stochastic control. This new algorithm may also be used to simplify computations when the plant is nonlinear, when the controls are constrained, or when the cost is nonquadratic. Simulation results are presented which show the superiority of the new algorithm over the open-loop-optimal feedback control.     

A quasi-newton differential dynamic programming algorithm for discrete-time optimal control

We develop a quasi-Newton differential dynamic programming algorithm (QDDP) for discrete-time optimal control problems. In the spirit of dynamic programming, the quasi-Newton approximations are performed in a stagewise manner. We establish the global convergence of the method and also show a superlinear convergence rate. Among other advantages of the QDDP method, second derivatives need not be calculated. In theory, the computational effort of each recursion grows proportionally to the number of stages N, whereas with conventional quasi-Newton techniques which do not take advantage of the optimal control problem structure, the growth is as N². Computational results are also reported.     

Optimal timing control of discrete-time linear switched stochastic systems

Optimal switch-time control is the study that investigates how best to switch between different modes. In this paper, we investigate the optimal switch-time control problem for discrete-time linear switched stochastic systems. In particular, under the assumption that the sequence of active subsystems is pre-specified, we focus on the problem where the objective is to minimize a cost functional defined on the states and the switching times are the only control variables. For systems with one switching time, using calculus of variations, we firstly derive the difference formulae of the cost functional with respect to the switching time, which can be directly used to find the optimal switching instant. Then, a method is presented to deal with the problem with multiple switching times case. Finally, the viability of the proposed method is illustrated through two numerical examples.     

Polynomial optimization of stochastic discrete-time control for unstable plants

The matrix polynomial method of optimization of linear multivariable discrete-time control for unstable plants with stationary stochastic inputs is derived in a constructive manner. New sufficient conditions are obtained and Ku?era's second diophantine equation is found directly from the requirement that the system be stable. Although the second equation must be satisfied, it is proved that it need not be used in the optimization when a certain coprimeness exists.     

Networked iterative learning control design for discrete-time systems with stochastic communication delay in input and output channels

This paper develops two kinds of derivative-type networked iterative learning control (NILC) schemes for repetitive discrete-time systems with stochastic communication delay occurred in input and output channels and modelled as 0-1 Bernoulli-type stochastic variable. In the two schemes, the delayed signal of the current control input is replaced by the synchronous input utilised at the previous iteration, whilst for the delayed signal of the system output the one scheme substitutes it by the synchronous predetermined desired trajectory and the other takes it by the synchronous output at the previous operation, respectively. In virtue of the mathematical expectation, the tracking performance is analysed which exhibits that for both the linear time-invariant and nonlinear affine systems the two kinds of NILCs are convergent under the assumptions that the probabilities of communication delays are adequately constrained and the product of the input–output coupling matrices is full-column rank. Last, two illustrative examples are presented to demonstrate the effectiveness and validity of the proposed NILC schemes.     

带Markov跳的离散时间随机控制系统的最大值原理

本文研究一类同时含有Markov跳过程和乘性噪声的离散时间非线性随机系统的最优控制问题, 给出并证明了相应的最大值原理. 首先, 利用条件期望的平滑性, 通过引入具有适应解的倒向随机差分方程, 给出了带有线性差分方程约束的线性泛函的表示形式, 并利用Riesz定理证明其唯一性. 其次, 对带Markov跳的非线性随机控制系统, 利用针状变分法, 对状态方程进行一阶变分, 获得其变分所满足的线性差分方程. 然后, 在引入Hamilton函数的基础上, 通过一对由倒向随机差分方程刻画的伴随方程, 给出并证明了带有Markov跳的离散时间非线性随机最优控制问题的最大值原理, 并给出该最优控制问题的一个充分条件和相应的Hamilton-Jacobi-Bellman方程. 最后, 通过 一个实际例子说明了所提理论的实用性和可行性.     

基于随机牛顿算法的离散系统自适应参数估计

针对一类离散系统,提出一种基于随机牛顿算法的自适应参数估计新框架,相较于已有的参数估计算法,所提出方法仅要求系统满足有限激励条件,而非传统的持续激励条件.所提出算法的核心思想在于通过对原始代价函数的修正,在使用当前时刻误差信息的基础上融入历史误差信息,进而通过对历史信息和历史激励的复用使得持续激励条件转化为有限激励条件;然后,为了解决传统算法收敛速度慢的问题并避免潜在的病态问题,采用随机牛顿算法推导出参数自适应律,并引入含有历史信息的海森矩阵作为时变学习增益,保证参数估计误差指数收敛;最后,基于李雅普诺夫稳定性理论给出不同激励条件下所提出算法的收敛性结论和证明,并通过对比仿真验证所提出算法的有效性和优越性.     

一种基于齐次系统理论的二阶离散超螺旋控制算法

相比于传统滑模控制算法,超螺旋控制算法可以对系统的干扰进行精确估计并补偿,因此可以显著提高闭环系统的抗干扰能力.然而,对于采样控制系统,由于采样频率的限制,离散超螺旋控制算法在性能方面受到限制.本文基于齐次系统理论提出了一种改进的离散超螺旋控制算法.通过引入一个可自由调节的分数幂参数,基于齐次系统理论,证明了所提出的改进控制算法可以使得闭环系统具有更高的控制精度.仿真实例验证了理论的正确性.     

Asynchronous control of discrete-time stochastic bilinear systems with Markovian switchings

This paper is concerned with the problem of asynchronous control for a class of discrete-time Markov systems with multiplicative stochastic white noises. Based on a stability analysis scheme developed from mode-dependent Lyapunov function method, we first derive testable conditions in linear matrix inequality (LMI) setting to ensure the robust stability of the closed-loop system. We then recast the proposed stability conditions into equivalent forms that are later utilised to design a multi-mode asynchronous state-feedback controller (ASFC) that makes the closed-loop system stable. An extension to the case of deficient mode information (i.e. transition rates of the system and the controller are not fully accessible) is also presented. Finally, a model of networked control with DC devices is given to demonstrate the efficacy of the proposed design scheme.     

Infinite horizon indefinite stochastic linear quadratic control for discrete-time systems

This paper discusses discrete-time stochastic linear quadratic (LQ) problem in the infinite horizon with state and control dependent noise, where the weighting matrices in the cost function are assumed to be indefinite. The problem gives rise to a generalized algebraic Riccati equation (GARE) that involves equality and inequality constraints. The well-posedness of the indefinite LQ problem is shown to be equivalent to the feasibility of a linear matrix inequality (LMI). Moreover, the existence of a stabilizing solution to the GARE is equivalent to the attainability of the LQ problem. All the optimal controls are obtained in terms of the solution to the GARE. Finally, we give an LMI -based approach to solve the GARE via a semidefinite programming.     

Nonstationary discrete-time deterministic and stochastic control systems with infinite horizon

This article is about nonstationary nonlinear discrete-time deterministic and stochastic control systems with Borel state and control spaces, possibly noncompact control constraint sets, and unbounded costs. The control problem is to minimise an infinite-horizon total cost performance index. Using dynamic programming arguments we show that, under suitable assumptions, the optimal cost functions satisfy optimality equations, which in turn give a procedure to find optimal control policies.     

H∞-type control for discrete-time stochastic systems

In this paper we consider discrete-time, linear stochastic systems with random state and input matrices which are subjected to stochastic disturbances and controlled by dynamic output feedback. The aim is to develop an H^∞-type theory for such systems. For this class of systems a stochastic bounded real lemma is derived which provides the basis for a linear matrix inequality (LMI) approach similar to, but more general than the one presented in Reference 1 for stochastic differential systems. Necessary and sufficient conditions are derived for the existence of a stabilizing controller which reduces the norm of the closed-loop perturbation operator to a level below a given threshold γ. These conditions take the form of coupled nonlinear matrix inequalities. In the absence of the stochastic terms they get reduced to the linear matrix inequalities of deterministic H^∞-theory for discrete time systems. Copyright © 1999 John Wiley & Sons, Ltd.