Nonlinear stochastic receding horizon control: stability,robustness and Monte Carlo methods for control approximation

This work considers the stability of nonlinear stochastic receding horizon control when the optimal controller is only computed approximately. A number of general classes of controller approximation error are analysed including deterministic and probabilistic errors and even controller sample and hold errors. In each case, it is shown that the controller approximation errors do not accumulate (even over an infinite time frame) and the process converges exponentially fast to a small neighbourhood of the origin. In addition to this analysis, an approximation method for receding horizon optimal control is proposed based on Monte Carlo simulation. This method is derived via the Feynman–Kac formula which gives a stochastic interpretation for the solution of a Hamilton–Jacobi–Bellman equation associated with the true optimal controller. It is shown, and it is a prime motivation for this study, that this particular controller approximation method practically stabilises the underlying nonlinear process.     

Quantum quasi-Markov processes in eventum mechanics dynamics, observation, filtering and control

Quantum mechanical systems exhibit an inherently probabilistic behavior upon measurement which excludes in principle the singular case of direct observability. The theory of quantum stochastic time continuous measurements and quantum filtering was earlier developed by the author on the basis of non-Markov conditionally-independent increment models for quantum noise and quantum nondemolition observability. Here this theory is generalized to the case of demolition indirect measurements of quantum unstable systems satisfying the microcausality principle. The exposition of the theory is given in the most general algebraic setting unifying quantum and classical theories as particular cases. The reduced quantum feedback-controlled dynamics is described equivalently by linear quasi-Markov and nonlinear conditionally-Markov stochastic master equations. Using this scheme for diffusive and counting measurements to describe the stochastic evolution of the open quantum system under the continuous indirect observation and working in parallel with classical indeterministic control theory, we derive the Bellman equations for optimal feedback control of the a posteriori stochastic quantum states conditioned upon these measurements. The resulting Bellman equation for the diffusive observation is then applied to the explicitly solvable quantum linear-quadratic-Gaussian problem which emphasizes many similarities with the corresponding classical control problem.     

Nonlinear–nonquadratic optimal and inverse optimal control for stochastic dynamical systems

In this paper, we develop a unified framework to address the problem of optimal nonlinear analysis and feedback control for nonlinear stochastic dynamical systems. Specifically, we provide a simplified and tutorial framework for stochastic optimal control and focus on connections between stochastic Lyapunov theory and stochastic Hamilton–Jacobi–Bellman theory. In particular, we show that asymptotic stability in probability of the closed‐loop nonlinear system is guaranteed by means of a Lyapunov function that can clearly be seen to be the solution to the steady‐state form of the stochastic Hamilton–Jacobi–Bellman equation and, hence, guaranteeing both stochastic stability and optimality. In addition, we develop optimal feedback controllers for affine nonlinear systems using an inverse optimality framework tailored to the stochastic stabilization problem. These results are then used to provide extensions of the nonlinear feedback controllers obtained in the literature that minimize general polynomial and multilinear performance criteria. Copyright © 2017 John Wiley & Sons, Ltd.     

The optimal control related to Riemannian manifolds and the viscosity solutions to Hamilton–Jacobi–Bellman equations

In this paper we study the optimal stochastic control problem for stochastic differential equations on Riemannian manifolds. The cost functional is specified by controlled backward stochastic differential equations in Euclidean space. Under some suitable assumptions, we conclude that the value function is the unique viscosity solution to the associated Hamilton–Jacobi–Bellman equation which is a fully nonlinear parabolic partial differential equation on Riemannian manifolds.     

Iterative Path Integral Approach to Nonlinear Stochastic Optimal Control Under Compound Poisson Noise

Nonlinear stochastic optimal control theory has played an important role in many fields. In this theory, uncertainties of dynamics have usually been represented by Brownian motion, which is Gaussian white noise. However, there are many stochastic phenomena whose probability density has a long tail, which suggests the necessity to study the effect of non‐Gaussianity. This paper employs Lévy processes, which cause outliers with a significantly higher probability than Brownian motion, to describe such uncertainties. In general, the optimal control law is obtained by solving the Hamilton–Jacobi–Bellman equation. This paper shows that the path‐integral approach combined with the policy iteration method is efficiently applicable to solve the Hamilton–Jacobi–Bellman equation in the Lévy problem setting. Finally, numerical simulations illustrate the usefulness of this method.     

带Markov跳的离散时间随机控制系统的最大值原理

本文研究一类同时含有Markov跳过程和乘性噪声的离散时间非线性随机系统的最优控制问题, 给出并证明了相应的最大值原理. 首先, 利用条件期望的平滑性, 通过引入具有适应解的倒向随机差分方程, 给出了带有线性差分方程约束的线性泛函的表示形式, 并利用Riesz定理证明其唯一性. 其次, 对带Markov跳的非线性随机控制系统, 利用针状变分法, 对状态方程进行一阶变分, 获得其变分所满足的线性差分方程. 然后, 在引入Hamilton函数的基础上, 通过一对由倒向随机差分方程刻画的伴随方程, 给出并证明了带有Markov跳的离散时间非线性随机最优控制问题的最大值原理, 并给出该最优控制问题的一个充分条件和相应的Hamilton-Jacobi-Bellman方程. 最后, 通过 一个实际例子说明了所提理论的实用性和可行性.     

Numerical solution of continuous-time mean–variance portfolio selection with nonlinear constraints

An investment problem is considered with dynamic mean–variance (M–V) portfolio criterion under discontinuous prices described by jump-diffusion processes. Some investment strategies are restricted in the study. This M–V portfolio with restrictions can lead to a stochastic optimal control model. The corresponding stochastic Hamilton–Jacobi–Bellman equation of the problem with linear and nonlinear constraints is derived. Numerical algorithms are presented for finding the optimal solution in this article. Finally, a computational experiment is to illustrate the proposed methods by comparing with M–V portfolio problem which does not have any constraints.     

一类随机系统完全统计特征控制

非线性随机系统完全统计特征控制优于低阶矩控制,但往往因为算法的复杂性难以实际应用.本文针对受高斯白噪声激励的标量非线性随机系统,针对状态响应提出了一种完全统计特征控制方法.首先将刻画完全统计特征的概率密度函数表示成指数函数,利用FPK(Fokker-Planck-Kolmogorov)方程求出概率密度函数的各阶导数,进而建立指数函数Taylor展开的系数与待求反馈控制增益间的关系.然后,依据控制目标给出了求解反馈增益的优化问题.针对目标概率密度函数的不同情况,分别给出了跟踪控制策略:对于指数函数Taylor展开为有限项形式的情况,能够直接得到控制增益并完全跟踪目标概率密度函数;其他情况下,也能够达到较好的控制效果.仿真验证了本文方法的有效性.     

具有条件马尔科夫结构的离散随机系统最优控制

基于Bellman随机非线性动态规划法, 提出了具有条件马尔科夫跳变结构的离散随机系统的最优控制方法, 应用随机变结构系统的性质对最优控制算法进行了简化处理, 并将后验概率密度函数用条件高斯函数来逼近, 针对一类具有条件马尔科夫跳变结构的线性离散随机系统, 给出了其逼近最优控制算法.     

First-passage failure of MDOF nonlinear oscillator

First-passage failure of multiple-degree-of-freedom nonlinear oscillators with lightly nonlinear dampings and strongly nonlinear stiffness subject to additive and/or parametric Gaussian white noise excitations is studied. First, by using the stochastic averaging method based on the generalized harmonic functions, the averaged It stochastic differential equation for the amplitudes of the nonlinear oscillators can be derived. Then the associated backward Kolmogorov equation of the conditional reliability func...     

A stability criterion for a class of discrete nonlinear systems with random parameters

In this paper, discrete nonlinear models with random parameters are studied. A new frame to classify and analyze discrete stochastic nonlinear systems has been developed from deterministic nonlinear systems to stochastic nonlinear systems. This frame is broad and includes a large class of stochastic nonlinear systems. A stability criterion developed for this frame is a non-Lyapunov method of stability analysis and is easily applied. In addition, this derived sufficient condition of stability is obtained without the assumption of stationarity for the random noise as frequently assumed in the literature.     

Suboptimal control of linear discrete stochastic systems with linear input constraints

Optimal control of linear discrete stochastic systems with linear input constraints is considered. A lower bound on the achievable minimum of the loss function is given. A suboptimal control strategy is derived by using a truncated Taylor series to approximate the expected future loss in the Bellman equation. The performance of the suboptimal controller is studied using Monte Carlo simulation and the obtained loss is compared with the lower bound.     

Aircraft Trajectory Optimization for Collision Avoidance Using Stochastic Optimal Control

Optimizing aircraft collision avoidance and performing trajectory optimization are the key problems in an air transportation system. This paper is focused on solving these problems by using a stochastic optimal control approach. The major contribution of this paper is a proposed stochastic optimal control algorithm to dynamically adjust and optimize aircraft trajectory. In addition, this algorithm accounts for random wind dynamics and convective weather areas with changing size. Although the system is modeled by a stochastic differential equation, the optimal feedback control for this equation can be computed as a solution of a partial differential equation, namely, an elliptic Hamilton‐Jacobi‐Bellman equation. In this paper, we solve this equation numerically using a Markov Chain approximation approach, where a comparison of three different iterative methods and two different optimization search methods are presented. Simulations show that the proposed method provides better performance in reducing conflict probability in the system and that it is feasible for real applications.     

A Computational Method for Stochastic Optimal Control Problems in Financial Mathematics

Principle of optimality or dynamic programming leads to derivation of a partial differential equation (PDE) for solving optimal control problems, namely the Hamilton‐Jacobi‐Bellman (HJB) equation. In general, this equation cannot be solved analytically; thus many computing strategies have been developed for optimal control problems. Many problems in financial mathematics involve the solution of stochastic optimal control (SOC) problems. In this work, the variational iteration method (VIM) is applied for solving SOC problems. In fact, solutions for the value function and the corresponding optimal strategies are obtained numerically. We solve a stochastic linear regulator problem to investigate the applicability and simplicity of the presented method and prove its convergence. In particular, for Merton's portfolio selection model as a problem of portfolio optimization, the proposed numerical method is applied for the first time and its usefulness is demonstrated. For the nonlinear case, we investigate its convergence using Banach's fixed point theorem. The numerical results confirm the simplicity and efficiency of our method.     

Suboptimal mean controllers for bounded and dynamic stochastic distributions

Based on the recently developed algorithms for the modelling and control of bounded dynamic stochastic systems (H. Wang, J. Zhang, Bounded stochastic distributions control for pseudo ARMAX stochastic systems, IEEE Transactions on Automatic control, 486–490), this paper presents the design of a subotpimal nonlinear mean controller for bounded dynamic stochastic systems with guaranteed stability. The B-spline functional expansion based square root model is used to represent the output probability density function of the system. This is then followed by the design of a mean controller of the output distribution of the system using nonlinear output tracking concept. A nonlinear quadratic optimization is performed using the well known Hamilton–Jacobi–Bellman equation. This leads to a controller which consists of a static unit, a state feedback part and an equivalent output feedback loop. In order to achieve high precision for the output tracking, the output feedback gain is determined by a learning process, where the Lyapunov stability analysis is performed to show the asymptotic stability of the closed loop system under some conditions. A simulation example is included to demonstrate the use of the algorithm and encouraging results have been obtained.     

On the application of minimum principle for solving partially observable risk-sensitive control problems

This paper is concerned with the application of a minimum principle derived for general nonlinear partially observable exponential-of-integral control problems, to solve linear-exponential-quadratic-Gaussian problems. This minimum principle is the stochastic analog of Pontryagin's minimum principle for deterministic systems. It consists of an information state equation, an adjoint process governed by a stochastic partial differential equation with terminal condition, and a Hamiltonian functional. Two methods are employed to obtain the optimal control law. The first method appeals to the well-known approach of completing the squares, by first determining the optimal control law that minimizes the Hamiltonian functional. The second method provides significant insight into relations with the HamiltoniJacobi approach associated with completely observable exponential-of-integral control problems. These methods of solution are particularly attractive because they do not assume a certainty equivalence principle, hence they can be used to solve nonlinear problems as well.     

Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning

In this paper, a new formulation for the optimal tracking control problem (OTCP) of continuous-time nonlinear systems is presented. This formulation extends the integral reinforcement learning (IRL) technique, a method for solving optimal regulation problems, to learn the solution to the OTCP. Unlike existing solutions to the OTCP, the proposed method does not need to have or to identify knowledge of the system drift dynamics, and it also takes into account the input constraints a priori. An augmented system composed of the error system dynamics and the command generator dynamics is used to introduce a new nonquadratic discounted performance function for the OTCP. This encodes the input constrains into the optimization problem. A tracking Hamilton–Jacobi–Bellman (HJB) equation associated with this nonquadratic performance function is derived which gives the optimal control solution. An online IRL algorithm is presented to learn the solution to the tracking HJB equation without knowing the system drift dynamics. Convergence to a near-optimal control solution and stability of the whole system are shown under a persistence of excitation condition. Simulation examples are provided to show the effectiveness of the proposed method.     

具有加性和乘性噪声的线性离散时间随机系统的无模型最优跟踪控制

本文研究一类同时受加性和乘性噪声影响的离散时间随机系统的最优跟踪控制问题.通过构造由原始系统和参考轨迹组成的增广系统,将随机线性二次跟踪控制(SLQT)的成本函数转化为与增广状态相关的二次型函数,由此推导出用于求解SLQT的贝尔曼方程和增广随机代数黎卡提方程(SARE),而后进一步针对系统和参考轨迹动力学信息完全未知的情形,提出一种Q-学习算法来在线求解增广SARE,证明了该算法的收敛性,并采用批处理最小二乘法(BLS)解决该在线无模型控制算法的实现问题.通过对单相电压源UPS逆变器的仿真,验证了所提出控制方案的有效性.     

Probabilistic solutions of some multi-degree-of-freedom nonlinear stochastic dynamical systems excited by filtered Gaussian white noise

In this paper, the state-space-split method is extended for the dimension reduction of some high-dimensional Fokker–Planck–Kolmogorov equations or the nonlinear stochastic dynamical systems in high dimensions subject to external excitation which is the filtered Gaussian white noise governed by the second order stochastic differential equation. The selection of sub state variables and then the dimension-reduction procedure for a class of nonlinear stochastic dynamical systems is given when the external excitation is the filtered Gaussian white noise. The stretched Euler–Bernoulli beam with hinge support at two ends, point-spring supports, and excited by uniformly distributed load being filtered Gaussian white noise governed by the second-order stochastic differential equation is analyzed and numerical results are presented. The results obtained with the presented procedure are compared with those obtained with the Monte Carlo simulation and equivalent linearization method to show the effectiveness and advantage of the state-space-split method and exponential polynomial closure method in analyzing the stationary probabilistic solutions of the multi-degree-of-freedom nonlinear stochastic dynamical systems excited by filtered Gaussian white noise.     

Adaptive critic methods for stochastic systems with input-dependent noise

In this paper, a novel dual heuristic programming (DHP) adaptive-critic-based cautious controller is proposed. The proposed controller avoids the pre-identification training phase of the forward model by taking into consideration model uncertainty when calculating the control law. It is suitable for linear and nonlinear, deterministic and stochastic control systems that are characterized by functional uncertainty. Convergence of the proposed DHP adaptive critic method to the correct value of the cost function is proven by evaluating analytically the correct value of the cost function, which satisfies the Bellman equation, and compares it to that calculated by the proposed method in a simple linear quadratic example. Moreover, the performance of the proposed cautious controller is demonstrated on linear one-dimensional and multidimensional examples.