Necessary and sufficient conditions of risk‐sensitive optimal control and differential games for stochastic differential delayed equations

In this paper, we consider risk‐sensitive optimal control and differential games for stochastic differential delayed equations driven by Brownian motion. The problems are related to robust stochastic optimization with delay due to the inherent feature of the risk‐sensitive objective functional. For both problems, by using the logarithmic transformation of the associated risk‐neutral problem, the necessary and sufficient conditions for the risk‐sensitive maximum principle are obtained. We show that these conditions are characterized in terms of the variational inequality and the coupled anticipated backward stochastic differential equations (ABSDEs). The coupled ABSDEs consist of the first‐order adjoint equation and an additional scalar ABSDE, where the latter is induced due to the nonsmooth nonlinear transformation of the adjoint process of the associated risk‐neutral problem. For applications, we consider the risk‐sensitive linear‐quadratic control and game problems with delay, and the optimal consumption and production game, for which we obtain explicit optimal solutions.     

Data-driven policy iteration algorithm for continuous-time stochastic linear-quadratic optimal control problems

This paper studies a continuous-time stochastic linear-quadratic (SLQ) optimal control problem on infinite-horizon. Combining the Kronecker product theory with an existing policy iteration algorithm, a data-driven policy iteration algorithm is proposed to solve the problem. In contrast to most existing methods that need all information of system coefficients, the proposed algorithm eliminates the requirement of three system matrices by utilizing data of a stochastic system. More specifically, this algorithm uses the collected data to iteratively approximate the optimal control and a solution of the stochastic algebraic Riccati equation (SARE) corresponding to the SLQ optimal control problem. The convergence analysis of the obtained algorithm is given rigorously, and a simulation example is provided to illustrate the effectiveness and applicability of the algorithm.     

不确定拟哈密顿系统的随机最优控制

本文提出了不确定拟哈密顿系统、基于随机平均法、随机极大值原理和随机微分对策理论的一种随机极大极小最优控制策略.首先,运用拟哈密顿系统的随机平均法,将系统状态从速度和位移的快变量形式转化为能量的慢变量形式,得到部分平均的It随机微分方程;其次,给定控制性能指标,对于不确定拟哈密顿系统的随机最优控制,根据随机微分对策理论,将其转化为一个极小极大控制问题;再根据随机极大值原理,建立关于系统与伴随过程的前向-后向随机微分方程,随机最优控制表达为哈密顿控制函数的极大极小条件,由此得到最坏情形下的扰动参数与极大极小最优控制;然后,将最坏扰动参数与最优控制代入部分平均的It随机微分方程并完成平均,求解与完全平均的It随机微分方程相应的Fokker-Planck-Kolmogorov(FPK)方程,可得受控系统的响应量并计算控制效果;最后,将上述不确定拟哈密顿系统的随机最优控制策略应用于一个两自由度非线性系统,通过数值结果说明该随机极大极小控制策略的控制效果.     

基于混沌多项式的指令鲁棒优化及在飞行控制中的应用

本文提出一种新的方法对随机系统进行运动预测和控制指令设计, 该方法可以充分利用已知信息设计控 制指令以提高闭环随机系统的鲁棒性. 首先采用混沌多项式对随机信息进行数学表述, 并利用Galerkin投影法将随 机变量的混沌多项式引入常微分方程中. 然后, 将随机变量的均值和方差考虑至优化问题的成本函数中, 并利用伪 谱法对控制指令进行鲁棒优化. 最后, 将该方法应用于飞行器的动力学预测以及控制指令设计. 仿真结果表明, 该 方法能够预测飞行器飞行过程中不确定性的演化, 其精度与蒙特卡罗方法相当, 并且计算效率更高. 此外, 获得的 控制指令对存在不确定参数或初始条件的随机系统具有强鲁棒性.     

Optimal bang-bang control of linear stochastic systems with a small noise parameter

This study considers the problem of determining optimal feedback control laws for linear stochastic systems with amplitude-constrained control inputs. Two basic performance indices are considered, average time and average integral quadratic form. The optimization interval is random and defined as the first time a trajectory reaches the terminal regionR. The plant is modeled as a stochastic differential equation with an additive Wiener noise disturbance. The variance parameter of the Wiener noise process is assumed to be suitably small. A singular perturbation technique is presented for the solution of the stochastic optimization equations (second-order partial differential equation). A method for generating switching curves for the resulting optimal bang-bang control system is then developed. The results are applied to various problems associated with a second-order purely inertial system with additive noise at the control input. This problem is typical of satellite attitude control problems.     

Verification Theorem Of Stochastic Optimal Control With Mixed Delay And Applications To Finance

This paper focuses on a general model of a controlled stochastic differential equation with mixed delay in the state variable. Based on the Itô formula, stochastic analysis, convex analysis, and inequality technique, we obtain a semi‐coupled forward‐backward stochastic differential equation with mixed delay and mixed initial‐terminal conditions and prove that such forward‐backward system admits a unique adapted solution. The verification theorem for an optimal control of a system with mixed delay is established. The obtained results generalize and improve some recent results, and they are more easily verified and applied in practice. As an application, we conclude with finding explicitly the optimal consumption rate from the wealth process of a person given by a stochastic differential equation with mixed delay which fit into our general model.     

Stochastic Point‐to‐Point Iterative Learning Control Based on Stochastic Approximation

An iterative learning control algorithm with iteration decreasing gain is proposed for stochastic point‐to‐point tracking systems. The almost sure convergence and asymptotic properties of the proposed recursive algorithm are strictly proved. The selection of learning gain matrix is given. An illustrative example shows the effectiveness and asymptotic trajectory properties of the proposed approach.     

Model‐Reference Adaptive Moment Control of Uncertain Nonlinear Stochastic Systems

In this paper, a new model‐reference adaptive moment control method is proposed to control the first and second moments of an uncertain nonlinear system with additive external stochastic excitation. This method has established a closed‐loop control system that calculates an adaptive stochastic nonlinear input by introducing a Lyapunov function and adaptive update law. The proposed adaptive structure is innovative in trying to minimize two errors simultaneously: the moments tracking error and the error between the nonlinear system output and reference model. Furthermore, the proposed method can control the expected and covariance matrices of the states without needing to solve the complicated Fokker‐Planck‐Kolmogorov differential equation or using the approximate methods. Simulation has been performed on two practical examples, which show a good performance for the designed controller.     

A Computational Method for Stochastic Optimal Control Problems in Financial Mathematics

Principle of optimality or dynamic programming leads to derivation of a partial differential equation (PDE) for solving optimal control problems, namely the Hamilton‐Jacobi‐Bellman (HJB) equation. In general, this equation cannot be solved analytically; thus many computing strategies have been developed for optimal control problems. Many problems in financial mathematics involve the solution of stochastic optimal control (SOC) problems. In this work, the variational iteration method (VIM) is applied for solving SOC problems. In fact, solutions for the value function and the corresponding optimal strategies are obtained numerically. We solve a stochastic linear regulator problem to investigate the applicability and simplicity of the presented method and prove its convergence. In particular, for Merton's portfolio selection model as a problem of portfolio optimization, the proposed numerical method is applied for the first time and its usefulness is demonstrated. For the nonlinear case, we investigate its convergence using Banach's fixed point theorem. The numerical results confirm the simplicity and efficiency of our method.     

New optimization method to solve motion planning of dynamic systems: application on mechanical manipulators

In this paper, a new method is proposed to solve a nonlinear optimal control problem and determine the Dynamic Load-Carrying Capacity (DLCC) of fixed and mobile manipulators in point-to-point motion. Solution methods for designing nonlinear optimal controller in closed loop form are usually based on indirect methods, but the proposed method is a combination of direct and indirect methods. The optimal control law with state feedback form, for nonlinear dynamic systems, is given by the solution to the nonlinear Hamilton–Jacobi–Bellman (HJB) equation. The Galerkin procedure and a nonlinear optimization algorithm are used to solve this equation numerically. Another innovation of this paper is optimal trajectory planning, which is done simultaneously with the controller design procedure. Finally, a new algorithm is developed to find DLCC of manipulators and the related optimal trajectory using proposed method. The validity of the method is demonstrated via simulation and experimental tests for a fixed manipulator and two-link wheeled mobile manipulator named Scout.     

具有加性和乘性噪声的线性离散时间随机系统的无模型最优跟踪控制

本文研究一类同时受加性和乘性噪声影响的离散时间随机系统的最优跟踪控制问题.通过构造由原始系统和参考轨迹组成的增广系统,将随机线性二次跟踪控制(SLQT)的成本函数转化为与增广状态相关的二次型函数,由此推导出用于求解SLQT的贝尔曼方程和增广随机代数黎卡提方程(SARE),而后进一步针对系统和参考轨迹动力学信息完全未知的情形,提出一种Q-学习算法来在线求解增广SARE,证明了该算法的收敛性,并采用批处理最小二乘法(BLS)解决该在线无模型控制算法的实现问题.通过对单相电压源UPS逆变器的仿真,验证了所提出控制方案的有效性.     

Probability‐weighted nonlinear stochastic optimal control strategy of quasi‐integrable Hamiltonian systems with uncertain parameters

The nonlinear stochastic optimal control problem of quasi‐integrable Hamiltonian systems with uncertain parameters is investigated. The uncertain parameters are described by using a random vector with λ probability density function. First, the partially averaged Itô stochastic differential equations are derived by using the stochastic averaging method for quasi‐integrable Hamiltonian systems. Then, the dynamical programming equation is established based on stochastic dynamical programming principle. By minimizing the dynamical programming equation with respect to control forces, the optimal control forces can be derived, which are functions of the uncertain parameters. The final optimal control forces are then determined by probability‐weighted average of the obtained control forces with the probability density of the uncertain parameters as weighting function. The mean control effectiveness and mean control efficiency are used to evaluate the proposed control strategy. The robustness of the proposed control is measured by using the ratios of the variation coefficients of mean control effectiveness and mean control efficiency to the variation coefficients of uncertain parameters. Finally, two examples are given to illustrate the proposed control strategy and its effectiveness and robustness. Copyright © 2014 John Wiley & Sons, Ltd.     

Singular optimal control for stochastic linear quadratic singular system using ant colony programming

In this article, singular optimal control for stochastic linear singular system with quadratic performance is obtained using ant colony programming (ACP). To obtain the optimal control, the solution of matrix Riccati differential equation is computed by solving differential algebraic equation using a novel and nontraditional ACP approach. The obtained solution in this method is equivalent or very close to the exact solution of the problem. Accuracy of the solution computed by the ACP approach to the problem is qualitatively better. The solution of this novel method is compared with the traditional Runge Kutta method. An illustrative numerical example is presented for the proposed method.     

利用改进的差分进化算法求解等圆Packing问题

等圆Packing问题属于强约束的复杂组合优化问题之一,针对其强约束特点及难点,通过改进传统的差分进化算法,提出一种等圆Packing问题的求解方法。该改进算法特点是将有效解空间加入差分进化的变异约束中,并采用随机排序机制改进差分进化的选择机制。通过多次实验,表明此算法在求解小规模等圆Packing问题上取得的效果与目前所能找到的最优值相差不到0.6%,从而验证了演化计算在求解等圆Packing问题的可行性;与此同时,演化算法具有很好的收敛性,因此在其他强约束的复杂优化问题上将有很好的应用。     

Intelligent trajectory tracking of an aircraft in the presence of internal and external disturbances

This research deals with developing an intelligent trajectory tracking control approach for an aircraft in the presence of internal and external disturbances. Internal disturbances including actuators faults, unmodeled dynamics, and model uncertainties as well as the external disturbances such as wind turbulence significantly affect the performance of the common trajectory tracking control approaches. There are several fault‐tolerant control approaches in the literature to overcome the effects of specific actuator or sensor faults during the flight. However, trajectory tracking control of an air vehicle in the presence of unexpected faults and simultaneous presence of wind turbulence is still a challenging problem. In this paper, an intelligent neural network‐based model predictive control structure is proposed, where the prediction model is updated in each iteration based on a novel proposed online sequential multimodel structure. A hybrid offline‐online learning algorithm is adopted in the introduced online sequential multimodel structure to identify the time‐varying dynamics of the system. The proposed control structure can satisfactorily deal with unexpected actuator faults and structural damages as well as unmodeled dynamics and wind turbulence. The stability of the closed‐loop system is proved under some realistic assumptions. The simulation results demonstrate the high capability of the proposed approach for trajectory tracking control of a conventional aircraft in the simultaneous presence of system faults and external disturbances.     

基于伪谱法的滑翔轨迹多级迭代优化策略

伪谱法可实时求解具有高度非线性动态特性的飞行器最优轨迹;以X-51A相似飞行器模型为研究对象,采用增量法与查表插值建立纵向气动力模型,伪谱法与序列二次规划算法求解滑翔轨迹最优控制问题;提出使用多级迭代优化策略,为序列二次规划算法求解伪谱法参数化得到的大规模非线性规划问题提供初值,弥补序列二次规划算法在求解大规模非线性规划问题过程中,出现的初值敏感、收敛速度减慢等问题。通过与传统方法求解出的状态量与控制量仿真飞行状态进行对比,证明了多级迭代优化策略的有效性和高效性,该策略在实际工程应用中取得了良好效果。     

An efficient method for stochastic optimal control with joint chance constraints for nonlinear systems

Stochastic model predictive control hinges on the online solution of a stochastic optimal control problem. This paper presents a computationally efficient solution method for stochastic optimal control for nonlinear systems subject to (time‐varying) stochastic disturbances and (time‐invariant) probabilistic model uncertainty in initial conditions and parameters. To this end, new methods are presented for joint propagation of time‐varying and time‐invariant probabilistic uncertainty and the nonconservative approximation of joint chance constraint (JCC) on the system state. The proposed uncertainty propagation method relies on generalized polynomial chaos and conditional probability rules to obtain tractable expressions for the state mean and covariance matrix. A moment‐based surrogate is presented for JCC approximation to circumvent construction of the full probability distribution of the state or the use of integer variables as required when using the sample average approximation. The proposed solution method for stochastic optimal control is illustrated on a nonlinear semibatch reactor case study in the presence of probabilistic model uncertainty and stochastic disturbances. It is shown that the proposed solution method is significantly superior to a standard random sampling method for stochastic optimal control in terms of computational requirements. Furthermore, the moment‐based surrogate for the JCC is shown to be substantially less conservative than the widely used distributionally robust Cantelli‐Chebyshev inequality for chance constraint approximation.     

多自由度拟不可积哈密顿系统的随机分数阶最优控制

推广了适用于分数阶系统控制的随机分数阶最优控制策略,提出了高斯白噪声激励下多自由度拟不可积哈密顿系统以响应最小化为目标的随机分数阶最优控制策略.首先,应用拟不可积哈密顿系统随机平均法,将受控系统简化为关于能量的部分平均伊藤方程.然后,将控制性能指标中关于控制力的部分表示为分数阶形式,结合随机动态规划原理,建立并求解部分平均系统的无界遍历控制的随机动态规划方程,获得了随机分数阶最优控制律.最后,采用一个算例验证了随机分数阶控制策略的控制效果和控制效率.研究表明,随机分数阶最优控制策略对传统的整数阶随机动力学系统同样适用,能比传统的整数阶控制策略取得更好的控制效果.另外,随着激励强度增加,整数阶控制策略的控制效率显著降低;而分数阶控制策略的控制效率虽比整数阶控制策略的控制效率略低,但随着激励强度的增加,分数阶控制策略的控制效率缓慢上升并趋于平稳,可以有效地缓解控制效率与控制效果之间的矛盾.     

Exact optimal solution for a class of dual control problems

This paper considers a discrete-time stochastic optimal control problem for which only measurement equation is partially observed with unknown constant parameters taking value in a finite set of stochastic systems. Because of the fact that the cost-to-go function at each stage contains variance and the non-separability of the variance is so complicated that the dynamic programming cannot be successfully applied, the optimal solution has not been found. In this paper, a new approach to the optimal solution is proposed by embedding the original non-separable problem into a separable auxiliary problem. The theoretical condition on which the optimal solution of the original problem can be attained from a set of solutions of the auxiliary problem is established. In addition, the optimality of the interchanging algorithm is proved and the analytical solution of the optimal control is also obtained. The performance of this controller is illustrated with a simple example.     

带马尔科夫跳和乘积噪声的随机系统的最优控制

讨论了N个选手随机系统的最优控制问题. 设计了无限时间的带有马尔科夫跳和乘积噪声的随机系统的Pareto最优控制器. 应用推广的Lyapunov方法和解随机Riccati代数方程得到了系统的Pareto最优解, 证明了最优控制器是稳定的反馈控制器, 以及对应于最优控制器的反馈增益中的随机Riccati代数方程的解是最小解.