Time-optimal crossing of a sphere in a viscous medium

Multidimensional controlled motions of a material point in a homogeneous viscous medium are considered. The problem of steering this point to a fixed sphere (from the outside or from the inside) for a minimum time by a force with a bounded absolute value is solved. For an arbitrary initial position and any initial velocity, optimal control both in the open-loop form of the program and in the feedback form, optimal time and Bellman function, as well as optimal phase trajectory are constructed in an explicit form using the Pontryagin’s maximum principle. The solution is studied analytically and numerically, and qualitative mechanical properties of optimal characteristics of motion are found (nonmonotonic dependence of optimal time on the value of initial vector of velocity, discontinuity of the Bellman function, and so on).     

Bilateral Estimation of the Bellman Function in the Problems of Optimal Stochastic Control of Discrete Systems by the Probabilistic Performance Criterion

Consideration was given to the optimal control of discrete stochastic systems by the probabilistic quality criterion. The new characteristics of the Bellman equation for this class of problems were examined, and the two-sided estimate of the Bellman function was determined. The problem of optimal control of the security portfolio with one riskless and a given number of risk assets was considered by way of example. The class of strategies featuring asymptotic optimality was established using the two-sided estimate of the Bellman function.     

Constructive methods of control optimization in nonlinear systems

The paper investigates methods of optimal program and point-to-point control in nonlinear differential systems. The proposed methods are based on the idea of discretizing the continuous-time problem. In case of program control, the method of projected Lagrangian is used, which involves solution of an auxiliary problem with linearized constraints by the reduced gradient method; in case of point-to-point control, Bellman’s optimality principle is employed for a “grid” over an approximating solvability tube of the system for a specified goal set with approximation of the point-to-point control by families of controlling function or parameters. An example is given, where the optimal point-to-point control is calculated for a model of maneuvering aircraft.     

具有条件马尔科夫结构的离散随机系统最优控制

基于Bellman随机非线性动态规划法, 提出了具有条件马尔科夫跳变结构的离散随机系统的最优控制方法, 应用随机变结构系统的性质对最优控制算法进行了简化处理, 并将后验概率密度函数用条件高斯函数来逼近, 针对一类具有条件马尔科夫跳变结构的线性离散随机系统, 给出了其逼近最优控制算法.     

Two-loop reinforcement learning algorithm for finite-horizon optimal control of continuous-time affine nonlinear systems

This article proposes three novel time-varying policy iteration algorithms for finite-horizon optimal control problem of continuous-time affine nonlinear systems. We first propose a model-based time-varying policy iteration algorithm. The method considers time-varying solutions to the Hamiltonian–Jacobi–Bellman equation for finite-horizon optimal control. Based on this algorithm, value function approximation is applied to the Bellman equation by establishing neural networks with time-varying weights. A novel update law for time-varying weights is put forward based on the idea of iterative learning control, which obtains optimal solutions more efficiently compared to previous works. Considering that system models may be unknown in real applications, we propose a partially model-free time-varying policy iteration algorithm that applies integral reinforcement learning to acquiring the time-varying value function. Moreover, analysis of convergence, stability, and optimality is provided for every algorithm. Finally, simulations for different cases are given to verify the convenience and effectiveness of the proposed algorithms.     

Sequential estimation of states for non-linear lumped parameter systems†

This paper considers the sequential estimation of plants described by non-linear differential equations. The statistical nature of the disturbance being unknown, the problem is treated as an optimal control problem with least-squares criterion. It is seen that the criterion function satisfies the Bellman equation at terminal time, which is considered as a running variable. The sequential estimator equations are directly obtained by using Pearson's approximation solution.     

On the optimal design of elastic beams

In this paper we investigate the optimal dynamics of simply supported nonlinearly elastic beams with rectangular cross-sections. We consider the elastic beam under the assumption of time-dependent intensive transverse loading. The state of the beam is described by a system of partial differential equations of the fourth order. We deal with the problem of choosing the optimal shape for the beam. The optimal shape is determined in such a way that the deflection of the nonlinearly elastic beam for any given time is minimal. The problem of choosing the optimal shape is formulated as an optimal control problem. To solve the obtained problem effectively, we use the optimality principle of Bellman (Bellman and Dreyfus 1962; Bryson and Ho 1975) and the penalty function method (Polyak 1987). We present a constructive algorithm for the optimal design of nonlinearly elastic beams. Some simple examples of the implementation of the proposed numerical algorithm are given.     

Optimized tracking control using reinforcement learning strategy for a class of nonlinear systems

This paper is to develop a simplified optimized tracking control using reinforcement learning (RL) strategy for a class of nonlinear systems. Since the nonlinear control gain function is considered in the system modeling, it is challenging to extend the existing RL-based optimal methods to the tracking control. The main reasons are that these methods' algorithm are very complex; meanwhile, they also require to meet some strict conditions. Different with these exiting RL-based optimal methods that derive the actor and critic training laws from the square of Bellman residual error, which is a complex function consisting of multiple nonlinear terms, the proposed optimized scheme derives the two RL training laws from negative gradient of a simple positive function, so that the algorithm can be significantly simplified. Moreover, the actor and critic in RL are constructed by employing neural network (NN) to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation. Finally, the feasibility of the proposed method is demonstrated in accordance with both Lyapunov stability theory and simulation example.     

Optimal control of ultradiffusion processes with application to mathematical finance

We introduce the optimal control problem associated with ultradiffusion processes as a stochastic differential equation constrained optimization of the expected system performance over the set of feasible trajectories. The associated Bellman function is characterized as the solution to a Hamilton–Jacobi equation evaluated along an optimal process. For an important class of ultradiffusion processes, we define the value function in terms of the time and the natural state variables. Approximation solvability is shown and an application to mathematical finance demonstrates the applicability of the paradigm. In particular, we utilize a method-of-lines finite element method to approximate the value function of a European style call option in a market subject to asset liquidity risk (including limit orders) and brokerage fees.     

Wiener-Poisson control problems†

A one-dimensional Wiener plus independent Poisson control process has an integrated, discounted non-quadratic cost function with asymmetric bounds on the non-anticipative control, assumed to be a function of the current state. A Bellman equation and maximum principle for partial differential-difference equations may be used to obtain the optimal closed-loop control if some assumptions on the asymptotic behaviour of certain partial differential-difference equations are met. The finite and infinite integral cases are treated separately.     

Some nonlinear optimal control problems with closed‐form solutions

Optimal controllers guarantee many desirable properties including stability and robustness of the closed‐loop system. Unfortunately, the design of optimal controllers is generally very difficult because it requires solving an associated Hamilton–Jacobi–Bellman equation. In this paper we develop a new approach that allows the formulation of some nonlinear optimal control problems whose solution can be stated explicitly as a state‐feedback controller. The approach is based on using Young's inequality to derive explicit conditions by which the solution of the associated Hamilton–Jacobi–Bellman equation is simplified. This allows us to formulate large families of nonlinear optimal control problems with closed‐form solutions. We demonstrate this by developing optimal controllers for a Lotka–Volterra system. Copyright © 2001 John Wiley & Sons, Ltd.     

Necessary and sufficient optimality conditions for discrete time-invariant automaton-type systems

The optimal control of deterministic discrete time-invariant automaton-type systems is considered. Changes in the system’s state are governed by a recurrence equation. The switching times and their order are not specified in advance. They are found by optimizing a functional that takes into account the cost of each switching. This problem is a generalization of the classical optimal control problem for discrete time-invariant systems. It is proved that, in the time-invariant case, switchings of the optimal trajectory (may be multiple instantaneous switchings) are possible only at the initial and (or) terminal points in time. This fact is used in the derivation of equations for finding the value (Hamilton–Jacobi–Bellman) function and its generators. The necessary and sufficient optimality conditions are proved. It is shown that the generators of the value function in linear–quadratic problems are quadratic, and the value function itself is piecewise quadratic. Algorithms for the synthesis of the optimal closed-loop control are developed. The application of the optimality conditions is demonstrated by examples.     

基于伪谱法的自由采样实时最优反馈控制及应用

利用高斯伪谱法收敛速率快、精度高的特点,基于通用伪谱优化软件包在线求解非线性系统的最优控制问题.将伪谱反馈控制理论与非线性最优控制理论结合起来,给出了一种自由采样实时最优反馈控制算法,该算法通过连续在线生成开环最优控制的方式提供闭环反馈.考虑计算误差、模型参数不确定性和干扰的作用,假定系统状态方程右侧的非线性向量函数关于状态、控制和系统参数是Lipschitz连续的,利用Bellman最优性原理对闭环控制系统的有界稳定性进行了分析和理论证明.最后,以高超声速再入飞行器为应用对象,研究了其再入制导问题,仿真结果验证了该算法的可行性和有效性.     

Weakly monotone solutions of the Hamilton-Jacobi inequality and optimality conditions with positional controls

We obtain necessary global optimality conditions for classical optimal control problems based on positional controls. These controls are constructed with classical dynamical programming but with respect to upper (weakly monotone) solutions of the Hamilton-Jacobi equation instead of a Bellman function. We put special emphasis on the positional minimum condition in Pontryagin formalism that significantly strengthens the Maximum Principle for a wide class of problems and can be naturally combined with first order sufficient optimality conditions with linear Krotov’s function. We compare the positional minimum condition with the modified nonsmooth Ka?kosz-Lojasiewicz Maximum Principle. All results are illustrated with specific examples.     

Asymptotic optimal control of uncertain nonlinear Euler–Lagrange systems

A sufficient condition to solve an optimal control problem is to solve the Hamilton–Jacobi–Bellman (HJB) equation. However, finding a value function that satisfies the HJB equation for a nonlinear system is challenging. For an optimal control problem when a cost function is provided a priori, previous efforts have utilized feedback linearization methods which assume exact model knowledge, or have developed neural network (NN) approximations of the HJB value function. The result in this paper uses the implicit learning capabilities of the RISE control structure to learn the dynamics asymptotically. Specifically, a Lyapunov stability analysis is performed to show that the RISE feedback term asymptotically identifies the unknown dynamics, yielding semi-global asymptotic tracking. In addition, it is shown that the system converges to a state space system that has a quadratic performance index which has been optimized by an additional control element. An extension is included to illustrate how a NN can be combined with the previous results. Experimental results are given to demonstrate the proposed controllers.     

Nonlinear–nonquadratic optimal and inverse optimal control for stochastic dynamical systems

In this paper, we develop a unified framework to address the problem of optimal nonlinear analysis and feedback control for nonlinear stochastic dynamical systems. Specifically, we provide a simplified and tutorial framework for stochastic optimal control and focus on connections between stochastic Lyapunov theory and stochastic Hamilton–Jacobi–Bellman theory. In particular, we show that asymptotic stability in probability of the closed‐loop nonlinear system is guaranteed by means of a Lyapunov function that can clearly be seen to be the solution to the steady‐state form of the stochastic Hamilton–Jacobi–Bellman equation and, hence, guaranteeing both stochastic stability and optimality. In addition, we develop optimal feedback controllers for affine nonlinear systems using an inverse optimality framework tailored to the stochastic stabilization problem. These results are then used to provide extensions of the nonlinear feedback controllers obtained in the literature that minimize general polynomial and multilinear performance criteria. Copyright © 2017 John Wiley & Sons, Ltd.     

Analysis of a high-order finite difference detector for discontinuities

High-order finite difference discontinuity detectors are essential for the location of discontinuities on discretized functions, especially in the application of high-order numerical methods for high-speed compressible flows for shock detection. The detectors are used mainly for switching between numerical schemes in regions of discontinuity to include artificial dissipation and avoid spurious oscillations. In this work a discontinuity detector is analysed by the construction of a piecewise polynomial function that incorporates jump discontinuities present on the function or its derivatives (up to third order) and the discussion on the selection of a cut-off value required by the detector. The detector function is also compared with other discontinuity detectors through numerical examples.     

A Computational Method for Stochastic Optimal Control Problems in Financial Mathematics

Principle of optimality or dynamic programming leads to derivation of a partial differential equation (PDE) for solving optimal control problems, namely the Hamilton‐Jacobi‐Bellman (HJB) equation. In general, this equation cannot be solved analytically; thus many computing strategies have been developed for optimal control problems. Many problems in financial mathematics involve the solution of stochastic optimal control (SOC) problems. In this work, the variational iteration method (VIM) is applied for solving SOC problems. In fact, solutions for the value function and the corresponding optimal strategies are obtained numerically. We solve a stochastic linear regulator problem to investigate the applicability and simplicity of the presented method and prove its convergence. In particular, for Merton's portfolio selection model as a problem of portfolio optimization, the proposed numerical method is applied for the first time and its usefulness is demonstrated. For the nonlinear case, we investigate its convergence using Banach's fixed point theorem. The numerical results confirm the simplicity and efficiency of our method.     

离散非线性零和博弈的事件驱动最优控制方案

在求解离散非线性零和博弈问题时,为了在有效降低网络通讯和控制器执行次数的同时保证良好的控制效果,本文提出了一种基于事件驱动机制的最优控制方案.首先,设计了一个采用新型事件驱动阈值的事件驱动条件,并根据贝尔曼最优性原理获得了最优控制对的表达式.为了求解该表达式中的最优值函数,提出了一种单网络值迭代算法.利用一个神经网络构建评价网.设计了新的评价网权值更新规则.通过在评价网、控制策略及扰动策略之间不断迭代,最终获得零和博弈问题的最优值函数和最优控制对.然后,利用Lyapunov稳定性理论证明了闭环系统的稳定性.最后,将该事件驱动最优控制方案应用到了两个仿真例子中,验证了所提方法的有效性.     

Approximate dynamic programming via iterated Bellman inequalities

In this paper, we introduce new methods for finding functions that lower bound the value function of a stochastic control problem, using an iterated form of the Bellman inequality. Our method is based on solving linear or semidefinite programs, and produces both a bound on the optimal objective, as well as a suboptimal policy that appears to works very well. These results extend and improve bounds obtained in a previous paper using a single Bellman inequality condition. We describe the methods in a general setting and show how they can be applied in specific cases including the finite state case, constrained linear quadratic control, switched affine control, and multi‐period portfolio investment. Copyright © 2014 John Wiley & Sons, Ltd.