Degenerate problems of optimal control. III

Consideration was given to the methods of studying the degenerate problems of optimal control from the standpoint of the Krotov general sufficient optimality conditions: the method of multiple maxima as a special way of defining the Krotov function, generalized Bellman equations and their counterparts for estimation of the attainability sets of the systems with unlimited controls. Indicated was a class of problems economic dynamics and stable development for which the method of multiple maxima is most efficient. Solution with allowance for the innovation processes for one of the new problems of this kind was given by way of example. The possibilities of using these methods to seek approximate globally optimal solutions were discussed.     

On the optimal design of elastic beams

In this paper we investigate the optimal dynamics of simply supported nonlinearly elastic beams with rectangular cross-sections. We consider the elastic beam under the assumption of time-dependent intensive transverse loading. The state of the beam is described by a system of partial differential equations of the fourth order. We deal with the problem of choosing the optimal shape for the beam. The optimal shape is determined in such a way that the deflection of the nonlinearly elastic beam for any given time is minimal. The problem of choosing the optimal shape is formulated as an optimal control problem. To solve the obtained problem effectively, we use the optimality principle of Bellman (Bellman and Dreyfus 1962; Bryson and Ho 1975) and the penalty function method (Polyak 1987). We present a constructive algorithm for the optimal design of nonlinearly elastic beams. Some simple examples of the implementation of the proposed numerical algorithm are given.     

具有条件马尔科夫结构的离散随机系统最优控制

基于Bellman随机非线性动态规划法, 提出了具有条件马尔科夫跳变结构的离散随机系统的最优控制方法, 应用随机变结构系统的性质对最优控制算法进行了简化处理, 并将后验概率密度函数用条件高斯函数来逼近, 针对一类具有条件马尔科夫跳变结构的线性离散随机系统, 给出了其逼近最优控制算法.     

Analytical design of optimal controllers in the class of logic-dynamic (hybrid) systems

The classical problem of analytical design of the optimal controllers which was stated and solved by A.M. Letov for continuous linear systems was considered for the class of logicdynamic (hybrid) systems. Equations to determine the optimal feedback control were derived on basis of the sufficient optimality conditions. In distinction to classical case, the optimal positional control is realized by the piecewise-linear controller with the piecewise-quadratic Bellman function. In contrast to the continuous, discrete or discrete-continuous systems, the optimal processes of logic-dynamic systems, can have multiple switchings in the logic block, thus hindering the optimal control design. Therefore it was suggested to consider a simplified problem where the logic-dynamic system is replaced by a discrete-continuous system admitting instantaneous multiple switchings of the discrete part. The resulting control for the logic-dynamic system becomes suboptimal. Application of the optimality and suboptimality conditions was demonstrated by way of several examples.     

Two-loop reinforcement learning algorithm for finite-horizon optimal control of continuous-time affine nonlinear systems

This article proposes three novel time-varying policy iteration algorithms for finite-horizon optimal control problem of continuous-time affine nonlinear systems. We first propose a model-based time-varying policy iteration algorithm. The method considers time-varying solutions to the Hamiltonian–Jacobi–Bellman equation for finite-horizon optimal control. Based on this algorithm, value function approximation is applied to the Bellman equation by establishing neural networks with time-varying weights. A novel update law for time-varying weights is put forward based on the idea of iterative learning control, which obtains optimal solutions more efficiently compared to previous works. Considering that system models may be unknown in real applications, we propose a partially model-free time-varying policy iteration algorithm that applies integral reinforcement learning to acquiring the time-varying value function. Moreover, analysis of convergence, stability, and optimality is provided for every algorithm. Finally, simulations for different cases are given to verify the convenience and effectiveness of the proposed algorithms.     

Transfer of a Dynamic Object onto the Surface of an Ellipsoid

Using Pontryagin’s maximum principle, the problem of the quickest transfer of a multidimensional object onto the surface of an ellipsoid is reduced to solving a scalar algebraic equation. The concentration of the endpoints of optimal trajectories in the vicinity of the points forming the boundary in the case of a degenerate ellipsoid is demonstrated. An example in which the optimal control has a jump and the Bellman function has a discontinuity when the magnitude of the initial velocity vector undergoes a small change is constructed. It is also shown that the jump in the optimal control can occur without the discontinuity of the Bellman function.     

The finite-horizon optimal control for a class of time-delay affine nonlinear system

In this paper, a new iteration algorithm is proposed to solve the finite-horizon optimal control problem for a class of time-delay affine nonlinear systems with known system dynamic. First, we prove that the algorithm is convergent as the iteration step increases. Then, a theorem is presented to demonstrate that the limit of the iteration performance index function satisfies discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation, and the finite-horizon iteration algorithm is presented with satisfactory accuracy error. At last, two neural networks are used to approximate the iteration performance index function and the corresponding control policy. In simulation part, an example is given to demonstrate the effectiveness of the proposed iteration algorithm.     

Robust nonlinear feedback control for uncertain linear systems with nonquadratic performance criteria

In this paper we develop a unified framework to address the problem of optimal nonlinear robust control for linear uncertain systems. Specifically, we transform a given robust control problem into an optimal control problem by properly modifying the cost functional to account for the system uncertainty. As a consequence, the resulting solution to the modified optimal control problem guarantees robust stability and performance for a class of nonlinear uncertain systems. The overall framework generalizes the classical Hamilton–Jacobi–Bellman conditions to address the design of robust nonlinear optimal controllers for uncertain linear systems. © 1998 Elsevier Science B.V.     

Robust nonlinear feedback control for uncertain linear systems with nonquadratic performance criteria

In this paper we develop a unified framework to address the problem of optimal nonlinear robust control for linear uncertain systems. Specifically, we transform a given robust control problem into an optimal control problem by properly modifying the cost functional to account for the system uncertainty. As a consequence, the resulting solution to the modified optimal control problem guarantees robust stability and performance for a class of nonlinear uncertain systems. The overall framework generalizes the classical Hamilton–Jacobi–Bellman conditions to address the design of robust nonlinear optimal controllers for uncertain linear systems. © 1998 Elsevier Science B.V.     

Optimized tracking control using reinforcement learning strategy for a class of nonlinear systems

This paper is to develop a simplified optimized tracking control using reinforcement learning (RL) strategy for a class of nonlinear systems. Since the nonlinear control gain function is considered in the system modeling, it is challenging to extend the existing RL-based optimal methods to the tracking control. The main reasons are that these methods' algorithm are very complex; meanwhile, they also require to meet some strict conditions. Different with these exiting RL-based optimal methods that derive the actor and critic training laws from the square of Bellman residual error, which is a complex function consisting of multiple nonlinear terms, the proposed optimized scheme derives the two RL training laws from negative gradient of a simple positive function, so that the algorithm can be significantly simplified. Moreover, the actor and critic in RL are constructed by employing neural network (NN) to approximate the solution of Hamilton–Jacobi–Bellman (HJB) equation. Finally, the feasibility of the proposed method is demonstrated in accordance with both Lyapunov stability theory and simulation example.     

On the value function for nonautonomous optimal control problems with infinite horizon

In this paper we consider nonautonomous optimal control problems of infinite horizon type, whose control actions are given by L¹-functions. We verify that the value function is locally Lipschitz. The equivalence between dynamic programming inequalities and Hamilton–Jacobi–Bellman (HJB) inequalities for proximal sub (super) gradients is proven. Using this result we show that the value function is a Dini solution of the HJB equation. We obtain a verification result for the class of Dini sub-solutions of the HJB equation and also prove a minimax property of the value function with respect to the sets of Dini semi-solutions of the HJB equation. We introduce the concept of viscosity solutions of the HJB equation in infinite horizon and prove the equivalence between this and the concept of Dini solutions. In the Appendix we provide an existence theorem.     

Constructive methods of control optimization in nonlinear systems

The paper investigates methods of optimal program and point-to-point control in nonlinear differential systems. The proposed methods are based on the idea of discretizing the continuous-time problem. In case of program control, the method of projected Lagrangian is used, which involves solution of an auxiliary problem with linearized constraints by the reduced gradient method; in case of point-to-point control, Bellman’s optimality principle is employed for a “grid” over an approximating solvability tube of the system for a specified goal set with approximation of the point-to-point control by families of controlling function or parameters. An example is given, where the optimal point-to-point control is calculated for a model of maneuvering aircraft.     

Optimal generated numerical sequences in problems of placement of tracking segments in evaluating algorithms of operator activity, I

In designing algorithms of activity of an operator of an anthropocentric object, the problem of evaluating its load by tracking algorithms arises. The mathematical formulation of this problem is as follows: for a given positive numerical sequence with a finite number of terms and a given piecewise-linear function, estimating this sequence by the sum of estimates of its terms, find an optimal sequence among the set of sequences (generated sequences) obtained by adding at any single place or in several places any number of adjacent terms. For an arbitrary sequence (from a certain class) under an a priori known piecewise-linear function, a computer oriented technique of embedded sliding sections is found that makes it possible to construct a complete set of generated sequences. An algorithm for selecting optimal generated sequences in this set, recalling the algorithm of the R. Bellman dynamic programming, is proposed.     

Optimal control problems for stochastic delay evolution equations in Banach spaces

We consider the optimal control for a Banach space valued stochastic delay evolution equation. The existence and uniqueness of the mild solution for the associated Hamilton–Jacobi–Bellman equations are obtained by means of backward stochastic differential equations. An application to optimal control of stochastic delay partial differential equations is also given.     

Approximate optimal solution of the DTHJB equation for a class of nonlinear affine systems with unknown dead-zone constraints

In this paper, an optimal control scheme of a class of unknown discrete-time nonlinear systems with dead-zone control constraints is developed using adaptive dynamic programming (ADP). First, the discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation is derived. Then, an improved iterative ADP algorithm is constructed which can solve the DTHJB equation approximately. Combining with Riemann integral, detailed proofs of existence and uniqueness of the solution are also presented. It is emphasized that this algorithm allows the implementation of optimal control without knowing internal system dynamics. Moreover, the approach removes the requirements of precise parameters of the dead-zone. Finally, simulation studies are given to demonstrate the performance of the present approach using neural networks.     

Time-optimal crossing of a sphere in a viscous medium

Multidimensional controlled motions of a material point in a homogeneous viscous medium are considered. The problem of steering this point to a fixed sphere (from the outside or from the inside) for a minimum time by a force with a bounded absolute value is solved. For an arbitrary initial position and any initial velocity, optimal control both in the open-loop form of the program and in the feedback form, optimal time and Bellman function, as well as optimal phase trajectory are constructed in an explicit form using the Pontryagin’s maximum principle. The solution is studied analytically and numerically, and qualitative mechanical properties of optimal characteristics of motion are found (nonmonotonic dependence of optimal time on the value of initial vector of velocity, discontinuity of the Bellman function, and so on).     

Weakly monotone solutions of the Hamilton-Jacobi inequality and optimality conditions with positional controls

We obtain necessary global optimality conditions for classical optimal control problems based on positional controls. These controls are constructed with classical dynamical programming but with respect to upper (weakly monotone) solutions of the Hamilton-Jacobi equation instead of a Bellman function. We put special emphasis on the positional minimum condition in Pontryagin formalism that significantly strengthens the Maximum Principle for a wide class of problems and can be naturally combined with first order sufficient optimality conditions with linear Krotov’s function. We compare the positional minimum condition with the modified nonsmooth Ka?kosz-Lojasiewicz Maximum Principle. All results are illustrated with specific examples.     

Optimal control of ultradiffusion processes with application to mathematical finance

We introduce the optimal control problem associated with ultradiffusion processes as a stochastic differential equation constrained optimization of the expected system performance over the set of feasible trajectories. The associated Bellman function is characterized as the solution to a Hamilton–Jacobi equation evaluated along an optimal process. For an important class of ultradiffusion processes, we define the value function in terms of the time and the natural state variables. Approximation solvability is shown and an application to mathematical finance demonstrates the applicability of the paradigm. In particular, we utilize a method-of-lines finite element method to approximate the value function of a European style call option in a market subject to asset liquidity risk (including limit orders) and brokerage fees.     

Piecewise linear quadratic optimal control

The use of piecewise quadratic cost functions is extended from stability analysis of piecewise linear systems to performance analysis and optimal control. Lower bounds on the optimal control cost are obtained by semidefinite programming based on the Bellman inequality. This also gives an approximation to the optimal control law. An upper bound to the optimal cost is obtained by another convex optimization problem using the given control law. A compact matrix notation is introduced to support the calculations and it is proved that the framework of piecewise linear systems can be used to analyze smooth nonlinear dynamics with arbitrary accuracy     

Asymptotic optimal control of uncertain nonlinear Euler–Lagrange systems

A sufficient condition to solve an optimal control problem is to solve the Hamilton–Jacobi–Bellman (HJB) equation. However, finding a value function that satisfies the HJB equation for a nonlinear system is challenging. For an optimal control problem when a cost function is provided a priori, previous efforts have utilized feedback linearization methods which assume exact model knowledge, or have developed neural network (NN) approximations of the HJB value function. The result in this paper uses the implicit learning capabilities of the RISE control structure to learn the dynamics asymptotically. Specifically, a Lyapunov stability analysis is performed to show that the RISE feedback term asymptotically identifies the unknown dynamics, yielding semi-global asymptotic tracking. In addition, it is shown that the system converges to a state space system that has a quadratic performance index which has been optimized by an additional control element. An extension is included to illustrate how a NN can be combined with the previous results. Experimental results are given to demonstrate the proposed controllers.