Two-loop reinforcement learning algorithm for finite-horizon optimal control of continuous-time affine nonlinear systems

This article proposes three novel time-varying policy iteration algorithms for finite-horizon optimal control problem of continuous-time affine nonlinear systems. We first propose a model-based time-varying policy iteration algorithm. The method considers time-varying solutions to the Hamiltonian–Jacobi–Bellman equation for finite-horizon optimal control. Based on this algorithm, value function approximation is applied to the Bellman equation by establishing neural networks with time-varying weights. A novel update law for time-varying weights is put forward based on the idea of iterative learning control, which obtains optimal solutions more efficiently compared to previous works. Considering that system models may be unknown in real applications, we propose a partially model-free time-varying policy iteration algorithm that applies integral reinforcement learning to acquiring the time-varying value function. Moreover, analysis of convergence, stability, and optimality is provided for every algorithm. Finally, simulations for different cases are given to verify the convenience and effectiveness of the proposed algorithms.     

Optimal shapes of parametrically excited beams

Straight elastically supported beams of variable width under the action of a periodic axial force are considered. Two shape optimization problems for reducing parametric resonance zones are studied. In the first problem, the minimal (critical) amplitude of the excitation force is maximized. In the second problem, the range of resonant frequencies is minimized for a given parametric resonance zone and a fixed amplitude of excitation. These two optimization problems are proved to be equivalent in the case of small external damping and small excitation force amplitude. It is shown that optimal designs have strong universal character, i.e. they depend only on the natural modes involved in the parametric resonance and boundary conditions. An efficient numerical method of optimization is developed. Optimal beam shapes are found for different boundary conditions and resonant modes. Experiments for uniform and optimal simply supported elastic beams have been conducted demonstrating a very good agreement with theoretical prediction.     

Transverse impact problems by higher order beam finite element

A higher order beam finite element is derived and shown to be very efficient in solving the transient dynamic problem. The finite element thus developed is then applied to impact problems concerning the response of beams to striking elastic masses. Due to the local indentation the motion of the beam is nonlinearly coupled with the motion of the mass. An efficient iterative procedure is proposed in this paper to integrate the time variable. Both elastic impact and impact with permanent indentations are considered. The finite element solutions are found in good agreement with some existing solutions.     

Optimality of affine control system of several species in competition on a sequential batch reactor

In this paper, we analyse the optimality of affine control system of several species in competition for a single substrate on a sequential batch reactor, with the objective being to reach a given (low) level of the substrate. We allow controls to be bounded measurable functions of time plus possible impulses. A suitable modification of the dynamics leads to a slightly different optimal control problem, without impulsive controls, for which we apply different optimality conditions derived from Pontryagin principle and the Hamilton–Jacobi–Bellman equation. We thus characterise the singular trajectories of our problem as the extremal trajectories keeping the substrate at a constant level. We also establish conditions for which an immediate one impulse (IOI) strategy is optimal. Some numerical experiences are then included in order to illustrate our study and show that those conditions are also necessary to ensure the optimality of the IOI strategy.     

具有条件马尔科夫结构的离散随机系统最优控制

基于Bellman随机非线性动态规划法, 提出了具有条件马尔科夫跳变结构的离散随机系统的最优控制方法, 应用随机变结构系统的性质对最优控制算法进行了简化处理, 并将后验概率密度函数用条件高斯函数来逼近, 针对一类具有条件马尔科夫跳变结构的线性离散随机系统, 给出了其逼近最优控制算法.     

基于每阶段平均费用最优的激励学习算法

文中利用求解最优费用函数的方法给出了一种新的激励学习算法，即基于每阶段平均费用最优的激励学习算法。这种学习算法是求解信息不完全Markov决策问题的一种有效激励学习方法，它从求解分阶段最优平均费用函数的方法出发，分析了最优解的存在性，分阶段最优平均费用函数与初始状态的关系以及与之相关的Bellman方程。这种方法的建立，可以使得动态规划（DP）算法中的许多结论直接应用到激励学习的研究中来。     

智能隔震结构序列最优控制Simulink仿真分析

介绍了现有经典控制算法特点及Simulink S-函数实现主动控制仿真的方法。针对地震引发结构受迫振动最优控制问题,借鉴离散系统最优控制Bellman最优法则,提出了基于全状态反馈的主动序列最优控制律。与现有经典控制算法相比,序列最优控制算法增益矩阵具有时变的特点。通过对五层基础隔震结构地震反应控制仿真分析表明,经典最优控制算法和序列最优控制算法都能有效地减少地震反应,但序列最优控制算法具有较强的抗噪声能力且具有良好的鲁棒性能。     

Distributed event-triggered optimal bipartite consensus control for multiagent systems with input delay via reinforcement learning method

In this article, an optimal bipartite consensus control (OBCC) scheme is proposed for heterogeneous multiagent systems (MASs) with input delay by reinforcement learning (RL) algorithm. A directed signed graph is established to construct MASs with competitive and cooperative relationships, and model reduction method is developed to tackle input delay problem. Then, based on the Hamilton–Jacobi–Bellman (HJB) equation, policy iteration method is utilized to design the bipartite consensus controller, which consists of value function and optimal controller. Further, a distributed event-triggered function is proposed to increase control efficiency, which only requires information from its own agent and neighboring agents. Based on the input-to-state stability (ISS) function and Lyapunov function, sufficient conditions for the stability of MASs can be derived. Apart from that, RL algorithm is employed to solve the event-triggered OBCC problem in MASs, where critic neural networks (NNs) and actor NNs estimate value function and control policy, respectively. Finally, simulation results are given to validate the feasibility and efficiency of the proposed algorithm.     

一类DEDS最优调度问题的解法

本文提出了带存储器生产线的一类新的最优调度问题,给出了最优调度目标函数的 具体形式,指出它不是凸函数;在一个变量时给出了最优调度的公式解,在多个变量时得到了 一个迭代寻优的算法.     

Optimal control of large space structures governed by a coupled system of ordinary and partial differential equations

In this paper we consider the problem of optimal regulation of large space structures in the presence of flexible appendages. For simplicity of presentation, we consider a spacecraft consisting of a rigid bus and a flexible beam. The complete dynamics of the system is given by a coupled set of ordinary and partial differential equations. We show that the solution of this hybrid system is defined in a product space of appropriate finite- and infinite-dimensional spaces. We develop necessary conditions for determining the control torque and forces for optimal regulation of attitude maneuvers of the satellite along with simultaneous suppression of elastic vibrations of the flexible beam.     

Finite topology variations in optimal design of structures

The method of optimal design of structures by finite topology modification is presented in the paper. This approach is similar to growth models of biological structures, but in the present case, topology modification is described by the finite variation of a topological parameter. The conditions for introducing topology modification and the method for determining finite values of topological parameters characterizing the modified structure are specified. The present approach is applied to the optimal design of truss, beam, and frame structures. For trusses, the heuristic algorithm of bar exchange is proposed for minimizing the global compliance subject to a material volume constraint and it is extended to volume minimization with stress and buckling constraints. The optimal design problem for beam and frame structures with elastic or rigid supports, aimed at minimizing the structure cost for a specified global compliance, is also considered.     

The finite-horizon optimal control for a class of time-delay affine nonlinear system

In this paper, a new iteration algorithm is proposed to solve the finite-horizon optimal control problem for a class of time-delay affine nonlinear systems with known system dynamic. First, we prove that the algorithm is convergent as the iteration step increases. Then, a theorem is presented to demonstrate that the limit of the iteration performance index function satisfies discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation, and the finite-horizon iteration algorithm is presented with satisfactory accuracy error. At last, two neural networks are used to approximate the iteration performance index function and the corresponding control policy. In simulation part, an example is given to demonstrate the effectiveness of the proposed iteration algorithm.     

Robust shape control of beams with load uncertainties by optimally placed piezo actuators

The shape of a laminated beam is controlled by an optimally placed piezo actuator so as to minimize its maximum deflection. The locations and magnitudes of the external loads are not known a priori and belong to a specified load uncertainty domain. The optimal actuator location is obtained for any load combinations and locations which are determined so as to produce the maximum deflection corresponding to the worst case of loading. In this sense, loading uncertainties lead to an anti-optimization problem which is coupled to the optimization problem via the design parameter and loading. The resulting coupled problems are solved simultaneously to compute the optimal actuator location and the least favourable loading condition. Multiple load cases are also considered. Numerical results are given to assess the effect of load uncertainty and actuator length on the actuator location and the design efficiency which is defined with respect to the corresponding uncontrolled beam.     

水上行走机器人支撑腿的设计与布局

研究了水上行走机器人支撑腿形状设计与布局问题.首先将支撑腿视为欧拉-伯努利弹性曲梁,通过对其柔性变形的几何分析与应力应变分析,提出了-种支撑腿形状优化设计方法,方法的目标是获得最大支撑力.对方法的正确性和有效性进行了仿真和实验验证.其次通过分析支撑腿间距对支撑力的影响,以及支撑腿布局与机器人抗翻转能力的关系,提出了支撑...     

An HMM approach for optimal investment of an insurer

In this paper, we introduce a Hidden Markov Model (HMM) for studying an optimal investment problem of an insurer when model uncertainty is present. More specifically, the financial price and insurance risk processes are modulated by a continuous‐time, finite‐state, hidden Markov chain. The states of the chain represent different modes of the model. The HMM approach is viewed as a ‘dynamic’ version of the Bayesian approach to model uncertainty. The optimal investment problem is formulated as a stochastic optimal control problem with partial observations. The innovations approach in the filtering theory is then used to transform the problem into one with complete observations. New robust filters of the chain and estimates of key parameters are derived. We discuss the optimal investment problem using the Hamilton–Jacobi–Bellman (HJB) dynamic programming approach and derive a closed‐form solution in the case of an exponential utility and zero interest rate. Copyright © 2011 John Wiley & Sons, Ltd.     

Table design in dynamic programming

Dynamic Programming solves combinatorial optimization problems by recursive decomposition and tabulation of intermediate results. The first step in the design of a dynamic programming algorithm is to decide on the set of tables that will hold optimal solutions to subproblems. This step predetermines the shape of the dynamic programming recurrences as well as the asymptotic efficiency of the algorithm in time and space. We study dynamic programming in a formal framework where design of tables and problem decomposition can be done independently. Our main result shows that choosing a good table design for a given decomposition is an NP-complete problem. A heuristic or approximate approach is therefore needed to automate good table design. We report on a strategy that combines user annotation and a brute force algorithm, which is shown to perform well in a large application.     

Optimal control of ultradiffusion processes with application to mathematical finance

We introduce the optimal control problem associated with ultradiffusion processes as a stochastic differential equation constrained optimization of the expected system performance over the set of feasible trajectories. The associated Bellman function is characterized as the solution to a Hamilton–Jacobi equation evaluated along an optimal process. For an important class of ultradiffusion processes, we define the value function in terms of the time and the natural state variables. Approximation solvability is shown and an application to mathematical finance demonstrates the applicability of the paradigm. In particular, we utilize a method-of-lines finite element method to approximate the value function of a European style call option in a market subject to asset liquidity risk (including limit orders) and brokerage fees.     

Optimal control problems for stochastic delay evolution equations in Banach spaces

We consider the optimal control for a Banach space valued stochastic delay evolution equation. The existence and uniqueness of the mild solution for the associated Hamilton–Jacobi–Bellman equations are obtained by means of backward stochastic differential equations. An application to optimal control of stochastic delay partial differential equations is also given.     

Bilateral Estimation of the Bellman Function in the Problems of Optimal Stochastic Control of Discrete Systems by the Probabilistic Performance Criterion

Consideration was given to the optimal control of discrete stochastic systems by the probabilistic quality criterion. The new characteristics of the Bellman equation for this class of problems were examined, and the two-sided estimate of the Bellman function was determined. The problem of optimal control of the security portfolio with one riskless and a given number of risk assets was considered by way of example. The class of strategies featuring asymptotic optimality was established using the two-sided estimate of the Bellman function.     

Optimal design of rigid-plastic beams subjected to dynamical loading

In the present paper the optimization problem of dynamically loaded simply supported beams is handled. The concept of a rigid-plastic body is used. The shape of the beam is sought, for which the integral residual deflection for a given time-instant and load is minimal. Two numerical methods for solving the problem are proposed. The numerical results are compared with those obtained by Lepik (1982).