Adaptive dynamic programming and optimal control of nonlinear nonaffine systems

In this paper, a novel optimal control design scheme is proposed for continuous-time nonaffine nonlinear dynamic systems with unknown dynamics by adaptive dynamic programming (ADP). The proposed methodology iteratively updates the control policy online by using the state and input information without identifying the system dynamics. An ADP algorithm is developed, and can be applied to a general class of nonlinear control design problems. The convergence analysis for the designed control scheme is presented, along with rigorous stability analysis for the closed-loop system. The effectiveness of this new algorithm is illustrated by two simulation examples.     

飞机纵向飞行轨迹的优化与实现*

本文研究了飞机纵向飞行轨迹的优化技术,选用质点运动能量状态方程为飞机运动模型,直接操作成本为优化指标函数;飞机纵向飞行剖面被假定分为三个飞行段:爬升、巡航和下降;将能量状态引入指标函数,使其成为哈密顿函数中的独立变量,则由动态变分法和极小值原理得到对各飞行轨迹段的优化算法;尔后用Fibonacci单参数搜索法来予以实现。文中给出了部分仿真结果。     

Adaptive configuration control of multiple UAVs

Several missions including surveillance, exploration, search-and-track, and lifting of heavy loads are best accomplished by multiple unmanned aerial vehicles (UAVs). Another important advantage to utilizing multiple vehicles is a reduction in the risk to successful completion of a mission due to the loss of a single vehicle. This increased robustness can lead to a commensurate decrease in vehicle specifications and cost, further improving the argument for swarm operations. This paper describes the development of an adaptive configuration controller for multiple vehicles executing a cooperative task in the presence of parametric uncertainty. A novel adaptive outer-loop controller that uses both local and global information is presented.     

Self-tuning control of a pH-neutralization process

A self-tuning regulator has been used in control of a pH-neutralization process. The process is difficult to control due to great variations in process sensitivity with pH and due to process nonlinearities. Experiments on a pilot plant demonstrate that a self-tuning pH-regulator including exponential forgetting and quadratic optimal control has satisfactory static and dynamic properties even though the process is nonlinear. It is also demonstrated that the self-tuning algorithm can adapt to changes in process conditions, and it may be a useful alternative to traditional PI-controllers.     

On type 1 systems and the clarke-gawthrop regulator

The Clarke-Gawthrop regulator is known to stabilize nonminimum-phase plants if the poles are stable. It is shown here that it is also possible to include an integrator in the plant. A comparison of variance is made with the optimal LQ solution, for a simple nonminimum-phase system.     

Variance constrained self-tuning control

A self-tuning regulator for a variance constrained optimal control problem is given. The criterion for control is to minimize the stationary variance of the output. In the cases when the regulator which gives minimum variance requires too large control signals an inequality constraint on the input variance is introduced. In practice it is easier to select a constraint on the variance of the input than to choose the relative weights in a quadratic loss function. The self-tuning regulator applies the Robbins-Monro scheme to adjust the Lagrange multiplier of the variance constrained control problem. The behaviour of the algorithm is illustrated by a simulated example. The asymptotic behaviour of the regulator is studied using a set of associated ordinary differential equations.     

Adaptive steering of ships—A model reference approach

This paper describes the application of model reference adaptive control (MRAS) to automatic steering of ships. The main advantages in this case are the simplified controller adjustment which yields safer operation and the decreased fuel cost. After discussion of the mathematical models of process and disturbances, criteria for optimal steering are defined. Algorithms are given for direct adaptation of the controller gains, applicable after setpoint changes, as well as for identification and adaptive state estimation, to be used when the input is constant. Solutions for applying MRAS to a certain class of nonlinear systems are dealt with. Full-scale trials at sea and tests with a scale model in a towing tank are described. It is shown that the autopilot designed indeed has the desired properties. Fuel savings up to 5% in comparison to conventional PID control are demonstrated. These savings are mainly possible because of the adaptive state estimator.     

Optimality and convergence of adaptive optimal control by reinforcement synthesis

Adaptive Optimal Control (AOC) by reinforcement synthesis is proposed to facilitate the application of optimal control theory in feedback controls. Reinforcement synthesis uses the critic–actor architecture of reinforcement learning to carry out sequential optimization. Optimality conditions for AOC are formulated using the discrete minimum principle. A proof of the convergence conditions for the reinforcement synthesis algorithm is presented. As the final time extends to infinity, the reinforcement synthesis algorithm is equivalent to the Dual Heuristic dynamic Programming (DHP) algorithm, a version of approximate dynamic programming. Thus, formulating DHP with the AOC approach has rigorous proofs of optimality and convergence. The efficacy of AOC by reinforcement synthesis is demonstrated by solving a linear quadratic regulator problem.     

基于自适应动态规划的非线性鲁棒近似最优跟踪控制

为克服现有近似最优跟踪控制方法只能跟踪连续可微参考输入的局限,本文针对一类具有未知动态的连续时间非线性时不变仿射系统,提出了一种新的基于自适应动态规划的鲁棒近似最优跟踪控制方法.首先采用递归神经网络建立系统模型,然后建立评价神经网络对最优性能指标进行估计,从而得到最优性能指标偏导数的估计值,进而得到近似最优跟踪控制器,最后利用系统输出与参考输入之间的跟踪误差设计鲁棒项对神经网络建模误差进行补偿.分别针对两个非线性系统进行仿真实验,仿真结果表明了所提方法的有效性和优越性.     

State-of-the-art and prospects of adaptive systems

The theory and applications of adaptive control have developed very intensively. There are two reasons for this: variable parameters of controlled plants change within a broad range, on the one hand, and accuracy requirements to process equipment, aircraft systems etc. increase, on the other. Moreover, the wide use of digital computers permits now even complex adaptation algorithms to be easily realized. The paper attempts to review methods of design analysis and synthesis of adaptive systems (AS). Extreme systems, systems with passive adaptation and identification problems have not been considered. Since contributions of Soviet authors has been largely overlooked by the writers of earlier reviews (Åström, 1983; Fujii, 1981; and others) we have decided to focus our attention mainly on the results obtained in the U.S.S.R.     

基于强化学习的浓密机底流浓度在线控制算法

复杂过程工业控制一直是控制应用领域研究的前沿问题. 浓密机作为一种复杂大型工业设备广泛用于冶金、采矿等领域. 由于其在运行过程中具有多变量、非线性、高时滞等特点, 浓密机的底流浓度控制技术一直是学界、工业界的研究难点与热点. 本文提出了一种基于强化学习技术的浓密机在线控制算法. 该算法在传统启发式动态规划 (Heuristic dynamic programming, HDP)算法的基础上, 设计融合了评价网络与模型网络的双网结构, 并提出了基于短期经验回放的方法用于增强评价网络的训练准确性, 实现了对浓密机底流浓度的稳定控制, 并保持控制输入稳定在设定范围之内. 最后, 通过浓密机仿真实验的方式验证了算法的有效性, 实验结果表明本文提出的方法在时间消耗、控制精度上优于其他算法.     

Independent tracking and regulation adaptive control with forgetting factor

This paper considers a discrete-time adaptive control algorithm with a forgetting factor applicable to minimum phase plants. The tracking and regulation objectives are independently specified. The relevance of the eigenvalues of the gain matrix (F_k) used in the updating equation for the adaptive parameters ($\text{}\text{?}\text{gq(k)}$) is shown. It is proved that if the maximum eigenvalue of the inverse of the gain matrix F_k has an upper bound and a non-zero lower bound then the global convergence of the control algorithm is insured. The result of the design is a simple control scheme using a linear constant feedforward controller and a nonlinear feedback controller. The performance of the control structure in tracking and regulation are evaluated by simulations.     

基于自适应动态规划的导弹制导律研究综述

自适应动态规划（Adaptive dynamic programming,ADP）作为最优控制领域的近似优化方法,是求解复杂非线性系统最优控制问题的有力工具.近年来,已成为控制理论与计算智能领域的研究热点.本文着重介绍ADP算法的理论研究进展及其在航空航天领域的应用.分析了几种典型的制导律优化设计方法,以及ADP方法在导弹制导律设计中的应用现状和前景.     

Altitude transitions in energy climbs

The aircraft energy-climb trajectory for configurations with a sharp transonic drag rise is well known to possess two branches in the altitude/Mach-number plane. Transition in altitude between the two branches occurs instantaneously, a ‘corner’ in the minimum-time solution obtained with the energy-state model. If the initial and final values of altitude do not lie on the energy-climb trajectory, then additional jumps (crude approximations to dives and zooms) are required at the initial and terminal points. With a singular-perturbation approach, a ‘boundary-layer’ correction is obtained for each altitude jump, the transonic jump being a so-called ‘internal’ boundary layer, different in character from the initial and terminal layers. The determination of this internal boundary layer is examined and some computational results for an example presented.