Optimality and convergence of adaptive optimal control by reinforcement synthesis

Adaptive Optimal Control (AOC) by reinforcement synthesis is proposed to facilitate the application of optimal control theory in feedback controls. Reinforcement synthesis uses the critic–actor architecture of reinforcement learning to carry out sequential optimization. Optimality conditions for AOC are formulated using the discrete minimum principle. A proof of the convergence conditions for the reinforcement synthesis algorithm is presented. As the final time extends to infinity, the reinforcement synthesis algorithm is equivalent to the Dual Heuristic dynamic Programming (DHP) algorithm, a version of approximate dynamic programming. Thus, formulating DHP with the AOC approach has rigorous proofs of optimality and convergence. The efficacy of AOC by reinforcement synthesis is demonstrated by solving a linear quadratic regulator problem.     

Finite-time optimal tracking control for a class of nonlinear systems with prescribed performance

In this paper, a finite-time optimal tracking control scheme based on integral reinforcement learning is developed for partially unknown nonlinear systems. In order to realize the prescribed performance, the original system is transformed into an equivalent unconstrained system so as to a composite system is constructed. Subsequently, a modified nonlinear quadratic performance function containing the auxiliary tracking error is designed. Furthermore, the technique of experience replay is used to update the critic neural network, which eliminates the persistent of excitation condition in traditional optimal methods. By combining the prescribed performance control with the finite-time optimization control technique, the tracking error is driven to a desired performance in finite time. Consequently, it has been shown that all signals in the partially unknown nonlinear system are semiglobally practical finite-time stable by stability analysis. Finally, the provided comparative simulation results verify the effectiveness of the developed control scheme.     

MPC-based Co-design of Control and Routing for Wireless Sensor and Actuator Networks

A wireless sensor and actuator network (WSAN) is a class of networked control systems. In WSANs, sensors and actuators are located in a distributed way, and communicate to controllers through a wireless communication network such as a multi-hop network. In this paper, we propose a model predictive control (MPC) method for co-design of control and routing of WSANs. MPC is an optimal control strategy based on numerical optimization. The control input is calculated by solving the finite-time optimal control problem at each discrete time. In the proposed method, a WSAN is modeled by a switched linear system. In the finite-time optimal control problem, a control input and a mode corresponding to a communication path are optimized simultaneously. The proposed method is demonstrated by a numerical example.     

A distributed finite-time optimization algorithm for directed networks of continuous-time agents

This article proposes a new distributed finite-time optimization algorithm for agents under directed graphs. By employing the nonsmooth technique and graph theory, a distributed discontinuous algorithm for continuous-time agents subject to strongly convex local cost functions is first designed with a finite-time distributed estimator, where the gradients of the local cost functions are estimated in finite time. It is shown that for a strongly connected graph and arbitrary initial conditions, the proposed algorithms can achieve consensus, and the systems can converge to the optimal point in finite time. Then, a two-step approach is proposed to achieve finite-time optimization of high-order agents with disturbances under directed graphs. Finally, the validity of the proposed finite-time optimization algorithm is verified by two numerical examples.     

网联车辆有限时间分布式轨迹优化和协同控制

提出一种基于分层控制框架的网联车辆有限时间轨迹优化和协同控制方法.首先,在上层,为了最小化车辆间的间距误差,采用有限时间分布式优化算法构造车辆参考模型,产生最优轨迹信号;在下层,设计基于终端滑模的跟踪控制器使得车辆在有限时间内跟踪最优轨迹信号,并克服扰动等不确定因素的影响.然后,通过李雅普诺夫分析,严格验证控制系统的稳定性.最后,通过数值仿真结果表明了所提出方法的有效性.     

基于梯度估计的多智能体系统有限时间分布式优化

现有多智能体系统分布式优化算法大多具有渐近收敛速度,且要求系统的网络拓扑图为无向图或有向平衡图,在实际应用中具有一定的保守性.本文研究了具有强连通拓扑的多智能体系统有限时间分布式优化问题.首先,基于非光滑分析和Lyapunov稳定性理论设计了一个有限时间分布式梯度估计器.然后,基于该梯度估计器提出了一种适用于强连通有向图的有限时间分布式优化算法,实现了多智能体系统中智能体的状态在有限时间内一致收敛到全局最优状态值.与现有的有限时间分布式优化算法相比,新提出的有限时间优化算法适用于具有强连通拓扑的多智能体系统,放宽了系统对网络拓扑结构的要求.此外,本文基于Nussbaum函数方法对上述优化算法进行了拓展解决了含有未知高频增益符号的多智能体系统分布式优化问题.最后,通过仿真实例对提出的分布式优化算法的有效性进行了验证.     

自适应动态规划综述

自适应动态规划(Adaptive dynamic programming, ADP)是最优控制领域新兴起的一种近似最优方法, 是当前国际最优化领域的研究热点. ADP方法 利用函数近似结构来近似哈密顿--雅可比--贝尔曼(Hamilton-Jacobi-Bellman, HJB)方程的解, 采用离线迭代或者在线更新的方法, 来获得系统的近似最优控制策略, 从而能够有效地解决非线性系统的优化控制问题. 本文按照ADP的结构变化、算法的发展和应用三个方面介绍ADP方法. 对目前ADP方法的研究成果加以总结, 并对这 一研究领域仍需解决的问题和未来的发展方向作了进一步的展望.     

The Optimization of Control Parameters: Finite-time and Fixed-time Synchronization of Inertial Memristive Neural Networks with Proportional Delays and Switching Jumps Mismatch

This thesis’s object is inertial memristive neural networks (IMNNs) with proportional delays and switching jumps mismatch. Different from the traditional bounded delay, the proportional delay will be infinite as t → ∞. The finite-time synchronization (FN-TS) and fixed-time synchronization (FX-TS) can be realized with the devised controllers for the drive-response systems (D-RSs). Along with the Lyapunov function and some inequalities, the synchronization criteria of D-RSs are given. This paper presents an optimization model with minimum control energy and dynamic error as objective functions, aiming to obtain more accurate and optimized controller parameters. An intelligent algorithm: particle swarm optimization with stochastic inertia weight (SIWPSO) algorithm is introduced to solve the optimization model. Meanwhile, an integrated algorithm for selecting optimal control parameters is presented as well. In this method, the optimal control parameters and the setting time of synchronization can be obtained directly. At last, some simulations are presented to verify the theorems and the optimization model.

     

Local-homogeneity-based global continuous control for mechanical systems with constrained inputs: finite-time and exponential stabilisation

A global continuous control scheme for the finite-time or (local) exponential stabilisation of mechanical systems with constrained inputs is proposed. The approach is formally developed within the theoretical framework of local homogeneity. This has permitted to solve the formulated problem not only guaranteeing input saturation avoidance but also giving a wide range of design flexibility. The proposed scheme is characterised by a saturating-proportional-derivative type term with generalised saturating and locally homogeneous structure that permits multiple design choices on both aspects. The work includes a simulation implementation section where the veracity of the so-cited argument claiming that finite-time stabilisers are faster than asymptotical ones is studied. In particular, a way to carry out the design so as to, indeed, guarantee faster stabilisation through finite-time controllers (beyond their finite-time convergence) is shown.     

Recursive Learning Automata Approach to Markov Decision Processes

In this note, we present a sampling algorithm, called recursive automata sampling algorithm (RASA), for control of finite-horizon Markov decision processes (MDPs). By extending in a recursive manner Sastry's learning automata pursuit algorithm designed for solving nonsequential stochastic optimization problems, RASA returns an estimate of both the optimal action from a given state and the corresponding optimal value. Based on the finite-time analysis of the pursuit algorithm by Rajaraman and Sastry, we provide an analysis for the finite-time behavior of RASA. Specifically, for a given initial state, we derive the following probability bounds as a function of the number of samples: 1) a lower bound on the probability that RASA will sample the optimal action and 2) an upper bound on the probability that the deviation between the true optimal value and the RASA estimate exceeds a given error.     

Linear quadratic optimal learning control (LQL)

A learning control solution to the problem of finding a finite-time optimal control history that minimizes a quadratic cost is presented. Learning achieves optimization without requiring detailed knowledge of the system, which may be affected by unknown but repetitive disturbances. The optimal solution is synthesized one basis function at a time, reaching optimality in a finite number of trials. These system-dependent basis functions are special in that (1) each newly added basis function is learned without interfering with the previously optimized ones, and (2) it is extracted using data from previous learning trials. Numerical and experimental results are used to illustrate the algorithm.     

Receding-horizon optimal control of the current profile evolution during the ramp-up phase of a tokamak discharge

The control of the toroidal current density spatial profile in tokamak plasmas will be absolutely critical in future commercial-grade reactors to enable high fusion gain, non-inductive sustainment of the plasma current for steady-state operation, and magnetohydrodynamic (MHD) instability-free performance. The evolution in time of the current profile is related to the evolution of the poloidal magnetic flux, which is modeled in normalized cylindrical coordinates using a partial differential equation (PDE) usually referred to as the magnetic flux diffusion equation. The control objective during the ramp-up phase is to drive an arbitrary initial profile to approximately match, in a short time windows during the early flattop phase, a predefined target profile that will be maintained during the subsequent phases of the discharge. Thus, such a matching problem can be treated as an optimal control problem for a PDE system. A distinctive characteristic of the current profile control problem in tokamaks is that it admits interior, boundary and diffusivity actuation. A receding-horizon control scheme is proposed in this work to exploit this unique characteristic and to solve the associated open-loop finite-time optimal control problem using different optimization techniques. The efficiency of the proposed scheme is shown in simulations.     

Event-driven optimization-based control of hybrid systems with integral continuous-time dynamics

In this paper we introduce a class of continuous-time hybrid dynamical systems called integral continuous-time hybrid automata (icHA) for which we propose an event-driven optimization-based control strategy. Events include both external actions applied to the system and changes of continuous dynamics (mode switches). The icHA formalism subsumes a number of hybrid dynamical systems with practical interest, e.g., linear hybrid automata. Different cost functions, including minimum-time and minimum-effort criteria, and constraints are examined in the event-driven optimal control formulation. This is translated into a finite-dimensional mixed-integer optimization problem, in which the event instants and the corresponding values of the control input are the optimization variables. As a consequence, the proposed approach has the advantage of automatically adjusting the attention of the controller to the frequency of event occurrence in the hybrid process. A receding horizon control scheme exploiting the event-based optimal control formulation is proposed as a feedback control strategy and proved to ensure either finite-time or asymptotic convergence of the closed-loop.     

Generalized Thermodynamics of Maximum Work in Finite Time

We consider thermodynamic behaviour of thermal machines founded on kinetic rather than static origins. Their models, which are formulated for finite time transitions, simplify to models of classical thermodynamics in the limiting case of an infinite duration. An extended exergy is derived as a finite-time extension of the classical thermodynamic work delivered from a system of a body and its environment. With this quantity enhanced bounds can be determined for active continuous and cascade processes, in which there is an indirect energy exchange between two sybsystems through the working fluid of an engine, a refrigerator or a heat pump. These bounds refer to systems with finite exchange area or with a finite contact time. An economic framework of this theory is outlined.For both continuous and discrete processes, nonlinear thermodynamic models are derived from a combination of the energy balance and transfer equations. These models serve as constraints in the problem of work optimization. Variational and optimal control approaches are developed which are analogous to those found in analytical mechanics. Variational calculus is used along with some aspect of the canonical transformation theory to maximize work and discuss the role of a finite process intensity and of a finite duration.The optimality of a definite irreversible process for a finite-time transition of a controlled fluid is pointed out as well as a connection between the process duration, optimal dissipation and the optimal process intensity measured in terms of a hamiltonian, a dissipative quantity. It is shown that limits of the classical availability theory should be replaced by stronger limits which are obtained for finite time processes, and which are closer to reality. A hysteretic property of the generalized exergy describes a decrease of the maximum work received from an engine system and an increase of work added to a heat pump system, the features which are particularly important in high-rate regions of thermodynamic processes. For an infinite sequence of infinitesimal thermal machines, an optimal temperature strategy is obtained in the form similar to that known in the theory of simulated annealing.     

Parametric optimization and optimal control using algebraic geometry methods

We present two algebraic methods to solve the parametric optimization problem that arises in non-linear model predictive control. We consider constrained discrete-time polynomial systems and the corresponding constrained finite-time optimal control problem. The first method is based on cylindrical algebraic decomposition. The second uses Gröbner bases and the eigenvalue method for solving systems of polynomial equations. Both methods aim at moving most of the computational burden associated with the optimization problem off-line, by pre-computing certain algebraic objects. Then, an on-line algorithm uses this pre-computed information to obtain the solution of the original optimization problem in real time fast and efficiently. Introductory material is provided as appropriate and the algorithms are accompanied by illustrative examples.     

Robust finite-time guaranteed cost control for impulsive switched systems with time-varying delay

This paper is concerned with the problem of robust finite-time guaranteed cost control for a class of impulsive switched systems with time-varying delay. Firstly, the definitions of finite-time boundedness, finite-time stabilization, and robust finite-time guaranteed cost control are presented. Next, based on the average dwell-time approach, sufficient conditions on robust finite-time guaranteed cost control are obtained for the uncertain impulsive switched systems. Then, by using multiple Lyapunov-like functional method and linear matrix equality (LMI) technique, the state feedback controller is designed to guarantee that the uncertain impulsive switched system is finite-time stabilized. Furthermore, a finite-time guaranteed cost bound is given. Finally, a numerical example is shown to illustrate the effectiveness of the proposed results.     

Finite-time consensus of time-varying nonlinear multi-agent systems

This paper investigates the problem of leader–follower finite-time consensus for a class of time-varying nonlinear multi-agent systems. The dynamics of each agent is assumed to be represented by a strict feedback nonlinear system, where nonlinearities satisfy Lipschitz growth conditions with time-varying gains. The main design procedure is outlined as follows. First, it is shown that the leader–follower consensus problem is equivalent to a conventional control problem of multi-variable high-dimension systems. Second, by introducing a state transformation, the control problem is converted into the construction problem of two dynamic equations. Third, based on the Lyapunov stability theorem, the global finite-time stability of the closed-loop control system is proved, and the finite-time consensus of the concerned multi-agent systems is thus guaranteed. An example is given to verify the effectiveness of the proposed consensus protocol algorithm.     

改进蚁群算法在配货发送方面的研究

蚁群算法是一种用来在图中寻找优化路径的机率型算法,由于蚁群算法的多样性和反馈性会有可能过早的收敛于局部最优解,这样得到的最优解精度不高,该文改进了这一点,取开始的各条路径信息量为最大,让每条路径都有遍历,从而得到准确的最优解而不是局部最优解。对准确性方面进行了比较,得出改进后的算法是确实可行的。     

改进蚁群算法在配货发送方面的研究

蚁群算法是一种用来在图中寻找优化路径的机率型算法,由于蚁群算法的多样性和反馈性会有可能过早的收敛于局部最优解,这样得到的最优解精度不高,该文改进了这一点,取开始的各条路径信息量为最大,让每条路径都有遍历,从而得到准确的最优解而不是局部最优解。对准确性方面进行了比较,得出改进后的算法是确实可行的。     

Optimal control of distributed multiagent systems with finite-time group flocking

The flocking of multiple intelligent agents, inspired by the swarm behavior of natural phenomena, has been widely used in the engineering fields such as in unmanned aerial vehicle (UAV) and robots system. However, the performance of the system (such as response time, network throughput, and resource utilization) may be greatly affected while the intelligent agents are engaged in cooperative work. Therefore, it is concerned to accomplish the distributed cooperation while ensuring the optimal performance of the intelligent system. In this paper, we investigated the optimal control problem of distributed multiagent systems (MASs) with finite-time group flocking movement. Specifically, we propose two optimal group flocking algorithms of MASs with single-integrator model and double-integrator model. Then, we study the group consensus of distributed MASs by using modern control theory and finite-time convergence theory, where the proposed optimal control algorithms can drive MASs to achieve the group convergence in finite-time while minimizing the performance index of the intelligence system. Finally, experimental simulation shows that MASs can keep the minimum energy function under the effect of optimal control algorithm, while the intelligent agents can follow the optimal trajectory to achieve group flocking in finite time.