Quantum‑enhanced reinforcement learning for control: a preliminary study

Reinforcement learning is one of the fastest growing areas in machine learning, and has obtained great achievements in biomedicine, Internet of Things (IoT), logistics, robotic control, etc. However, there are still many challenges for engineering applications, such as how to speed up the learning process, how to balance the trade-off between exploration and exploitation. Quantum technology, which can solve complex problems faster than classical methods, especially in supercomputers, provides us a new paradigm to overcome these challenges in reinforcement learning. In this paper, a quantum-enhanced reinforcement learning is pictured for optimal control. In this algorithm, the states and actions of reinforcement learning are quantized by quantum technology. And then, a probability amplification method, which can effectively avoid the trade-off between exploration and exploitation via quantized technology, is presented. Finally, the optimal control policy is learnt during the process of reinforcement learning. The performance of this quantized algorithm is demonstrated in both MountainCar reinforcement learning environment and CartPole reinforcement learning environment—one kind of classical control reinforcement learning environment in the OpenAI Gym. The preliminary study results validate that, compared with Q-learning, this quantized reinforcement learning method has better control performance without considering the trade-off between exploration and exploitation. The learning performance of this new algorithm is stable with different learning rates from 0.01 to 0.10, which means it is promising to be employed in unknown dynamics systems.     

时变机器人系统的重复学习控制: 一种混合学习方案

Repetitive learning control is presented for finite-time-trajectory tracking of uncertain time-varying robotic systems. A hybrid learning scheme is given to cope with the constant and time-varying unknowns in system dynamics, where the time functions are learned in an iterative learning way, without the aid of Taylor expression, while the conventional differential learning method is suggested for estimating the constant ones. It is distinct that the presented repetitive learning control avoids the requirement for initial repositioning at the beginning of each cycle, and the time-varying unknowns are not necessary to be periodic. It is shown that with the adoption of hybrid learning, the boundedness of state variables of the closed-loop system is guaranteed and the tracking error is ensured to converge to zero as iteration increases. The effectiveness of the proposed scheme is demonstrated through numerical simulation.     

Quantum learning control using differential evolution with equally-mixed strategies

Learning control has been recognized as a powerful approach in quantum information technology. In this paper, we extend the application of differential evolution (DE) to design optimal control for various quantum systems. Various DE methods are introduced and analyzed, and EMSDE featuring in equally mixed strategies is employed for quantum control. Two classes of quantum control problems, including control of four-level open quantum ensembles and quantum superconducting systems, are investigated to demonstrate the performance of EMSDE for learning control of quantum systems. Numerical results verify the effectiveness of the EMSDE method for various quantum systems and show the potential for complex quantum control problems.     

Robust Optimization-Based Iterative Learning Control for Nonlinear Systems With Nonrepetitive Uncertainties

This paper aims to solve the robust iterative learning control(ILC)problems for nonlinear time-varying systems in the presence of nonrepetitive uncertainties.A new optimization-based method is proposed to design and analyze adaptive ILC,for which robust convergence analysis via a contraction mapping approach is realized by leveraging properties of substochastic matrices.It is shown that robust tracking tasks can be realized for optimization-based adaptive ILC,where the boundedness of system trajectories and estimated parameters can be ensured,regardless of unknown time-varying nonlinearities and nonrepetitive uncertainties.Two simulation tests,especially implemented for an injection molding process,demonstrate the effectiveness of our robust optimization-based ILC results.     

Iterative learning control for nonlinear systems with uncertain state delay and arbitrary initial error

Most of the existing iterative learning control algorithms proposed for time-delay systems are based on the condition that the time-delay is precisely available, and the initial state is reset to the desired one or a fixed value at the start of each operation, which makes great limitation on the practical application of corresponding results. In this paper, a new iterative learning control algorithm is studied for a class of nonlinear system with uncertain state delay and arbitrary initial error. This algorithm needs to know only the boundary estimation of the state delay, and the initial state is updated, while the convergence of the system is guaranteed. Without state disturbance and output measurement noise, the system output will strictly track the desired trajectory after successive iteration. Furthermore, in the presence of state disturbance and measurement noise, the tracking error will be bounded uniformly. The convergence is strictly proved mathematically, and sufficient conditions are obtained. A numerical example is shown to demonstrate the effectiveness of the proposed approach.     

Batch Process Modelling and Optimal Control Based on Neural Network Models

This paper presents several neural network based modelling, reliable optimal control, and iterative learning control methods for batch processes. In order to overcome the lack of robustness of a single neural network, bootstrap aggregated neural networks are used to build reliable data based empirical models. Apart from improving the model generalisation capability, a bootstrap aggregated neural network can also provide model prediction confidence bounds. A reliable optimal control method by incorporating model prediction confidence bounds into the optimisation objective function is presented. A neural network based iterative learning control strategy is presented to overcome the problem due to unknown disturbances and model-plant mismatches. The proposed methods are demonstrated on a simulated batch polymerisation process.     

Simple adaptive air-fuel ratio control of a port injection SI engine with a cylinder pressure sensor

The problem of air-fuel ratio(AFR) control of the port injection spark ignition(SI) engine is still of considerable importance because of stringent demands on emission control. In this paper, the static AFR calculation model based on in-cylinder pressure data and on the adaptive AFR control strategy is presented. The model utilises the intake manifold pressure, engine speed, total heat release, and the rapid burn angle, as input variables for the AFR computation. The combustion parameters, total heat release,and rapid burn angle, are calculated from in-cylinder pressure data. This proposed AFR model can be applied to the virtual lambda sensor for the feedback control system. In practical applications, simple adaptive control(SAC) is applied in conjunction with the AFR model for port-injected fuel control. The experimental results show that the proposed model can estimate the AFR, and the accuracy of the estimated value is applicable to the feedback control system. Additionally, the adaptive controller with the AFR model can be applied to regulate the AFR of the port injection SI engine.     

未知相对度非线性系统的一阶D型迭代学习控制

The classical D-type iterative learning control law depends crucially on the relative degree of the controlled system, high order differential iterative learning law must be taken for systems with high order relative degree. It is very difficult to ascertain the relative degree of the controlled system for uncertain nonlinear systems. A first-order D-type iterative learning control design method is presented for a class of nonlinear systems with unknown relative degree based on dummy model in this paper. A dummy model with relative degree 1 is constructed for a class of nonlinear systems with unknown relative degree. A first-order D-type iterative learning control law is designed based on the dummy model, so that the dummy model can track the desired trajectory perfectly, and the controlled system can track the desired trajectory within a certain error. The simulation example demonstrates the feasibility and effectiveness of the presented method.     

An adaptive series control for an interior permanent magnet synchronous motor with actuator compensation

An adaptive series speed control system for an interior permanent magnet synchronous motor (IPMSM) drive is presented in this paper. This control system consists of a current and a speed control loop, and it is intended to improve the drive’s speed tracking performance as well as to compensate for voltage distortions caused by non-ideal characteristics of the drive’s actuator, which is a voltage source inverter (VSI). To achieve these goals, a simple model that captures these characteristics of the VSI is developed and embedded in the motor’s electrical model. Then, based on the resulting model, an adaptive proportional-integral (PI) control for the current loops is designed, allowing for state regulation and actuator compensation. Additionally, to improve the drive’s speed tracking performance, a proportional-model-reference adaptive controller (MRAC) is designed for the speed loop. Techniques from machine learning are used for designing the MRAC to effectively address nonlinearities and uncertainties in the speed dynamic. Finally, simulation results are presented to illustrate the outstanding performance of the proposed multi-loop controller.     

Set-point-related indirect iterative learning control for multi-input multi-output systems

A form of iterative learning control (ILC) is used to update the set-point for the local controller. It is referred to as set-point-related (SPR) indirect ILC. SPR indirect ILC has shown excellent performance: as a supervision module for the local controller, ILC can improve the tracking performance of the closed-loop system along the batch direction. In this study, an ILC-based P-type controller is proposed for multi-input multi-output (MIMO) linear batch processes, where a P-type controller is used to design the control signal directly and an ILC module is used to update the set-point for the P-type controller. Under the proposed ILC-based P-type controller, the closed-loop system can be transformed to a 2-dimensional (2D) Roesser s system. Based on the 2D system framework, a sufficient condition for asymptotic stability of the closed-loop system is derived in this paper. In terms of the average tracking error (ATE), the closed-loop control performance under the proposed algorithm can be improved from batch to batch, even though there are repetitive disturbances. A numerical example is used to validate the proposed results.     

Initial state iterative learning for final state control in motion systems

In this work, an initial state iterative learning control (ILC) approach is proposed for final state control of motion systems. ILC is applied to learn the desired initial states in the presence of system uncertainties. Four cases are considered where the initial position or speed is a manipulated variable and the final displacement or speed is a controlled variable. Since the control task is specified spatially in states, a state transformation is introduced such that the final state control problems are formulated in the phase plane to facilitate spatial ILC design and analysis. An illustrative example is provided to verify the validity of the proposed ILC algorithms.     

多步k最近邻初值寻优的气压模拟系统遗忘迭代学习控制

高速列车车内压力波动过大会对乘客舒适性造成影响,而气压模拟系统是一套通过对车内模拟气压跟踪控制,实现对乘客舒适性进行研究的装置.为解决系统历史运行数据利用率低以及存在迭代初始误差导致系统收敛速度慢的问题,采用k最近邻(kNN)算法,建立一种基于历史控制信息的最优初次控制信号提取方法,并根据迭代学习控制的基本原理,将最优控制初值输入到带遗忘因子的迭代学习控制器中,通过不断迭代来实现车内期望气压轨迹的跟踪控制,并和基于大数据的迭代学习控制以及传统PID迭代学习控制进行对比分析.仿真结果表明:基于多步kNN的遗忘迭代学习控制收敛速度更快、系统抖动程度更小、控制精度更高以及算法鲁棒性更好.     

时变非线性系统迭代学习控制遗忘因子算法

针对迭代学习P型控制算法对初始偏差和输出误差扰动的敏感性问题,研究了一种带有遗忘因子的时变非线性系统的迭代学习控制方法.在有扰动的情况下,利用迭代学习过程记忆的期望轨迹,期望控制以及跟踪误差,通过有界学习增益和批次时变因子设计学习控制器,并基于算子理论给出了控制算法存在的充分必要条件及其收敛性分析,改善了系统的鲁棒性和动态特性.最后以注塑机的注射速度控制仿真验证了本文算法的有效性.     

带有初态学习的可变增益迭代学习控制

针对一类非线性系统提出一种新的学习控制算法,该算法在可变学习增益的迭代学习控制律基础上,增加了系统初态的迭代学习律.利用算子理论证明了系统在存在初态偏移时经过迭代学习后,其输出能够完全跟踪期望轨迹,同时得到了该算法谱半径形式的收敛条件.将该算法与传统迭代学习控制相比较可以看出,前者的收敛速度得到了较大提高,而且解决了可变学习增益迭代学习控制的初态偏移问题.仿真结果验证了该算法的有效性.     

Analysis of Nonlinear Discrete-Time Systems with Higher-Order Iterative Learning Control

In this paper, an iterative learning control method is proposed for a class of nonlinear discrete-time systems with well-defined relative degree, which uses the output data from several previous operation cycles to enhance tracking performance. A new analysis approach is developed, by which the iterative learning control is shown to guarantee the convergence of the output trajectory to the desired one within bound and the bound is proportional to the bound on resetting errors. It is further proved effective to overcome initial shifts and the resultant output trajectory can be assessed as iteration increases. Numerical simulation is carried out to verify the theoretical results and exhibits that the proposed updating law possesses good transient behavior of learning process so that the convergence speed is improved.     

Iterative learning control for nonlinear differential inclusion systems

In this paper, we propose an iterative learning control strategy to track a desired trajectory for a class of uncertain systems governed by nonlinear differential inclusions. By imposing Lipschitz continuous condition on a set‐valued mapping described by a closure of the convex hull of a set and using D‐type and PD‐type updating laws with initial iterative learning, we establish the iterative learning process and give a new convergence analysis with the help of Steiner‐type selector. Finally, numerical examples are provided to verify the effectiveness of the proposed method with suitable selection of set‐valued mappings. An application to the speed control of robotic fish is also given.     

迭代学习控制的指数变增益加速学习算法及其在机器人控制上的应用

迭代学习控制作为智能控制的一个分支，近年来得到了很大的发展，在各个领域都有广泛的运用。为提高迭代学习速度，本文给出了指数变增益加速算法。机器人系统的仿真结果表明，该方法能大大提高学习速度，具有良好的控制性能。     

迭代学习控制技术回顾与长期学习控制展望

本文首先回顾了迭代学习控制中初始状态漂移问题和单调收敛性分析的研究技术.其次,综述了高阶迭代学习控制机制及其收敛速度比较和有效性.再次,评述了重复运行大系统和变幅值大工业过程的迭代学习控制机理.最后,展望了长期学习控制的研究趋势等.     

同步发电机初态学习与PSS结合的励磁控制

为了解决开环迭代学习励磁控制器对大干扰耐受力差、对初始定位要求严格的问题,针对多机电力系统,设计了具有初态学习的开闭环迭代学习控制(ILC)+PSS的励磁控制器。该控制器在开环ILC的基础上,引入机端电压闭环反馈信号,另外为放宽对初态的要求,允许有一定的初始定位误差,对算法还进一步改进,初态也进行学习,同时还结合PSS提供的正阻尼能力,以增强系统的功角稳定性。最后,仿真结果表明,该励磁控制器不但能快速达到机端电压的调节精度,而且改善了系统的功角稳定性。