Dynamic intermittent Q‐learning–based model‐free suboptimal co‐design of ‐stabilization

This paper proposes an intermittent model‐free learning algorithm for linear time‐invariant systems, where the control policy and transmission decisions are co‐designed simultaneously while also being subjected to worst‐case disturbances. The control policy is designed by introducing an internal dynamical system to further reduce the transmission rate and provide bandwidth flexibility in cyber‐physical systems. Moreover, a Q‐learning algorithm with two actors and a single critic structure is developed to learn the optimal parameters of a Q‐function. It is shown by using an impulsive system approach that the closed‐loop system has an asymptotically stable equilibrium and that no Zeno behavior occurs. Furthermore, a qualitative performance analysis of the model‐free dynamic intermittent framework is given and shows the degree of suboptimality concerning the optimal continuous updated controller. Finally, a numerical simulation of an unknown system is carried out to highlight the efficacy of the proposed framework.     

Event‐triggered optimal tracking control of nonlinear systems

We propose a novel event‐triggered optimal tracking control algorithm for nonlinear systems with an infinite horizon discounted cost. The problem is formulated by appropriately augmenting the system and the reference dynamics and then using ideas from reinforcement learning to provide a solution. Namely, a critic network is used to estimate the optimal cost while an actor network is used to approximate the optimal event‐triggered controller. Because the actor network updates only when an event occurs, we shall use a zero‐order hold along with appropriate tuning laws to encounter for this behavior. Because we have dynamics that evolve in continuous and discrete time, we write the closed‐loop system as an impulsive model and prove asymptotic stability of the equilibrium point and Zeno behavior exclusion. Simulation results of a helicopter, a one‐link rigid robot under gravitation field, and a controlled Van‐der‐Pol oscillator are presented to show the efficacy of the proposed approach. Copyright © 2016 John Wiley & Sons, Ltd.     

H∞ tracking control for linear discrete‐time systems via reinforcement learning

In this paper, the H_∞ tracking control of linear discrete‐time systems is studied via reinforcement learning. By defining an improved value function, the tracking game algebraic Riccati equation with a discount factor is obtained, which is solved by iteration learning algorithms. In particular, Q‐learning based on value iteration is presented for H_∞ tracking control, which does not require the system model information and the initial allowable control policy. In addition, to improve the practicability of algorithm, the convergence analysis of proposed algorithm with a discount factor is given. Finally, the feasibility of proposed algorithms is verified by simulation examples.     

Q‐learning for continuous‐time graphical games on large networks with completely unknown linear system dynamics

In this paper, we consider the problem of leader synchronization in systems with interacting agents in large networks while simultaneously satisfying energy‐related user‐defined distributed optimization criteria. But modeling in large networks is very difficult, and for that reason, we derive a model‐free formulation that is based on a separate distributed Q‐learning function for every agent. Every Q‐function is a parametrization of each agent's control, of the neighborhood controls, and of the neighborhood tracking error. It is also evident that none of the agents has any information on where the leader is connected to and from where she spreads the desired information. The proposed algorithm uses an integral reinforcement learning approach with a separate distributed actor/critic network for each agent: a critic approximator to approximate each value function and an actor approximator to approximate each optimal control law. The derived tuning laws for each actor and critic approximators are designed appropriately by using gradient descent laws. We provide rigorous stability and convergence proofs to show that the closed‐loop system has an asymptotically stable equilibrium point and that the control policies form a graphical Nash equilibrium. We demonstrate the effectiveness of the proposed method on a network consisting of 10 agents. Copyright © 2016 John Wiley & Sons, Ltd.     

Discrete‐time repetitive controller for time‐delay systems with disturbance observer

A novel discrete‐time repetitive controller design for time‐delay systems subject to a periodic reference and exogenous periodic disturbances is presented. The main idea behind the proposed approach is to take advantage of the plant delay in the controller design, and not to compensate for the effect of this delay. To facilitate this concept, we introduce an appropriate time‐delay and a compensator in a positive feedback connection with the plant, such that a generator for periodic signals is constructed. Then a proportional controller is used to stabilize the closed‐loop system. The tracking control capability is thus guaranteed according to the internal model principle (IMP). In addition, to attenuate external periodic disturbances, a disturbance observer (DO) is developed to simultaneously achieve reference tracking and disturbance rejection. The possible fractional delay due to the digital discretization is handled by using a fractional delay filter approximation. The proposed controller has a simple structure, in which only a proportional parameter and a low‐pass filter are required to be chosen. The closed‐loop stability conditions and a robustness analysis under model uncertainties are studied. Numerical simulations and practical experiments on a servo motor system are conducted to verify the feasibility and simplicity of the proposed controller. Copyright © 2011 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Output‐feedback tracking in causal nonminimum‐phase nonlinear systems using higher‐order sliding modes

Asymptotic output‐feedback tracking in a class of causal nonminimum phase uncertain nonlinear systems is addressed via sliding mode techniques. Sliding mode control is proposed for robust stabilization of the output tracking error in the presence of a bounded disturbance. The output reference profile and the unknown input/disturbance are supposed to be described by unknown linear exogenous systems of a given order. Local asymptotic stability of the output tracking error dynamics along with the boundedness of the internal states are proven. The unstable internal states are estimated asymptotically via the proposed multistage observer that is based on the method of extended system center. A higher‐order sliding mode observer/differentiator is used for the exact estimation of the input–output states in a finite time. The bounded disturbance is reconstructed asymptotically. A numerical example illustrates the efficiency of the proposed output‐feedback tracking approach developed for causal nonminimum phase nonlinear systems. Copyright © 2009 John Wiley & Sons, Ltd.     

Neural‐network‐based predictive controller design: An application to temperature control of a plastic injection molding process

This paper presents a neural‐network‐based predictive control (NPC) method for a class of discrete‐time multi‐input multi‐output (MIMO) systems. A discrete‐time mathematical model using a recurrent neural network (RNN) is constructed and a learning algorithm adopting an adaptive learning rate (ALR) approach is employed to identify the unknown parameters in the recurrent neural network model (RNNM). The NPC controller is derived based on a modified predictive performance criterion, and its convergence is guaranteed by adopting an optimal algorithm with an adaptive optimal rate (AOR) approach. The stability analysis of the overall MIMO control system is well proven by the Lyapunov stability theory. A real‐time control algorithm is proposed which has been implemented using a digital signal processor, TMS320C31 from Texas Instruments. Two examples, including the control of a MIMO nonlinear system and the control of a plastic injection molding process, are used to demonstrate the effectiveness of the proposed strategy. Results from both numerical simulations and experiments show that the proposed method is capable of controlling MIMO systems with satisfactory tracking performance under setpoint and load changes. Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Tracking control for sampled‐data systems with uncertain time‐varying sampling intervals and delays

In this paper, a solution to the approximate tracking problem of sampled‐data systems with uncertain, time‐varying sampling intervals and delays is presented. Such time‐varying sampling intervals and delays can typically occur in the field of networked control systems. The uncertain, time‐varying sampling and network delays cause inexact feedforward, which induces a perturbation on the tracking error dynamics, for which a model is presented in this paper. Sufficient conditions for the input‐to‐state stability (ISS) of the tracking error dynamics with respect to this perturbation are given. Hereto, two analysis approaches are developed: a discrete‐time approach and an approach in terms of delay impulsive differential equations. These ISS results provide bounds on the steady‐state tracking error as a function of the plant properties, the control design and the network properties. Moreover, it is shown that feedforward preview can significantly improve the tracking performance and an online extremum seeking (nonlinear programming) algorithm is proposed to online estimate the optimal preview time. The results are illustrated on a mechanical motion control example showing the effectiveness of the proposed strategy and providing insight into the differences and commonalities between the two analysis approaches. Copyright © 2009 John Wiley & Sons, Ltd.     

Online solution of nonlinear two‐player zero‐sum games using synchronous policy iteration

The two‐player zero‐sum (ZS) game problem provides the solution to the bounded L₂‐gain problem and so is important for robust control. However, its solution depends on solving a design Hamilton–Jacobi–Isaacs (HJI) equation, which is generally intractable for nonlinear systems. In this paper, we present an online adaptive learning algorithm based on policy iteration to solve the continuous‐time two‐player ZS game with infinite horizon cost for nonlinear systems with known dynamics. That is, the algorithm learns online in real time an approximate local solution to the game HJI equation. This method finds, in real time, suitable approximations of the optimal value and the saddle point feedback control policy and disturbance policy, while also guaranteeing closed‐loop stability. The adaptive algorithm is implemented as an actor/critic/disturbance structure that involves simultaneous continuous‐time adaptation of critic, actor, and disturbance neural networks. We call this online gaming algorithm ‘synchronous’ ZS game policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for critic, actor, and disturbance networks. The convergence to the optimal saddle point solution is proven, and stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm in solving the HJI equation online for a linear system and a complex nonlinear system. Copyright © 2011 John Wiley & Sons, Ltd.     

Data‐driven high‐order terminal iterative learning control with a faster convergence speed

In this paper, a novel high‐order optimal terminal iterative learning control (high‐order OTILC) is proposed via a data‐driven approach for nonlinear discrete‐time systems with unknown orders in the input and output. The objective is to track the desired values at the endpoint of the operation cycle. The terminal tracking errors over more than one previous iterations are used to enhance the high‐order OTILC's performance with faster convergence. From rigor of the analysis, the monotonic convergence of the terminal tracking error is proved along the iteration direction. More importantly, the condition for a high‐order OTILC to outperform the low‐order ones is first established by this work. The learning gain is not fixed but iteratively updated by using the input and output (I/O) data, which enhances the flexibility of the proposed controller for modifications and expansions. The proposed method is data‐driven in which no explicit models are used except for the input and output data. The applications to a highly nonlinear continuous stirred tank reactor and a highly nonlinear fed‐batch fermentater demonstrate the effectiveness of the proposed high‐order OTILC design.     

Event-triggered optimal tracking control of multiplayer unknown nonlinear systems via adaptive critic designs

In this article, the event-triggered optimal tracking control problem for multiplayer unknown nonlinear systems is investigated by using adaptive critic designs. By constructing a neural network (NN)-based observer with input–output data, the system dynamics of multiplayer unknown nonlinear systems is obtained. Subsequently, the optimal tracking control problem is converted to an optimal regulation problem by establishing a tracking error system. Then, the optimal tracking control policy for each player is derived by solving coupled event-triggered Hamilton-Jacobi (HJ) equation via a critic NN. Meanwhile, a novel weight updating rule is designed by adopting concurrent learning method to relax the persistence of excitation (PE) condition. Moreover, an event-triggering condition is designed by using Lyapunov's direct method to guarantee the uniform ultimate boundedness (UUB) of the closed-loop multiplayer systems. Finally, the effectiveness of the developed method is verified by two different multiplayer nonlinear systems.     

Observer‐based tracking control for switched stochastic systems based on a hybrid 2‐D model

This paper is concerned with the observer‐based output tracking problem for a class of linear switched stochastic systems with time delay and disturbance by using repetitive control approach. More precisely, a two‐dimensional hybrid model is incorporated to obtain and optimize the repetitive controller. In particular, the repetitive controller is used to improve the tracking performance through its continuous learning actions. In addition, an equivalent‐input‐disturbance estimator is incorporated into the repetitive control design approach to reduce the effect of the external disturbances. The main aim of the control design is to track the periodic reference signal with the measured output of the system under consideration even in the presence of an unknown bounded disturbance. By constructing a suitable Lyapunov‐Krasovskii functional and using average dwell time approach and Jensen inequality, sufficient conditions are obtained in terms of linear matrix inequalities to guarantee the mean‐square exponential stability of the considered system. Eventually, a numerical example is provided to demonstrate the effectiveness of the developed method.     

A STABLE ON‐LINE SELF‐TUNING OPTIMAL PID CONTROLLER FOR A CLASS OF UNKNOWN SYSTEMS

In this paper, a novel self‐tuning method of optimal PID control laws is proposed for both continuous‐time systems and discrete‐time systems. The controlled plant is assumed to be unknown except the system order (or system delay) and the direction of transmitting control input. Through the minimization of PID gains subject to the Lyapunov stability based reaching condition, the tuning of the three PID control gains is transformed to solve the inequality constraint optimization problem. An unknown SISO nonlinear system subject to a unit step input, and the tracking control problem of the piezoelectric actuator (PZA) with unknown dynamics are simulated. The simulation results show that the excellent tracking performance can be achieved.     

Optimal grade transition control for liquid‐propylene polymerization reactor

A nonlinear control system integrating an off‐line optimizer and a nonlinear model‐based controller is developed to perform optimal grade transition operations in a continuous pilot plant reactor. A simple black‐box model is developed and used to determine optimal trajectories of inputs and outputs for a series of three polypropylene grades. The simplified model is also used to develop a nonlinear controller. This controller is similar to generic model control; however, the integral action is omitted and an on‐line updating scheme is incorporated to update pre‐specified model parameters using delayed process measurements. The time optimal inputs, which are calculated by the off‐line optimizer, are introduced to the plant in a feedforward manner. At the same time, the deviations from the optimal output are corrected using the feedback nonlinear controller. The simulations on a complex mechanistic model of the process reveal that the nonlinear control scheme performs well for both set point tracking and disturbance rejection. This paper integrates well‐known methodologies, such as the generic‐model control algorithm, parameter update schemes, and off‐line optimization, together to develop an applicable and robust control technique for continuous polymerization reactors. Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Composite stratified anti‐disturbance control for a class of MIMO discrete‐time systems with nonlinearity

Anti‐disturbance control and estimation problem are investigated for nonlinear system subject to multi‐source disturbances. The disturbances classified model is proposed based on the error and noise analysis of priori knowledge. The disturbance observers are constructed separately from the controller design to estimate the disturbance with partial known information. By integrating disturbance‐observer‐based control with discrete‐time sliding‐mode control (DSMC), a novel type of composite stratified anti‐disturbance control scheme is presented for a class of multiple‐input–multiple‐output discrete‐time systems with known and unknown nonlinear dynamics, respectively. Simulations for a flight control system are given to demonstrate the effectiveness of the results compared with the previous schemes. Copyright © 2011 John Wiley & Sons, Ltd.     

OPTIMAL REJECTION WITH ZERO STEADY‐STATE ERROR OF SINUSOIDAL DISTURBANCES FOR TIME‐DELAY SYSTEMS

This paper studies the problem of optimal rejection with zero steady‐state error of sinusoidal disturbances for linear systems with time‐delay. Based on the internal model principle, a disturbance compensator is constructed to counterbalance the external sinusoidal disturbances, so that the original system can be transformed into an augmented system without disturbances. Then, with the introduction of a sensitivity parameter and expanding power series around it, the optimal disturbance rejection problem can be simplified to the problem of solving an infinite sum of a linear optimal control series without time‐delay or disturbance. The optimal control law for disturbance rejection with zero steady‐state error consists of accurate linear state feedback terms and a time‐delay compensating term, which is an infinite sum of an adjoint vector series. In the presented approach, iteration is required only for the time‐delay compensation series. By intercepting a finite sum of the compensation series, we obtain an approximate physically realizable optimal control law that avoids complex calculation. A numerical simulation shows that the algorithm is effective and easy to implement.     

Adaptive disturbance observer‐based finite‐time continuous fault‐tolerant control for reentry RLV

The control effectors of reusable launch vehicle (RLV) can produce significant perturbations and faults in reentry phase. Such a challenge imposes tight requirements to enhance the robustness of vehicle autopilot. Focusing on this problem, a novel finite‐time fault‐tolerant control strategy is proposed for reentry RLV in this paper. The key of this strategy is to design an adaptive‐gain multivariable finite‐time disturbance observer (FDO) to estimate the synthetical perturbation with unknown bounds, which is composed of model uncertainty, external disturbance, and actuator fault considered as the partial loss of actuator effectiveness in this work. Then, combined with the finite‐time high‐order observer and differentiator, a continuous homogeneous second‐order sliding mode controller based on the terminal sliding mode and super‐twisting algorithm is designed to achieve a fast and accurate RLV attitude tracking with chattering attenuation. The main features of the integrated control strategy are that the adaptation algorithm of FDO can achieve non‐overestimating values of the observer gains and the second‐order super‐twisting sliding mode approach can obtain a more elegant solution in finite time. Finally, simulation results of classical RLV (X‐33) are provided to verify the effectiveness and robustness of the proposed fault‐tolerant controller in tracking the guidance commands. Copyright © 2017 John Wiley & Sons, Ltd.     

Cooperative robust output regulation problem for discrete‐time linear time‐delay multi‐agent systems

In this paper, we study the cooperative robust output regulation problem for discrete‐time linear multi‐agent systems with both communication and input delays by a distributed internal model approach. We first introduce the distributed internal model for discrete‐time multi‐agent systems with both communication and input delays. Then, we define the so‐called auxiliary system and auxiliary augmented system. Finally, we solve our problem by showing, under some standard assumptions, that if a distributed state feedback control or a distributed output feedback control solves the robust output regulation problem of the auxiliary system, then the same control law solves the cooperative robust output regulation problem of the original multi‐agent systems.     

Stochastic Point‐to‐Point Iterative Learning Control Based on Stochastic Approximation

An iterative learning control algorithm with iteration decreasing gain is proposed for stochastic point‐to‐point tracking systems. The almost sure convergence and asymptotic properties of the proposed recursive algorithm are strictly proved. The selection of learning gain matrix is given. An illustrative example shows the effectiveness and asymptotic trajectory properties of the proposed approach.     

Precise Tracking for Continuous‐Time Non‐Minimum Phase Systems

Stable inversion based precise tracking for continuous‐time square or nonsquare non‐minimum phase systems is studied. However, high precision trajectory tracking of non‐minimum phase systems can be obtained by the stable inversion method but requiring large enough extended time interval. In order to solve this problem of large extended time restriction, a novel approach to precise trajectory tracking of non‐minimum phase systems is proposed, it is called the improved stable inversion (ISI) method, using an optimal integration of the pre‐actuation and the optimal state transition (OST) techniques. The ISI method can obtain precise trajectory tracking with a smaller extended time interval as compared to the stable inversion method. The proposed method achieves better validation through numerical simulations for the non‐minimum phase system.