参数不确定马尔可夫跳变系统的鲁棒适应控制

针对具有参数不确定的线性马尔可夫跳变系统的鲁棒适应控制问题进行了研究.分析了切换系统切换律的可观测和不可观测情形.对于可观测的切换律,利用线性矩阵不等式和共同二次Lyapunov函数方法,得出的具有参数不确定的切换系统是鲁棒可镇定的;对于切换律符合不可观测的马尔可夫随机过程的情况,通过设计恰当的采样适应控制器得到系统随机可镇定的充分条件.并通过例子给出适应镇定控制器的算法.     

Stabilization of stochastic systems with hidden Markovian jumps

This paper considers the adaptive control of discrete-time hybrid stochastic systems with unknown randomly jumping parameters described by a finite-state hidden Markov chain. An intuitive yet longstanding conjecture in this area is that such hybrid systems can be adaptively stabilized whenever the rate of transition of the hidden Markov chain is small enough. This paper provides a rigorous positive answer to this conjecture by establishing the global stability of a gradient-algorithm-based adaptive linear-quadratic control.     

认知无线电网络一种自适应切换机制的Markov建模分析

为了提高认知无线电网络(CRN)的利用率和吞吐量性能,提出了一种新的主用户优先的自适应频谱切换机制。强调主用户的优先地位,并考虑主用户到达率对次用户的通信影响和限制次用户对主用户的干扰功率,次用户以此自适应地决定执行主动切换机制或被动切换机制。在此自适应切换机制下建立了主次用户之间的Markov链,求出了相应的稳态概率,由稳态概率和不同状态下的吞吐量推导出系统吞吐量和信道利用率的解析表达式。又对次用户之间的通信建立了一个Markov链,推导出次用户之间传递控制信息的时间。数值结果表明提出的新的自适应切换机制比基于CSMA的接入方法具有更高的系统吞吐量和信道利用率,并且可以求出次用户之间的传递时间。     

一种可信的自适应服务组合机制

提出一种可信的自适应服务组合机制.首先,将组合服务的可信性保证问题转换为自适应控制问题,可信性保证策略作为可调节控制器,组合服务作为被控对象,并设计了相应的系统结构;其次,在马尔可夫决策过程框架下建模和优化组合服务的可信维护过程和策略,并设计了相应的算法,实现了基于强化学习的直接自适应控制机制;最后,通过仿真实验,将组合服务的自适应维护与随机维护策略比较,表明组合服务的自适应维护具有明显的优越性.     

Almost Sure Stabilization for Feedback Controls of Regime-Switching Linear Systems With a Hidden Markov Chain

This work is devoted to the almost sure stabilization of adaptive control systems that involve an unknown Markov chain. The control system displays continuous dynamics represented by differential equations and discrete events given by a hidden Markov chain. In the previous investigation on this class of problems, averaging criteria were used, which provides only the system behavior in some expectation sense. A closer scrutiny of the system behavior necessarily requires the consideration of sample path properties. Different from previous work on stabilization of adaptive controlled systems with a hidden Markov chain, where average criteria were considered, this work focuses on the almost sure stabilization or sample path stabilization of the underlying processes. Under simple conditions, it is shown that as long as the feedback controls have linear growth in the continuous component, the resulting process is regular. Moreover, by appropriate choice of the Lyapunov functions, it is shown that the adaptive system is stabilizable almost surely. As a by-product, it is also established that the controlled process is positive recurrent.     

An Adaptive Learning Control and Its Application to Vibration Isolation Systems

We present adaptive learning control algorithms which use Markov parameters. Since in the algorithm, the control input and the model parameter are updated at each iteration, it is applicable to unknown systems in a class. In order to overcome the effect of a disturbance, Markov parameters of the inverse model is also introduced. The effectiveness of the algorithm is illustrated by means of a numerical simulation and experimental application to an active vibration isolation system.     

Optimality analysis of adaptive sampled control of hybrid systems with quadratic index

This note is concerned with the sampled-data based linear quadratic (LQ) adaptive control of continuous-time systems with unknown Markov jump parameters. A parameter estimator and a control design method are given. It is shown that when the sample step size is small, the sampled-data based adaptive control is suboptimal under LQ index. The result is illustrated by a simulation example.     

A Markov Game-Adaptive Fuzzy Controller for Robot Manipulators

This paper develops an adaptive fuzzy controller for robot manipulators using a Markov game formulation. The Markov game framework offers a promising platform for robust control of robot manipulators in the presence of bounded external disturbances and unknown parameter variations. We propose fuzzy Markov games as an adaptation of fuzzy Q-learning (FQL) to a continuous-action variation of Markov games, wherein the reinforcement signal is used to tune online the conclusion part of a fuzzy Markov game controller. The proposed Markov game-adaptive fuzzy controller uses a simple fuzzy inference system (FIS), is computationally efficient, generates a swift control, and requires no exact dynamics of the robot system. To illustrate the superiority of Markov game-adaptive fuzzy control, we compare the performance of the controller against a) the Markov game-based robust neural controller, b) the reinforcement learning (RL)-adaptive fuzzy controller, c) the FQL controller, d) the H_infin theory-based robust neural game controller, and e) a standard RL-based robust neural controller, on two highly nonlinear robot arm control problems of i) a standard two-link rigid robot arm and ii) a 2-DOF SCARA robot manipulator. The proposed Markov game-adaptive fuzzy controller outperformed other controllers in terms of tracking errors and control torque requirements, over different desired trajectories. The results also demonstrate the viability of FISs for accelerating learning in Markov games and extending Markov game-based control to continuous state-action space problems.     

A reinforcement learning neural network for adaptive control ofMarkov chains

In this paper we consider the problem of reinforcement learning in a dynamically changing environment. In this context, we study the problem of adaptive control of finite-state Markov chains with a finite number of controls. The transition and payoff structures are unknown. The objective is to find an optimal policy which maximizes the expected total discounted payoff over the infinite horizon. A stochastic neural network model is suggested for the controller. The parameters of the neural net, which determine a random control strategy, are updated at each instant using a simple learning scheme. This learning scheme involves estimation of some relevant parameters using an adaptive critic. It is proved that the controller asymptotically chooses an optimal action in each state of the Markov chain with a high probability     

Adaptive Identification of Nonstationary Technological Processes with Markov Parameters in Stochastic Control Problems

An adaptive approach to the identification of nonstationary technological processes with Markov parameters is put forward, which relies on the use of the local identification in the problems of stochastic control. A two-loop system of adaptive control with a group of experts (decision-makers) in the outer loop is considered, which accounts for specific features of the applied field of investigations.     

Embedding adaptive JLQG into LQ martingale control with acompletely observable stochastic control matrix

With jump linear quadratic Gaussian (JLQG) control, one refers to the control under a quadratic performance criterion of a linear Gaussian system, the coefficients of which are completely observable, while they are jumping according to a finite-state Markov process. With adaptive JLQG, one refers to the more complicated situation that the finite-state process is only partially observable. Although many practically applicable results have been developed, JLQG and adaptive JLQG control are lagging behind those for linear quadratic Gaussian (LQG) and adaptive LQG. The aim of this paper is to help improve the situation by introducing an exact transformation which embeds adaptive JLQG control into LQM (linear quadratic Martingale) control with a completely observable stochastic control matrix. By LQM control, the authors mean the control of a martingale driven linear system under a quadratic performance criterion. With the LQM transformation, the adaptive JLQG control can be studied within the framework of robust or minimax control without the need for the usual approach of averaging or approximating the adaptive JLQG dynamics. To show the effectiveness of the authors' transformation, it is used to characterize the open-loop-optimal feedback (OLOF) policy for adaptive JLQG control     

Adaptive estimation of doubly stochastic poisson processes

The optimal adaptive estimator structures for a class of doubly stochastic Poisson processes (DSPP) are presented. The structure is used along with a moment assumption to obtain implementable estimators. The class of DSPP considered is that of a linear Markov diffusion process modulating a linear intensity rate. The uncertainty for which the adaptation process is developed includes both structures uncertainty in the Markov diffusion process and parameter uncertainty in the Markov diffusion process and the intensity rate process. Results are given on the problem of adaptation of which of a finite number of Markov realizations is modulating the intensity process. The nonlinear adaptive estimator structures are obtained by use of a particular theorem that yields an optimal structure for the adaptive estimator. The structure is used to obtain a quasi-optimal adaptive estimator for the problem by use of a zero third central moment assumption. The estimator structure consists of a nonlinear, nonadaptive part, and a nonlinear, adaptive part which contains the parameter structure adaptations. The necessary covariance equations for performance evaluation are obtained. The theory is applied to the problem of wavefront estimation in adaptive optics for use in high-energy lasers and in imaging through atomospheric turbulence. Other examples are given.     

Ergodic and adaptive control of nearest-neighbor motions

The self-tuning approach to adaptive control is applied to a class of Markov chains called nearest-neighbor motions. These have a countable state space and move from any state to at most finitely many neighboring states. For compact parameter and control spaces, the almost-sure optimality of the self-tuner for an ergodic cost criterion is established under two sets of assumptions.     

Aero‐Engine DCS Fault‐Tolerant Control with Markov Time Delay Based On Augmented Adaptive Sliding Mode Observer

With a focus on aero‐engine distributed control systems (DCSs) with Markov time delay, unknown input disturbance, and sensor and actuator simultaneous faults, a combined fault tolerant algorithm based on the adaptive sliding mode observer is studied. First, an uncertain augmented model of distributed control system is established under the condition of simultaneous sensor and actuator faults, which also considers the influence of the output disturbances. Second, an augmented adaptive sliding mode observer is designed and the linear matrix inequality (LMI) form stability condition of the combined closed‐loop system is deduced. Third, a robust sliding mode fault tolerant controller is designed based on fault estimation of the sliding mode observer, where the theory of predictive control is adopted to suppress the influence of random time delay on system stability. Simulation results indicate that the proposed sliding mode fault tolerant controller can be very effective despite the existence of faults and output disturbances, and is suitable for the simultaneous sensor and actuator faults condition.     

Adaptive control of Markov chains with average cost

We present an adaptive control scheme for the control of Markov chains to minimize long-run average cost when the system transition and reward structures are unknown. Q-factors are estimated along a single sample path of the system and control actions are applied based on the latest estimates. We prove that an optimal policy is obtained asymptotically with probability one. More importantly, we prove that optimal system performance is achieved as well, which means that the performance of the system can not be bettered even if the system transition and reward structures are known. An example is given to illustrate our adaptive control scheme     

Adaptive visual tracking using the prioritized Q-learning algorithm: MDP-based parameter learning approach

This paper introduces an adaptive visual tracking method that combines the adaptive appearance model and the optimization capability of the Markov decision process. Most tracking algorithms are limited due to variations in object appearance from changes in illumination, viewing angle, object scale, and object shape. This paper is motivated by the fact that tracking performance degradation is caused not only by changes in object appearance but also by the inflexible controls of tracker parameters. To the best of our knowledge, optimization of tracker parameters has not been thoroughly investigated, even though it critically influences tracking performance. The challenge is to equip an adaptive tracking algorithm with an optimization capability for a more flexible and robust appearance model. In this paper, the Markov decision process, which has been applied successfully in many dynamic systems, is employed to optimize an adaptive appearance model-based tracking algorithm. The adaptive visual tracking is formulated as a Markov decision process based dynamic parameter optimization problem with uncertain and incomplete information. The high computation requirements of the Markov decision process formulation are solved by the proposed prioritized Q-learning approach. We carried out extensive experiments using realistic video sets, and achieved very encouraging and competitive results.     

基于混合马尔科夫树模型的ICS异常检测算法

针对工业控制系统中现有异常检测算法在语义攻击检测方面存在的不足,提出一种基于混合马尔科夫树模型的异常检测算法,充分利用工业控制系统的阶段性和周期性特征,构建系统正常运行时的行为模型|混合马尔科夫树.该模型包含合法的状态事件、合法的状态转移、正常的概率分布以及正常的转移时间间隔等4种信息,基于动态自适应的方法增强状态事件的关联度并引入时间间隔信息以实现对复杂语义攻击的检测,语义建模时设计一种剪枝策略以去除模型中的低频事件、低转移事件以及冗余节点,当被检测行为使得模型的以上4种信息产生的偏差超过阈值时,判定该行为异常.最后,基于OMNeT++网络仿真环境构建一个简化的污水处理系统对本文算法进行功能性验证,并利用真实物理测试床的数据集对算法的检测准确度进行性能验证.验证结果表明,本文算法能有效消除人机交互和常规诊断等操作带来的噪声影响,对复杂语义攻击具有较高的检出率,且能识别传统的非语义攻击.     

Adaptive CUSUM procedures with Markovian mean estimation

This paper proposes using a Markovian-type mean estimating procedure in the conventional cumulative sum (CUSUM) control scheme to update its reference value in an adaptive way. This generalizes a class of Markovian adaptive CUSUM (ACUSUM) schemes to achieve the aim of providing an overall good performance over a range of future expected but unknown mean shifts. A two-dimensional Markov chain model is developed to analyze the run length performance of the new scheme. A comparison of run length performance of the proposed ACUSUM scheme and other control charts is shown favorable to the former.     

Direct Adaptive Dynamic Compensation for Minimum Phase Systems With Unknown Relative Degree

We consider parameter-monotonic direct adaptive control for single-input-single-output minimum-phase linear time-invariant systems with knowledge of the sign of the high-frequency gain (first nonzero Markov parameter) and an upper bound on the magnitude of the high-frequency gain. The first part of the paper is devoted to fixed-gain analysis of single-parameter high-gain-stabilizing controllers. Two novel fixed-gain dynamic compensators are presented for stabilizing minimum-phase systems. One compensator stabilizes systems with arbitrary-but-known relative degree, while the other utilizes a Fibonacci series construction to stabilize systems with unknown-but-bounded relative degree. Next, we provide a general treatment of parameter-monotonic adaptive control, including a result that guarantees state convergence to zero. This result is then combined with the high-gain-stabilizing controllers to yield parameter-monotonic direct adaptive dynamic compensation for minimum-phase systems with either arbitrary-but-known or unknown-but-bounded relative degree     

Monotonic convergence and robustness of higher‐order gain‐adaptive iterative learning control

For a class of repetitive linear discrete time‐invariant systems with higher relative degree, a higher‐order gain‐adaptive iterative learning control (HOGAILC) is developed while minimizing the energy increment of two adjacent tracking errors with the argument being the iteration‐time‐variable learning‐gain vector (ITVLGV). By taking advantage of rows/columns exchanging transformation of matrix, the ITVLGV is achieved in an explicit form which is dependent upon the system Markov parameters and adaptive to the iterationwise tracking‐error vector. Algebraic derivation demonstrates that the HOGAILC is strictly monotonously convergent. On the basis of the adaptive mode, a damping quasi‐HOGAILC strategy is exploited while the uncertainties of the system Markov parameters exist. Rigorous analysis delivers that the damping quasi‐scheme is strictly monotonically convergent and thus the HOGAILC mechanism is robust to a wider range of uncertainty of system parameters and the damping factor may relax the uncertainty range. Numerical simulations are made to illustrate the validity and the effectiveness.