Estimation of convergence rate for multi-regression learning algorithm

In many applications, the pre-information on regression function is always unknown. Therefore, it is necessary to learn regression function by means of some valid tools. In this paper we investigate the regression problem in learning theory, i.e., convergence rate of regression learning algorithm with least square schemes in multi-dimensional polynomial space. Our main aim is to analyze the generalization error for multi-regression problems in learning theory. By using the famous Jackson operators in approximation theory, covering number, entropy number and relative probability inequalities, we obtain the estimates of upper and lower bounds for the convergence rate of learning algorithm. In particular, it is shown that for multi-variable smooth regression functions, the estimates are able to achieve almost optimal rate of convergence except for a logarithmic factor. Our results are significant for the research of convergence, stability and complexity of regression learning algorithm.     

Smooth sliding-mode observers for specific growth rate and substrate from biomass measurement

This paper addresses the estimation of specific growth rate and substrate concentration from biomass measurements in fermentation processes. Specifically, sliding-mode observers are proposed, for which finite-time global convergence is demonstrated using Lyapunov stability theory and concepts of variable structure systems. Two observers are developed for specific growth rate estimation, one producing a discontinuous estimation which is used afterwards for substrate estimation, and the other one – based on high-gain observers – that generates a smooth estimation with first-order dynamics and finite-time bounded convergence error. In the case of substrate estimation, an observer that increases the convergence rate to a vicinity of the real substrate concentration while achieving asymptotic convergence despite kinetic model uncertainties in properly excited processes is designed. This observer also exhibits first-order dynamics.     

多元回归学习算法收敛速度的估计

在许多应用中,回归函数的先验信息往往不能事先获取.因此,有必要利用有效的方法学习回归函数.本文研究学习理论中的回归问题,即研究多项式空间上具有最小二乘平方损失正则学习算法的收敛速度问题.主要目的在于分析学习理论中多维回归问题的泛化误差.利用逼近论中著名Jackson算子、覆盖数理论、集合的熵数以及有关概率不等式,得到学习算法收敛速度的上、下界估计.特别地,对于满足一定条件的多元光滑回归函数,除一个对数因子外,所获的收敛速度是最优的.本文结果对研究回归学习算法的收敛性、稳定性及复杂性等有着重要的意义.     

反馈非线性系统随机梯度辨识算法及其收敛性

反馈非线性受控自回归系统是由前向通道的受控自回归模型和反馈通道的静态非线性构成, 这类系统经过参数化后得到双线性参数辨识模型. 本文通过对辨识模型中双线性参数乘积项进行分解, 基于梯度搜索原理, 提 出了反馈非线性系统的随机梯度辨识算法. 为了改善随机梯度算法的收敛速度, 引入遗忘因子, 文章给出了遗忘因子随机梯度算法, 利用随机过程理论, 建立了随机梯度算法的参数估计收敛定理, 证明了算法的收敛性. 最后, 通过数值仿真验证了算法的有效性.     

Convergence of the generalized dual control algorithm

In this paper the convergence properties of the generalized dual controller proposed by Chan and Zarrop (1985;) are analysed. Using the martingale convergence theory, we present solutions for global stability of the algorithm, strong consistency of the estimates, and their convergence rate. It is demonstrated that the passivity of two time-varying operators, and strict passivity of at least one of them, is sufficient for global stability of the algorithm. In the context of the attenuating excitation technique, we show that under certain conditions related to the properties of the optimal regulator and order of the adaptive regulator, the regression vector satisfies the persistency exciting condition. Bearing this in mind, we derive a relation describing the convergence rate of the parameter estimates.     

Identification of discrete Hammerstein systems using kernel regression estimates

In this note a discrete-time Hammerstein system is identified. The weighting function of the dynamical subsystem is recovered by the correlation method. The main results concern estimation of the nonlinear memoryless subsystem. No conditions concerning functional form of the transform characteristic of the subsystem are made and an algorithm for estimation of the characteristic is presented.The algorithm is a nonparametric kernel estimate of regression functions calculated from dependent data. It is shown that the algorithm converges to the characteristic as the number of observations tend to infinity. For sufficiently smooth characteristics, the rate of convergence isO(n^{-2/5})in probability.     

非线性多功能传感器信号重构的抗差估计方法研究

在非线性多功能传感器的信号重构过程中,训练样本集不可避免地夹杂粗差数据。为了得到既有较强抗差性又有较高效率的估值,采用抗差最小二乘法在粗差干扰条件下进行参数估计。抗差最小二乘估计通过等价权把抗差估计原理与加权最小二乘形式结合在一起,因此在抵抗粗差的同时保持了最小二乘法的优点。非线性信号重构的粗差抑制结果表明,抗差最小二乘法计算速度快,且具有良好的抗粗差能力和收敛可靠性。     

Simulation-based two-stage estimation for multiple nonparametric regression

To reduce the curse of dimensionality arising from nonparametric estimation procedures for multiple nonparametric regression, in this paper we suggest a simulation-based two-stage estimation. We first introduce a simulation-based method to decompose the multiple nonparametric regression into two parts. The first part can be estimated with the parametric convergence rate and the second part is small enough so that it can be approximated by orthogonal basis functions with a small trade-off parameter. Then the linear combination of the first and second step estimators results in a two-stage estimator for the multiple regression function. Our method does not need any specified structural assumption on the regression function and it is proved that the newly proposed estimation is always consistent even if the trade-off parameter is designed to be small. Thus when the common nonparametric estimator such as local linear smoothing collapses because of the curse of dimensionality, our estimator still works well.     

New adaptive vector control methods for induction motors with simpler structure and better performance

This paper deals with the vector control, including both the direct vector control(DVC) and the indirect vector control(Id VC),of induction motors. It is well known that the estimation of rotor flux plays a fundamental role in the DVC and the estimation of rotor resistance is vital in the slip compensation of the Id VC. In these estimations, the precision is significantly affected by the motor resistances. Therefore, online estimation of motor resistances is indispensable in practice.For a fast estimation of motor resistances, it is necessary to slow down the convergence rate of the current estimate. On the other hand, for a fast estimation of the rotor flux, it is necessary to speed up its convergence rate. It is very difficult to realize such a trade-off in convergence rates in a full order observer.In this paper, we propose to decouple the current observer from the flux observer so as to realize independent convergence rates. Then, the resistance estimation algorithm is applied to both DVC and Id VC. In particular, in the application to Id VC the flux observer needs not be used, which leads to a simpler structure. Meanwhile, independent convergence rates of current observer and flux observer yield an improved performance. A superior performance in the torque and flux responses in both cases is verified by numerous simulations.     

Asymptotic efficiency of kernel support vector machines (SVM)

The paper analyzes the asymptotic properties of Vapnik’s SVM-estimates of a regression function as the size of the training sample tends to infinity. The estimation problem is considered as infinite-dimensional minimization of a regularized empirical risk functional in a reproducing kernel Hilbert space. The rate of convergence of the risk functional on SVM-estimates to its minimum value is established. The sufficient conditions for the uniform convergence of SVM-estimates to a true regression function with unit probability are given. Translated from Kibernetika i Sistemnyi Analiz, No. 4, pp. 81–97, July–August 2009     

Recursive parameter estimation methods and convergence analysis for a special class of nonlinear systems

This paper is concerned with the joint estimation of states and parameters of a special class of nonlinear systems, ie, bilinear systems. The key is to investigate new estimation methods for interactive state and parameter estimation of the considered system based on the interactive estimation theory. Because the system states are unknown, a bilinear state observer is established based on the Kalman filtering principle. Then, the unavailable states are updated by the state observer outputs recursively. Once the state estimates are obtained, the bilinear state observer–based hierarchical stochastic gradient algorithm is developed by using the gradient search. For the purpose of improving the convergence rate and the parameter estimation accuracy, a bilinear state observer–based hierarchical multi‐innovation stochastic gradient algorithm is proposed by expanding a scalar innovation to an innovation vector. The convergence analysis indicates that the parameter estimates can converge to their true values. The numerical example illustrates the effectiveness of the proposed algorithms.     

基于改进差分进化算法的非线性系统模型参数辨识

针对非线性模型的参数估计寻优较为困难的问题,提出一种基于改进的差分进化算法的非线性系统模型参数辨识新方法。通过引入一个自适应变异率,随着迭代的进行自适应调整缩放因子,从而在初期保持种群多样性以避免早熟,并在后期逐步降低变异率,保留优良信息,避免最优解遭到破坏。交叉概率采用动态非线性增加的方法,提高了收敛速度。为了验证算法性能,针对几类典型的非线性模型参数辨识问题进行了仿真研究,并将其应用于一类发酵动力学模型参数的估计中。结果表明改进算法的参数辨识精度高,收敛速度也比较快,有效提高了模型建立的精度与效率,为解决实际系统中参数估计问题提供了一条可行的途径。     

How exciting can a signal really be?

The rate of parameter convergence in a number of adaptive estimation schemes is related to the smallest eigenvalue of the average information matrix determined by the regression vector. Using a very simple examples, we illustrate that the input signals that maximize this minimum eigenvalue may be quite different from the input signals that optimize more classical input design criteria, e.g. D-optimal criterion.     

Elementwise decoupling and convergence of the Riccati equation in the SG algorithm

It is shown that the difference Riccati equation of the Stenlund-Gustafsson (SG) algorithm for estimation of linear regression models can be solved elementwise. Convergence estimates for the elements of the solution to the Riccati equation are provided, directly relating convergence rate to the signal-to-noise ratio in the regression model. It is demonstrated that the elements of the solution lying in the direction of excitation exponentially converge to a stationary point while the other elements experience bounded excursions around their current values.     

Design of Asymptotically Convergent Frequency Estimator Using Contraction Theory

The problem of estimating unknown frequencies of a sinusoidal signal simultaneously is a classical problem in signal and system theory. Many approaches and algorithms are proposed in literature to develop estimators for a measurable sinusoidal signal having multiple sinusoids with unknown amplitudes, frequencies and phases. In this work, an asymptotically convergent frequency estimator is given for estimation of n-unknown frequencies of a measurable sinusoidal signal. The contraction theory approach is adopted to show the asymptotic convergence of the proposed estimator in quite a simplified manner. Approach given here exploits the results of contraction theory related to semi-contracting systems. The nonlinear estimator based on dynamic system approach, guarantees global boundedness and convergence of the state and frequency estimation for all initial conditions and frequency values. It further ensures simultaneous globally convergent estimation of states and frequencies of a sinusoid involving multiple frequencies. Numerical simulations are presented for different combinations of frequencies to justify the claim.     

A three-dimensional robust nonlinear terminal guidance law with ISS finite-time convergence

This paper presents a novel three-dimensional nonlinear terminal guidance law with finite-time convergence for intercepting manoeuvring targets. Different from the usual method of decoupling the missile-target relative motion into two-dimensional planes, this law is designed via using the coupled dynamics. The guidance law is derived based on the theory of finite-time input-to-state stability (ISS), which needs no assumption of the linearisation and the estimation of target accelerations. Under this law, the line-of-sight angular rates can be stabilised to a small domain of convergence around zero in finite time. The convergence rate and convergence domain can be adjusted by changing the guidance parameters. First, a sufficient condition on finite-time ISS of the guidance system is given, and is subsequently used to design the guidance law. Finally, simulation results are provided to show that the proposed guidance law possesses fast convergence rate and strong robustness to target manoeuvres.     

Set-valued observer design for a class of uncertain linear systems with persistent disturbance and measurement noise

In this paper, a class of linear systems affected by parameter variations, additive noise and persistent disturbances is considered. The problem of designing a set-valued state observer, which estimates a region containing the real state for each time instant, is investigated. The techniques for designing the observer are based on positive invariant set theory. By constructing a set-induced Lyapunov function, it is shown that the estimation error converges exponentially to a given compact set with an assigned rate of convergence.     

基于数据滤波的随机梯度辨识方法

针对有色噪声干扰下的随机系统,利用数据滤波技术,对输入输出数据进行滤波,将具有滑动平均噪声的原始系统转换为白噪声干扰下的系统,提出有限脉冲响应滑动平均系统的滤波增广随机梯度算法,并对该算法进行收敛性分析.此外,为了提高参数估计的精度和加快算法的收敛速度,使用多新息辨识理论提出滤波多新息增广随机梯度算法,并分析其收敛性.与增广随机梯度算法相比,所提出的滤波增广随机梯度算法和滤波多新息增广随机梯度算法可以得到更高精度的参数估计.最后,通过仿真实例表明了所提出算法的有效性.     

辅助变量最小二乘辨识的均方收敛性

利用随机过程理论,首次证明了递推辅助变量最小二乘(RIVLS)的收敛性,研究了RIVLS算法的收敛速率,给出估算RIVLS算法均方参数估计误差上界的计算公式.分析表明,当辅助矩阵与信息矩阵的乘积是非奇异阵,且关于辅助向量的弱持续激励条件成立时,均方参数估计误差以(1/t)的速率收敛于零.这一研究结果对于提高RIVLS算法的实际应用效果具有重要意义.数字仿真例子表明了该结论的正确性.     

基于AdaBelief的Heavy-Ball动量方法

同时使用动量和自适应步长技巧的自适应矩估计(Adaptive Moment Estimation,Adam)型算法广泛应用于深度学习中.针对此方法不能同时在理论和实验上达到最优这一问题,文中结合AdaBelief灵活调整步长提高实验性能的技巧,以及仅采用指数移动平均(Exponential Moving Average,EMA)策略调整步长的Heavy-Ball动量方法加速收敛的优点,提出基于AdaBelief的Heavy-Ball动量方法.借鉴AdaBelief和Heavy-Ball动量方法收敛性分析的技巧,巧妙选取时变步长、动量系数,并利用添加动量项和自适应矩阵的方法,证明文中方法对于非光滑一般凸优化问题具有最优的个体收敛速率.最后,在凸优化问题和深度神经网络上的实验验证理论分析的正确性,并且证实文中方法可在理论上达到最优收敛性的同时提高性能.