Outliers, inliers and the generalized least trimmed squares estimator in system identification

The least trimmed squares estimator (LTS) is a well known robust estinaator in terms of protecting the estimatefrom the outliers. Its high computational complexity is however a problem in practice. We show that the LTS estimate can be obtained by a simple algorithm with the complexity O( N In N) for large N, where N is the number of measurements. We also showthat though the LTS is robust in terms of the outliers, it is sensitive to the inliers. The concept of the inliers is introduced. Moreover, the Generalized Least Trimmed Squares estimator (GLTS) together with its solution are presented that reduces the effect of both the outliers and the inliers.     

A random least-trimmed-squares identification algorithm

The Least-trimmed-squares (LTS) estimator is a well known robust estimator in terms of protecting the estimate from the outliers. Its high computational complexity is however a problem in practice. In this paper, we propose a random LTS algorithm which has a low computational complexity that can be calculated a priori as a function of the required error bound and the confidence interval. Moreover, if the number of data points goes to infinite, the algorithm becomes a deterministic one that converges to the true LTS in some probability sense.     

基于鲁棒学习的最小二乘支持向量机及其应用

鉴于最小二乘支持向量机比标准支持向量机具有更高的计算效率和拟合精度,但缺少标准支持向量机的鲁棒性,即当采样数据存在奇异点或者误差变量的高斯分布假设不成立时,会导致不稳健的估计结果,提出了一种鲁棒最小二乘支持向量机方法.该方法在最小二乘支持向量机基础上,通过引入鲁棒学习方法来获得鲁棒估计.仿真分析及某湿法冶金厂的应用实例验证了该方法的可行性和有效性.     

The least squares algorithm, parametric system identification and bounded noise

The least squares parametric system identification algorithm is analyzed assuming that the noise is a bounded signal. A bound on the worst-case parameter estimation error is derived. This bound shows that the worst-case parameter estimation error decreases to zero as the bound on the noise is decreased to zero.     

Recursive least squares parameter identification algorithms for systems with colored noise using the filtering technique and the auxilary model

This paper focuses on the parameter estimation problems of output error autoregressive systems and output error autoregressive moving average systems (i.e., the Box–Jenkins systems). Two recursive least squares parameter estimation algorithms are proposed by using the data filtering technique and the auxiliary model identification idea. The key is to use a linear filter to filter the input–output data. The proposed algorithms can identify the parameters of the system models and the noise models interactively and can generate more accurate parameter estimates than the auxiliary model based recursive least squares algorithms. Two examples are given to test the proposed algorithms.     

Robust least squares methods under bounded data uncertainties

We study the problem of estimating an unknown deterministic signal that is observed through an unknown deterministic data matrix under additive noise. In particular, we present a minimax optimization framework to the least squares problems, where the estimator has imperfect data matrix and output vector information. We define the performance of an estimator relative to the performance of the optimal least squares (LS) estimator tuned to the underlying unknown data matrix and output vector, which is defined as the regret of the estimator. We then introduce an efficient robust LS estimation approach that minimizes this regret for the worst possible data matrix and output vector, where we refrain from any structural assumptions on the data. We demonstrate that minimizing this worst-case regret can be cast as a semi-definite programming (SDP) problem. We then consider the regularized and structured LS problems and present novel robust estimation methods by demonstrating that these problems can also be cast as SDP problems. We illustrate the merits of the proposed algorithms with respect to the well-known alternatives in the literature through our simulations.     

Accuracy analysis of bias-eliminating least squares estimates for errors-in-variables systems

The bias-eliminating least squares (BELS) method is one of the consistent estimators for identifying dynamic errors-in-variables systems. In this paper, we investigate the accuracy properties of the BELS estimates. An explicit expression for the normalized asymptotic covariance matrix of the estimated parameters is derived and supported by some numerical examples.     

On strong consistency of least squares identification algorithms

In this paper almost sure convergence results are derived for least squares identification algorithms. The convergence conditions expressed in terms of the measurable signal model states derived for asymptotically stable signal models and possibly nonstationary processes are in essence the same as those previously given, but are derived more directly. Strong consistency results are derived for the case of signal models with unstable modes and exponential rates of convergence to the unstable modes are demonstrated. These latter convergence results are stronger than those earlier ones in which weak consistency conditions are given and there is also less restriction on the noise disturbances than in earlier theories. The derivations in the paper appeal to martingale convergence theorems and the Toeplitz lemma.     

Least squares based iterative identification for a class of multirate systems

This paper studies modeling and identification problems for multi-input multirate systems with colored noises. The state-space models are derived for the systems with different input updating periods and furthermore the corresponding transfer functions are obtained. To solve the difficulty of identification models with unmeasurable noises terms, the least squares based iterative algorithm is presented by replacing the unmeasurable variables with their iterative estimates. Finally, the simulation results indicate that the proposed iterative algorithm has advantages over the recursive algorithms.     

On parameter convergence in least squares identification and adaptive control

Least squares estimation is appealing in performance and robustness improvements of adaptive control. A strict condition termed persistent excitation (PE) needs to be satisfied to achieve parameter convergence in least squares estimation. This paper proposes a least squares identification and adaptive control strategy to achieve parameter convergence without the PE condition. A modified modeling error that utilizes online historical data together with instant data is constructed as additional feedback to update parameter estimates, and an integral transformation is introduced to avoid the time derivation of plant states in the modified modeling error. On the basis of these results, a regressor filtering–free least squares estimation law is proposed to guarantee exponential parameter convergence by an interval excitation condition, which is much weaker than the PE condition. And then, an identification‐based indirect adaptive control law is proposed to establish exponential stability of the closed‐loop system under the interval excitation condition. Illustrative results considering both identification and control problems have verified the effectiveness and superiority of the proposed approach.     

Extended Kalman filtering and weighted least squares dynamic identification of robot

This paper presents an experimental comparison between the weighted least squares (WLS) estimation and the extended Kalman filtering (EKF) methods for robot dynamic identification. Comparative results and discussion are presented for a SCARA robot, depending on a priori knowledge and data filtering.     

Detection techniques in least squares identification

Parameter estimation schemes based on least squares identification and detection ideas are proposed for ease of computation, reduced numerical difficulties, and bias reduction in the presence of colored noise correlated with the states of the signal generating system. The algorithms are simpler because in the calculations, the state vector is at one point replaced by a quantized version. This technique avoids to some extent numerical difficulties associated with ill-conditioning in least squares schemes and thus obviates the need for square root algorithms and the need for high order precision calculations. In recursive form, the schemes are designed to yield parameter estimates with negligible bias without the additional computational effort or instability risks associated with generalized and extended least squares, recursive maximum likelihood schemes, or the method of instrumental variables. Nonrecursive schemes are designed to minimize computational effort in a batch processing situation while at the same time giving some reduction of bias in the state dependent colored noise situation.

The novel algorithms have the limitation that they are suboptimal and there is thus a consequent reduction in the speed of convergence for some applications. The merits of the proposed schemes are assessed via simulation studies in this paper and an adaptive equalization application in a companion paper.     

A continuous-time framework for least squares parameter estimation

This paper proposes a continuous-time framework for the least-squares parameter estimation method through evolution equations. Nonlinear systems in the standard state space representation that are linear in the unknown, constant parameters are investigated. Two estimators are studied. The first one consists of a linear evolution equation while the second one consists of an impulsive linear evolution equation. The paper discusses some theoretical aspects related to the proposed estimators: uniqueness of a solution and an attractive equilibrium point which solves for the unknown parameters. A deterministic framework for the estimation under noisy measurements is proposed using a Sobolev space with negative index to model the noise. The noise can be of large magnitude. Concrete signals issued from an electronic device are used to discuss numerical aspects.     

Instability detection of ARMA systems based on AR system identification

The ability of parametric autoregressive (AR) system identification methods to detect the instability of an autoregressive moving average (ARMA) system of an unknown order is investigated. The collection of least squares AR estimators of various orders is shown to have the capacity to detect the instability of the underlying system. Necessary information is not the order of the system but, instead, an upper bound of the number of unstable poles with the maximal magnitude outside the unit circle.     

Direct and indirect least squares methods in continuous-time parameter estimation

The discrete-time least squares approach is extended to the estimation of parameters in continuous nonlinear models. The resulting direct integral least squares (DILS) method is both simple and numerically efficient and it usually improves the mean-squared error of the estimates compared with the conventional indirect least squares (ILS) method. The biasedness of the DILS estimates may become serious if the sample points are widely spaced in time and/or the signal-to-noise ratio is low and so a continuous-time symmetric bootstrap (SB) estimator which removes this problem is described. The DILS, SB and ILS methods form a three-stage procedure combining the robustness and numerical efficiency of direct methods with the asymptotic unbiasedness of ILS procedures.     

Computing the least quartile difference estimator in the plane

A common problem in linear regression is that largely aberrant values can strongly influence the results. The least quartile difference (LQD) regression estimator is highly robust, since it can resist up to almost 50% largely deviant data values without becoming extremely biased. Additionally, it shows good behavior on Gaussian data—in contrast to many other robust regression methods. However, the LQD is not widely used yet due to the high computational effort needed when using common algorithms. It is shown that it is possible to compute the LQD estimator for n bivariate data points in expected running time O(n²logn) or deterministic running time . Additionally, two easy to implement algorithms with slightly inferior time bounds are presented. All of these algorithms are also applicable to least quantile of squares and least median of squares regression through the origin, improving the known time bounds to expected time O(nlogn) and deterministic time . The proposed algorithms improve on known results of existing LQD algorithms and hence increase the practical relevance of the LQD estimator.     

On noncausal weighted least squares identification of nonstationary stochastic systems

In this paper, we consider the problem of noncausal identification of nonstationary, linear stochastic systems, i.e., identification based on prerecorded input/output data. We show how several competing weighted (windowed) least squares parameter smoothers, differing in memory settings, can be combined together to yield a better and more reliable smoothing algorithm. The resulting parallel estimation scheme automatically adjusts its smoothing bandwidth to the unknown, and possibly time-varying, rate of nonstationarity of the identified system. We optimize the window shape for a certain class of parameter variations and we derive computationally attractive recursive smoothing algorithms for such an optimized case.     

基于辅助模型的递推增广最小二乘辨识方法

针对有色噪声干扰的输出误差滑动平均系统, 将辅助模型与递推增广最小二乘算法相结合: 用辅助模型的输出代替辨识模型信息向量中的未知真实输出项, 用估计残差代替信息向量中的不可测噪声项, 从而提出了基于辅助模型的递推增广最小二乘辨识方法. 为了展示所提方法的特点, 文中还给出了经过模型变换的递推增广最小二乘算法. 理论分析和仿真研究表明, 提出的方法原理简单、计算量小, 可以给出高精度参数估计, 且能够用于在线辨识.     

Matrix strategies for computing the least trimmed squares estimation of the general linear and SUR models

An algorithm for computing the exact least trimmed squares (LTS) estimator of the standard regression model has recently been proposed. The LTS algorithm is adapted to the general linear and seemingly unrelated regressions models with possible singular dispersion matrices. It searches through a regression tree to find the optimal estimates and has combinatorial complexity. The model is formulated as a generalized linear least squares problem. Efficient matrix techniques are employed to update the generalized residual sum of squares of a subset model. Specifically, the new algorithm utilizes previous computations to update a generalized QR decomposition by a single row. The sparse structure of the model is exploited. Theoretical measures of computational complexity are provided. Experimental results confirm the ability of the new algorithms to identify outlying observations.     

Decomposition-based recursive least squares identification methods for multivariate pseudo-linear systems using the multi-innovation

This paper studies the parameter estimation algorithms of multivariate pseudo-linear autoregressive systems. A decomposition-based recursive generalised least squares algorithm is deduced for estimating the system parameters by decomposing the multivariate pseudo-linear autoregressive system into two subsystems. In order to further improve the parameter accuracy, a decomposition based multi-innovation recursive generalised least squares algorithm is developed by means of the multi-innovation theory. The simulation results confirm that these two algorithms are effective.