Outliers, inliers and the generalized least trinuned squares estimator in system identification

The least trimmed squares estimator (LTS) is a well known robust estimator in terms of protecting the estimate from the outliers. Its high computational complexity is however a problem in practice. We show that the LTS estimate can be obtained by a simple algorithm with the complexity O( N In N) for large N, where N is the number of measurements. We also show that though the LTS is robust in terms of the outliers, it is sensitive to the inliers. The concept of the inliers is introduced. Moreover, the Generalized Least Trimmed Squares estimator (GLTS) together with its solution are presented that reduces the effect of both the outliers and the inliers.     

Outliers, inliers and the generalized least trimmed squares estimator in system identification

The least trimmed squares estimator (LTS) is a well known robust estinaator in terms of protecting the estimatefrom the outliers. Its high computational complexity is however a problem in practice. We show that the LTS estimate can be obtained by a simple algorithm with the complexity O( N In N) for large N, where N is the number of measurements. We also showthat though the LTS is robust in terms of the outliers, it is sensitive to the inliers. The concept of the inliers is introduced. Moreover, the Generalized Least Trimmed Squares estimator (GLTS) together with its solution are presented that reduces the effect of both the outliers and the inliers.     

Instability detection of ARMA systems based on AR system identification

The ability of parametric autoregressive (AR) system identification methods to detect the instability of an autoregressive moving average (ARMA) system of an unknown order is investigated. The collection of least squares AR estimators of various orders is shown to have the capacity to detect the instability of the underlying system. Necessary information is not the order of the system but, instead, an upper bound of the number of unstable poles with the maximal magnitude outside the unit circle.     

Least squares based iterative identification for a class of multirate systems

This paper studies modeling and identification problems for multi-input multirate systems with colored noises. The state-space models are derived for the systems with different input updating periods and furthermore the corresponding transfer functions are obtained. To solve the difficulty of identification models with unmeasurable noises terms, the least squares based iterative algorithm is presented by replacing the unmeasurable variables with their iterative estimates. Finally, the simulation results indicate that the proposed iterative algorithm has advantages over the recursive algorithms.     

An efficient hierarchical identification method for general dual-rate sampled-data systems

For the lifted input–output representation of general dual-rate sampled-data systems, this paper presents a decomposition based recursive least squares (D-LS) identification algorithm using the hierarchical identification principle. Compared with the recursive least squares (RLS) algorithm, the proposed D-LS algorithm does not require computing the covariance matrices with large sizes and matrix inverses in each recursion step, and thus has a higher computational efficiency than the RLS algorithm. The performance analysis of the D-LS algorithm indicates that the parameter estimates can converge to their true values. A simulation example is given to confirm the convergence results.     

An evolutionary algorithm for robust regression

A drawback of robust statistical techniques is the increased computational effort often needed as compared to non-robust methods. Particularly, robust estimators possessing the exact fit property are NP-hard to compute. This means that—under the widely believed assumption that the computational complexity classes NP and P are not equal—there is no hope to compute exact solutions for large high dimensional data sets. To tackle this problem, search heuristics are used to compute NP-hard estimators in high dimensions. A new evolutionary algorithm that is applicable to different robust estimators is presented. Further, variants of this evolutionary algorithm for selected estimators—most prominently least trimmed squares and least median of squares—are introduced and shown to outperform existing popular search heuristics in difficult data situations. The results increase the applicability of robust methods and underline the usefulness of evolutionary algorithms for computational statistics.     

Recursive least squares parameter identification algorithms for systems with colored noise using the filtering technique and the auxilary model

This paper focuses on the parameter estimation problems of output error autoregressive systems and output error autoregressive moving average systems (i.e., the Box–Jenkins systems). Two recursive least squares parameter estimation algorithms are proposed by using the data filtering technique and the auxiliary model identification idea. The key is to use a linear filter to filter the input–output data. The proposed algorithms can identify the parameters of the system models and the noise models interactively and can generate more accurate parameter estimates than the auxiliary model based recursive least squares algorithms. Two examples are given to test the proposed algorithms.     

A user perspective on errors-in-variables methods in system identification

The paper starts with a brief survey of errors-in-variables methods in system identification. Background and motivation are given, and it is illustrated why the identification problem can be difficult. Under general weak assumptions, the system is not identifiable, but can be parameterized using one degree of freedom. Examples where identifiability is achieved under additional assumptions are also provided. A number of approaches for parameter estimation of errors-in-variables models are reviewed. The underlying assumptions and principles for each approach are highlighted. The paper then continues by discussing from a user’s perspective on how to proceed when practical problems are handled.     

Adaptive parameter identification of linear SISO systems with unknown time-delay

An adaptive online parameter identification is proposed for linear single-input-single-output (SISO) time-delay systems to simultaneously estimate the unknown time-delay and other parameters. After representing the system as a parameterized form, a novel adaptive law is developed, which is driven by appropriate parameter estimation error information. Consequently, the identification error convergence can be proved under the conventional persistent excitation (PE) condition, which can be online tested in this paper. A finite-time (FT) identification scheme is further studied by incorporating the sliding mode scheme into the adaptation to achieve FT error convergence. The previously imposed constraint on the system relative degree is removed and the derivatives of the input and output are not required. Comparative simulation examples are provided to demonstrate the validity and efficacy of the proposed algorithms.     

Noise-induced bias in last principal component modeling of linear system

Last principal component (LPC) modeling relies on principal component transformation, and utilizes the eigenvectors associated with the last (smallest) principal components. When applied to experimental data, it may be considered an alternative to least squares based estimation of model parameters. Experimental results in the literature (cited in the body of the paper) suggest that LPC modeling is inferior to LS, in terms of estimation bias, in the presence of noise. Other results show that LPC produces unbiased estimates only in a very special case. In this paper, we derive explicit expressions for noise-induced bias in LPC-based identification. We investigate static systems with input actuator and measurement noise, and discrete dynamic systems with output measurement noise. We show that, indeed, LPC-based estimates are biased even when LS-based ones are not, and when the LS estimate is also biased, the LPC estimate has the LS bias plus an additional term. The theoretical results are supported by simulation studies.     

Errors-in-variables methods in system identification

The paper gives a survey of errors-in-variables methods in system identification. Background and motivation are given, and examples illustrate why the identification problem can be difficult. Under general weak assumptions, the systems are not identifiable, but can be parameterized using one degree-of-freedom. Examples where identifiability is achieved under additional assumptions are also provided. A number of approaches for parameter estimation of errors-in-variables models are presented. The underlying assumptions and principles for each approach are highlighted.     

Frequency domain identification of Wiener models

This paper proposes a frequency domain algorithm for Wiener model identifications based on exploring the fundamental frequency and harmonics generated by the unknown nonlinearity. The convergence of the algorithm is established in the presence of white noise. No a priori knowledge of the structure of the nonlinearity is required and the linear part can be nonparametric.     

The least squares algorithm, parametric system identification and bounded noise

The least squares parametric system identification algorithm is analyzed assuming that the noise is a bounded signal. A bound on the worst-case parameter estimation error is derived. This bound shows that the worst-case parameter estimation error decreases to zero as the bound on the noise is decreased to zero.     

A multiple-kernel LSSVR method for separable nonlinear system identification

In some nonlinear dynamic systems, the state variables function usually can be separated from the control variables function, which brings much trouble to the identification of such systems. To well solve this problem, an improved least squares support vector regression （LSSVR） model with multiple-kernel is proposed and the model is applied to the nonlinear separable system identification. This method utilizes the excellent nonlinear mapping ability of Morlet wavelet kernel function and combines the state and control variables information into a kernel matrix. Using the composite wavelet kernel, the LSSVR includes two nonlinear functions, whose variables are the state variables and the control ones respectively, in this way, the regression function can gain better nonlinear mapping ability, and it can simulate almost any curve in quadratic continuous integral space. Then, they are used to identify the two functions in the separable nonlinear dynamic system. Simulation results show that the multiple-kernel LSSVR method can greatly improve the identification accuracy than the single kernel method, and the Morlet wavelet kernel is more efficient than the other kernels.     

A cumulant based algorithm for the identification of input-output quadratic systems

The identification of a special class of polynomial models is pursued in this paper. In particular a parameter estimation algorithm is developed for the identification of an input-output quadratic model excited by a zero mean white Gaussian input and with the output corrupted by additive measurement noise. Input-output crosscumulants up to the fifth order are employed and the identification problem of the unknown model parameters is reduced to the solution of successive triangular linear systems of equations that are solved at each step of the algorithm. Simulation studies are carried out and the proposed methodology is compared with two least squares type identification algorithms, the output error method and a combination of the instrumental variables and the output error approach. The proposed cumulant based algorithm and the output error method are tested with real data produced by a robotic manipulator.     

Bayesian system identification via Markov chain Monte Carlo techniques

The work here explores new numerical methods for supporting a Bayesian approach to parameter estimation of dynamic systems. This is primarily motivated by the goal of providing accurate quantification of estimation error that is valid for arbitrary, and hence even very short length data records. The main innovation is the employment of the Metropolis-Hastings algorithm to construct an ergodic Markov chain with invariant density equal to the required posterior density. Monte Carlo analysis of samples from this chain then provides a means for efficiently and accurately computing posteriors for model parameters and arbitrary functions of them.     

Rate of convergence of the LMS method

It is shown under reasonable assumptions that for any q the L_q-norm of the estimation error generated by the LMS algorithm is of the order of magnitude , where T is the length of the observation period.     

Optimal multilinear estimation of a random vector under constraints of causality and limited memory

A new technique is provided for random vector estimation from noisy data under the constraints that the estimator is causal and dependent on at most a finite number p of observations. Nonlinear estimators defined by multilinear operators of degree r are employed, the choice of r allowing a trade-off between the accuracy of the optimal filter and the complexity of the calculations. The techniques utilise an exact correspondence of the nonlinear problem to a corresponding linear one. This is then solved by a new procedure, the least squares singular pivot algorithm, whereby the linear problem can be repeated reduced to smaller structurally similar problems. Invertibility of the relevant covariance matrices is not assumed. Numerical experiments with real data are used to illustrate the efficacy of the new algorithm.     

Analysis of the variability of joint input-output estimation methods

It has been recently established that, when estimating parametric models on the basis of closed loop data, the frequency domain variability of direct and various indirect methods may significantly differ from one another. This paper continues this work by analysing the performance of certain common joint input-output estimation methods.