Outliers, inliers and the generalized least trinuned squares estimator in system identification

The least trimmed squares estimator (LTS) is a well known robust estimator in terms of protecting the estimate from the outliers. Its high computational complexity is however a problem in practice. We show that the LTS estimate can be obtained by a simple algorithm with the complexity O( N In N) for large N, where N is the number of measurements. We also show that though the LTS is robust in terms of the outliers, it is sensitive to the inliers. The concept of the inliers is introduced. Moreover, the Generalized Least Trimmed Squares estimator (GLTS) together with its solution are presented that reduces the effect of both the outliers and the inliers.     

Outliers, inliers and the generalized least trimmed squares estimator in system identification

The least trimmed squares estimator (LTS) is a well known robust estinaator in terms of protecting the estimatefrom the outliers. Its high computational complexity is however a problem in practice. We show that the LTS estimate can be obtained by a simple algorithm with the complexity O( N In N) for large N, where N is the number of measurements. We also showthat though the LTS is robust in terms of the outliers, it is sensitive to the inliers. The concept of the inliers is introduced. Moreover, the Generalized Least Trimmed Squares estimator (GLTS) together with its solution are presented that reduces the effect of both the outliers and the inliers.     

Instability detection of ARMA systems based on AR system identification

The ability of parametric autoregressive (AR) system identification methods to detect the instability of an autoregressive moving average (ARMA) system of an unknown order is investigated. The collection of least squares AR estimators of various orders is shown to have the capacity to detect the instability of the underlying system. Necessary information is not the order of the system but, instead, an upper bound of the number of unstable poles with the maximal magnitude outside the unit circle.     

Least squares based iterative identification for a class of multirate systems

This paper studies modeling and identification problems for multi-input multirate systems with colored noises. The state-space models are derived for the systems with different input updating periods and furthermore the corresponding transfer functions are obtained. To solve the difficulty of identification models with unmeasurable noises terms, the least squares based iterative algorithm is presented by replacing the unmeasurable variables with their iterative estimates. Finally, the simulation results indicate that the proposed iterative algorithm has advantages over the recursive algorithms.     

An evolutionary algorithm for robust regression

A drawback of robust statistical techniques is the increased computational effort often needed as compared to non-robust methods. Particularly, robust estimators possessing the exact fit property are NP-hard to compute. This means that—under the widely believed assumption that the computational complexity classes NP and P are not equal—there is no hope to compute exact solutions for large high dimensional data sets. To tackle this problem, search heuristics are used to compute NP-hard estimators in high dimensions. A new evolutionary algorithm that is applicable to different robust estimators is presented. Further, variants of this evolutionary algorithm for selected estimators—most prominently least trimmed squares and least median of squares—are introduced and shown to outperform existing popular search heuristics in difficult data situations. The results increase the applicability of robust methods and underline the usefulness of evolutionary algorithms for computational statistics.     

An efficient hierarchical identification method for general dual-rate sampled-data systems

For the lifted input–output representation of general dual-rate sampled-data systems, this paper presents a decomposition based recursive least squares (D-LS) identification algorithm using the hierarchical identification principle. Compared with the recursive least squares (RLS) algorithm, the proposed D-LS algorithm does not require computing the covariance matrices with large sizes and matrix inverses in each recursion step, and thus has a higher computational efficiency than the RLS algorithm. The performance analysis of the D-LS algorithm indicates that the parameter estimates can converge to their true values. A simulation example is given to confirm the convergence results.     

Errors-in-variables methods in system identification

The paper gives a survey of errors-in-variables methods in system identification. Background and motivation are given, and examples illustrate why the identification problem can be difficult. Under general weak assumptions, the systems are not identifiable, but can be parameterized using one degree-of-freedom. Examples where identifiability is achieved under additional assumptions are also provided. A number of approaches for parameter estimation of errors-in-variables models are presented. The underlying assumptions and principles for each approach are highlighted.     

The least squares algorithm, parametric system identification and bounded noise

The least squares parametric system identification algorithm is analyzed assuming that the noise is a bounded signal. A bound on the worst-case parameter estimation error is derived. This bound shows that the worst-case parameter estimation error decreases to zero as the bound on the noise is decreased to zero.     

Frequency domain identification of Wiener models

This paper proposes a frequency domain algorithm for Wiener model identifications based on exploring the fundamental frequency and harmonics generated by the unknown nonlinearity. The convergence of the algorithm is established in the presence of white noise. No a priori knowledge of the structure of the nonlinearity is required and the linear part can be nonparametric.     

A cumulant based algorithm for the identification of input-output quadratic systems

The identification of a special class of polynomial models is pursued in this paper. In particular a parameter estimation algorithm is developed for the identification of an input-output quadratic model excited by a zero mean white Gaussian input and with the output corrupted by additive measurement noise. Input-output crosscumulants up to the fifth order are employed and the identification problem of the unknown model parameters is reduced to the solution of successive triangular linear systems of equations that are solved at each step of the algorithm. Simulation studies are carried out and the proposed methodology is compared with two least squares type identification algorithms, the output error method and a combination of the instrumental variables and the output error approach. The proposed cumulant based algorithm and the output error method are tested with real data produced by a robotic manipulator.     

Rate of convergence of the LMS method

It is shown under reasonable assumptions that for any q the L_q-norm of the estimation error generated by the LMS algorithm is of the order of magnitude , where T is the length of the observation period.     

Optimal multilinear estimation of a random vector under constraints of causality and limited memory

A new technique is provided for random vector estimation from noisy data under the constraints that the estimator is causal and dependent on at most a finite number p of observations. Nonlinear estimators defined by multilinear operators of degree r are employed, the choice of r allowing a trade-off between the accuracy of the optimal filter and the complexity of the calculations. The techniques utilise an exact correspondence of the nonlinear problem to a corresponding linear one. This is then solved by a new procedure, the least squares singular pivot algorithm, whereby the linear problem can be repeated reduced to smaller structurally similar problems. Invertibility of the relevant covariance matrices is not assumed. Numerical experiments with real data are used to illustrate the efficacy of the new algorithm.     

Bayesian system identification via Markov chain Monte Carlo techniques

The work here explores new numerical methods for supporting a Bayesian approach to parameter estimation of dynamic systems. This is primarily motivated by the goal of providing accurate quantification of estimation error that is valid for arbitrary, and hence even very short length data records. The main innovation is the employment of the Metropolis-Hastings algorithm to construct an ergodic Markov chain with invariant density equal to the required posterior density. Monte Carlo analysis of samples from this chain then provides a means for efficiently and accurately computing posteriors for model parameters and arbitrary functions of them.     

Robust maximum-likelihood estimation of multivariable dynamic systems

This paper examines the problem of estimating linear time-invariant state-space system models. In particular, it addresses the parametrization and numerical robustness concerns that arise in the multivariable case. These difficulties are well recognised in the literature, resulting (for example) in extensive study of subspace-based techniques, as well as recent interest in ‘data driven’ local co-ordinate approaches to gradient search solutions. The paper here proposes a different strategy that employs the expectation-maximisation (EM) technique. The consequence is an algorithm that is iterative, with associated likelihood values that are locally convergent to stationary points of the (Gaussian) likelihood function. Furthermore, theoretical and empirical evidence presented here establishes additional attractive properties such as numerical robustness, avoidance of difficult parametrization choices, the ability to naturally and easily estimate non-zero initial conditions, and moderate computational cost. Moreover, since the methods here are maximum-likelihood based, they have associated known and asymptotically optimal statistical properties.     

On the frequency domain accuracy of closed-loop estimates

It has been argued that the frequency domain accuracy of high model-order estimates obtained on the basis of closed-loop data is largely invariant to whether direct or indirect approaches are used. The analysis underlying this conclusion has employed variance expressions that are asymptotic both in the data length and the model order, and hence are approximations when either of these are finite. However, recent work has provided variance expressions that are exact for finite (possibly low) model order, and hence can potentially deliver more accurate quantification of estimation accuracy. This paper, and a companion one, revisits the study of identification from closed-loop data in light of these new quantifications and establishes that, under certain assumptions, there can be significant differences in the accuracy of frequency response estimates. These discrepencies are established here and in the companion paper to be dependent on what type of direct, indirect or joint input-output identification strategy is pursued.     

A blind approach to the Hammerstein-Wiener model identification

In this paper, we propose a blind approach to the sampled Hammerstein-Wiener model identification. By using the blind approach, it is shown that all internal variables can be recovered solely based on the output measurements. Then, identification of linear and nonlinear parts can be carried out. No a priori structural knowledge about the input nonlinearity is assumed and no white noise assumption is imposed on the input.     

A self-organizing random immigrants genetic algorithm for dynamic optimization problems

In this paper a genetic algorithm is proposed where the worst individual and individuals with indices close to its index are replaced in every generation by randomly generated individuals for dynamic optimization problems. In the proposed genetic algorithm, the replacement of an individual can affect other individuals in a chain reaction. The new individuals are preserved in a subpopulation which is defined by the number of individuals created in the current chain reaction. If the values of fitness are similar, as is the case with small diversity, one single replacement can affect a large number of individuals in the population. This simple approach can take the system to a self-organizing behavior, which can be useful to control the diversity level of the population and hence allows the genetic algorithm to escape from local optima once the problem changes due to the dynamics.     

Accuracy analysis of bias-eliminating least squares estimates for errors-in-variables systems

The bias-eliminating least squares (BELS) method is one of the consistent estimators for identifying dynamic errors-in-variables systems. In this paper, we investigate the accuracy properties of the BELS estimates. An explicit expression for the normalized asymptotic covariance matrix of the estimated parameters is derived and supported by some numerical examples.