Accuracy analysis of a covariance matching approach for identifying errors-in-variables systems

A system identification method for errors-in-variables problems based on covariance matching was recently proposed. In the first step, a small amount of covariances of noisy input–output data are computed, and then a parametric model is fitted to these covariances. In this paper, the method is further analyzed and the asymptotic accuracy of the parameter estimates is derived. An explicit algorithm for computing the asymptotic covariance matrix of the parameter estimates is given, and the identification method is shown to be asymptotically statistically efficient assuming that the given information is the computed covariances. As an important byproduct, an efficient algorithm is presented for computing the covariance matrix of the computed input–output covariances.     

Frequency-domain subspace system identification using non-parametric noise models

In the general case of non-uniformly spaced frequency-domain data and/or arbitrarily coloured disturbing noise, the frequency-domain subspace identification algorithms described in McKelvey, Akçay, and Ljung (IEEE Trans. Automatic Control 41(7) (1996) 960) and Van Overschee and De Moor (Signal Processing 52(2) (1996) 179) are consistent only if the covariance matrix of the disturbing noise is known. This paper studies the asymptotic properties (strong convergence, convergence rate, asymptotic normality, strong consistency and loss in efficiency) of these algorithms when the true noise covariance matrix is replaced by the sample noise covariance matrix obtained from a small number of independent repeated experiments. As an additional result the strong convergence (in case of model errors), the convergence rate and the asymptotic normality of the subspace algorithms with known noise covariance matrix follows.     

Variance results for identification of cascade systems

The objective of this contribution is to analyze statistical properties of estimated models of cascade systems. Models of such systems are important in for example cascade control applications. The aim is to present and analyze some fundamental limitations in the quality of an identified model of a cascade system under the condition that the true subsystems have certain common dynamics. The model quality is analyzed by studying the asymptotic (large data) covariance matrix of the Prediction Error Method parameter estimate. The analysis will focus on cascade systems with three subsystems. The main result is that if the true transfer functions of the first and second subsystem are identical, the output signal information from the second and third subsystems will not affect the asymptotic variance of the estimated model of the first subsystem. This result implies that for a cascade system with two subsystems, where the dynamics of the first subsystem is a factor of the dynamics of the second one, the output signal information from the second subsystem will not improve the asymptotic quality of the estimate of the first subsystem. The results are illustrated by some simple FIR examples.     

On the accuracy of a covariance matching method for continuous-time errors-in-variables identification

An analysis of a covariance matching method for continuous-time errors-in-variables system identification from discrete-time data is made. In the covariance matching method, the noise-free input signal is not explicitly modeled and only assumed to be a stationary process. The asymptotic normalized covariance matrix, valid for a large number of data and a small sampling interval, is derived. This involves the evaluation of a covariance matrix of estimated covariance elements and estimated derivatives of such elements, and large parts of the paper are devoted to this task. The latter covariance matrix consists of two parts, where the first part contains integrals that are approximations of Riemann sums, and the second part depends on the measurement noise variances.     

Stochastic system identification with noisy input-outputmeasurements using polyspectra

Two new classes of parametric, frequency domain approaches are proposed for estimation of the parameters of scalar, linear “errors-in-variables” models, i.e., linear systems where measurements of both input and output of the system are noise contaminated. The first approach consists of linear estimators where using the bispectrum or the integrated polyspectrum of the input and the cross-bispectrum or the integrated cross-polyspectrum, the system transfer function is first estimated at a number of frequencies exceeding one-half the number of unknown parameters. The estimated transfer function is then used to estimate the unknown parameters using an overdetermined linear system of equations. In the second class of approaches, quadratic transfer function matching criteria are optimized by using the results of the linear estimators as initial guesses. Both classes of the parameter estimators are shown to be consistent in any measurement noise that has symmetric probability density function when the bispectral approaches are used. The proposed parameter estimators are shown to be consistent in Gaussian measurement noise when trispectral approaches are used     

The Steiglitz-McBride identification algorithm revisited--Convergence analysis and accuracy aspects

The convergence and accuracy properties of the Steiglitz and McBride identification method are examined. The analysis is valid for a sufficiently large number of data. It is shown that the method can converge to the true parameter vector only when the additive output noise is white. In that case the method is proved to be locally convergent to the true parameters. The global convergence properties are also investigated. It is pointed out that the method is not always globally convergent. Some sufficient conditions guaranteeing global convergence are given. Assuming convergence takes place the estimates are shown to be asymptotically Gaussian distributed. An explicit expression is given for their asymptotic covariance matrix.     

Blind maximum likelihood identification of Hammerstein systems

This paper is about the identification of discrete-time Hammerstein systems from output measurements only (blind identification). Assuming that the unobserved input is white Gaussian noise, that the static nonlinearity is invertible, and that the output is observed without errors, a Gaussian maximum likelihood estimator is constructed. Its asymptotic properties are analyzed and the Cramér–Rao lower bound is calculated. In practice, the latter can be computed accurately without using the strong law of large numbers. A two-step procedure is described that allows to find high quality initial estimates to start up the iterative Gauss–Newton based optimization scheme. The paper includes the illustration of the method on a simulation example. A theoretical analysis demonstrates that additive output measurement noise introduces a bias that is proportional to the variance of that additive, unmodeled noise source. The simulations support this result, and show that this bias is insignificant beyond a certain Signal-to-Noise Ratio (40 dB in the example).     

Parameter consistency and quadratically constrained errors-in-variables least-squares identification

In this article, we investigate the consistency of parameter estimates obtained from least-squares identification with a quadratic parameter constraint. For generality, we consider infinite impulse-response systems with coloured input and output noise. In the case of finite data, we show that there always exists a possibly indefinite quadratic constraint depending on the noise realisation that results in a constrained optimisation problem that yields the true parameters of the system when a persistency condition is satisfied. When the noise covariance matrix is known to within a scalar multiple, we prove that solutions of the quadratically constrained least-squares (QCLs) estimator with a semidefinite constraint matrix are both unbiased and consistent in the sense that the averaged problem and limiting problem produce, respectively, unbiased and true (with probability 1) estimators. In addition, we provide numerical results that illustrate these properties of the QCLS estimator.     

Noise covariance identification for time-varying and nonlinear systems

Kalman-based state estimators assume a priori knowledge of the covariance matrices of the process and observation noise. However, in most practical situations, noise statistics and initial conditions are often unknown and need to be estimated from measurement data. This paper presents an auto-covariance least-squares-based algorithm for noise and initial state error covariance estimation of large-scale linear time-varying (LTV) and nonlinear systems. Compared to existing auto-covariance least-squares based-algorithms, our method does not involve any approximations for LTV systems, has fewer parameters to determine and is more memory/computationally efficient for large-scale systems. For nonlinear systems, our algorithm uses full information estimation/moving horizon estimation instead of the extended Kalman filter, so that the stability and accuracy of noise covariance estimation for nonlinear systems can be guaranteed or improved, respectively.     

Linear filtering with adaptive adjustment of the disturbance covariation matrices in the plant and measurement noise

For filtering a nonstationary linear plant under the unknown intensities of input signals such as plant disturbances and measurement noise, a new algorithm was presented. It is based on selecting the vectors of values of these signals compatible with the observed plant output and minimizing the error variances of the last predicted measurement. The measurement prediction is determined from the Kalman filter where the input signals are assumed to be white noise and the covariance matrix coincides with the empirical covariance matrix of the selected vectors. Numerical modeling demonstrated that the so-calculated filter coefficients are close to the optimal ones constructed from the true covariance matrices of plant disturbances and measurement noise. The approximate Newton method for minimization of the prediction error variance was shown to agree with the solution of the auxiliary optimal control problem, which allows to make one or some few iterations to find the point of minimum.     

自校正解耦信息融合Wiener状态估值器

对含未知噪声方差阵的多传感器系统,用现代时间序列分析方法.基于滑动平均(MA)新息模型的在线辨识和求解相关函数矩阵方程组,可得到估计噪声方差阵估值器,进而在按分量标量加权线性最小方差最优信息融合则下,提出了自校正解耦信息融合Wiener状态估值器.它的精度比每个局部自校正Wiener状态估值器精度高.它实现了状态分量的解耦局部Wiener估值器和解耦融合Wiener估值器.证明了它的收敛性,即若MA新息模型参数估计是一致的,则它将收敛于噪声统计已知时的最优解耦信息融合Wiener状态估值器,因而它具有渐近最优性.一个带3传感器的目标跟踪系统的仿真例子说明了其有效性.     

Subspace model identification Part 2. Analysis of the elementary output-error state-space model identification algorithm

The elementary MOESP algorithm presented in the first part of this series of papers is analysed in this paper. This is done in three different ways. First, we study the asymptotic properties of the estimated state-space model when only considering zero-mean white noise perturbations on the output sequence. It is shown that, in this case, the MOESPl implementation yields asymptotically unbiased estimates. An important constraint to this result is that the underlying system must have a finite impulse response and subsequently the size of the Hankel matrices, constructed from the input and output data at the beginning of the computations, depends on the number of non-zero Markov parameters. This analysis, however, leads to a second implementation of the elementary MOESP scheme, namely MOESP2. The latter implementation has the same asymptotic properties without the finite impulse response constraint. Secondly, we compare the MOESP2 algorithm with a classical state space model identification scheme. The latter scheme, referred to as the CLASSIC algorithm, is based on the Ho and Kalman realization scheme and estimated Markov parameters. The comparison is done by a sensitivity study, where the effect is studied of the errors on the data on the calculated column space of the shift-invariant subspace. This study demonstrates that the elementary MOESP2 scheme is more robust with respect to the errors considered than the CLASSIC algorithm. In the third part, the model reduction capabilities of the elementary MOESP schemes are analysed when the observations are error-free. We demonstrate in which sense the reduced order model is optimal when acquired with the MOESP schemes. The optimality is expressed by the difference between the 2-norm of the errors on the state (or output) sequence of the reduced-order model and the 2-norm of the matrix containing the rejected singular values being as small as possible. The insights obtained in these three parts are evaluated in a simulation study, and validated in this paper. They lead to the assertion that the MOESP2 implementation allows identification of a compact, low-dimensional, state-space model accurately describing the input -output behaviour of the system to be identified, while making use of ‘perturbed’ input-output data. This can be done efficiently.     

Convergence analysis of recursive identification algorithms with forgetting factor

A convergence analysis of a modified version of the least-squares recursive identification algorithm with forgetting factor is given. It is shown that the parametric distance converges to a zero mean random variable. It is also shown that, under persistent excitation condition on both system input and output, the condition number of the adaptation gain matrix is bounded. The variance of the parametric distance is bounded by the product of the noise variance times the upper bound of the condition number of the gain matrix. This is done by normalizing the measurement vector entering in the identification algorithm and by using a forgetting factor verifying λ_t ? 1 ? ε; ε >0.     

Accuracy Analysis of the Frisch Scheme for Identifying Errors-in-Variables Systems

Several estimation methods have been proposed for identifying errors-in-variables systems, where both input and output measurements are corrupted by noise. One of the promising approaches is the so-called Frisch scheme. This paper provides an accuracy analysis of the Frisch scheme applied to system identification. The estimates of the system parameters and the noise variances are shown to be asymptotically Gaussian distributed. An explicit expression for the covariance matrix of the asymptotic distribution is given as well. Numerical simulations support the theoretical results. A comparison with the Cramer-Rao lower bound is also given in the examples, and it is shown that the Frisch scheme gives a performance close to the Cramer-Rao bound for large signal-to-noise ratios (SNRs).     

Estimation of the autoregressive parameters of a mixed autoregressive moving-average time series

The problem of estimating the autoregressive parameters of a mixed autoregressive moving-average (ARMA) time series (of known order) using the output data alone is treated. This problem is equivalent to the estimation of the denominator terms of the scalar transfer function of a stationary, linear discrete time system excited by an unobserved unenrrelated sequence input by employing only the observations of the scalar output. The solution of this problem solves the problem of the identification of the dynamics of a white-noise excited continuous-time linear stationary system using sampled data. The latter problem was suggested by Bartlett in 1946. The problem treated here has appeared before in the engineering literature. The earlier treatment yielded biased parameter estimates. An asymptotically unbiased estimator of the autoregressive parameters is obtained as the solution of a modified set of Yule-Walker equations. The asymptotic estimator covariance matrix behaves like a least-squares parameter estimate of an observation set with unknown error covariances. The estimators are also shown to be unbiased in the presence of additive independent observation noise of arbitrary finite correlation time. An example illustrates the performance of the estimating procedures.     

An improved bias-compensation approach for errors-in-variables model identification

Parametric estimation of the dynamic errors-in-variables models is considered in this paper. In particular, a bias compensation approach is examined in a generalized framework. Sufficient conditions for uniqueness of the identified model are presented. Subsequently, a statistical accuracy analysis of the estimation algorithm is carried out. The asymptotic covariance matrix of the system parameter estimates depends on a user chosen filter and a certain weighting matrix. It is shown how these can be tuned to boost the estimation performance. The numerical simulation results suggest that the covariance matrix of the estimated parameter vector is very close to the Cramér-Rao lower bound for the estimation problem.     

Direct and two-step methods for closed-loop identification: a comparison of asymptotic and finite data set performance

The asymptotic and finite data behavior of some closed-loop identification methods are investigated. It is shown that, when the output power is limited, closed-loop identification can generally identify models with smaller variance than open-loop identification. Several variations on some two-step identification methods are compared with the direct identification method. High order FIR models are used as process models to avoid bias issues arising from inadequate model structures for the processes. Comparisons are, therefore, made based on the variance of the identified process models both for asymptotic situations and for finite data sets. Process model bias resulting from improper selection of the noise and sensitivity function models is also investigated. In this context, the results support the use of direct identification methods on closed-loop data.     

Restricted regression estimation in measurement error models

The problem of consistent estimation of the regression coefficients when some prior information about the regression coefficients is available is considered. Such prior information is expressed in the form of exact linear restrictions. The knowledge of covariance matrix of measurement errors that is associated with explanatory variables is used to construct the consistent estimators. Some consistent estimators are suggested which satisfy the exact linear restrictions also. Their asymptotic properties are derived and analytically analyzed under a multivariate ultrastructural model with not necessarily normally distributed measurement errors. The finite sample properties of the estimators are studied through a Monte-Carlo simulation experiment.     

Population coding and decoding in a neural field: a computational study

This study uses a neural field model to investigate computational aspects of population coding and decoding when the stimulus is a single variable. A general prototype model for the encoding process is proposed, in which neural responses are correlated, with strength specified by a gaussian function of their difference in preferred stimuli. Based on the model, we study the effect of correlation on the Fisher information, compare the performances of three decoding methods that differ in the amount of encoding information being used, and investigate the implementation of the three methods by using a recurrent network. This study not only rediscovers main results in existing literatures in a unified way, but also reveals important new features, especially when the neural correlation is strong. As the neural correlation of firing becomes larger, the Fisher information decreases drastically. We confirm that as the width of correlation increases, the Fisher information saturates and no longer increases in proportion to the number of neurons. However, we prove that as the width increases further--wider than (sqrt)2 times the effective width of the turning function--the Fisher information increases again, and it increases without limit in proportion to the number of neurons. Furthermore, we clarify the asymptotic efficiency of the maximum likelihood inference (MLI) type of decoding methods for correlated neural signals. It shows that when the correlation covers a nonlocal range of population (excepting the uniform correlation and when the noise is extremely small), the MLI type of method, whose decoding error satisfies the Cauchy-type distribution, is not asymptotically efficient. This implies that the variance is no longer adequate to measure decoding accuracy.     

Box-Jenkins identification revisited—Part I: Theory

In classical time domain Box-Jenkins identification discrete-time plant and noise models are estimated using sampled input/output signals. The frequency content of the input/output samples covers uniformly the whole unit circle in a natural way, even in case of prefiltering. Recently, the classical time domain Box-Jenkins framework has been extended to frequency domain data captured in open loop. The proposed frequency domain maximum likelihood (ML) solution can handle (i) discrete-time models using data that only covers a part of the unit circle, and (ii) continuous-time models. Part I of this series of two papers (i) generalizes the frequency domain ML solution to the closed loop case, and (ii) proves the properties of the ML estimator under non-standard conditions. Contrary to the classical time domain case it is shown that the controller should be either known or estimated. The proposed ML estimators are applicable to frequency domain data as well as time domain data.