An efficient hierarchical identification method for general dual-rate sampled-data systems

For the lifted input–output representation of general dual-rate sampled-data systems, this paper presents a decomposition based recursive least squares (D-LS) identification algorithm using the hierarchical identification principle. Compared with the recursive least squares (RLS) algorithm, the proposed D-LS algorithm does not require computing the covariance matrices with large sizes and matrix inverses in each recursion step, and thus has a higher computational efficiency than the RLS algorithm. The performance analysis of the D-LS algorithm indicates that the parameter estimates can converge to their true values. A simulation example is given to confirm the convergence results.     

A covariance matching approach for identifying errors-in-variables systems

The errors-in-variables identification problem concerns dynamic systems whose input and output variables are affected by additive noise. Several estimation methods have been proposed for identifying dynamic errors-in-variables models. In this paper a covariance matching approach is proposed to solve the identification problem. It applies for general types of input signals. The method utilizes a small set of covariances of the measured input–output data. This property applies also for some other methods, such as the Frisch scheme and the bias-eliminating least squares method. Algorithmic details for the proposed method are provided. User choices, for example specification of which input–output covariances to utilize, are discussed in some detail. The method is evaluated by using numerical examples, and is shown to have competitive properties as compared to alternative methods.     

On the accuracy in errors-in-variables identification compared to prediction-error identification

Errors-in-variables estimation problems for single-input–single-output systems with Gaussian signals are considered in this contribution. It is shown that the Fisher information matrix is monotonically increasing as a function of the input noise variance when the noise spectrum at the input is known and the corresponding noise variance is estimated. Furthermore, it is shown that Whittle’s formula for the Fisher information matrix can be represented as a Gramian and this is used to provide a geometric representation of the asymptotic covariance matrix for asymptotically efficient estimators. Finally, the asymptotic covariance of the parameter estimates for the system dynamics is compared for the two cases: (i) when the model includes white measurement noise on the input and the variance of the noise is estimated, and (ii) when the model includes only measurement noise on the output. In both cases, asymptotically efficient estimators are assumed. An explicit expression for the difference is derived when the underlying system is subject only to measurement noise on the output.     

Errors-in-variables identification of dynamic systems excited by arbitrary non-white input

This work deals with the identification of dynamic systems from noisy input–output observations, where the noise-free input is not parameterized. The basic assumptions made are (1) the dynamic system can be modeled by a (discrete- or continuous-time) rational transfer function model, (2) the temporal input–output disturbances are mutually independent, identically distributed noises, and (3) the input power spectrum is non-white (not necessarily rational) and is modeled nonparametrically. The system identifiability is guaranteed by exploiting the non-white spectrum property of the noise-free input. A frequency domain identification strategy is developed to estimate consistently the plant model parameters and the input–output noise variances. The uncertainty bound of the estimates is calculated and compared to the Cramér–Rao lower bound. The efficiency of the proposed algorithm is illustrated on numerical examples.     

The Steiglitz-McBride identification algorithm revisited--Convergence analysis and accuracy aspects

The convergence and accuracy properties of the Steiglitz and McBride identification method are examined. The analysis is valid for a sufficiently large number of data. It is shown that the method can converge to the true parameter vector only when the additive output noise is white. In that case the method is proved to be locally convergent to the true parameters. The global convergence properties are also investigated. It is pointed out that the method is not always globally convergent. Some sufficient conditions guaranteeing global convergence are given. Assuming convergence takes place the estimates are shown to be asymptotically Gaussian distributed. An explicit expression is given for their asymptotic covariance matrix.     

Multivariate mixture modeling using skew-normal independent distributions

In this paper we consider a flexible class of models, with elements that are finite mixtures of multivariate skew-normal independent distributions. A general EM-type algorithm is employed for iteratively computing parameter estimates and this is discussed with emphasis on finite mixtures of skew-normal, skew-t, skew-slash and skew-contaminated normal distributions. Further, a general information-based method for approximating the asymptotic covariance matrix of the estimates is also presented. The accuracy of the associated estimates and the efficiency of some information criteria are evaluated via simulation studies. Results obtained from the analysis of artificial and real data sets are reported illustrating the usefulness of the proposed methodology. The proposed EM-type algorithm and methods are implemented in the R package mixsmsn.     

Bounds on the accuracy of Gaussian ARMA parameter estimation methods based on sample covariances

The asymptotic accuracy of Gaussian ARMA parameter estimation methods based on a fixed number of sample covariances is considered. Several key results are briefly reviewed, including: i) a general asymptotic expression for the error covariance of the ARMA parameter estimates; ii) the fact that this error covariance is always greater than a certain lower bound; iii) the fact that this lower bound is strictly greater than the Cramer-Rao bound; iv) an explicit ARMA estimation technique that asymptotically achieves the bound. The key result of this note is a proof that this lower bound approaches the Cramer-Rao bound as the number of sample covariances tends to infinity.     

A generalized autocovariance least-squares method for Kalman filter tuning

This paper discusses a method for estimating noise covariances from process data. In linear stochastic state-space representations the true noise covariances are generally unknown in practical applications. Using estimated covariances a Kalman filter can be tuned in order to increase the accuracy of the state estimates. There is a linear relationship between covariances and autocovariance. Therefore, the covariance estimation problem can be stated as a least-squares problem, which can be solved as a symmetric semidefinite least-squares problem. This problem is convex and can be solved efficiently by interior-point methods. A numerical algorithm for solving the symmetric is able to handle systems with mutually correlated process noise and measurement noise.     

A new autocovariance least-squares method for estimating noise covariances

Industrial implementation of model-based control methods, such as model predictive control, is often complicated by the lack of knowledge about the disturbances entering the system. In this paper, we present a new method (constrained ALS) to estimate the variances of the disturbances entering the process using routine operating data. A variety of methods have been proposed to solve this problem. Of note, we compare ALS to the classic approach presented in Mehra [(1970). On the identification of variances and adaptive Kalman filtering. IEEE Transactions on Automatic Control, 15(12), 175-184]. This classic method, and those based on it, use a three-step procedure to compute the covariances. The method presented in this paper is a one-step procedure, which yields covariance estimates with lower variance on all examples tested. The formulation used in this paper provides necessary and sufficient conditions for uniqueness of the estimated covariances, previously not available in the literature. We show that the estimated covariances are unbiased and converge to the true values with increasing sample size. The proposed method also guarantees positive semidefinite covariance estimates by adding constraints to the ALS problem. The resulting convex program can be solved efficiently.     

On the arithmetic complexity of matrix Kronecker powers

In this paper we study the arithmetic complexity of computing the pth Kronecker power of an n × n matrix. We first analyze a straightforward inductive computation which requires an asymptotic average of p multiplications and p – 1 additions per computed output. We then apply efficient methods for matrix multiplication to obtain an algorithm that achieves the optimal rate of one multiplication per output at the expense of increasing the number of additions, and an algorithm that requires O(log p) multiplications and O(log²p) additions per output.     

On instrumental variable and total least squares approaches for identification of noisy systems

Many practical system identification problems can be formulated as linear regression problems. The parameter estimates can be computed using instrumental variables (IV) or total least squares (TLS) estimators, both of which have moderate computational complexity. In this work, explicit expressions for the asymptotic covariance matrix of the TLS estimates is derived and is shown to be same as that of the IV method. The accuracy of the parameter estimates for an errors-in-variables model using the above methods has been treated in particular, as standard analysis does not apply. The results obtained from the numerical simulations show that the practical behaviour of the estimators is well predicted by the theoretical results. We provide an explanation why for finite samples, the IV approach is found to be somewhat more robust than the TLS approach. On the other hand, the TLS approach has lower computational load than the IV method.     

Ensemble data assimilation in the presence of cloud

In numerical weather prediction (NWP) data assimilation (DA) methods are used to combine available observations with numerical model estimates. This is done by minimising measures of error on both observations and model estimates with more weight given to data that can be more trusted. For any DA method an estimate of the initial forecast error covariance matrix is required. For convective scale data assimilation, however, the properties of the error covariances are not well understood.An effective way to investigate covariance properties in the presence of convection is to use an ensemble-based method for which an estimate of the error covariance is readily available at each time step. In this work, we investigate the performance of the ensemble square root filter (EnSRF) in the presence of cloud growth applied to an idealised 1D convective column model of the atmosphere. We show that the EnSRF performs well in capturing cloud growth, but the ensemble does not cope well with discontinuities introduced into the system by parameterised rain. The state estimates lose accuracy, and more importantly the ensemble is unable to capture the spread (variance) of the estimates correctly. We also find, counter-intuitively, that by reducing the spatial frequency of observations and/or the accuracy of the observations, the ensemble is able to capture the states and their variability successfully across all regimes.     

Tracking randomly varying parameters: Analysis of a standard algorithm

In linear stochastic system identification, when the unknown parameters are randomly time varying and can be represented by a Markov model, a natural estimation algorithm to use is the Kalman filter. In seeking an understanding of the properties of this algorithm, existing Kalman-filter theory yields useful results only for the case where the noises are gaussian with covariances precisely known. In other cases, the stochastic and unbounded nature of the regression vector (which is regarded as the output gain matrix in state-space terminology) precludes application of standard theory. Here we develop asymptotic properties of the algorithm. In particular, we establish the tracking error bounds for the unknown randomly varying parameters, and some results on sample path deviations of the estimates.     

Estimation of the autoregressive parameters of a mixed autoregressive moving-average time series

The problem of estimating the autoregressive parameters of a mixed autoregressive moving-average (ARMA) time series (of known order) using the output data alone is treated. This problem is equivalent to the estimation of the denominator terms of the scalar transfer function of a stationary, linear discrete time system excited by an unobserved unenrrelated sequence input by employing only the observations of the scalar output. The solution of this problem solves the problem of the identification of the dynamics of a white-noise excited continuous-time linear stationary system using sampled data. The latter problem was suggested by Bartlett in 1946. The problem treated here has appeared before in the engineering literature. The earlier treatment yielded biased parameter estimates. An asymptotically unbiased estimator of the autoregressive parameters is obtained as the solution of a modified set of Yule-Walker equations. The asymptotic estimator covariance matrix behaves like a least-squares parameter estimate of an observation set with unknown error covariances. The estimators are also shown to be unbiased in the presence of additive independent observation noise of arbitrary finite correlation time. An example illustrates the performance of the estimating procedures.     

Auxiliary model-based interval-varying multi-innovation least squares identification for multivariable OE-like systems with scarce measurements

The identification problem of multivariable OE-like systems with scarce measurements is considered in this paper. By replacing the unknown inner variables in the information matrix with the outputs of the auxiliary model and by expanding the scalar innovation to an innovation vector, an auxiliary model-based multi-innovation least squares (AM-MILS) algorithm is proposed. In order to deal with the scarce measurement pattern, the algorithm takes the form of interval-varying recursive computation to skip the unavailable measurements including outliers. The introduction of the multi-innovation concept improves the parameter estimation accuracy and makes the identification algorithm more efficient. The convergence analysis shows that for the proposed algorithm, the parameter estimates can converge to their true values in the scarce output measurement pattern. Illustrative examples are given to demonstrate the effectiveness and accuracy of the proposed method.     

Model order selection of nonlinear input–output models––a clustering based approach

Selecting the order of an input–output model of a dynamical system is a key step toward the goal of system identification. The false nearest neighbors algorithm (FNN) is a useful tool for the estimation of the order of linear and nonlinear systems. While advanced FNN uses nonlinear input–output data-based models for the model-based selection of the threshold constant that is used to compute the percentage of false neighbors, the computational effort of the method increases along with the number of data and the dimension of the model. To increase the efficiency of this method, in this paper we propose a clustering-based algorithm. Clustering is applied to the product space of the input and output variables. The model structure is then estimated on the basis of the cluster covariance matrix eigenvalues. The main advantage of the proposed solution is that it is model-free. This means that no particular model needs to be constructed in order to select the order of the model, while most other techniques are ‘wrapped' around a particular model construction method. This saves the computational effort and avoids a possible bias due to the particular construction method used. Three simulation examples are given to illustrate the proposed technique: estimation of the model structure for a linear system, a polymerization reactor and the van der Vusse reactor.     

A generalized instrumental variable estimation method for errors-in-variables identification problems

The errors-in-variables identification problem concerns dynamic systems whose input and output variables are affected by additive noise. Several estimation methods have been proposed for identifying dynamic errors-in-variables models. In this paper it is shown how a number of common methods for errors-in-variables methods can be put into a general framework, resulting into a Generalized Instrumental Variable Estimator (GIVE). Various computational aspects of GIVE are presented, and the asymptotic distribution of the parameter estimates is derived.     

Parameter estimation with scarce measurements

In this paper, the problems of parameter estimation are addressed for systems with scarce measurements. A gradient-based algorithm is derived to estimate the parameters of the input–output representation with scarce measurements, and the convergence properties of the parameter estimation and unavailable output estimation are established using the Kronecker lemma and the deterministic version of the martingale convergence theorem. Finally, an example is provided to demonstrate the effectiveness of the proposed algorithm.     

On the accuracy of a covariance matching method for continuous-time errors-in-variables identification

An analysis of a covariance matching method for continuous-time errors-in-variables system identification from discrete-time data is made. In the covariance matching method, the noise-free input signal is not explicitly modeled and only assumed to be a stationary process. The asymptotic normalized covariance matrix, valid for a large number of data and a small sampling interval, is derived. This involves the evaluation of a covariance matrix of estimated covariance elements and estimated derivatives of such elements, and large parts of the paper are devoted to this task. The latter covariance matrix consists of two parts, where the first part contains integrals that are approximations of Riemann sums, and the second part depends on the measurement noise variances.     

Estimation of the disturbance structure from data using semidefinite programming and optimal weighting

Designing a state estimator for a linear state-space model requires knowledge of the characteristics of the disturbances entering the states and the measurements. In [Odelson, B. J., Rajamani, M. R., & Rawlings, J. B. (2006). A new autocovariance least squares method for estimating noise covariances. Automatica, 42(2), 303-308], the correlations between the innovations data were used to form a least-squares problem to determine the covariances for the disturbances. In this paper we present new and simpler necessary and sufficient conditions for the uniqueness of the covariance estimates. We also formulate the optimal weighting to be used in the least-squares objective in the covariance estimation problem to ensure minimum variance in the estimates. A modification to the above technique is then presented to estimate the number of independent stochastic disturbances affecting the states. This minimum number of disturbances is usually unknown and must be determined from data. A semidefinite optimization problem is solved to estimate the number of independent disturbances entering the system and their covariances.