Maximum likelihood based identification methods for rational models

ABSTRACT

In a rational model, some terms of the information vector are correlated with the noise, which makes the traditional least squares based iterative algorithms biased. In order to overcome this shortcoming, this paper develops two recursive algorithms for estimating the rational model parameters. These two algorithms, based on the maximum likelihood principle, have three integrated key features: (1) to establish two unbiased maximum likelihood recursive algorithms, (2) to develop a maximum likelihood recursive least squares (ML-RLS) algorithm to decrease the computational efforts, (3) to update the parameter estimates by the ML-RLS based particle swarm optimisation (ML-RLS-PSO) algorithm when the noise-to-output ratio is large. Comparative studies demonstrate that (1) the ML-RLS algorithm is only valid for rational models when the noise-to-output ratio is small, (2) the ML-RLS-PSO algorithm is effective for rational models with random noise-to-output ratio, but at the cost of heavy computational efforts. Furthermore, the simulations provide cases for potential expansion and applications of the proposed algorithms.     

Maximum likelihood parameter estimation algorithm for controlled autoregressive autoregressive models

A maximum likelihood parameter estimation algorithm is derived for controlled autoregressive autoregressive (CARAR) models based on the maximum likelihood principle. In this derivation, we use an estimated noise transfer function to filter the input–output data. The simulation results show that the proposed estimation algorithm can effectively estimate the parameters of such class of CARAR systems and give more accurate parameter estimates than the recursive generalized least-squares algorithm.     

Maximum likelihood estimation of cascade point-process neural encoding models

Recent work has examined the estimation of models of stimulus-driven neural activity in which some linear filtering process is followed by a nonlinear, probabilistic spiking stage. We analyze the estimation of one such model for which this nonlinear step is implemented by a known parametric function; the assumption that this function is known speeds the estimation process considerably. We investigate the shape of the likelihood function for this type of model, give a simple condition on the nonlinearity ensuring that no non-global local maxima exist in the likelihood-leading, in turn, to efficient algorithms for the computation of the maximum likelihood estimator-and discuss the implications for the form of the allowed nonlinearities. Finally, we note some interesting connections between the likelihood-based estimators and the classical spike-triggered average estimator, discuss some useful extensions of the basic model structure, and provide two novel applications to physiological data.     

Maximum likelihood estimation of Gaussian mixture models using stochastic search

Gaussian mixture models (GMM), commonly used in pattern recognition and machine learning, provide a flexible probabilistic model for the data. The conventional expectation–maximization (EM) algorithm for the maximum likelihood estimation of the parameters of GMMs is very sensitive to initialization and easily gets trapped in local maxima. Stochastic search algorithms have been popular alternatives for global optimization but their uses for GMM estimation have been limited to constrained models using identity or diagonal covariance matrices. Our major contributions in this paper are twofold. First, we present a novel parametrization for arbitrary covariance matrices that allow independent updating of individual parameters while retaining validity of the resultant matrices. Second, we propose an effective parameter matching technique to mitigate the issues related with the existence of multiple candidate solutions that are equivalent under permutations of the GMM components. Experiments on synthetic and real data sets show that the proposed framework has a robust performance and achieves significantly higher likelihood values than the EM algorithm.     

Maximum likelihood estimation for generalized semi-markov processes

Parametric statistical inference for generalized semi-Markov processes is addressed. This class of processes encompasses a large number of real-world discrete-event stochastic systems. Because of its properties (e.g., consistency, asymptotic normality, etc.), maximum likelihood estimation is considered here. Under reasonable conditions on the process, we show that a maximum likelihood estimator exists, and that it converges to the true parameter at ratet ^–1/2, wheret is the length of the observation period. A related estimator, which is typically easier to compute, is also introduced. We show that the use of this estimator results in no loss of statistical efficiency. It is also shown that the estimation problem does decouple into separate subproblems when the process' transition probabilities and event distributions depend on different parameters.     

Maximum likelihood estimation method for dual-rate Hammerstein systems

International Journal of Control, Automation and Systems - For a dual-rate sampled Hammerstein controlled autoregressive moving average (CARMA) system, this paper uses the polynomial transformation...     

Maximum likelihood estimation for continuous-time autoregressive models by relaxation on residual variances ratio parameters

In this paper we derive an explicit expression for the log likelihood function of a continuous-time autoregressive model. Then, using earlier results relating the autoregressive coefficients to the set of positive parameters called residual variances ratios, we develop an iterative algorithm for computing the maximum likelihood estimator of the model, similar to one in the discrete-time case. A simple noniterative estimation method, which can be used to produce an initial estimate for the algorithm, is also proposed.     

Maximum likelihood estimation for constrained parameters of multinomial distributions—Application to Zipf-Mandelbrot models

A numerical maximum likelihood (ML) estimation procedure is developed for the constrained parameters of multinomial distributions. The main difficulty involved in computing the likelihood function is the precise and fast determination of the multinomial coefficients. For this the coefficients are rewritten into a telescopic product. The presented method is applied to the ML estimation of the Zipf-Mandelbrot (ZM) distribution, which provides a true model in many real-life cases. The examples discussed arise from ecological and medical observations. Based on the estimates, the hypothesis that the data is ZM distributed is tested using a chi-square test. The computer code of the presented procedure is available on request by the author.     

Maximum likelihood identification of Wiener models

The Wiener model is a block oriented model, having a linear dynamic system followed by a static nonlinearity. The dominating approach to estimate the components of this model has been to minimize the error between the simulated and the measured outputs. We show that this will, in general, lead to biased estimates if there are other disturbances present than measurement noise. The implications of Bussgang’s theorem in this context are also discussed. For the case with general disturbances, we derive the Maximum Likelihood method and show how it can be efficiently implemented. Comparisons between this new algorithm and the traditional approach, confirm that the new method is unbiased and also has superior accuracy.     

Nonlinear random effects mixture models: Maximum likelihood estimation via the EM algorithm

Nonlinear random effects models with finite mixture structures are used to identify polymorphism in pharmacokinetic/pharmacodynamic (PK/PD) phenotypes. An EM algorithm for maximum likelihood estimation approach is developed and uses sampling-based methods to implement the expectation step, that results in an analytically tractable maximization step. A benefit of the approach is that no model linearization is performed and the estimation precision can be arbitrarily controlled by the sampling process. A detailed simulation study illustrates the feasibility of the estimation approach and evaluates its performance. Applications of the proposed nonlinear random effects mixture model approach to other population PK/PD problems will be of interest for future investigation.     

Maximum likelihood and prediction error methods

The basic ideas behind the parameter estimation methods are discussed in a general setting. The application to estimation or parameters in dynamical systems is treated in detail using the prototype problem of estimating parameters in a continuous time system using discrete time measurements. Computational aspects are discussed. Theoretical results in consistency, asymptotic normality and efficiency are covered. Model validation and selection of model structures are discussed. An example is given which illustrates some properties of the methods and shows the usefulness of interactive computing. Additional examples illustrate what happens when the data has different artefacts.     

Maximum likelihood identification of noisy input-output models

This work deals with the identification of errors-in-variables models corrupted by white and uncorrelated Gaussian noises. By introducing an auxiliary process, it is possible to obtain a maximum likelihood solution of this identification problem, by means of a two-step iterative algorithm. This approach allows also to estimate, as a byproduct, the noise-free input and output sequences. Moreover, an analytic expression of the finite Cràmer-Rao lower bound is derived. The method does not require any particular assumption on the input process, however, the ratio of the noise variances is assumed as known. The effectiveness of the proposed algorithm has been verified by means of Monte Carlo simulations.     

Tensor methods for full-information maximum likelihood estimation: Unconstrained estimation

In this study, we present a new method, called a tensor method, for the computation of unconstrained Full-Information Maximum Likelihood (FIML) estimates. The new techniqus is based upon a fourth order approximation to the log-likelihood function, rather than the second order approximation used in standard methods. The higher order terms are low rank third and fourth order tensors that are computed, at very little storage or computation cost, using information from previous iterations. We form and solve the tensor model, then present test results showing that the tensor method is far more efficient than the standard Newton's method for a wide range of unconstrained FIML estimation problems.This paper is based upon part of my doctoral dissertation at George Washington University. I would like to thank my committee members, Professors Robert Phillips and Frederick Joutz of George Washington University and John R. Norsworthy of Renssalaer Polytechnic Institute for their support and suggestions. Any errors remaining are my own.     

Maximum likelihood estimation of mixtures of factor analyzers

Mixtures of factor analyzers have been receiving wide interest in statistics as a tool for performing clustering and dimension reduction simultaneously. In this model it is assumed that, within each component, the data are generated according to a factor model. Therefore, the number of parameters on which the covariance matrices depend is reduced. Several estimation methods have been proposed for this model, both in the classical and in the Bayesian framework. However, so far, a direct maximum likelihood procedure has not been developed. This direct estimation problem, which simultaneously allows one to derive the information matrix for the mixtures of factor analyzers, is solved. The effectiveness of the proposed procedure is shown on a simulation study and on a toy example.     

Maximum likelihood estimation using square root information filters

The maximum likelihood parameter estimation algorithm is known to provide optimal estimates for linear time-invariant dynamic systems. However, the algorithm is computationally expensive and requires evaluations of the gradient of a log likelihood function and the Fisher information matrix. By using the square-root information filter, a numerically reliable algorithm to compute the required gradient and the Fisher information matrix is developed. The algorithm is a significant improvement over the methods based on the conventional Kalman filter. The square-root information filter relies on the use of orthogonal transformations that are well known for numerical reliability. This algorithm can be extended to real-time system identification and adaptive control     

Maximum likelihood estimation from interval censored recurrent event data

In this paper, we consider the recurrent failures of several repairable units, which can only be observed at periodic inspection times. A unit is not aging over the period between a failure and its detection. The failure times are interval censored by the periodic assessment times. The observed data consists of censoring intervals of failure times and the unobserved data are the actual ages of the units at the failure times. We formulate the likelihood function and use several iterative algorithms to find the maximum likelihood estimate (MLE) of the parameters. The complete Expectation–Maximization (EM) algorithm, the EM gradient, full Newton–Raphson (NR), and the Simplex method are used. We derive recursive equations to calculate the expected values required in the algorithms. We estimate the parameters for four failure datasets, assuming that the failures follow a non-homogeneous Poisson process (NHPP). Three datasets are obtained from a hospital for the components of general infusion pump, and the fourth dataset is simulated. Since the estimation could take a long time, we compare the performance of the algorithms in terms of the required number of iterations to converge, the total execution time, and the precision of the estimated parameters. We also use Monte Carlo and Quasi-Monte Carlo simulation as the substitutes for the recursive procedures in the Expectation step of the EM gradient and compare the results.     

Maximum likelihood estimation of linear stochastic systems in the class of sequential square-root orthogonal filtering methods

Theoretical and applied aspects are considered for development of numerically stable adaptive methods of the parametric identification of linear discrete stochastic systems in the space of states. The unknown system parameters to be estimated can enter into any matrices specifying a system and into initial conditions. The class of gradient methods is first suggested, which is developed on the basis of orthogonal square-root implementations of the discrete Kalman filter with the use of the technology of sequential data processing. It is shown that the algorithms of the given type can be effectively used to solve ill-conditioned problems of parametric identification. A test example is drawn. The practical significance of the suggested class of methods is illustrated by an example of the solution for one of the financial mathematics problems — the identification of a multidimensional model of stochastic volatility.     

Maximum likelihood estimation of vessel parameters from scale space analysis

We describe a method of detecting features in retinal images using a model-based approach. The image is processed using a bank of filters in a scale space. A parametric model of the target feature is then proposed and the filter responses to the model calculated. A noise model is proposed, and incorporated into a maximum likelihood estimator to estimate model parameters. The estimator uses the generative parametric model to explore smoothly the scale space. This method is applied to the detection of retinal blood vessels, using a Gaussian-profiled valley as a model. A simple thresholding method is proposed as an example of using the rich estimated parameter maps to detect vessels and the results are compared against two existing vessel detectors. Our system is compared against ground truth and the output of existing systems. It is found to be comparable and, in addition, produces direct estimates of vessel calibres and contrasts. It does not use any form of region growing or vessel tracking, but thresholds a function of the estimated vessel parameters to determine vessel regions.     

Exact maximum likelihood estimation for non-stationary periodic time series models

Time series models with parameter values that depend on the seasonal index are commonly referred to as periodic models. Periodic formulations for two classes of time series models are considered: seasonal autoregressive integrated moving average and unobserved components models. Convenient state space representations of the periodic models are proposed to facilitate model identification, specification and exact maximum likelihood estimation of the periodic parameters. These formulations do not require a priori (seasonal) differencing of the time series. The time-varying state space representation is an attractive alternative to the time-invariant vector representation of periodic models which typically leads to a high dimensional state vector in monthly periodic time series models. A key development is our method for computing the variance-covariance matrix of the initial set of observations which is required for exact maximum likelihood estimation. The two classes of periodic models are illustrated for a monthly postwar US unemployment time series.