Robust distributed maximum likelihood estimation with dependent quantized data

In this paper, we consider the distributed maximum likelihood estimation (MLE) with dependent quantized data under the assumption that the structure of the joint probability density function (pdf) is known, but it contains unknown deterministic parameters. The parameters may include different vector parameters corresponding to marginal pdfs and parameters that describe the dependence of observations across sensors. Since MLE with a single quantizer is sensitive to the choice of thresholds due to the uncertainty of pdf, we concentrate on MLE with multiple groups of quantizers (which can be determined by the use of prior information or some heuristic approaches) to fend off against the risk of a poor/outlier quantizer. The asymptotic efficiency of the MLE scheme with multiple quantizers is proved under some regularity conditions and the asymptotic variance is derived to be the inverse of a weighted linear combination of Fisher information matrices based on multiple different quantizers which can be used to show the robustness of our approach. As an illustrative example, we consider an estimation problem with a bivariate non-Gaussian pdf that has applications in distributed constant false alarm rate (CFAR) detection systems. Simulations show the robustness of the proposed MLE scheme especially when the number of quantized measurements is small.     

A maximum likelihood approximation method for Dirichlet's parameter estimation

Dirichlet distributions are natural choices to analyse data described by frequencies or proportions since they are the simplest known distributions for such data apart from the uniform distribution. They are often used whenever proportions are involved, for example, in text-mining, image analysis, biology or as a prior of a multinomial distribution in Bayesian statistics. As the Dirichlet distribution belongs to the exponential family, its parameters can be easily inferred by maximum likelihood. Parameter estimation is usually performed with the Newton-Raphson algorithm after an initialisation step using either the moments or Ronning's methods. However this initialisation can result in parameters that lie outside the admissible region. A simple and very efficient alternative based on a maximum likelihood approximation is presented. The advantages of the presented method compared to two other methods are demonstrated on synthetic data sets as well as for a practical biological problem: the clustering of protein sequences based on their amino acid compositions.     

On the equivalence of time and frequency domain maximum likelihood estimation

Maximum likelihood estimation has a rich history. It has been successfully applied to many problems including dynamical system identification. Different approaches have been proposed in the time and frequency domains. In this paper we discuss the relationship between these approaches and we establish conditions under which the different formulations are equivalent for finite length data. A key point in this context is how initial (and final) conditions are considered and how they are introduced in the likelihood function.     

GSA-based maximum likelihood estimation for threshold vector error correction model

The log-likelihood function of threshold vector error correction models is neither differentiable, nor smooth with respect to some parameters. Therefore, it is very difficult to implement maximum likelihood estimation (MLE) of the model. A new estimation method, which is based on a hybrid algorithm and MLE, is proposed to resolve this problem. The hybrid algorithm, referred to as genetic-simulated annealing, not only inherits aspects of genetic-algorithms (GAs), but also avoids premature convergence by incorporating elements of simulated annealing (SA). Simulation experiments demonstrate that the proposed method allows to estimate the parameters of larger cointegrating systems. Additionally, numerical results show that the hybrid algorithm does a better job than either SA or GA alone.     

New maximum likelihood motion estimation schemes for noisy ultrasound images

When performing block-matching based motion estimation with the ML estimator, one would try to match blocks from the two images, within a predefined search area. The estimated motion vector is that which maximizes a likelihood function, formulated according to the image formation model. Two new maximum likelihood motion estimation schemes for ultrasound images are presented. The new likelihood functions are based on the assumption that both images are contaminated by a Rayleigh distributed multiplicative noise. The new approach enables motion estimation in cases where a noiseless reference image is not available. Experimental results show a motion estimation improvement with regards to other known ML estimation methods.     

Faster ARMA maximum likelihood estimation

A new likelihood based AR approximation is given for ARMA models. The usual algorithms for the computation of the likelihood of an ARMA model require O(n) flops per function evaluation. Using our new approximation, an algorithm is developed which requires only O(1) flops in repeated likelihood evaluations. In most cases, the new algorithm gives results identical to or very close to the exact maximum likelihood estimate (MLE). This algorithm is easily implemented in high level quantitative programming environments (QPEs) such as Mathematica, MatLab and R. In order to obtain reasonable speed, previous ARMA maximum likelihood algorithms are usually implemented in C or some other machine efficient language. With our algorithm it is easy to do maximum likelihood estimation for long time series directly in the QPE of your choice. The new algorithm is extended to obtain the MLE for the mean parameter. Simulation experiments which illustrate the effectiveness of the new algorithm are discussed. Mathematica and R packages which implement the algorithm discussed in this paper are available [McLeod, A.I., Zhang, Y., 2007. Online supplements to “Faster ARMA Maximum Likelihood Estimation”, 〈http://www.stats.uwo.ca/faculty/aim/2007/faster/〉]. Based on these package implementations, it is expected that the interested researcher would be able to implement this algorithm in other QPEs.     

Frequency domain maximum likelihood estimation of linear dynamic errors-in-variables models

This paper studies the linear dynamic errors-in-variables problem for filtered white noise excitations. First, a frequency domain Gaussian maximum likelihood (ML) estimator is constructed that can handle discrete-time as well as continuous-time models on (a) part(s) of the unit circle or imaginary axis. Next, the ML estimates are calculated via a computationally simple and numerically stable Gauss-Newton minimization scheme. Finally, the Cramér-Rao lower bound is derived.     

Recursive maximum likelihood parameter estimation for state space systems using polynomial chaos theory

This paper combines polynomial chaos theory with maximum likelihood estimation for a novel approach to recursive parameter estimation in state-space systems. A simulation study compares the proposed approach with the extended Kalman filter to estimate the value of an unknown damping coefficient of a nonlinear Van der Pol oscillator. The results of the simulation study suggest that the proposed polynomial chaos estimator gives comparable results to the filtering method but may be less sensitive to user-defined tuning parameters. Because this recursive estimator is applicable to linear and nonlinear dynamic systems, the authors portend that this novel formulation will be useful for a broad range of estimation problems.     

Frequency domain sample maximum likelihood estimation for spatially dependent parameter estimation in PDEs

The identification of the spatially dependent parameters in Partial Differential Equations (PDEs) is important in both physics and control problems. A methodology is presented to identify spatially dependent parameters from spatio-temporal measurements. Local non-rational transfer functions are derived based on three local measurements allowing for a local estimate of the parameters. A sample Maximum Likelihood Estimator (SMLE) in the frequency domain is used, because it takes noise properties into account and allows for high accuracy consistent parameter estimation. Confidence bounds on the parameters are estimated based on the noise properties of the measurements. This method is successfully applied to the simulations of a finite difference model of a parabolic PDE with piecewise constant parameters.     

GFREG: a computer program for maximum likelihood regression using the Generalized F distribution

A FORTRAN program is described for maximum likelihood estimation within the Generalized F family of distributions. It can be used to estimate regression parameters in a log-linear model for censored survival times with covariates, for which the error distribution may have a great variety of shapes, including most distributions of current use in biostatistics. The optimization is performed by an algorithm based on the generalized reduced gradient method. A stepwise variable search algorithm for covariate selection is included in the program. Output features include: model selection criteria, standard errors of parameter estimates, quantile and survival rates with their standard errors, residuals and several plots. An example based on data from Princess Margaret Hospital, Toronto, is discussed to illustrate the program's capabilities.     

Intrinsic dimension estimation by maximum likelihood in isotropic probabilistic PCA

A central issue in dimension reduction is choosing a sensible number of dimensions to be retained. This work demonstrates the surprising result of the asymptotic consistency of the maximum likelihood criterion for determining the intrinsic dimension of a dataset in an isotropic version of probabilistic principal component analysis (PPCA). Numerical experiments on simulated and real datasets show that the maximum likelihood criterion can actually be used in practice and outperforms existing intrinsic dimension selection criteria in various situations. This paper exhibits and outlines the limits of the maximum likelihood criterion. It leads to recommend the use of the AIC criterion in specific situations. A useful application of this work would be the automatic selection of intrinsic dimensions in mixtures of isotropic PPCA for classification.     

Causal effects in longitudinal studies: Definition and maximum likelihood estimation

Recently, a nonparametric marginal structural model (NPMSM) approach to Causal Inference has been proposed [Neugebauer, R., van der Laan, M., 2006. Nonparametric causal effects based on marginal structural models. J. Statist. Plann. Inference (in press), 〈www http://www.sciencedirect.com/science/journal/03783758〉.] as an appealing practical alternative to the original parametric MSM (PMSM) approach introduced by Robins [Robins, J., 1998a. Marginal structural models. In: 1997 Proceedings of the American Statistical Association, American Statistical Association, Alexandria, VA, pp. 1-10]. The new MSM-based causal inference methodology generalizes the concept of causal effects: the proposed nonparametric causal effects are interpreted as summary measures of the causal effects defined with PMSMs. In addition, causal inference with NPMSM does not rely on the assumed correct specification of a parametric MSM but instead defines causal effects based on a user-specified working causal model which can be willingly misspecified. The NPMSM approach was developed for studies with point treatment data or with longitudinal data where the outcome is not time-dependent (typically collected at the end of data collection). In this paper, we generalize this approach to longitudinal studies where the outcome is time-dependent, i.e. collected throughout the span of the studies, and address the subsequent estimation inconsistency which could easily arise from a hasty generalization of the algorithm for maximum likelihood estimation. More generally, we provide an overview of the multiple causal effect representations which have been developed based on MSMs in longitudinal studies.     

Maximum likelihood estimation for constrained parameters of multinomial distributions—Application to Zipf-Mandelbrot models

A numerical maximum likelihood (ML) estimation procedure is developed for the constrained parameters of multinomial distributions. The main difficulty involved in computing the likelihood function is the precise and fast determination of the multinomial coefficients. For this the coefficients are rewritten into a telescopic product. The presented method is applied to the ML estimation of the Zipf-Mandelbrot (ZM) distribution, which provides a true model in many real-life cases. The examples discussed arise from ecological and medical observations. Based on the estimates, the hypothesis that the data is ZM distributed is tested using a chi-square test. The computer code of the presented procedure is available on request by the author.     

基于无线传感器网络的极大似然定位法的分析

对现在普遍应用的极大似然估计定位方法进行了分析,发现极大似然估计法参考方程(第n个方程)的锚节点(参照锚节点)测距误差和参照锚节点在定位区域内的位置对定位误差有重要影响:参照锚节点的测距误差越小,计算得到的定位精度越高;当所有方程所含误差相同时,参照锚节点在定位区域中心时定位误差小,在定位区域边缘时定位误差大;当参照锚...     

Copula density estimation by total variation penalized likelihood with linear equality constraints

A copula density is the joint probability density function (PDF) of a random vector with uniform marginals. An approach to bivariate copula density estimation is introduced that is based on maximum penalized likelihood estimation (MPLE) with a total variation (TV) penalty term. The marginal unity and symmetry constraints for copula density are enforced by linear equality constraints. The TV-MPLE subject to linear equality constraints is solved by an augmented Lagrangian and operator-splitting algorithm. It offers an order of magnitude improvement in computational efficiency over another TV-MPLE method without constraints solved by the log-barrier method for the second order cone program. A data-driven selection of the regularization parameter is through K-fold cross-validation (CV). Simulation and real data application show the effectiveness of the proposed approach. The MATLAB code implementing the methodology is available online.     

Maximum likelihood estimation via the extended covariance and combined square-root filters

The method of maximum likelihood is a general method for parameter estimation and is often used in system identification. To implement it, it is necessary to maximize the likelihood function, which is usually done using the gradient approach. It involves the computation of the likelihood gradient with respect to unknown system parameters. For linear stochastic system models this leads to the implementation of the Kalman filter, which is known to be numerically unstable. The aim of this work is to present new efficient algorithms for likelihood gradient evaluation. They are more reliable in practice and improve robustness of computations against roundoff errors. All algorithms are derived in measurement and time updates form. The comparison with the conventional Kalman filter approach and results of numerical experiments are given.     

Blind maximum likelihood identification of Hammerstein systems

This paper is about the identification of discrete-time Hammerstein systems from output measurements only (blind identification). Assuming that the unobserved input is white Gaussian noise, that the static nonlinearity is invertible, and that the output is observed without errors, a Gaussian maximum likelihood estimator is constructed. Its asymptotic properties are analyzed and the Cramér–Rao lower bound is calculated. In practice, the latter can be computed accurately without using the strong law of large numbers. A two-step procedure is described that allows to find high quality initial estimates to start up the iterative Gauss–Newton based optimization scheme. The paper includes the illustration of the method on a simulation example. A theoretical analysis demonstrates that additive output measurement noise introduces a bias that is proportional to the variance of that additive, unmodeled noise source. The simulations support this result, and show that this bias is insignificant beyond a certain Signal-to-Noise Ratio (40 dB in the example).     

Tensor methods for full-information maximum likelihood estimation: Unconstrained estimation

In this study, we present a new method, called a tensor method, for the computation of unconstrained Full-Information Maximum Likelihood (FIML) estimates. The new techniqus is based upon a fourth order approximation to the log-likelihood function, rather than the second order approximation used in standard methods. The higher order terms are low rank third and fourth order tensors that are computed, at very little storage or computation cost, using information from previous iterations. We form and solve the tensor model, then present test results showing that the tensor method is far more efficient than the standard Newton's method for a wide range of unconstrained FIML estimation problems.This paper is based upon part of my doctoral dissertation at George Washington University. I would like to thank my committee members, Professors Robert Phillips and Frederick Joutz of George Washington University and John R. Norsworthy of Renssalaer Polytechnic Institute for their support and suggestions. Any errors remaining are my own.     

Maximum likelihood estimation of mixtures of factor analyzers

Mixtures of factor analyzers have been receiving wide interest in statistics as a tool for performing clustering and dimension reduction simultaneously. In this model it is assumed that, within each component, the data are generated according to a factor model. Therefore, the number of parameters on which the covariance matrices depend is reduced. Several estimation methods have been proposed for this model, both in the classical and in the Bayesian framework. However, so far, a direct maximum likelihood procedure has not been developed. This direct estimation problem, which simultaneously allows one to derive the information matrix for the mixtures of factor analyzers, is solved. The effectiveness of the proposed procedure is shown on a simulation study and on a toy example.     

An analysis of separable least squares data driven local coordinates for maximum likelihood estimation of linear systems

In this paper we study a novel parametrization for state-space systems, namely separable least squares data driven local coordinates (slsDDLC). The parametrization by slsDDLC has recently been successfully applied to maximum likelihood estimation of linear dynamic systems. In a simulation study, the use of slsDDLC has led to numerical advantages in comparison to the use of more conventional parametrizations, including data driven local coordinates (DDLC). However, an analysis of properties of slsDDLC, which are relevant to identification, has not been performed up to now. In this paper, we provide insights into the geometry and topology of the slsDDLC construction and show a number of results which are important for actual identification, in particular for maximum likelihood estimation. We also prove that the separable least squares methodology is indeed guaranteed to be applicable to maximum likelihood estimation of linear dynamic systems in typical situations.