Nonlinear random effects mixture models: Maximum likelihood estimation via the EM algorithm

Nonlinear random effects models with finite mixture structures are used to identify polymorphism in pharmacokinetic/pharmacodynamic (PK/PD) phenotypes. An EM algorithm for maximum likelihood estimation approach is developed and uses sampling-based methods to implement the expectation step, that results in an analytically tractable maximization step. A benefit of the approach is that no model linearization is performed and the estimation precision can be arbitrarily controlled by the sampling process. A detailed simulation study illustrates the feasibility of the estimation approach and evaluates its performance. Applications of the proposed nonlinear random effects mixture model approach to other population PK/PD problems will be of interest for future investigation.     

Maximum likelihood estimation of a stochastic integrate-and-fire neural encoding model

We examine a cascade encoding model for neural response in which a linear filtering stage is followed by a noisy, leaky, integrate-and-fire spike generation mechanism. This model provides a biophysically more realistic alternative to models based on Poisson (memoryless) spike generation, and can effectively reproduce a variety of spiking behaviors seen in vivo. We describe the maximum likelihood estimator for the model parameters, given only extracellular spike train responses (not intracellular voltage data). Specifically, we prove that the log-likelihood function is concave and thus has an essentially unique global maximum that can be found using gradient ascent techniques. We develop an efficient algorithm for computing the maximum likelihood solution, demonstrate the effectiveness of the resulting estimator with numerical simulations, and discuss a method of testing the model's validity using time-rescaling and density evolution techniques.     

Maximum likelihood estimation of cascade point-process neural encoding models

Recent work has examined the estimation of models of stimulus-driven neural activity in which some linear filtering process is followed by a nonlinear, probabilistic spiking stage. We analyze the estimation of one such model for which this nonlinear step is implemented by a known parametric function; the assumption that this function is known speeds the estimation process considerably. We investigate the shape of the likelihood function for this type of model, give a simple condition on the nonlinearity ensuring that no non-global local maxima exist in the likelihood-leading, in turn, to efficient algorithms for the computation of the maximum likelihood estimator-and discuss the implications for the form of the allowed nonlinearities. Finally, we note some interesting connections between the likelihood-based estimators and the classical spike-triggered average estimator, discuss some useful extensions of the basic model structure, and provide two novel applications to physiological data.     

Maximum likelihood estimation methods for multispectral randomfield image models

Considers the problem of estimating parameters of multispectral random field (RF) image models using maximum likelihood (ML) methods. For images with an assumed Gaussian distribution, analytical results are developed for multispectral simultaneous autoregressive (MSAR) and Markov random field (MMRF) models which lead to practical procedures for calculating ML estimates. Although previous work has provided least squares methods for parameter estimation, the superiority of the ML method is evidenced by experimental results provided in this work. The effectiveness of multispectral RF models using ML estimates in modeling color texture images is also demonstrated     

Maximum likelihood estimation in models with two systematic parts

Many standard univariate distributions have two or more parameters. In regression models, these may both depend on explanatory variables. General algorithms are developed for maximum likelihood estimation using a sequence of least squares calculations. The algorithms are an improvement over previously used methods. Applications to heteroscedastic normal models and to negative binomial models are described in detail. Other applications are indicated.     

Maximum likelihood estimation using square root information filters

The maximum likelihood parameter estimation algorithm is known to provide optimal estimates for linear time-invariant dynamic systems. However, the algorithm is computationally expensive and requires evaluations of the gradient of a log likelihood function and the Fisher information matrix. By using the square-root information filter, a numerically reliable algorithm to compute the required gradient and the Fisher information matrix is developed. The algorithm is a significant improvement over the methods based on the conventional Kalman filter. The square-root information filter relies on the use of orthogonal transformations that are well known for numerical reliability. This algorithm can be extended to real-time system identification and adaptive control     

Fast estimation of Gaussian mixture models for image segmentation

The expectation maximization algorithm has been classically used to find the maximum likelihood estimates of parameters in probabilistic models with unobserved data, for instance, mixture models. A key issue in such problems is the choice of the model complexity. The higher the number of components in the mixture, the higher will be the data likelihood, but also the higher will be the computational burden and data overfitting. In this work, we propose a clustering method based on the expectation maximization algorithm that adapts online the number of components of a finite Gaussian mixture model from multivariate data or method estimates the number of components and their means and covariances sequentially, without requiring any careful initialization. Our methodology starts from a single mixture component covering the whole data set and sequentially splits it incrementally during expectation maximization steps. The coarse to fine nature of the algorithm reduce the overall number of computations to achieve a solution, which makes the method particularly suited to image segmentation applications whenever computational time is an issue. We show the effectiveness of the method in a series of experiments and compare it with a state-of-the-art alternative technique both with synthetic data and real images, including experiments with images acquired from the iCub humanoid robot.     

Maximum likelihood parameter estimation algorithm for controlled autoregressive autoregressive models

A maximum likelihood parameter estimation algorithm is derived for controlled autoregressive autoregressive (CARAR) models based on the maximum likelihood principle. In this derivation, we use an estimated noise transfer function to filter the input–output data. The simulation results show that the proposed estimation algorithm can effectively estimate the parameters of such class of CARAR systems and give more accurate parameter estimates than the recursive generalized least-squares algorithm.     

Maximum likelihood estimation using probability density functions of order statistics

A variation of maximum likelihood estimation (MLE) of parameters that uses probability density functions of order statistic is presented. Results of this method are compared with traditional maximum likelihood estimation for complete and right-censored samples in a life test. Further, while the concept can be applied to most types of censored data sets, results are presented in the case of order statistic interval censoring, in which even a few order statistics estimate well, compared to estimates from complete and right-censored samples. Distributions investigated include the exponential, Rayleigh, and normal distributions. Computation methods using A Probability Programming Language running in Maple are more straightforward than existing methods using various numerical method algorithms.     

Maximum likelihood identification of stochastic linear systems

The maximum likelihood estimation of the coefficients of multiple output linear dynamical systems and the noise correlations from the noisy measurements of input and output are discussed. Conditions are derived under which the estimates converge to their true values as the number of measurements tend to infinity. The computational methods are illustrated by several numerical examples.     

Averaging, maximum penalized likelihood and Bayesian estimation forimproving Gaussian mixture probability density estimates

We apply the idea of averaging ensembles of estimators to probability density estimation. In particular, we use Gaussian mixture models which are important components in many neural-network applications. We investigate the performance of averaging using three data sets. For comparison, we employ two traditional regularization approaches, i.e., a maximum penalized likelihood approach and a Bayesian approach. In the maximum penalized likelihood approach we use penalty functions derived from conjugate Bayesian priors such that an expectation maximization (EM) algorithm can be used for training. In all experiments, the maximum penalized likelihood approach and averaging improved performance considerably if compared to a maximum likelihood approach. In two of the experiments, the maximum penalized likelihood approach outperformed averaging. In one experiment averaging was clearly superior. Our conclusion is that maximum penalized likelihood gives good results if the penalty term in the cost function is appropriate for the particular problem. If this is not the case, averaging is superior since it shows greater robustness by not relying on any particular prior assumption. The Bayesian approach worked very well on a low-dimensional toy problem but failed to give good performance in higher dimensional problems.     

Maximum likelihood identification of Wiener models

The Wiener model is a block oriented model, having a linear dynamic system followed by a static nonlinearity. The dominating approach to estimate the components of this model has been to minimize the error between the simulated and the measured outputs. We show that this will, in general, lead to biased estimates if there are other disturbances present than measurement noise. The implications of Bussgang’s theorem in this context are also discussed. For the case with general disturbances, we derive the Maximum Likelihood method and show how it can be efficiently implemented. Comparisons between this new algorithm and the traditional approach, confirm that the new method is unbiased and also has superior accuracy.     

Efficient greedy estimation of mixture models through a binary tree search

Unsupervised data clustering can be addressed by the estimation of mixture models, where the mixture components are associated to clusters in data space. In this paper we present a novel unsupervised classification algorithm based on the simultaneous estimation of the mixture’s parameters and the number of components (complexity). Its distinguishing aspect is the way the data space is searched. Our algorithm starts from a single component covering all the input space and iteratively splits components according to breadth first search on a binary tree structure that provides an efficient exploration of the possible solutions. The proposed scheme demonstrates important computational savings with respect to other state-of-the-art algorithms, making it particularly suited to scenarios where the performance time is an issue, such as in computer and robot vision applications. The initialization procedure is unique, allowing a deterministic evolution of the algorithm, while the parameter estimation is performed with a modification of the Expectation Maximization algorithm. To compare models with different complexity we use the Minimum Message Length information criteria that implement the trade-off between the number of components and data fit log-likelihood. We validate our new approach with experiments on synthetic data, and we test and compare to related approaches its computational efficiency in data-intensive image segmentation applications.     

Test for homogeneity in gamma mixture models using likelihood ratio

A testing problem of homogeneity in gamma mixture models is studied. It is found that there is a proportion of the penalized likelihood ratio test statistic that degenerates to zero. The limiting distribution of this statistic is found to be the chi-bar-square distributions. The degeneration is due to the negative-definiteness of a complicated random matrix, depending on the shape parameter under the null hypothesis. In light of this dependency, bounds on the distribution are introduced and a weighted average procedure is proposed. Simulation suggests that the results are accurate and consistent, and that the asymptotic result applies to the maximum likelihood estimator, obtained via an Expectation–Maximization algorithm.     

Fusion of possibly biased location estimates using Gaussian mixture models

A probabilistic framework for fusing location estimates, which may be biased and inconsistent, is presented. The proposed method, involving Gaussian mixture models (GMMs), utilizes prior information regarding the sensor bias, firstly, to reduce errors in the fused location estimate, and secondly, to produce a fused covariance matrix that better reflects the expected location error. Simulations are used to evaluate performance, relative to other techniques, such as the covariance union (CU) method. A passive geolocation application involving an airborne electronic support (ES) system is considered.     

Maximum likelihood estimation for generalized semi-markov processes

Parametric statistical inference for generalized semi-Markov processes is addressed. This class of processes encompasses a large number of real-world discrete-event stochastic systems. Because of its properties (e.g., consistency, asymptotic normality, etc.), maximum likelihood estimation is considered here. Under reasonable conditions on the process, we show that a maximum likelihood estimator exists, and that it converges to the true parameter at ratet ^–1/2, wheret is the length of the observation period. A related estimator, which is typically easier to compute, is also introduced. We show that the use of this estimator results in no loss of statistical efficiency. It is also shown that the estimation problem does decouple into separate subproblems when the process' transition probabilities and event distributions depend on different parameters.     

Maximum likelihood identification of noisy input-output models

This work deals with the identification of errors-in-variables models corrupted by white and uncorrelated Gaussian noises. By introducing an auxiliary process, it is possible to obtain a maximum likelihood solution of this identification problem, by means of a two-step iterative algorithm. This approach allows also to estimate, as a byproduct, the noise-free input and output sequences. Moreover, an analytic expression of the finite Cràmer-Rao lower bound is derived. The method does not require any particular assumption on the input process, however, the ratio of the noise variances is assumed as known. The effectiveness of the proposed algorithm has been verified by means of Monte Carlo simulations.     

Maximum likelihood estimation for constrained parameters of multinomial distributions—Application to Zipf-Mandelbrot models

A numerical maximum likelihood (ML) estimation procedure is developed for the constrained parameters of multinomial distributions. The main difficulty involved in computing the likelihood function is the precise and fast determination of the multinomial coefficients. For this the coefficients are rewritten into a telescopic product. The presented method is applied to the ML estimation of the Zipf-Mandelbrot (ZM) distribution, which provides a true model in many real-life cases. The examples discussed arise from ecological and medical observations. Based on the estimates, the hypothesis that the data is ZM distributed is tested using a chi-square test. The computer code of the presented procedure is available on request by the author.