Maximum likelihood estimation of mixtures of factor analyzers

Mixtures of factor analyzers have been receiving wide interest in statistics as a tool for performing clustering and dimension reduction simultaneously. In this model it is assumed that, within each component, the data are generated according to a factor model. Therefore, the number of parameters on which the covariance matrices depend is reduced. Several estimation methods have been proposed for this model, both in the classical and in the Bayesian framework. However, so far, a direct maximum likelihood procedure has not been developed. This direct estimation problem, which simultaneously allows one to derive the information matrix for the mixtures of factor analyzers, is solved. The effectiveness of the proposed procedure is shown on a simulation study and on a toy example.     

Maximum likelihood estimation for constrained parameters of multinomial distributions—Application to Zipf-Mandelbrot models

A numerical maximum likelihood (ML) estimation procedure is developed for the constrained parameters of multinomial distributions. The main difficulty involved in computing the likelihood function is the precise and fast determination of the multinomial coefficients. For this the coefficients are rewritten into a telescopic product. The presented method is applied to the ML estimation of the Zipf-Mandelbrot (ZM) distribution, which provides a true model in many real-life cases. The examples discussed arise from ecological and medical observations. Based on the estimates, the hypothesis that the data is ZM distributed is tested using a chi-square test. The computer code of the presented procedure is available on request by the author.     

Maximum likelihood estimation for continuous-time autoregressive models by relaxation on residual variances ratio parameters

In this paper we derive an explicit expression for the log likelihood function of a continuous-time autoregressive model. Then, using earlier results relating the autoregressive coefficients to the set of positive parameters called residual variances ratios, we develop an iterative algorithm for computing the maximum likelihood estimator of the model, similar to one in the discrete-time case. A simple noniterative estimation method, which can be used to produce an initial estimate for the algorithm, is also proposed.     

Maximum likelihood estimation via the extended covariance and combined square-root filters

The method of maximum likelihood is a general method for parameter estimation and is often used in system identification. To implement it, it is necessary to maximize the likelihood function, which is usually done using the gradient approach. It involves the computation of the likelihood gradient with respect to unknown system parameters. For linear stochastic system models this leads to the implementation of the Kalman filter, which is known to be numerically unstable. The aim of this work is to present new efficient algorithms for likelihood gradient evaluation. They are more reliable in practice and improve robustness of computations against roundoff errors. All algorithms are derived in measurement and time updates form. The comparison with the conventional Kalman filter approach and results of numerical experiments are given.     

Maximum likelihood analysis of algorithms and data structures

We present a new approach for an average-case analysis of algorithms and data structures that supports a non-uniform distribution of the inputs and is based on the maximum likelihood training of stochastic grammars. The approach is exemplified by an analysis of the expected size of binary tries as well as by three sorting algorithms and it is compared to the known results that were obtained by traditional techniques. Investigating traditional settings like the random permutation model, we rediscover well-known results formerly derived by pure analytic methods; changing to biased data yields original results. All but one step of our analysis can be automated on top of a computer-algebra system. Thus our new approach can reduce the effort required for an average-case analysis, allowing for the consideration of realistic input distributions with unknown distribution functions at the same time. As a by-product, our approach yields an easy way to generate random combinatorial objects according to various probability distributions.     

Maximum likelihood identification of Wiener models

The Wiener model is a block oriented model, having a linear dynamic system followed by a static nonlinearity. The dominating approach to estimate the components of this model has been to minimize the error between the simulated and the measured outputs. We show that this will, in general, lead to biased estimates if there are other disturbances present than measurement noise. The implications of Bussgang’s theorem in this context are also discussed. For the case with general disturbances, we derive the Maximum Likelihood method and show how it can be efficiently implemented. Comparisons between this new algorithm and the traditional approach, confirm that the new method is unbiased and also has superior accuracy.     

Robust maximum likelihood estimation for stochastic state space model with observation outliers

The objective of this paper is to develop a robust maximum likelihood estimation (MLE) for the stochastic state space model via the expectation maximisation algorithm to cope with observation outliers. Two types of outliers and their influence are studied in this paper: namely,the additive outlier (AO) and innovative outlier (IO). Due to the sensitivity of the MLE to AO and IO, we propose two techniques for robustifying the MLE: the weighted maximum likelihood estimation (WMLE) and the trimmed maximum likelihood estimation (TMLE). The WMLE is easy to implement with weights estimated from the data; however, it is still sensitive to IO and a patch of AO outliers. On the other hand, the TMLE is reduced to a combinatorial optimisation problem and hard to implement but it is efficient to both types of outliers presented here. To overcome the difficulty, we apply the parallel randomised algorithm that has a low computational cost. A Monte Carlo simulation result shows the efficiency of the proposed algorithms.     

Recursive maximum likelihood parameter estimation for state space systems using polynomial chaos theory

This paper combines polynomial chaos theory with maximum likelihood estimation for a novel approach to recursive parameter estimation in state-space systems. A simulation study compares the proposed approach with the extended Kalman filter to estimate the value of an unknown damping coefficient of a nonlinear Van der Pol oscillator. The results of the simulation study suggest that the proposed polynomial chaos estimator gives comparable results to the filtering method but may be less sensitive to user-defined tuning parameters. Because this recursive estimator is applicable to linear and nonlinear dynamic systems, the authors portend that this novel formulation will be useful for a broad range of estimation problems.     

多径相干信号到达角和衰落系数的最大似然估计

针对多径相干信号到达角(DOA)和衰落系数(FC)的估计问题,在已知信号波形条件下,提出了一种联合估计DOA和衰落系数的改进最大似然估计方法。通过改进最大似然估计求极值获得DOA和衰落系数的解,在联合估计参数时,使用交替极大值技术,降低了估计算法的复杂度,并且推导了最大似然估计的Cramér-Rao界。理论分析和实验表明,该方法与现有方法相比具有更好的估计性能,当来自不同辐射源的多径信号中存在相同DOA时,仍然可以得到很好的估计效果。     

Maximum likelihood identification of noisy input-output models

This work deals with the identification of errors-in-variables models corrupted by white and uncorrelated Gaussian noises. By introducing an auxiliary process, it is possible to obtain a maximum likelihood solution of this identification problem, by means of a two-step iterative algorithm. This approach allows also to estimate, as a byproduct, the noise-free input and output sequences. Moreover, an analytic expression of the finite Cràmer-Rao lower bound is derived. The method does not require any particular assumption on the input process, however, the ratio of the noise variances is assumed as known. The effectiveness of the proposed algorithm has been verified by means of Monte Carlo simulations.     

Maximum likelihood stochastic gradient estimation for Hammerstein systems with colored noise based on the key term separation technique

This paper considers the identification problems of Hammerstein finite impulse response moving average (FIR-MA) systems using the maximum likelihood principle and stochastic gradient method based on the key term separation technique. In order to improve the convergence rate, a maximum likelihood multi-innovation stochastic gradient algorithm is presented. The simulation results show that the proposed algorithms can effectively estimate the parameters of the Hammerstein FIR-MA systems.     

Frequency domain maximum likelihood estimation of linear dynamic errors-in-variables models

This paper studies the linear dynamic errors-in-variables problem for filtered white noise excitations. First, a frequency domain Gaussian maximum likelihood (ML) estimator is constructed that can handle discrete-time as well as continuous-time models on (a) part(s) of the unit circle or imaginary axis. Next, the ML estimates are calculated via a computationally simple and numerically stable Gauss-Newton minimization scheme. Finally, the Cramér-Rao lower bound is derived.     

On the equivalence of time and frequency domain maximum likelihood estimation

Maximum likelihood estimation has a rich history. It has been successfully applied to many problems including dynamical system identification. Different approaches have been proposed in the time and frequency domains. In this paper we discuss the relationship between these approaches and we establish conditions under which the different formulations are equivalent for finite length data. A key point in this context is how initial (and final) conditions are considered and how they are introduced in the likelihood function.     

高维数据固有维数的自适应极大似然估计

如何确定高维数据的固有维数是降维成功与否的关键。基于极大似然估计（MLE）的维数估计方法是一种新近出现的方法,实现简单,选择合适的近邻能取得不错的结果。但当近邻数过小或过大时,均有比较明显的偏差。其根本原因是没有考虑每个点对固有维数的不同贡献。在充分考虑数据集的分布信息之后,提出了一种改进的MLE——自适应极大似然估计(AMLE)。实验表明,无论在合成数据集还是真实数据集上,AMLE较MLE在估计准确度上均有很大的提高,对近邻数的变化也不甚敏感。     

An analysis of separable least squares data driven local coordinates for maximum likelihood estimation of linear systems

In this paper we study a novel parametrization for state-space systems, namely separable least squares data driven local coordinates (slsDDLC). The parametrization by slsDDLC has recently been successfully applied to maximum likelihood estimation of linear dynamic systems. In a simulation study, the use of slsDDLC has led to numerical advantages in comparison to the use of more conventional parametrizations, including data driven local coordinates (DDLC). However, an analysis of properties of slsDDLC, which are relevant to identification, has not been performed up to now. In this paper, we provide insights into the geometry and topology of the slsDDLC construction and show a number of results which are important for actual identification, in particular for maximum likelihood estimation. We also prove that the separable least squares methodology is indeed guaranteed to be applicable to maximum likelihood estimation of linear dynamic systems in typical situations.     

Maximum likelihood ratio test for the stability of sequence of Gaussian random processes

Based on the Karhunen-Loeve expansion, the maximum likelihood ratio test for the stability of sequence of Gaussian random processes is investigated. The likelihood function is based on the first p scores of eigenfunctions in the Karhunen-Loeve expansion for Gaussian random processes. Though the scores are unobservable, we show that the effect of the difference between scores and their estimators is negligible as the sample size tends to infinity. The asymptotic distribution is proved to be the Gumbel extreme value distribution. Under the alternative the test is shown to be consistent. For different choices of p, simulation results show that the test behaves quite well in finite samples. The test procedure is also applied to the annual temperature data of central England. The results show that the temperatures have risen in the last twenty years, however there is no evidence to show that the autocovariance functions of the temperatures have changed among the range of the observations.     

Microcalcification detection using fuzzy logic and scale space approaches

Breast cancer is one of the leading causes of women mortality in the world. Since the causes are unknown, breast cancer cannot be prevented. It is difficult for radiologists to provide both accurate and uniform evaluation over the enormous number of mammograms generated in widespread screening. Computer-aided mammography diagnosis is an important and challenging task. Microcalcifications and masses are the early signs of breast carcinomas and their detection is one of the key issues for breast cancer control. In this study, a novel approach to microcalcification detection based on fuzzy logic and scale space techniques is presented. First, we employ fuzzy entropy principal and fuzzy set theory to fuzzify the images. Then, we enhance the fuzzified image. Finally, scale-space and Laplacian-of-Gaussian filter techniques are used to detect the sizes and locations of microcalcifications. A free-response operating characteristic curve is used to evaluate the performance. The major advantage of the proposed method is its ability to detect microcalcifications even in the mammograms of very dense breasts. A data set of 40 mammograms (Nijmegen database) containing 105 clusters of microcalcifications is studied. Experimental results demonstrate that the microcalcifications can be accurately and efficiently detected using the proposed approach. It can produce lower false positives and false negatives than the existing methods.     

A quick method for estimation of parameters of individual pulses in a multipath signal

Consider a multipath signal whose individual signals are deterministic known increasing or decreasing pulses. The problem is to estimate the amplitudes and delay times of individual signals. Several methods have been devoted to the solution of these unknown parameters. The maximum likelihood (ML) estimation and the estimate maximize (EM) algorithm are commonly used, but they are computationally intensive and still insufficient to obtain accurate estimations. The method can provide a quick and accurate estimate of the amplitudes and arrival (delay) times, even in the closely spaced multipaths and heavy noise.     

Accuracy analysis of time domain maximum likelihood method and sample maximum likelihood method for errors-in-variables and output error identification

For identifying errors-in-variables models, the time domain maximum likelihood (TML) method and the sample maximum likelihood (SML) method are two approaches. Both methods give optimal estimation accuracy but under different assumptions. In the TML method, an important assumption is that the noise-free input signal is modelled as a stationary process with rational spectrum. For SML, the noise-free input needs to be periodic. It is interesting to know which of these assumptions contain more information to boost the estimation performance. In this paper, the estimation accuracy of the two methods is analyzed statistically for both errors-in-variables (EIV) and output error models (OEM). Numerical comparisons between these two estimates are also done under different signal-to-noise ratios (SNRs). The results suggest that TML and SML have similar estimation accuracy at moderate or high SNR for EIV. For OEM identification, these two methods have the same accuracy at any SNR.     

Mutual information aspects of scale space images

In image registration, mutual information is a well-performing measure based on principles of uncertainty. Similarly, in image analysis the Gaussian scale space, based on minimal assumptions of the image, is used to derive intrinsic properties of an image. This paper starts an investigation of a combination of both methods. This combination results in a double parameterized mutual information measure using local information of the image. For single modality matching best response is found for coinciding parameters. Then critical values are found for which the parameterized mutual information has extrema. First results on multi-modality matching show that different parameter values instead of coinciding values yield the best response for the parameterized mutual information.