Mean and variance adaptation within the MLLR framework

One of the key issues for adaptation algorithms is to modify a large number of parameters with only a small amount of adaptation data. Speaker adaptation techniques try to obtain near speaker-dependent (SD) performance with only small amounts of speaker-specific data, and are often based on initial speaker-independent (SI) recognition systems. Some of these speaker adaptation techniques may also be applied to the task of adaptation to a new acoustic environment. In this case an SI recognition system trained in, typically, a clean acoustic environment is adapted to operate in a new, noise-corrupted, acoustic environment. This paper examines the maximum likelihood linear regression (MLLR) adaptation technique. MLLR estimates linear transformations for groups of model parameters to maximize the likelihood of the adaptation data. Previously, MLLR has been applied to the mean parameters in mixture-Gaussian HMM systems. In this paper MLLR is extended to also update the Gaussian variances and re-estimation formulae are derived for these variance transforms. MLLR with variance compensation is evaluated on several large vocabulary recognition tasks. The use of mean and variance MLLR adaptation was found to give an additional 2% to 7% decrease in word error rate over mean-only MLLR adaptation.     

Robust diagnostics for the heteroscedastic regression model

The assumption of equal variance in the normal regression model is not always appropriate. Cook and Weisberg (1983) provide a score test to detect heteroscedasticity, while Patterson and Thompson (1971) propose the residual maximum likelihood (REML) estimation to estimate variance components in the context of an unbalanced incomplete-block design. REML is often preferred to the maximum likelihood estimation as a method of estimating covariance parameters in a linear model. However, outliers may have some effect on the estimate of the variance function. This paper incorporates the maximum trimming likelihood estimation ( [Hadi and Luce?o, 1997] and [Vandev and Neykov, 1998]) in REML to obtain a robust estimation of modelling variance heterogeneity. Both the forward search algorithm of Atkinson (1994) and the fast algorithm of Neykov et al. (2007) are employed to find the resulting estimator. Simulation and real data examples are used to illustrate the performance of the proposed approach.     

Noise variance estimation based on measured maximums of sampled subsets

In this paper, an estimation of the Gaussian noise variance based on observed (measured) maximums of subsets of samples is given. Circumstances of the measurement environment being limited, only maximums of subsets of samples are available and the non-constant variance of the Gaussian noise can be estimated. In the case of power line noise, the variance of the zero mean Gaussian noise is a periodic function of the a-priory known parameterization.Variance function parameters estimation is computed in two steps, first the estimation formula of the constant variance Gaussian noise is applied to a certain subset of samples and second, the least mean square (LMS) criterion is applied to fit the parametrized variance function to estimated variances.The maximum likelihood estimation (MLE) criterion is applied to derive estimators of the variance function parameters. Beside that, the quotient of the variance of the zero mean Gaussian noise and its maximums is evolved explicitly.Experimental results on real and simulated data are given to demonstrate their accuracy.     

MLMATERN: A computer program for maximum likelihood inference with the spatial Matérn covariance model

The Matérn covariance scheme is of great importance in many geostatistical applications where the smoothness or differentiability of the random field that models a natural phenomenon is of interest. In addition to the range and nugget parameters, the flexibility of the Matérn model is provided by the so-called smoothness parameter which controls the degree of smoothness of the random field. It has been the usual practice in geostatistics to fit theoretical semivariograms like the spherical or exponential, thus implicitly assuming the smoothness parameter to be known, without questioning if there is any theoretical or empirical basis to justify such assumption. On the other hand, if only a small number of sparse experimental data are available, it is more critical to ask if the smoothness parameter can be identified with statistical reliability. Maximum likelihood estimation of spatial covariance parameters of the Matérn model has been used to address the previous questions. We have developed a general algorithm for estimating the parameters of a Matérn covariance (or semivariogram) scheme, where the model may be isotropic or anisotropic, the nugget variance can be included in the model if desired, and the uncertainty of the estimates is provided in terms of variance–covariance matrix (or standard error-coefficient of correlation matrix) as well as likelihood profiles for each parameter in the covariance model. It is assumed that the empirical data are a realization of a Gaussian process. Our program allows the presence of a polynomial trend of order zero (constant global mean), one (linear trend) or two (quadratic trend). The restricted maximum likelihood method has also been implemented in the program as an alternative to the standard maximum likelihood. Simulation results are given in order to investigate the sampling distribution of the parameters for small samples. Furthermore, a case study is provided to show a real practical example where the smoothness parameter needs to be estimated.     

Adaptive control with recursive identification for stochastic linear systems

This paper presents a stochastic adaptive control algorithm which is shown to possess the following properties when applied to a possibly unstable, inverse stable, linear stochastic system with unknown parameters, whenever that system satisfies a certain positive real condition on its (moving average) noise dynamics. 1) The adaptive control part of the algorithm stabilizes and asymptotically optimizes the behavior of the system in the sense that the (limit of the) sample mean-square variation of the-output around a given demand level equals that of a minimum variance control strategy implemented with known parameters. This optimal behavior is subject to an offset μ²where μ²is the variance of a dither signal added to the control action in order to produce a "continually disturbed control." Formu^{2} > 0, it is shown that the input-output process satisfies a persistent excitation property, and hence, subject to a simple identifiability condition, the next property holds. 2) The observed input and output of the controlled system may be taken as inputs to an approximate maximum likelihood algorithm (AML) which generates strongly consistent estimates of the system's parameters. Results are presented for the scalar and multivariable cases.     

Likelihood ratio test for estimation of time-varying systems

A likelihood ratio test is used to detect the portion of the prediction error variance caused by changes in parameters. It results in a simple rule for adjusting the dispersion factor in the parameter covariance matrix. The adaptation rule given by Jazwinski (1970) is shown to be a special case of one derived     

Multilevel cumulative logistic regression model with random effects: Application to British social attitudes panel survey data

A multilevel model for ordinal data in generalized linear mixed models (GLMM) framework is developed to account for the inherent dependencies among observations within clusters. Motivated by a data set from the British Social Attitudes Panel Survey (BSAPS), the random district effects and respondent effects are incorporated into the linear predictor to accommodate the nested clusterings. The fixed (random) effects are estimated (predicted) by maximizing the penalized quasi likelihood (PQL) function, whereas the variance component parameters are obtained via the restricted maximum likelihood (REML) estimation method. The model is employed to analyze the BSAPS data. Simulation studies are conducted to assess the performance of estimators.     

Maximum likelihood estimation of Gaussian mixture models using stochastic search

Gaussian mixture models (GMM), commonly used in pattern recognition and machine learning, provide a flexible probabilistic model for the data. The conventional expectation–maximization (EM) algorithm for the maximum likelihood estimation of the parameters of GMMs is very sensitive to initialization and easily gets trapped in local maxima. Stochastic search algorithms have been popular alternatives for global optimization but their uses for GMM estimation have been limited to constrained models using identity or diagonal covariance matrices. Our major contributions in this paper are twofold. First, we present a novel parametrization for arbitrary covariance matrices that allow independent updating of individual parameters while retaining validity of the resultant matrices. Second, we propose an effective parameter matching technique to mitigate the issues related with the existence of multiple candidate solutions that are equivalent under permutations of the GMM components. Experiments on synthetic and real data sets show that the proposed framework has a robust performance and achieves significantly higher likelihood values than the EM algorithm.     

Bias and skewness in a general extreme-value regression model

In this paper we introduce a general extreme-value regression model and derive Cox and Snell’s (1968) general formulae for second-order biases of maximum likelihood estimates (MLEs) of the parameters. We obtain formulae which can be computed by means of weighted linear regressions. Furthermore, we give the skewness of order n^−1/2 of the maximum likelihood estimators of the parameters by using Bowman and Shenton’s (1988) formula. A simulation study with results obtained with the use of Cox and Snell’s (1968) formulae is discussed. Practical uses of this model and of the derived formulae for bias correction are also presented.     

Point and interval estimation for extreme-value regression model under Type-II censoring

Inference for the extreme-value regression model under Type-II censoring is discussed. The likelihood function and the score functions of the unknown parameters are presented. The asymptotic variance-covariance matrix is derived through the inverse of the expected Fisher information matrix. Since the maximum likelihood estimators (MLE) cannot be solved analytically, an approximation to these MLE are proposed. The variance-covariance matrix of these approximate estimators is also derived. Next, confidence intervals are proposed based on the MLE and the approximate estimators. An extensive simulation study is carried out in order to study the bias and variance of all these estimators. We also examine the coverage probabilities as well as the expected widths of the confidence intervals. Finally, all the inferential procedures discussed here are illustrated with practical data.     

Optimal approximation of signal priors

In signal restoration by Bayesian inference, one typically uses a parametric model of the prior distribution of the signal. Here, we consider how the parameters of a prior model should be estimated from observations of uncorrupted signals. A lot of recent work has implicitly assumed that maximum likelihood estimation is the optimal estimation method. Our results imply that this is not the case. We first obtain an objective function that approximates the error occurred in signal restoration due to an imperfect prior model. Next, we show that in an important special case (small gaussian noise), the error is the same as the score-matching objective function, which was previously proposed as an alternative for likelihood based on purely computational considerations. Our analysis thus shows that score matching combines computational simplicity with statistical optimality in signal restoration, providing a viable alternative to maximum likelihood methods. We also show how the method leads to a new intuitive and geometric interpretation of structure inherent in probability distributions.     

Learning to Importance Sample in Primary Sample Space

Importance sampling is one of the most widely used variance reduction strategies in Monte Carlo rendering. We propose a novel importance sampling technique that uses a neural network to learn how to sample from a desired density represented by a set of samples. Our approach considers an existing Monte Carlo rendering algorithm as a black box. During a scene‐dependent training phase, we learn to generate samples with a desired density in the primary sample space of the renderer using maximum likelihood estimation. We leverage a recent neural network architecture that was designed to represent real‐valued non‐volume preserving (“Real NVP”) transformations in high dimensional spaces. We use Real NVP to non‐linearly warp primary sample space and obtain desired densities. In addition, Real NVP efficiently computes the determinant of the Jacobian of the warp, which is required to implement the change of integration variables implied by the warp. A main advantage of our approach is that it is agnostic of underlying light transport effects, and can be combined with an existing rendering technique by treating it as a black box. We show that our approach leads to effective variance reduction in several practical scenarios.     

Gaussian mixture spline trajectory: learning from a dataset,generating trajectories without one

Most optimization-based motion planners use a naive linear initialization, which does not use previous planning experience. We present an algorithm called ‘Gaussian mixture spline trajectory’ (GMST) that leverages motion datasets for generating trajectories for new planning problems. Unlike other trajectory prediction algorithms, our method does not retrieve trajectories from a dataset. Instead, it first uses a Gaussian mixture model (GMM) to modelize the likelihood of the trajectories to be inside the dataset and then uses the GMM's parameters to generate new trajectories. As the use of the dataset is restricted only to the learning phase it can take advantage of very large datasets. Using both abstract and robot system planning problems, we show that the GMST algorithm decreases the computation time and number of iterations of optimization-based planners while increasing their success rates as compared to that obtained with linear initialization.     

Robust distributed maximum likelihood estimation with dependent quantized data

In this paper, we consider the distributed maximum likelihood estimation (MLE) with dependent quantized data under the assumption that the structure of the joint probability density function (pdf) is known, but it contains unknown deterministic parameters. The parameters may include different vector parameters corresponding to marginal pdfs and parameters that describe the dependence of observations across sensors. Since MLE with a single quantizer is sensitive to the choice of thresholds due to the uncertainty of pdf, we concentrate on MLE with multiple groups of quantizers (which can be determined by the use of prior information or some heuristic approaches) to fend off against the risk of a poor/outlier quantizer. The asymptotic efficiency of the MLE scheme with multiple quantizers is proved under some regularity conditions and the asymptotic variance is derived to be the inverse of a weighted linear combination of Fisher information matrices based on multiple different quantizers which can be used to show the robustness of our approach. As an illustrative example, we consider an estimation problem with a bivariate non-Gaussian pdf that has applications in distributed constant false alarm rate (CFAR) detection systems. Simulations show the robustness of the proposed MLE scheme especially when the number of quantized measurements is small.     

Identification methods in a unified framework

The paper derives a framework suitable to discuss the classical Koopmans-Levin (KL) and maximum likelihood (ML) algorithms to estimate parameters of errors-in-variables linear models in a unified way. Using the capability of the unified approach a new parameter estimation algorithm is presented offering flexibility to ensure acceptable variance in the estimated parameters. The developed algorithm is based on the application of Hankel matrices of variable size and can equally be considered as a generalized version of the KL method (GKL) or as a reduced version of the ML estimation. The methodology applied to derive the GKL algorithm is used to present a straightforward derivation of the subspace identification algorithm.     

Estimation in the probit normal model for binary outcomes using the SAEM algorithm

Generalized linear mixed models (GLMM) form a very general class of random effects models for discrete and continuous responses in the exponential family. They are useful in a variety of applications. The traditional likelihood approach for GLMM usually involves high dimensional integrations which are computationally intensive. In this work, we investigate the case of binary outcomes analyzed under a two stage probit normal model with random effects. First, it is shown how ML estimates of the fixed effects and variance components can be computed using a stochastic approximation of the EM algorithm (SAEM). The SAEM algorithm can be applied directly, or in conjunction with a parameter expansion version of EM to speed up the convergence. A procedure is also proposed to obtain REML estimates of variance components and REML-based estimates of fixed effects. Finally an application to a real data set involving a clinical trial is presented, in which these techniques are compared to other procedures (penalized quasi-likelihood, maximum likelihood, Bayesian inference) already available in classical softwares (SAS Glimmix, SAS Nlmixed, WinBUGS), as well as to a Monte Carlo EM (MCEM) algorithm.     

Software reliability model with optimal selection of failure data

The possibility of obtaining more accurate predictions of future failures by excluding or giving lower weight to the earlier failure counts is suggested. Although data aging techniques such as moving average and exponential smoothing are frequently used in other fields, such as inventory control, the author did not find use of data aging in the various models surveyed. A model that includes the concept of selecting a subset of the failure data is the Schneidewind nonhomogeneous Poisson process (NHPP) software reliability model. In order to use the concept of data aging, there must be a criterion for determining the optimal value of the starting failure count interval. Four criteria for identifying the optimal starting interval for estimating model parameters are evaluated The first two criteria treat the failure count interval index as a parameter by substituting model functions for data vectors and optimizing on functions obtained from maximum likelihood estimation techniques. The third uses weighted least squares to maintain constant variance in the presence of the decreasing failure rate assumed by the model. The fourth criterion is the familiar mean square error. It is shown that significantly improved reliability predictions can be obtained by using a subset of the failure data. The US Space Shuttle on-board software is used as an example     

Maximum likelihood estimation using probability density functions of order statistics

A variation of maximum likelihood estimation (MLE) of parameters that uses probability density functions of order statistic is presented. Results of this method are compared with traditional maximum likelihood estimation for complete and right-censored samples in a life test. Further, while the concept can be applied to most types of censored data sets, results are presented in the case of order statistic interval censoring, in which even a few order statistics estimate well, compared to estimates from complete and right-censored samples. Distributions investigated include the exponential, Rayleigh, and normal distributions. Computation methods using A Probability Programming Language running in Maple are more straightforward than existing methods using various numerical method algorithms.     

A statistical model for contours in images

In this paper, we describe a statistical model for the gradient vector field of the gray level in images validated by different experiments. Moreover, we present a global constrained Markov model for contours in images that uses this statistical model for the likelihood. Our model is amenable to an iterative conditional estimation (ICE) procedure for the estimation of the parameters; our model also allows segmentation by means of the simulated annealing (SA) algorithm, the iterated conditional modes (ICM) algorithm, or the modes of posterior marginals (MPM) Monte Carlo (MC) algorithm. This yields an original unsupervised statistical method for edge-detection, with three variants. The estimation and the segmentation procedures have been tested on a total of 160 images. Those tests indicate that the model and its estimation are valid for applications that require an energy term based on the log-likelihood ratio. Besides edge-detection, our model can be used for semiautomatic extraction of contours, localization of shapes, non-photo-realistic rendering; more generally, it might be useful in various problems that require a statistical likelihood for contours.     

Learning in Gibbsian fields: how accurate and how fast can it be?

Gibbsian fields or Markov random fields are widely used in Bayesian image analysis, but learning Gibbs models is computationally expensive. The computational complexity is pronounced by the recent minimax entropy (FRAME) models which use large neighborhoods and hundreds of parameters. In this paper, we present a common framework for learning Gibbs models. We identify two key factors that determine the accuracy and speed of learning Gibbs models: The efficiency of likelihood functions and the variance in approximating partition functions using Monte Carlo integration. We propose three new algorithms. In particular, we are interested in a maximum satellite likelihood estimator, which makes use of a set of precomputed Gibbs models called "satellites" to approximate likelihood functions. This algorithm can approximately estimate the minimax entropy model for textures in seconds in a HP workstation. The performances of various learning algorithms are compared in our experiments