Bayesian learning of inverted Dirichlet mixtures for SVM kernels generation

We describe approaches for positive data modeling and classification using both finite inverted Dirichlet mixture models and support vector machines (SVMs). Inverted Dirichlet mixture models are used to tackle an outstanding challenge in SVMs namely the generation of accurate kernels. The kernels generation approaches, grounded on ideas from information theory that we consider, allow the incorporation of data structure and its structural constraints. Inverted Dirichlet mixture models are learned within a principled Bayesian framework using both Gibbs sampler and Metropolis-Hastings for parameter estimation and Bayes factor for model selection (i.e., determining the number of mixture’s components). Our Bayesian learning approach uses priors, which we derive by showing that the inverted Dirichlet distribution belongs to the family of exponential distributions, over the model parameters, and then combines these priors with information from the data to build posterior distributions. We illustrate the merits and the effectiveness of the proposed method with two real-world challenging applications namely object detection and visual scenes analysis and classification.     

Expectation-maximization algorithms for inference in Dirichlet processes mixture

Mixture models are ubiquitous in applied science. In many real-world applications, the number of mixture components needs to be estimated from the data. A popular approach consists of using information criteria to perform model selection. Another approach which has become very popular over the past few years consists of using Dirichlet processes mixture (DPM) models. Both approaches are computationally intensive. The use of information criteria requires computing the maximum likelihood parameter estimates for each candidate model whereas DPM are usually trained using Markov chain Monte Carlo (MCMC) or variational Bayes (VB) methods. We propose here original batch and recursive expectation-maximization algorithms to estimate the parameters of DPM. The performance of our algorithms is demonstrated on several applications including image segmentation and image classification tasks. Our algorithms are computationally much more efficient than MCMC and VB and outperform VB on an example.     

Comparison of Methods for Identifying Phenotype Subgroups Using Categorical Features Data With Application to Autism Spectrum Disorder

We evaluate the performance of the Dirichlet process mixture (DPM) and the latent class model (LCM) in identifying autism phenotype subgroups based on categorical autism spectrum disorder (ASD) diagnostic features from the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition Text Revision. A simulation study is designed to mimic the diagnostic features in the ASD dataset in order to evaluate the LCM and DPM methods in this context. Likelihood based information criteria and DPM partitioning are used to identify the best fitting models. The Rand statistic is used to compare the performance of the methods in recovering simulated phenotype subgroups. Our results indicate excellent recovery of the simulated subgroup structure for both methods. The LCM performs slightly better than DPM when the correct number of latent subgroups is selected a priori. The DPM method utilizes a maximum a posteriori (MAP) criterion to estimate the number of classes, and yielded results in fair agreement with the LCM method. Comparison of model fit indices in identifying the best fitting LCM showed that adjusted Bayesian information criteria (ABIC) picks the correct number of classes over 90% of the time. Thus, when diagnostic features are categorical and there is some prior information regarding the number of latent classes, LCM in conjunction with ABIC is preferred.     

Generalized Dirichlet priors for Naïve Bayesian classifiers with multinomial models in document classification

The generalized Dirichlet distribution has been shown to be a more appropriate prior than the Dirichlet distribution for naïve Bayesian classifiers. When the dimension of a generalized Dirichlet random vector is large, the computational effort for calculating the expected value of a random variable can be high. In document classification, the number of distinct words that is the dimension of a prior for naïve Bayesian classifiers is generally more than ten thousand. Generalized Dirichlet priors can therefore be inapplicable for document classification from the viewpoint of computational efficiency. In this paper, some properties of the generalized Dirichlet distribution are established to accelerate the calculation of the expected values of random variables. Those properties are then used to construct noninformative generalized Dirichlet priors for naïve Bayesian classifiers with multinomial models. Our experimental results on two document sets show that generalized Dirichlet priors can achieve a significantly higher prediction accuracy and that the computational efficiency of naïve Bayesian classifiers is preserved.     

基于狄利克雷过程混合模型的内外先验融合

近年来,使用高斯混合模型作为块先验的贝叶斯方法取得了优秀的图像复原性能,针对这类模型分量固定及主要依赖外部学习的缺点,提出了一种新的基于狄利克雷过程混合模型的图像先验模型。该模型从干净图像数据库中学习外部通用先验,从退化图像中学习内部先验,借助模型中统计量的可累加性自然实现内外部先验融合。通过聚类的新增及归并机制,模型的复杂度随着数据的增大或缩小而自适应地变化,可以学习到可解释及紧凑的模型。为了求解所有隐变量的变分后验分布,提出了一种结合新增及归并机制的批次更新可扩展变分算法,解决了传统坐标上升算法在大数据集下效率较低、容易陷入局部最优解的问题。在图像去噪及填充实验中,相比传统方法,所提模型无论在客观质量评价还是视觉观感上都更有优势,验证了该模型的有效性。     

Bayesian analysis of the patterns of biological susceptibility via reversible jump MCMC sampling

In some biological experiments, it is quite common that laboratory subjects differ in their patterns of susceptibility to a treatment. Finite mixture models are useful in those situations. In this paper we model the number of components and the component parameters jointly, and base inference about these quantities on their posterior probabilities, making use of the reversible jump Markov chain Monte Carlo methods. In particular, we apply the methodology to the analysis of univariate normal mixtures with multidimensional parameters, using a hierarchical prior model that allows weak priors while avoiding improper priors in the mixture context. The practical significance of the proposed method is illustrated with a dose-response data set.     

A countably infinite mixture model for clustering and feature selection

Mixture modeling is one of the most useful tools in machine learning and data mining applications. An important challenge when applying finite mixture models is the selection of the number of clusters which best describes the data. Recent developments have shown that this problem can be handled by the application of non-parametric Bayesian techniques to mixture modeling. Another important crucial preprocessing step to mixture learning is the selection of the most relevant features. The main approach in this paper, to tackle these problems, consists on storing the knowledge in a generalized Dirichlet mixture model by applying non-parametric Bayesian estimation and inference techniques. Specifically, we extend finite generalized Dirichlet mixture models to the infinite case in which the number of components and relevant features do not need to be known a priori. This extension provides a natural representation of uncertainty regarding the challenging problem of model selection. We propose a Markov Chain Monte Carlo algorithm to learn the resulted infinite mixture. Through applications involving text and image categorization, we show that infinite mixture models offer a more powerful and robust performance than classic finite mixtures for both clustering and feature selection.     

Dirichlet Gaussian mixture model: Application to image segmentation

Gaussian mixture model based on the Dirichlet distribution (Dirichlet Gaussian mixture model) has recently received great attention for modeling and processing data. This paper studies the new Dirichlet Gaussian mixture model for image segmentation. First, we propose a new way to incorporate the local spatial information between neighboring pixels based on the Dirichlet distribution. The main advantage is its simplicity, ease of implementation and fast computational speed. Secondly, existing Dirichlet Gaussian model uses complex log-likelihood function and require many parameters that are difficult to estimate. The total parameters in the proposed model lesser and the log-likelihood function have a simpler form. Finally, to estimate the parameters of the proposed Dirichlet Gaussian mixture model, a gradient method is adopted to minimize the negative log-likelihood function. Numerical experiments are conducted using the proposed model on various synthetic, natural and color images. We demonstrate through extensive simulations that the proposed model is superior to other algorithms based on the model-based techniques for image segmentation.     

Variational learning for Dirichlet process mixtures of Dirichlet distributions and applications

In this paper, we propose a Bayesian nonparametric approach for modeling and selection based on a mixture of Dirichlet processes with Dirichlet distributions, which can also be seen as an infinite Dirichlet mixture model. The proposed model uses a stick-breaking representation and is learned by a variational inference method. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the obstacle of estimating the correct number of clusters is sidestepped by assuming an infinite number of clusters. Compared to other approximation techniques, such as Markov chain Monte Carlo (MCMC), which require high computational cost and whose convergence is difficult to diagnose, the whole inference process in the proposed variational learning framework is analytically tractable with closed-form solutions. Additionally, the proposed infinite Dirichlet mixture model with variational learning requires only a modest amount of computational power which makes it suitable to large applications. The effectiveness of our model is experimentally investigated through both synthetic data sets and challenging real-life multimedia applications namely image spam filtering and human action videos categorization.     

A nonparametric Bayesian approach toward robot learning by demonstration

In the past years, many authors have considered application of machine learning methodologies to effect robot learning by demonstration. Gaussian mixture regression (GMR) is one of the most successful methodologies used for this purpose. A major limitation of GMR models concerns automatic selection of the proper number of model states, i.e., the number of model component densities. Existing methods, including likelihood- or entropy-based criteria, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori. Under this motivation, to resolve the aforementioned issues of GMR-based methods for robot learning by demonstration, in this paper we introduce a nonparametric Bayesian formulation for the GMR model, the Dirichlet process GMR model. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally investigate its efficacy as a robot learning by demonstration methodology, considering a number of demanding robot learning by demonstration scenarios.     

Dynamic Speech Spectrum Representation and Tracking Variable Number of Vocal Tract Resonance Frequencies With Time-Varying Dirichlet Process Mixture Models

In this paper, we propose a new approach for dynamic speech spectrum representation and tracking vocal tract resonance (VTR) frequencies. The method involves representing the spectral density of the speech signals as a mixture of Gaussians with unknown number of components for which time-varying Dirichlet process mixture model (DPM) is utilized. In the resulting representation, the number of formants is allowed to vary in time. The paper first presents an analysis on the continuity of the formants in the spectrum during the speech utterance. The analysis is based on a new state space representation of concatenated tube model. We show that the number of formants which appear in the spectrum is directly related to the location of the constriction of the vocal tract (i.e., the location of the excitation). Moreover, the disappearance of the formants in the spectrum is explained by “uncontrollable modes” of the state space model. Under the assumption of existence of varying number of formants in the spectrum, we propose the use of a DPM model based multi-target tracking algorithm for tracking unknown number of formants. The tracking algorithm defines a hierarchical Bayesian model for the unknown formant states and the inference is done via Rao–Blackwellized particle filter.      

Variational Bayesian inference for infinite generalized inverted Dirichlet mixtures with feature selection and its application to clustering

We developed a variational Bayesian learning framework for the infinite generalized Dirichlet mixture model (i.e. a weighted mixture of Dirichlet process priors based on the generalized inverted Dirichlet distribution) that has proven its capability to model complex multidimensional data. We also integrate a “feature selection” approach to highlight the features that are most informative in order to construct an appropriate model in terms of clustering accuracy. Experiments on synthetic data as well as real data generated from visual scenes and handwritten digits datasets illustrate and validate the proposed approach.     

Heterogeneous data analysis using a mixture of Laplace models with conjugate priors

The development of flexible parametric classes of probability models in Bayesian analysis is a very popular approach. This study is designed for heterogeneous population for a two-component mixture of the Laplace probability distribution. When a process initially starts, the researcher expects that the failure components will be very high but after some improvement/inspection it is assumed that the failure components will decrease sufficiently. That is why in such situation the Laplace model is more suitable as compared to the normal distribution due to its fatter tails behaviour. We considered the derivation of the posterior distribution for censored data assuming different conjugate informative priors. Various kinds of loss functions are used to derive these Bayes estimators and their posterior risks. A method of elicitation of hyperparameter is discussed based on a prior predictive approach. The results are also compared with the non-informative priors. To examine the performance of these estimators we have evaluated their properties for different sample sizes, censoring rates and proportions of the component of the mixture through the simulation study. To highlight the practical significance we have included an illustrative application example based on real-life mixture data.     

Online short text clustering using infinite extensions of discrete mixture models

Short text clustering is one of the fundamental tasks in natural language processing. Different from traditional documents, short texts are ambiguous and sparse due to their short form and the lack of recurrence in word usage from one text to another, making it very challenging to apply conventional machine learning algorithms directly. In this article, we propose two novel approaches for short texts clustering: collapsed Gibbs sampling infinite generalized Dirichlet multinomial mixture model infinite GSGDMM) and collapsed Gibbs sampling infinite Beta-Liouville multinomial mixture model (infinite GSBLMM). We adopt two flexible and practical priors to the multinomial distribution where in the first one the generalized Dirichlet distribution is integrated, while the second one is based on the Beta-Liouville distribution. We evaluate the proposed approaches on two famous benchmark datasets, namely, Google News and Tweet. The experimental results demonstrate the effectiveness of our models compared to basic approaches that use Dirichlet priors. We further propose to improve the performance of our methods with an online clustering procedure. We also evaluate the performance of our methods for the outlier detection task, in which we achieve accurate results.     

High-dimensional unsupervised selection and estimation of a finite generalized Dirichlet mixture model based on minimum message length

We consider the problem of determining the structure of high-dimensional data, without prior knowledge of the number of clusters. Data are represented by a finite mixture model based on the generalized Dirichlet distribution. The generalized Dirichlet distribution has a more general covariance structure than the Dirichlet distribution and offers high flexibility and ease of use for the approximation of both symmetric and asymmetric distributions. This makes the generalized Dirichlet distribution more practical and useful. An important problem in mixture modeling is the determination of the number of clusters. Indeed, a mixture with too many or too few components may not be appropriate to approximate the true model. Here, we consider the application of the minimum message length (MML) principle to determine the number of clusters. The MML is derived so as to choose the number of clusters in the mixture model which best describes the data. A comparison with other selection criteria is performed. The validation involves synthetic data, real data clustering, and two interesting real applications: classification of web pages, and texture database summarization for efficient retrieval.     

Bayesian mixture of autoregressive models

An infinite mixture of autoregressive models is developed. The unknown parameters in the mixture autoregressive model follow a mixture distribution, which is governed by a Dirichlet process prior. One main feature of our approach is the generalization of a finite mixture model by having the number of components unspecified. A Bayesian sampling scheme based on a weighted Chinese restaurant process is proposed to generate partitions of observations. Using the partitions, Bayesian prediction, while accounting for possible model uncertainty, determining the most probable number of mixture components, clustering of time series and outlier detection in time series, can be done. Numerical results from simulated and real data are presented to illustrate the methodology.     

Simplifying mixture models using the unscented transform

Mixture of Gaussians (MoG) model is a useful tool in statistical learning. In many learning processes that are based on mixture models, computational requirements are very demanding due to the large number of components involved in the model. We propose a novel algorithm for learning a simplified representation of a Gaussian mixture, that is based on the Unscented Transform which was introduced for filtering nonlinear dynamical systems. The superiority of the proposed method is validated on both simulation experiments and categorization of a real image database. The proposed categorization methodology is based on modeling each image using a Gaussian mixture model. A category model is obtained by learning a simplified mixture model from all the images in the category.     

Bayesian mixture of autoregressive models

An infinite mixture of autoregressive models is developed. The unknown parameters in the mixture autoregressive model follow a mixture distribution, which is governed by a Dirichlet process prior. One main feature of our approach is the generalization of a finite mixture model by having the number of components unspecified. A Bayesian sampling scheme based on a weighted Chinese restaurant process is proposed to generate partitions of observations. Using the partitions, Bayesian prediction, while accounting for possible model uncertainty, determining the most probable number of mixture components, clustering of time series and outlier detection in time series, can be done. Numerical results from simulated and real data are presented to illustrate the methodology.     

Gaussian mixture models with covariances or precisions in shared multiple subspaces

We introduce a class of Gaussian mixture models (GMMs) in which the covariances or the precisions (inverse covariances) are restricted to lie in subspaces spanned by rank-one symmetric matrices. The rank-one basis are shared between the Gaussians according to a sharing structure. We describe an algorithm for estimating the parameters of the GMM in a maximum likelihood framework given a sharing structure. We employ these models for modeling the observations in the hidden-states of a hidden Markov model based speech recognition system. We show that this class of models provide improvement in accuracy and computational efficiency over well-known covariance modeling techniques such as classical factor analysis, shared factor analysis and maximum likelihood linear transformation based models which are special instances of this class of models. We also investigate different sharing mechanisms. We show that for the same number of parameters, modeling precisions leads to better performance when compared to modeling covariances. Modeling precisions also gives a distinct advantage in computational and memory requirements.