Bayesian learning of inverted Dirichlet mixtures for SVM kernels generation

We describe approaches for positive data modeling and classification using both finite inverted Dirichlet mixture models and support vector machines (SVMs). Inverted Dirichlet mixture models are used to tackle an outstanding challenge in SVMs namely the generation of accurate kernels. The kernels generation approaches, grounded on ideas from information theory that we consider, allow the incorporation of data structure and its structural constraints. Inverted Dirichlet mixture models are learned within a principled Bayesian framework using both Gibbs sampler and Metropolis-Hastings for parameter estimation and Bayes factor for model selection (i.e., determining the number of mixture’s components). Our Bayesian learning approach uses priors, which we derive by showing that the inverted Dirichlet distribution belongs to the family of exponential distributions, over the model parameters, and then combines these priors with information from the data to build posterior distributions. We illustrate the merits and the effectiveness of the proposed method with two real-world challenging applications namely object detection and visual scenes analysis and classification.     

Positive vectors clustering using inverted Dirichlet finite mixture models

In this work we present an unsupervised algorithm for learning finite mixture models from multivariate positive data. Indeed, this kind of data appears naturally in many applications, yet it has not been adequately addressed in the past. This mixture model is based on the inverted Dirichlet distribution, which offers a good representation and modeling of positive non-Gaussian data. The proposed approach for estimating the parameters of an inverted Dirichlet mixture is based on the maximum likelihood (ML) using Newton Raphson method. We also develop an approach, based on the minimum message length (MML) criterion, to select the optimal number of clusters to represent the data using such a mixture. Experimental results are presented using artificial histograms and real data sets. The challenging problem of software modules classification is investigated within the proposed statistical framework, also.     

Variational Bayesian inference for infinite generalized inverted Dirichlet mixtures with feature selection and its application to clustering

We developed a variational Bayesian learning framework for the infinite generalized Dirichlet mixture model (i.e. a weighted mixture of Dirichlet process priors based on the generalized inverted Dirichlet distribution) that has proven its capability to model complex multidimensional data. We also integrate a “feature selection” approach to highlight the features that are most informative in order to construct an appropriate model in terms of clustering accuracy. Experiments on synthetic data as well as real data generated from visual scenes and handwritten digits datasets illustrate and validate the proposed approach.     

Dirichlet Process Gaussian Mixture Models: Choice of the Base Distribution

In the Bayesian mixture modeling framework it is possible to infer the necessary number of components to model the data and therefore it is unnecessary to explicitly restrict the number of components. Nonparametric mixture models sidestep the problem of finding the “correct” number of mixture components by assuming infinitely many components. In this paper Dirichlet process mixture (DPM) models are cast as infinite mixture models and inference using Markov chain Monte Carlo is described. The specification of the priors on the model parameters is often guided by mathematical and practical convenience. The primary goal of this paper is to compare the choice of conjugate and non-conjugate base distributions on a particular class of DPM models which is widely used in applications, the Dirichlet process Gaussian mixture model (DPGMM). We compare computational efficiency and modeling performance of DPGMM defined using a conjugate and a conditionally conjugate base distribution. We show that better density models can result from using a wider class of priors with no or only a modest increase in computational effort.     

Dirichlet Gaussian mixture model: Application to image segmentation

Gaussian mixture model based on the Dirichlet distribution (Dirichlet Gaussian mixture model) has recently received great attention for modeling and processing data. This paper studies the new Dirichlet Gaussian mixture model for image segmentation. First, we propose a new way to incorporate the local spatial information between neighboring pixels based on the Dirichlet distribution. The main advantage is its simplicity, ease of implementation and fast computational speed. Secondly, existing Dirichlet Gaussian model uses complex log-likelihood function and require many parameters that are difficult to estimate. The total parameters in the proposed model lesser and the log-likelihood function have a simpler form. Finally, to estimate the parameters of the proposed Dirichlet Gaussian mixture model, a gradient method is adopted to minimize the negative log-likelihood function. Numerical experiments are conducted using the proposed model on various synthetic, natural and color images. We demonstrate through extensive simulations that the proposed model is superior to other algorithms based on the model-based techniques for image segmentation.     

Alternative prior assumptions for improving the performance of naïve Bayesian classifiers

The prior distribution of an attribute in a naïve Bayesian classifier is typically assumed to be a Dirichlet distribution, and this is called the Dirichlet assumption. The variables in a Dirichlet random vector can never be positively correlated and must have the same confidence level as measured by normalized variance. Both the generalized Dirichlet and the Liouville distributions include the Dirichlet distribution as a special case. These two multivariate distributions, also defined on the unit simplex, are employed to investigate the impact of the Dirichlet assumption in naïve Bayesian classifiers. We propose methods to construct appropriate generalized Dirichlet and Liouville priors for naïve Bayesian classifiers. Our experimental results on 18 data sets reveal that the generalized Dirichlet distribution has the best performance among the three distribution families. Not only is the Dirichlet assumption inappropriate, but also forcing the variables in a prior to be all positively correlated can deteriorate the performance of the naïve Bayesian classifier.     

基于数据关联狄利克雷混合模型的电网净负荷不确定性表征研究EI北大核心CSCD

针对电网净负荷时序数据关联的特点,提出基于数据关联的狄利克雷混合模型(Data-relevance Dirichlet process mixture model,DDPMM)来表征净负荷的不确定性.首先,使用狄利克雷混合模型对净负荷的观测数据与预测数据进行拟合,得到其混合概率模型;然后,提出考虑数据关联的变分贝叶斯推断方法,改进后验分布对该混合概率模型进行求解,从而得到混合模型的最优参数;最后,根据净负荷预测值的大小得到其对应的预测误差边缘概率分布,实现不确定性表征.本文基于比利时电网的净负荷数据进行检验,算例结果表明:与传统的狄利克雷混合模型和高斯混合模型(Gaussian mixture model,GMM)等方法相比,所提出的基于数据关联狄利克雷混合模型可以更为有效地表征净负荷的不确定性.     

A new fault diagnosis method of multimode processes using Bayesian inference based Gaussian mixture contribution decomposition

This paper presents a novel Bayesian inference based Gaussian mixture contribution (BIGMC) method to isolate and diagnose the faulty variables in chemical processes with multiple operating modes. The statistical confidence intervals of traditional principal component analysis (PCA) based T² and SPE diagnostics rely upon the assumption that the operating data follow a multivariate Gaussian distribution approximately and therefore may not be able to determine the faulty variables in multimode non-Gaussian processes accurately. As an alternative solution, the proposed BIGMC method first identifies the multiple Gaussian modes corresponding to different operating conditions and then integrates the Mahalanobis distance based variable contributions across all the Gaussian clusters through Bayesian inference strategy. The derived BIGMC index is of probabilistic feature and includes all operation scenarios with posterior probabilities as weighting factors. The Tennessee Eastman process (TEP) is used to demonstrate the utility of the proposed BIGMC method for fault diagnosis of multimode processes. The comparison of the single-PCA and multi-PCA based contribution approaches shows that the BIGMC method can effectively identify the leading faulty variables with superior diagnosis capability.     

A countably infinite mixture model for clustering and feature selection

Mixture modeling is one of the most useful tools in machine learning and data mining applications. An important challenge when applying finite mixture models is the selection of the number of clusters which best describes the data. Recent developments have shown that this problem can be handled by the application of non-parametric Bayesian techniques to mixture modeling. Another important crucial preprocessing step to mixture learning is the selection of the most relevant features. The main approach in this paper, to tackle these problems, consists on storing the knowledge in a generalized Dirichlet mixture model by applying non-parametric Bayesian estimation and inference techniques. Specifically, we extend finite generalized Dirichlet mixture models to the infinite case in which the number of components and relevant features do not need to be known a priori. This extension provides a natural representation of uncertainty regarding the challenging problem of model selection. We propose a Markov Chain Monte Carlo algorithm to learn the resulted infinite mixture. Through applications involving text and image categorization, we show that infinite mixture models offer a more powerful and robust performance than classic finite mixtures for both clustering and feature selection.     

Non-homogeneous dynamic Bayesian networks for continuous data

Classical dynamic Bayesian networks (DBNs) are based on the homogeneous Markov assumption and cannot deal with non-homogeneous temporal processes. Various approaches to relax the homogeneity assumption have recently been proposed. The present paper presents a combination of a Bayesian network with conditional probabilities in the linear Gaussian family, and a Bayesian multiple changepoint process, where the number and location of the changepoints are sampled from the posterior distribution with MCMC. Our work improves four aspects of an earlier conference paper: it contains a comprehensive and self-contained exposition of the methodology; it discusses the problem of spurious feedback loops in network reconstruction; it contains a comprehensive comparative evaluation of the network reconstruction accuracy on a set of synthetic and real-world benchmark problems, based on a novel discrete changepoint process; and it suggests new and improved MCMC schemes for sampling both the network structures and the changepoint configurations from the posterior distribution. The latter study compares RJMCMC, based on changepoint birth and death moves, with two dynamic programming schemes that were originally devised for Bayesian mixture models. We demonstrate the modifications that have to be made to allow for changing network structures, and the critical impact that the prior distribution on changepoint configurations has on the overall computational complexity.     

概率假设密度高斯混合实现的分量删减

针对概率假设密度(Probability hypothesis density, PHD)高斯混合实现算法中的分量删减问题, 提出了基于Dirichlet分布的分量删减算法以改进概率假设密度高斯混合实现算法的性能. 算法采用极大后验准则估计混合参数, 采用仅依赖于混合权重的负指数Dirichlet分布作为混合参数的先验分布, 利用拉格朗日乘子推导了混合权重的更新公式. 算法利用负指数Dirichlet分布的不稳定性,在极大后验迭代过程中驱使与目标强度不相关的分量消亡. 该不稳定性还能够解决多个相近分量共同描述一个强度峰值的问题, 有利于后续多目标状态的提取. 仿真结果表明, 基于Dirichlet分布的分量删减算法优于典型高斯混合实现中的删减算法.     

Generalized linear mixed model with a penalized Gaussian mixture as a random effects distribution

Generalized linear mixed models are popular for regressing a discrete response when there is clustering, e.g. in longitudinal studies or in hierarchical data structures. It is standard to assume that the random effects have a normal distribution. Recently, it has been examined whether wrongly assuming a normal distribution for the random effects is important for the estimation of the fixed effects parameters. While it has been shown that misspecifying the distribution of the random effects has a minor effect in the context of linear mixed models, the conclusion for generalized mixed models is less clear. Some studies report a minor impact, while others report that the assumption of normality really matters especially when the variance of the random effect is relatively high. Since it is unclear whether the normality assumption is truly satisfied in practice, it is important that generalized mixed models are available which relax the normality assumption. A replacement of the normal distribution with a mixture of Gaussian distributions specified on a grid whereby only the weights of the mixture components are estimated using a penalized approach ensuring a smooth distribution for the random effects is proposed. The parameters of the model are estimated in a Bayesian context using MCMC techniques. The usefulness of the approach is illustrated on two longitudinal studies using R-functions.     

基于GMM的多工况过程监测方法

传统基于主元分析的故障检测方法大多假设工业过程只运行在1个稳定工况,数据服从单一的高斯分布。若这些方法直接用于多工况过程则将会产生大量的误检。为此,本文提出了1种基于高斯混合模型的多工况过程监测方法。首先利用PCA变换对过程数据集进行降维,在主元空间建立高斯混合模型对过程数据进行聚类,自动获取工况数和相关分布特性。然后对每个工况建立主元分析(principal component analysis,PCA)模型来描述整个运行过程数据分布的统计特性。最后在过程监测中,根据监测样本属于各个工况的概率构造综合统计量,实现对多工况过程的故障检测。TE过程的仿真结果表明,本文提出的方法与传统的PCA方法相比,能自动获取工况和精确估计各个工况的统计特性,从而能更准确及时地检测出多工况过程的各种故障。     

Bayesian mixture of autoregressive models

An infinite mixture of autoregressive models is developed. The unknown parameters in the mixture autoregressive model follow a mixture distribution, which is governed by a Dirichlet process prior. One main feature of our approach is the generalization of a finite mixture model by having the number of components unspecified. A Bayesian sampling scheme based on a weighted Chinese restaurant process is proposed to generate partitions of observations. Using the partitions, Bayesian prediction, while accounting for possible model uncertainty, determining the most probable number of mixture components, clustering of time series and outlier detection in time series, can be done. Numerical results from simulated and real data are presented to illustrate the methodology.     

Spatial color image segmentation based on finite non-Gaussian mixture models

Finite mixture models are one of the most widely and commonly used probabilistic techniques for image segmentation. Although the most well known and commonly used distribution when considering mixture models is the Gaussian, it is certainly not the best approximation for image segmentation and other related image processing problems. In this paper, we propose and investigate the use of several other mixture models based namely on Dirichlet, generalized Dirichlet and Beta–Liouville distributions, which offer more flexibility in data modeling, for image segmentation. A maximum likelihood (ML) based algorithm is applied for estimating the resulted segmentation model’s parameters. Spatial information is also employed for figuring out the number of regions in an image and several color spaces are investigated and compared. The experimental results show that the proposed segmentation framework yields good overall performance, on various color scenes, that is better than comparable techniques.     

Bayesian mixture of autoregressive models

An infinite mixture of autoregressive models is developed. The unknown parameters in the mixture autoregressive model follow a mixture distribution, which is governed by a Dirichlet process prior. One main feature of our approach is the generalization of a finite mixture model by having the number of components unspecified. A Bayesian sampling scheme based on a weighted Chinese restaurant process is proposed to generate partitions of observations. Using the partitions, Bayesian prediction, while accounting for possible model uncertainty, determining the most probable number of mixture components, clustering of time series and outlier detection in time series, can be done. Numerical results from simulated and real data are presented to illustrate the methodology.     

分层Dirichlet过程及其应用综述

Dirichlet过程是一种应用于非参数贝叶斯模型中的随机过程, 特别是作为先验分布应用在概率图模型中. 与传统的参数模型相比, Dirichlet过程的应用更加广泛且模型更加灵活, 特别是应用于聚类问题时, 该过程能够自动确定聚类数目和生成聚类中心的分布参数. 因此, 近年来, 在理论和应用上均得到了迅速的发展, 引起越来越多的关注. 本文首先介绍Dirichlet过程, 而后描述了以Dirichlet过程为先验分布的Dirichlet过程混合模型及其应用, 接着概述分层Dirichlet过程及其在相关算法构造中的应用, 最后对分层Dirichlet过程的理论和应用进行了总结, 并对未来的发展方向作了探讨.     

A nonparametric Bayesian approach toward robot learning by demonstration

In the past years, many authors have considered application of machine learning methodologies to effect robot learning by demonstration. Gaussian mixture regression (GMR) is one of the most successful methodologies used for this purpose. A major limitation of GMR models concerns automatic selection of the proper number of model states, i.e., the number of model component densities. Existing methods, including likelihood- or entropy-based criteria, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori. Under this motivation, to resolve the aforementioned issues of GMR-based methods for robot learning by demonstration, in this paper we introduce a nonparametric Bayesian formulation for the GMR model, the Dirichlet process GMR model. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally investigate its efficacy as a robot learning by demonstration methodology, considering a number of demanding robot learning by demonstration scenarios.     

Bayesian hybrid generative discriminative learning based on finite Liouville mixture models

Recently hybrid generative discriminative approaches have emerged as an efficient knowledge representation and data classification engine. However, little attention has been devoted to the modeling and classification of non-Gaussian and especially proportional vectors. Our main goal, in this paper, is to discover the true structure of this kind of data by building probabilistic kernels from generative mixture models based on Liouville family, from which we develop the Beta-Liouville distribution, and which includes the well-known Dirichlet as a special case. The Beta-Liouville has a more general covariance structure than the Dirichlet which makes it more practical and useful. Our learning technique is based on a principled purely Bayesian approach which resulted models are used to generate support vector machine (SVM) probabilistic kernels based on information divergence. In particular, we show the existence of closed-form expressions of the Kullback-Leibler and Rényi divergences between two Beta-Liouville distributions and then between two Dirichlet distributions as a special case. Through extensive simulations and a number of experiments involving synthetic data, visual scenes and texture images classification, we demonstrate the effectiveness of the proposed approaches.     

A Bayesian nonparametric approach to modeling motion patterns

The most difficult??and often most essential??aspect of many interception and tracking tasks is constructing motion models of the targets. Experts rarely can provide complete information about a target??s expected motion pattern, and fitting parameters for complex motion patterns can require large amounts of training data. Specifying how to parameterize complex motion patterns is in itself a difficult task. In contrast, Bayesian nonparametric models of target motion are very flexible and generalize well with relatively little training data. We propose modeling target motion patterns as a mixture of Gaussian processes (GP) with a Dirichlet process (DP) prior over mixture weights. The GP provides an adaptive representation for each individual motion pattern, while the DP prior allows us to represent an unknown number of motion patterns. Both automatically adjust the complexity of the motion model based on the available data. Our approach outperforms several parametric models on a helicopter-based car-tracking task on data collected from the greater Boston area.