A countably infinite mixture model for clustering and feature selection

Mixture modeling is one of the most useful tools in machine learning and data mining applications. An important challenge when applying finite mixture models is the selection of the number of clusters which best describes the data. Recent developments have shown that this problem can be handled by the application of non-parametric Bayesian techniques to mixture modeling. Another important crucial preprocessing step to mixture learning is the selection of the most relevant features. The main approach in this paper, to tackle these problems, consists on storing the knowledge in a generalized Dirichlet mixture model by applying non-parametric Bayesian estimation and inference techniques. Specifically, we extend finite generalized Dirichlet mixture models to the infinite case in which the number of components and relevant features do not need to be known a priori. This extension provides a natural representation of uncertainty regarding the challenging problem of model selection. We propose a Markov Chain Monte Carlo algorithm to learn the resulted infinite mixture. Through applications involving text and image categorization, we show that infinite mixture models offer a more powerful and robust performance than classic finite mixtures for both clustering and feature selection.     

Parameter estimation of Poisson mixture with automated model selection through BYY harmony learning

Finite mixture is widely used in the fields of information processing and data analysis. However, its model selection, i.e., the selection of components in the mixture for a given sample data set, has been still a rather difficult task. Recently, the Bayesian Ying-Yang (BYY) harmony learning has provided a new approach to the Gaussian mixture modeling with a favorite feature that model selection can be made automatically during parameter learning. In this paper, based on the same BYY harmony learning framework for finite mixture, we propose an adaptive gradient BYY learning algorithm for Poisson mixture with automated model selection. It is demonstrated well by the simulation experiments that this adaptive gradient BYY learning algorithm can automatically determine the number of actual Poisson components for a sample data set, with a good estimation of the parameters in the original or true mixture where the components are separated in a certain degree. Moreover, the adaptive gradient BYY learning algorithm is successfully applied to texture classification.     

On multivariate binary data clustering and feature weighting

This paper presents an approach that partitions data sets of unlabeled binary vectors without a priori information about the number of clusters or the saliency of the features. The unsupervised binary feature selection problem is approached using finite mixture models of multivariate Bernoulli distributions. Using stochastic complexity, the proposed model determines simultaneously the number of clusters in a given data set composed of binary vectors and the saliency of the features used. We conduct different applications involving real data, document classification and images categorization to show the merits of the proposed approach.     

Unsupervised Learning of Gaussian Mixtures Based on Variational Component Splitting

In this paper, we present an incremental method for model selection and learning of Gaussian mixtures based on the recently proposed variational Bayes approach. The method adds components to the mixture using a Bayesian splitting test procedure: a component is split into two components and then variational update equations are applied only to the parameters of the two components. As a result, either both components are retained in the model or one of them is found to be redundant and is eliminated from the model. In our approach, the model selection problem is treated locally, in a region of the data space, so we can set more informative priors based on the local data distribution. A modified Bayesian mixture model is presented to implement this approach, along with a learning algorithm that iteratively applies a splitting test on each mixture component. Experimental results and comparisons with two other techniques testify for the adequacy of the proposed approach     

子空间聚类的非参数模型及变分贝叶斯学习

子空间聚类的目标是在不同的特征子集上对给定的一组数据归类.此非监督学习方法试图发现数据"在不同表达下的相似"模式,并且引起了相关领域大量的关注和研究.首先扩展Hoff提出的"均值与方差平移"模型为一个新的基于特征子集的非参数聚类模型,其优点是能应用变分贝叶斯方法学习模型参数.此模型结合Dirichlet过程混合模型和选择特征子集的非参数模型,能自动选择聚类个数和进行子空间聚类.然后给出基于马尔可夫链蒙特卡罗的参数后验推断算法.出于计算速度上的考虑,提出应用变分贝叶斯方法学习模型参数.在仿真数据上的实验结果及在人脸聚类问题上的应用均表明了此模型能同时选择相关特征和在这些特征上具有相似模式的数据点.在UCI"多特征数据库"上应用无需抽样的变分贝叶斯方法,其实验结果说明此方法能快速推断模型参数.     

Simultaneous feature selection and clustering using mixture models

Clustering is a common unsupervised learning technique used to discover group structure in a set of data. While there exist many algorithms for clustering, the important issue of feature selection, that is, what attributes of the data should be used by the clustering algorithms, is rarely touched upon. Feature selection for clustering is difficult because, unlike in supervised learning, there are no class labels for the data and, thus, no obvious criteria to guide the search. Another important problem in clustering is the determination of the number of clusters, which clearly impacts and is influenced by the feature selection issue. In this paper, we propose the concept of feature saliency and introduce an expectation-maximization (EM) algorithm to estimate it, in the context of mixture-based clustering. Due to the introduction of a minimum message length model selection criterion, the saliency of irrelevant features is driven toward zero, which corresponds to performing feature selection. The criterion and algorithm are then extended to simultaneously estimate the feature saliencies and the number of clusters.     

基于稀疏字典的听觉显著性计算

听觉注意显著性计算模型是研究听觉注意模型的基本问题,显著性计算中选择合适的特征是关键,本文从特征选择的角度提出了一种基于稀疏字典学习的听觉显著性计算模型.该模型首先通过K-SVD字典学习算法学习各种声学信号的特征,然后对字典集进行归类整合,以选取的特征字典为基础,采用OMP算法对信号进行稀疏表示,并直接将稀疏系数按帧合并得到声学信号的听觉显著图.仿真结果表明该听觉显著性计算模型在特征选择上更符合声学信号的自然属性,基于基础特征字典的显著图可以突出噪声中具有结构特征的声信号,基于特定信号特征字典的显著图可以实现对特定声信号的选择性关注.     

Bayesian learning of inverted Dirichlet mixtures for SVM kernels generation

We describe approaches for positive data modeling and classification using both finite inverted Dirichlet mixture models and support vector machines (SVMs). Inverted Dirichlet mixture models are used to tackle an outstanding challenge in SVMs namely the generation of accurate kernels. The kernels generation approaches, grounded on ideas from information theory that we consider, allow the incorporation of data structure and its structural constraints. Inverted Dirichlet mixture models are learned within a principled Bayesian framework using both Gibbs sampler and Metropolis-Hastings for parameter estimation and Bayes factor for model selection (i.e., determining the number of mixture’s components). Our Bayesian learning approach uses priors, which we derive by showing that the inverted Dirichlet distribution belongs to the family of exponential distributions, over the model parameters, and then combines these priors with information from the data to build posterior distributions. We illustrate the merits and the effectiveness of the proposed method with two real-world challenging applications namely object detection and visual scenes analysis and classification.     

Recursive unsupervised learning of finite mixture models

There are two open problems when finite mixture densities are used to model multivariate data: the selection of the number of components and the initialization. In this paper, we propose an online (recursive) algorithm that estimates the parameters of the mixture and that simultaneously selects the number of components. The new algorithm starts with a large number of randomly initialized components. A prior is used as a bias for maximally structured models. A stochastic approximation recursive learning algorithm is proposed to search for the maximum a posteriori (MAP) solution and to discard the irrelevant components.     

Bayesian feature evaluation for visual saliency estimation

This paper presents a computational method of feature evaluation for modeling saliency in visual scenes. This is highly relevant in visual search studies since visual saliency is at the basis of visual attention deployment. Visual saliency can also become important in computer vision applications as it can be used to reduce the computational requirements by permitting processing only in those regions of the scenes containing relevant information. The method is based on Bayesian theory to describe the interaction between top-down and bottom-up information. Unlike other approaches, it evaluates and selects visual features before saliency estimation. This can reduce the complexity and, potentially, improve the accuracy of the saliency computation. To this end, we present an algorithm for feature evaluation and selection. A two-color conjunction search experiment has been applied to illustrate the theoretical framework of the proposed model. The practical value of the method is demonstrated with video segmentation of instruments in a laparoscopic cholecystectomy operation.     

基于背景评估的贝叶斯模型显著性检测

针对自然图像中,复杂背景信息对显著性目标检测的影响,提出一种利用背景信息进行预测和贝叶斯模型选择优化的显著性检测方法。首先,为了提取完整的先验信息,根据背景信息与图像边界的连通性,以及对图像边界是否为背景进行评估来生成先验显著图。其次,为了降低背景信息的干扰,通过对流行排序算法生成的显著图进行角点检测,选择较为准确的显著点构造凸包。最后,利用贝叶斯模型进行选择优化来抑制和显著目标具有相同特征的背景信息。在2个公开的数据集上进行测试,并与4种性能较好的显著性检测算法对比,结果显示本文算法可提高显著性检测的准确性和区域的完整性。     

基于混合概率模型的无监督离散化算法

现实应用中常常涉及许多连续的数值属性，而且前许多机器学习算法则要求所处理的属性取离散值，根据在对数值属性的离散化过程中，是否考虑相关类别属性的值，离散化算法可分为有监督算法和无监督算法两类。基于混合概率模型，该文提出了一种理论严格的无监督离散化算法，它能够在无先验知识，无类别是属性的前提下，将数值属性的值域划分为若干子区间，再通过贝叶斯信息准则自动地寻求最佳的子区间数目和区间划分方法。     

Simultaneous Localized Feature Selection and Model Detection for Gaussian Mixtures

In this paper, we propose a novel approach of simultaneous localized feature selection and model detection for unsupervised learning. In our approach, local feature saliency, together with other parameters of Gaussian mixtures, are estimated by Bayesian variational learning. Experiments performed on both synthetic and real-world data sets demonstrate that our approach is superior over both global feature selection and subspace clustering methods.     

Variational learning for Dirichlet process mixtures of Dirichlet distributions and applications

In this paper, we propose a Bayesian nonparametric approach for modeling and selection based on a mixture of Dirichlet processes with Dirichlet distributions, which can also be seen as an infinite Dirichlet mixture model. The proposed model uses a stick-breaking representation and is learned by a variational inference method. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the obstacle of estimating the correct number of clusters is sidestepped by assuming an infinite number of clusters. Compared to other approximation techniques, such as Markov chain Monte Carlo (MCMC), which require high computational cost and whose convergence is difficult to diagnose, the whole inference process in the proposed variational learning framework is analytically tractable with closed-form solutions. Additionally, the proposed infinite Dirichlet mixture model with variational learning requires only a modest amount of computational power which makes it suitable to large applications. The effectiveness of our model is experimentally investigated through both synthetic data sets and challenging real-life multimedia applications namely image spam filtering and human action videos categorization.     

BYY Harmony Learning on Finite Mixture: Adaptive Gradient Implementation and A Floating RPCL Mechanism

In tackling the learning problem on a set of finite samples, Bayesian Ying-Yang (BYY) harmony learning has developed a new learning mechanism that makes model selection implemented either automatically during parameter learning or in help of evaluating a new class of model selection criteria. In this paper, parameter learning with automated model selection has been studied for finite mixture model via an adaptive gradient learning algorithm for BYY harmony learning on a specific bidirectional architecture (BI-architecture). Via theoretical analysis, it has shown that the adaptive gradient learning implements a mechanism of floating rival penalized competitive learning (RPCL) among the components in the mixture. Also, the simulation results are demonstrated well for the adaptive gradient algorithm on the sample data sets from Gaussian mixtures with certain degree of overlap. Moreover, the adaptive gradient algorithm is applied to classification of the Iris data and unsupervised color image segmentation.     

A new feature selection method for Gaussian mixture clustering

With the wide applications of Gaussian mixture clustering, e.g., in semantic video classification [H. Luo, J. Fan, J. Xiao, X. Zhu, Semantic principal video shot classification via mixture Gaussian, in: Proceedings of the 2003 International Conference on Multimedia and Expo, vol. 2, 2003, pp. 189-192], it is a nontrivial task to select the useful features in Gaussian mixture clustering without class labels. This paper, therefore, proposes a new feature selection method, through which not only the most relevant features are identified, but the redundant features are also eliminated so that the smallest relevant feature subset can be found. We integrate this method with our recently proposed Gaussian mixture clustering approach, namely rival penalized expectation-maximization (RPEM) algorithm [Y.M. Cheung, A rival penalized EM algorithm towards maximizing weighted likelihood for density mixture clustering with automatic model selection, in: Proceedings of the 17th International Conference on Pattern Recognition, 2004, pp. 633-636; Y.M. Cheung, Maximum weighted likelihood via rival penalized EM for density mixture clustering with automatic model selection, IEEE Trans. Knowl. Data Eng. 17(6) (2005) 750-761], which is able to determine the number of components (i.e., the model order selection) in a Gaussian mixture automatically. Subsequently, the data clustering, model selection, and the feature selection are all performed in a single learning process. Experimental results have shown the efficacy of the proposed approach.     

基于改进动态系统稳定估计器的机器人技能学习方法

提出一种基于改进动态系统稳定估计器的机器人技能学习方法. 现有的动态系统稳定估计器方法可以通过非线性优化来确保学习系统的全局稳定性, 但是存在确定高斯混合分量个数困难以及稳定性和精度无法兼顾的问题. 因此, 根据贝叶斯非参数模型可以自动确定合适分量个数的特性, 采用狄利克雷过程高斯混合模型对演示进行初始拟合. 随后利用参数化二次李雅普诺夫函数重新推导新的稳定性约束, 有效地解决了动态系统稳定估计器方法中稳定性和精度难以兼顾的问题. 最后, 在LASA数据库和Franka-panda机器人上的实验验证了新方法的有效性和优越性.     

A nonparametric Bayesian approach toward robot learning by demonstration

In the past years, many authors have considered application of machine learning methodologies to effect robot learning by demonstration. Gaussian mixture regression (GMR) is one of the most successful methodologies used for this purpose. A major limitation of GMR models concerns automatic selection of the proper number of model states, i.e., the number of model component densities. Existing methods, including likelihood- or entropy-based criteria, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori. Under this motivation, to resolve the aforementioned issues of GMR-based methods for robot learning by demonstration, in this paper we introduce a nonparametric Bayesian formulation for the GMR model, the Dirichlet process GMR model. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally investigate its efficacy as a robot learning by demonstration methodology, considering a number of demanding robot learning by demonstration scenarios.     

基于简单帧选择的显著性检测方法

提出了一种新颖的视频显著性检测方法。为了提取视频序列中具有高置信度的特征,根据输入帧和输入帧的初始显著图提出一种简单帧选择标准,并使用该简单选择标准挑选出视频序列中比较容易且准确提取前景对象的帧,从简单帧中获得鲁棒的前景背景标签;将图像进行超像素分割,提取时空特征与前景标签输入集成学习模型,经过多核SVM集成学习,最终生成像素级别的显著图,并且由运动特征扩散到整个视频集。各种视频序列的实验结果表明,该算法在定性和定量上优于传统的显着性检测算法。     

Probabilistic Multi-Task Learning for Visual Saliency Estimation in Video

In this paper, we present a probabilistic multi-task learning approach for visual saliency estimation in video. In our approach, the problem of visual saliency estimation is modeled by simultaneously considering the stimulus-driven and task-related factors in a probabilistic framework. In this framework, a stimulus-driven component simulates the low-level processes in human vision system using multi-scale wavelet decomposition and unbiased feature competition; while a task-related component simulates the high-level processes to bias the competition of the input features. Different from existing approaches, we propose a multi-task learning algorithm to learn the task-related “stimulus-saliency” mapping functions for each scene. The algorithm also learns various fusion strategies, which are used to integrate the stimulus-driven and task-related components to obtain the visual saliency. Extensive experiments were carried out on two public eye-fixation datasets and one regional saliency dataset. Experimental results show that our approach outperforms eight state-of-the-art approaches remarkably.