Bayesian estimation of Dirichlet mixture model with variational inference

In statistical modeling, parameter estimation is an essential and challengeable task. Estimation of the parameters in the Dirichlet mixture model (DMM) is analytically intractable, due to the integral expressions of the gamma function and its corresponding derivatives. We introduce a Bayesian estimation strategy to estimate the posterior distribution of the parameters in DMM. By assuming the gamma distribution as the prior to each parameter, we approximate both the prior and the posterior distribution of the parameters with a product of several mutually independent gamma distributions. The extended factorized approximation method is applied to introduce a single lower-bound to the variational objective function and an analytically tractable estimation solution is derived. Moreover, there is only one function that is maximized during iterations and, therefore, the convergence of the proposed algorithm is theoretically guaranteed. With synthesized data, the proposed method shows the advantages over the EM-based method and the previously proposed Bayesian estimation method. With two important multimedia signal processing applications, the good performance of the proposed Bayesian estimation method is demonstrated.     

Semantics-based inference algorithms for adaptive visualenvironments

The paper presents a grammatical inference methodology for the generation of visual languages, that benefits from the availability of semantic information about the sample sentences. Several well-known syntactic inference algorithms are shown to obey a general inference scheme, which the authors call the Gen-Inf scheme. Then, all the algorithms of the Gen-Inf scheme are modified in agreement with the introduced semantics-based inference methodology. The use of grammatical inference techniques in the design of adaptive user interfaces was previously experimented with the VLG system for visual language generation. The system is a powerful tool for specifying, designing, and interpreting customized visual languages for different applications. They enhance the adaptivity of the VLG system to any visual environment by exploiting the proposed semantics-based inference methodology. As a matter of fact, a more general model of visual language generation is achieved, based on the Gen-Inf scheme, where the end-user is allowed to choose the algorithm which best fits his/her requirements within the particular application environment     

Linear response algorithms for approximate inference in graphical models

Belief propagation (BP) on cyclic graphs is an efficient algorithm for computing approximate marginal probability distributions over single nodes and neighboring nodes in the graph. However, it does not prescribe a way to compute joint distributions over pairs of distant nodes in the graph. In this article, we propose two new algorithms for approximating these pairwise probabilities, based on the linear response theorem. The first is a propagation algorithm that is shown to converge if BP converges to a stable fixed point. The second algorithm is based on matrix inversion. Applying these ideas to gaussian random fields, we derive a propagation algorithm for computing the inverse of a matrix.     

Dependent mixtures of Dirichlet processes

An approach to modeling dependent nonparametric random density functions is presented. This is based on the well known mixture of Dirichlet process model. The idea is to use a technique for constructing dependent random variables, first used for dependent gamma random variables. While the methodology works for an arbitrary number of dependent random densities, with each pair having their own dependent structure, the mathematics and estimation algorithm is focused on two dependent random density functions. Simulations and a real data example are presented.     

Comparison of fuzzy inference algorithms for stream flow prediction

Neural Computing and Applications - Fuzzy logic is, inter alia, a simple and flexible approach of modelling that can be used in river basins where adequate hydrological data are unavailable. In...     

Swendsen-Wang Cuts sampling for spatially constrained Dirichlet process mixture models

Spatially constrained Dirichlet process mixture models are springing up in image processing in recent years. However, inference for the model is NP-hard. Gibbs sampling which is a generic Markov chain Monte Carlo technique is commonly employed for the model inference. It needs to traverse all the nodes of the constructed graph in each iteration. The sampling process hardly crosses over the intermediate low probabilistic state. In addition, it is not well informed by the spatial relationship in the sampling process. In this paper, a spatially dependent split-merge algorithm for sampling the MRF/DPMM model based on Swendsen-Wang Cuts is proposed. It is a state of the art algorithm which combines the spatial relationship to direct the sampling, and lessen the mixing time drastically. In this algorithm, a set of nodes are being frozen together according to the discriminative probability of the edges between neighboring nodes. The frozen nodes update their states simultaneously in contrast to the single node update in a Gibbs sampling. The final step of the algorithm is to accept the proposed new state according to the Metropolis Hasting scheme, in which only the ratio of posterior distribution needs to be calculated in each iteration. Experimental results demonstrated that the proposed sampling algorithm is able to reduce the mixing time considerably. At the same time, it can obtain comparably stable results with a random initial state.     

Iterative optimization algorithms for discrete-continuous processes

We propose a series of approximate and iterative optimizational methods for discrete-continuous processes based on Krotov’s sufficient optimality conditions. Iterations are constructed by localizing the global optimality conditions and Krotov’s minimax scheme with various approximations. The very concept of a discrete-continuous system and the corresponding optimality conditions and algorithms represent a convenient formalism to study a wide class of complex systems and processes, in particular, magistral solutions of singular problems that are significantly non-uniform in structure.     

A new fault diagnosis method of multimode processes using Bayesian inference based Gaussian mixture contribution decomposition

This paper presents a novel Bayesian inference based Gaussian mixture contribution (BIGMC) method to isolate and diagnose the faulty variables in chemical processes with multiple operating modes. The statistical confidence intervals of traditional principal component analysis (PCA) based T² and SPE diagnostics rely upon the assumption that the operating data follow a multivariate Gaussian distribution approximately and therefore may not be able to determine the faulty variables in multimode non-Gaussian processes accurately. As an alternative solution, the proposed BIGMC method first identifies the multiple Gaussian modes corresponding to different operating conditions and then integrates the Mahalanobis distance based variable contributions across all the Gaussian clusters through Bayesian inference strategy. The derived BIGMC index is of probabilistic feature and includes all operation scenarios with posterior probabilities as weighting factors. The Tennessee Eastman process (TEP) is used to demonstrate the utility of the proposed BIGMC method for fault diagnosis of multimode processes. The comparison of the single-PCA and multi-PCA based contribution approaches shows that the BIGMC method can effectively identify the leading faulty variables with superior diagnosis capability.     

Entropy-type classification maximum likelihood algorithms for mixture models

Mixtures of distributions are popularly used as probability models for analyzing grouped data. Classification maximum likelihood (CML) is an important maximum likelihood approach to clustering with mixture models. Yang et al. extended CML to fuzzy CML. Although fuzzy CML presents better results than CML, it is always affected by the fuzziness index parameter. In this paper, we consider fuzzy CML with an entropy-regularization term to create an entropy-type CML algorithm. The proposed entropy-type CML is a parameter-free algorithm for mixture models. Some numerical and real-data comparisons show that the proposed method provides better results than some existing methods.     

Two time-efficient gibbs sampling inference algorithms for biterm topic model

Biterm Topic Model (BTM) is an effective topic model proposed to handle short texts. However, its standard gibbs sampling inference method (StdBTM) costs much more time than that (StdLDA) of Latent Dirichlet Allocation (LDA). To solve this problem we propose two time-efficient gibbs sampling inference methods, SparseBTM and ESparseBTM, for BTM by making a tradeoff between space and time consumption in this paper. The idea of SparseBTM is to reduce the computation in StdBTM by both recycling intermediate results and utilizing the sparsity of count matrix \(\mathbf {N}^{\mathbf {T}}_{\mathbf {W}}\). Theoretically, SparseBTM reduces the time complexity of StdBTM from O(|B| K) to O(|B| K _w ) which scales linearly with the sparsity of count matrix \(\mathbf {N}^{\mathbf {T}}_{\mathbf {W}}\) (K _w ) instead of the number of topics (K) (K _w < K, K _w is the average number of non-zero topics per word type in count matrix \(\mathbf {N}^{\mathbf {T}}_{\mathbf {W}}\)). Experimental results have shown that in good conditions SparseBTM is approximately 18 times faster than StdBTM. Compared with SparseBTM, ESparseBTM is a more time-efficient gibbs sampling inference method proposed based on SparseBTM. The idea of ESparseBTM is to reduce more computation by recycling more intermediate results through rearranging biterm sequence. In theory, ESparseBTM reduces the time complexity of SparseBTM from O(|B|K _w ) to O(R|B|K _w ) (0 < R < 1, R is the ratio of the number of biterm types to the number of biterms). Experimental results have shown that the percentage of the time efficiency improved by ESparseBTM on SparseBTM is between 6.4% and 39.5% according to different datasets.     

Gaussian processes for classification: mean-field algorithms

We derive a mean-field algorithm for binary classification with gaussian processes that is based on the TAP approach originally proposed in statistical physics of disordered systems. The theory also yields an approximate leave-one-out estimator for the generalization error, which is computed with no extra computational cost. We show that from the TAP approach, it is possible to derive both a simpler "naive" mean-field theory and support vector machines (SVMs) as limiting cases. For both mean-field algorithms and support vector machines, simulation results for three small benchmark data sets are presented. They show that one may get state-of-the-art performance by using the leave-one-out estimator for model selection and the built-in leave-one-out estimators are extremely precise when compared to the exact leave-one-out estimate. The second result is taken as strong support for the internal consistency of the mean-field approach.     

Actor-critic algorithms for hierarchical Markov decision processes

We consider the problem of control of hierarchical Markov decision processes and develop a simulation based two-timescale actor-critic algorithm in a general framework. We also develop certain approximation algorithms that require less computation and satisfy a performance bound. One of the approximation algorithms is a three-timescale actor-critic algorithm while the other is a two-timescale algorithm, however, which operates in two separate stages. All our algorithms recursively update randomized policies using the simultaneous perturbation stochastic approximation (SPSA) methodology. We briefly present the convergence analysis of our algorithms. We then present numerical experiments on a problem of production planning in semiconductor fabs on which we compare the performance of all algorithms together with policy iteration. Algorithms based on certain Hadamard matrix based deterministic perturbations are found to show the best results.     

A comparison of algorithms for inference and learning in probabilistic graphical models

Research into methods for reasoning under uncertainty is currently one of the most exciting areas of artificial intelligence, largely because it has recently become possible to record, store, and process large amounts of data. While impressive achievements have been made in pattern classification problems such as handwritten character recognition, face detection, speaker identification, and prediction of gene function, it is even more exciting that researchers are on the verge of introducing systems that can perform large-scale combinatorial analyses of data, decomposing the data into interacting components. For example, computational methods for automatic scene analysis are now emerging in the computer vision community. These methods decompose an input image into its constituent objects, lighting conditions, motion patterns, etc. Two of the main challenges are finding effective representations and models in specific applications and finding efficient algorithms for inference and learning in these models. In this paper, we advocate the use of graph-based probability models and their associated inference and learning algorithms. We review exact techniques and various approximate, computationally efficient techniques, including iterated conditional modes, the expectation maximization (EM) algorithm, Gibbs sampling, the mean field method, variational techniques, structured variational techniques and the sum-product algorithm ("loopy” belief propagation). We describe how each technique can be applied in a vision model of multiple, occluding objects and contrast the behaviors and performances of the techniques using a unifying cost function, free energy.     

Extended sequential algorithms for detecting changes in actingstochastic processes

We present, analyze, and numerically evaluate extended algorithms for detecting changes from an acting stochastic process to a number of possible alternatives. The algorithms are sequential, requiring minimal memory capacity and operational complexity, and they incorporate decision thresholds. The performance of the algorithms is controlled by the selection of the thresholds. Asymptotically, the first algorithmic extension detects the acting process correctly in an expected stopping time sense. In addition, the probability of error induced by a reinitialization algorithmic extension converges asymptotically to zero, when the acting process changes infrequently (with order inversely proportional to the value of the decision thresholds). The presented algorithmic systems are quite powerful and their applications are numerous, ranging from industrial quality control to traffic and performance monitoring in highspeed networks     

Hierarchical evolving Dirichlet processes for modeling nonlinear evolutionary traces in temporal data

Clustering analysis aims to group a set of similar data objects into the same cluster. Topic models, which belong to the soft clustering methods, are powerful tools to discover latent clusters/topics behind large data sets. Due to the dynamic nature of temporal data, clusters often exhibit complicated patterns such as birth, branch and death. However, most existing temporal clustering models assume that clusters evolve as a linear chain, and they cannot model and detect branching of clusters. In this paper, we present evolving Dirichlet processes (EDP for short) to model nonlinear evolutionary traces behind temporal data, especially for temporal text collections. In the setting of EDP, temporal collections are divided into epochs. In order to model cluster branching over time, EDP allows each cluster in an epoch to form Dirichlet processes (DP) and uses a combination of the cluster-specific DPs as the prior for cluster distributions in the next epoch. To model hierarchical temporal data, such as online document collections, we propose a new class of evolving hierarchical Dirichlet processes (EHDP for short) which extends the hierarchical Dirichlet processes (HDP) to model evolving temporal data. We design an online learning framework based on Gibbs sampling to infer the evolutionary traces of clusters over time. In experiments, we validate that EDP and EHDP can capture nonlinear evolutionary traces of clusters on both synthetic and real-world text collections and achieve better results than its peers.     

A Dirichlet process mixture model for the analysis of correlated binary responses

The multivariate probit model is a popular choice for modelling correlated binary responses. It assumes an underlying multivariate normal distribution dichotomized to yield a binary response vector. Other choices for the latent distribution have been suggested, but basically all models assume homogeneity in the correlation structure across the subjects. When interest lies in the association structure, relaxing this homogeneity assumption could be useful. The latent multivariate normal model is replaced by a location and association mixture model defined by a Dirichlet process. Attention is paid to the parameterization of the covariance matrix in order to make the Bayesian computations convenient. The approach is illustrated on a simulated data set and applied to oral health data from the Signal Tandmobiel^® study to examine the hypothesis that caries is mainly a spatially local disease.