Quantitative Inference by Qualitative Semantic Knowledge Mining with Bayesian Model Averaging

In this paper, we consider the problem of performing quantitative Bayesian inference and model averaging based on a set of qualitative statements about relationships. Statements are transformed into parameter constraints which are imposed onto a set of Bayesian networks. Recurrent relationship structures are resolved by unfolding in time to Dynamic Bayesian networks. The approach enables probabilistic inference by model averaging, i.e. it allows to predict probabilistic quantities from a set of qualitative constraints without probability assignment on the model parameters. Model averaging is performed by Monte Carlo integration techniques. The method is applied to a problem in a molecular medical context: We show how the rate of breast cancer metastasis formation can be predicted based solely on a set of qualitative biological statements about the involvement of proteins in metastatic processes.     

Robust L1 principal component analysis and its Bayesian variational inference

We introduce a robust probabilistic L1-PCA model in which the conventional gaussian distribution for the noise in the observed data was replaced by the Laplacian distribution (or L1 distribution). Due to the heavy tail characteristics of the L1 distribution, the proposed model is supposed to be more robust against data outliers. In this letter, we demonstrate how a variational approximation scheme enables effective inference of key parameters in the probabilistic L1-PCA model. As the L1 density can be expanded as a superposition of infinite number of gaussian densities, we express the L1-PCA model as a marginalized model over the superpositions. By doing so, a tractable Bayesian inference can be achieved based on the variational expectation-maximization-type algorithm.     

Variational learning for Dirichlet process mixtures of Dirichlet distributions and applications

In this paper, we propose a Bayesian nonparametric approach for modeling and selection based on a mixture of Dirichlet processes with Dirichlet distributions, which can also be seen as an infinite Dirichlet mixture model. The proposed model uses a stick-breaking representation and is learned by a variational inference method. Due to the nature of Bayesian nonparametric approach, the problems of overfitting and underfitting are prevented. Moreover, the obstacle of estimating the correct number of clusters is sidestepped by assuming an infinite number of clusters. Compared to other approximation techniques, such as Markov chain Monte Carlo (MCMC), which require high computational cost and whose convergence is difficult to diagnose, the whole inference process in the proposed variational learning framework is analytically tractable with closed-form solutions. Additionally, the proposed infinite Dirichlet mixture model with variational learning requires only a modest amount of computational power which makes it suitable to large applications. The effectiveness of our model is experimentally investigated through both synthetic data sets and challenging real-life multimedia applications namely image spam filtering and human action videos categorization.     

Probabilistic temporal reasoning and its application to fossil power plant operation

Many real-world applications, such as industrial diagnosis, require an adequate representation and inference mechanism that combines uncertainty and time. In this work, we propose a novel approach for representing dynamic domains under uncertainty based on a probabilistic framework, called temporal nodes Bayesian networks (TNBN). The TNBN model is an extension of a standard Bayesian network, in which each temporal node represents an event or state change of a variable and the arcs represent causal–temporal relationships between nodes. A temporal node has associated a probability distribution for its time of occurrence, where time is discretized in a finite number of temporal intervals; allowing a different number of intervals for each node and a different duration for the intervals within a node (multiple granularity). The main difference with previous probabilistic temporal models is that the representation is based on state changes at different times instead of state values at different times. Given this model, we can reason about the probability of occurrence of certain events, for diagnosis or prediction, using standard probability propagation techniques developed for Bayesian networks. The proposed approach is applied to fossil power plant diagnosis through two detailed case studies: power load increment and control level system failure. The results show that the proposed formalism could help to improve power plant availability through early diagnosis of events and disturbances.     

Evolutionary attribute ordering in Bayesian networks for predicting the metabolic syndrome

The metabolic syndrome is a set of risk factors that include abdominal obesity, insulin resistance, dyslipidemia and hypertension. It has affected around 25% of adults in the US and become a serious problem in Asian countries recently due to the change in dietary habit and life style. On the other hand, Bayesian networks that are the models to solve the problems of uncertainty provide a robust and transparent formalism for probabilistic modeling, so they have been used as a method for diagnostic or prognostic model in medical domain. Since the K2 algorithm, a well-known algorithm for Bayesian networks structure learning, is influenced by an input order of the attributes, an optimization of BN attribute ordering has been studied as a research issue. This paper proposes a novel ordering optimization method using a genetic algorithm based on medical expert knowledge in order to solve this problem. For experiments, we use the dataset examined twice in 1993 and 1995 in Yonchon County of Korea. It has 18 attributes of 1193 subjects participated in both surveys. Using this dataset, we make the prognostic model of the metabolic syndrome using Bayesian networks with an optimized ordering by evolutionary approach. Through an ordering optimization, the prognostic model of higher performance is constructed, and the optimized Bayesian network model by the proposed method outperforms the conventional BN model as well as neural networks and k-nearest neighbors. Finally, we present the application program using the prognostic model of the metabolic syndrome in order to show the usefulness of the proposed method.     

Mobile context inference using two-layered Bayesian networks for smartphones

Recently, mobile context inference becomes an important issue. Bayesian probabilistic model is one of the most popular probabilistic approaches for context inference. It efficiently represents and exploits the conditional independence of propositions. However, there are some limitations for probabilistic context inference in mobile devices. Mobile devices relatively lacks of sufficient memory. In this paper, we present a novel method for efficient Bayesian inference on a mobile phone. In order to overcome the constraints of the mobile environment, the method uses two-layered Bayesian networks with tree structure. In contrast to the conventional techniques, this method attempts to use probabilistic models with fixed tree structures and intermediate nodes. It can reduce the inference time by eliminating junction tree creation. To evaluate the performance of this method, an experiment is conducted with data collected over a month. The result shows the efficiency and effectiveness of the proposed method.     

Bayes网络学习的MCMC方法

基于Bayes统计理论, 提出了一种从数据样本中学习Bayes网络的Markov链Monte Carlo(MCMC)方法. 首先通过先验概率和数据样本的结合得到未归一化的后验概率, 然后使用此后验概率指导随机搜索算法寻找“好”的网络结构模型. 通过对Alarm网络的学习表明了本算法具有较好的性能.     

Latent-space variational bayes

Variational Bayesian Expectation-Maximization (VBEM), an approximate inference method for probabilistic models based on factorizing over latent variables and model parameters, has been a standard technique for practical Bayesian inference. In this paper, we introduce a more general approximate inference framework for conjugate-exponential family models, which we call Latent-Space Variational Bayes (LSVB). In this approach, we integrate out model parameters in an exact way, leaving only the latent variables. It can be shown that the LSVB approach gives better estimates of the model evidence as well as the distribution over latent variables than the VBEM approach, but in practice, the distribution over latent variables has to be approximated. As a practical implementation, we present a First-order LSVB (FoLSVB) algorithm to approximate this distribution over latent variables. From this approximate distribution, one can estimate the model evidence and the posterior over model parameters. The FoLSVB algorithm is directly comparable to the VBEM algorithm and has the same computational complexity. We discuss how LSVB generalizes the recently proposed collapsed variational methods [20] to general conjugate-exponential families. Examples based on mixtures of Gaussians and mixtures of Bernoullis with synthetic and real-world data sets are used to illustrate some advantages of our method over VBEM.     

Variational Bayesian learning of nonlinear hidden state-space models for model predictive control

This paper studies the identification and model predictive control in nonlinear hidden state-space models. Nonlinearities are modelled with neural networks and system identification is done with variational Bayesian learning. In addition to the robustness of control, the stochastic approach allows for various control schemes, including combinations of direct and indirect controls, as well as using probabilistic inference for control. We study the noise-robustness, speed, and accuracy of three different control schemes as well as the effect of changing horizon lengths and initialisation methods using a simulated cart–pole system. The simulations indicate that the proposed method is able to find a representation of the system state that makes control easier especially under high noise.     

Adaptive diagnosis in distributed systems

Real-time problem diagnosis in large distributed computer systems and networks is a challenging task that requires fast and accurate inferences from potentially huge data volumes. In this paper, we propose a cost-efficient, adaptive diagnostic technique called active probing . Probes are end-to-end test transactions that collect information about the performance of a distributed system. Active probing uses probabilistic reasoning techniques combined with information-theoretic approach, and allows a fast online inference about the current system state via active selection of only a small number of most-informative tests. We demonstrate empirically that the active probing scheme greatly reduces both the number of probes (from 60% to 75% in most of our real-life applications), and the time needed for localizing the problem when compared with nonadaptive (preplanned) probing schemes. We also provide some theoretical results on the complexity of probe selection, and the effect of "noisy" probes on the accuracy of diagnosis. Finally, we discuss how to model the system's dynamics using dynamic Bayesian networks (DBNs), and an efficient approximate approach called sequential multifault; empirical results demonstrate clear advantage of such approaches over "static" techniques that do not handle system's changes.     

Encoding probability propagation in belief networks

Complexity reduction is an important task in Bayesian networks. Recently, an approach known as the linear potential function (LPF) model has been proposed for approximating Bayesian computations. The LPF model can effectively compress a conditional probability table into a linear function. This correspondence extends the LPF model to approximate propagation in Bayesian networks. The extension focuses on encoding probability propagation as a polynomial function for a class of tractable problems.     

A variational Bayesian methodology for hidden Markov models utilizing Student's-t mixtures

The Student's-t hidden Markov model (SHMM) has been recently proposed as a robust to outliers form of conventional continuous density hidden Markov models, trained by means of the expectation-maximization algorithm. In this paper, we derive a tractable variational Bayesian inference algorithm for this model. Our innovative approach provides an efficient and more robust alternative to EM-based methods, tackling their singularity and overfitting proneness, while allowing for the automatic determination of the optimal model size without cross-validation. We highlight the superiority of the proposed model over the competition using synthetic and real data. We also demonstrate the merits of our methodology in applications from diverse research fields, such as human computer interaction, robotics and semantic audio analysis.     

Bayesian mixture of AR models for time series clustering

In this paper, we propose a Bayesian framework for estimation of parameters of a mixture of autoregressive models for time series clustering. The proposed approach is based on variational principles and provides a tractable approximation to the true posterior density that minimizes Kullback–Liebler (KL) divergence with respect to prior distribution. This method simultaneously addresses the model complexity and parameter estimation problems. The proposed approach is applied both on simulated and real-world time series datasets. It is found to be useful in exploring and finding the true number of underlying clusters, starting from an arbitrarily large number of clusters.     

When and where to transfer for Bayesian network parameter learning

Learning Bayesian networks from scarce data is a major challenge in real-world applications where data are hard to acquire. Transfer learning techniques attempt to address this by leveraging data from different but related problems. For example, it may be possible to exploit medical diagnosis data from a different country. A challenge with this approach is heterogeneous relatedness to the target, both within and across source networks. In this paper we introduce the Bayesian network parameter transfer learning (BNPTL) algorithm to reason about both network and fragment (sub-graph) relatedness. BNPTL addresses (i) how to find the most relevant source network and network fragments to transfer, and (ii) how to fuse source and target parameters in a robust way. In addition to improving target task performance, explicit reasoning allows us to diagnose network and fragment relatedness across Bayesian networks, even if latent variables are present, or if their state space is heterogeneous. This is important in some applications where relatedness itself is an output of interest. Experimental results demonstrate the superiority of BNPTL at various scarcities and source relevance levels compared to single task learning and other state-of-the-art parameter transfer methods. Moreover, we demonstrate successful application to real-world medical case studies.     

Bayesian Networks for Data Mining

A Bayesian network is a graphical model that encodesprobabilistic relationships among variables of interest. When used inconjunction with statistical techniques, the graphical model hasseveral advantages for data modeling. One, because the model encodesdependencies among all variables, it readily handles situations wheresome data entries are missing. Two, a Bayesian network can be used tolearn causal relationships, and hence can be used to gain understanding about a problem domain and to predict the consequencesof intervention. Three, because the model has both a causal andprobabilistic semantics, it is an ideal representation for combiningprior knowledge (which often comes in causal form) and data. Four,Bayesian statistical methods in conjunction with Bayesian networksoffer an efficient and principled approach for avoiding theoverfitting of data. In this paper, we discuss methods for constructing Bayesian networks from prior knowledge and summarizeBayesian statistical methods for using data to improve these models.With regard to the latter task, we describe methods for learning boththe parameters and structure of a Bayesian network, includingtechniques for learning with incomplete data. In addition, we relateBayesian-network methods for learning to techniques for supervised andunsupervised learning. We illustrate the graphical-modeling approachusing a real-world case study.     

A Bayesian Method for the Induction of Probabilistic Networks from Data

This paper presents a Bayesian method for constructing probabilistic networks from databases. In particular, we focus on constructing Bayesian belief networks. Potential applications include computer-assisted hypothesis testing, automated scientific discovery, and automated construction of probabilistic expert systems. We extend the basic method to handle missing data and hidden (latent) variables. We show how to perform probabilistic inference by averaging over the inferences of multiple belief networks. Results are presented of a preliminary evaluation of an algorithm for constructing a belief network from a database of cases. Finally, we relate the methods in this paper to previous work, and we discuss open problems.     

Adaptive anti-spam filtering for agglutinative languages: a special case for Turkish

We propose anti-spam filtering methods for agglutinative languages in general and for Turkish in particular. The methods are dynamic and are based on Artificial Neural Networks (ANN) and Bayesian Networks. The developed algorithms are user-specific and adapt themselves with the characteristics of the incoming e-mails. The algorithms have two main components. The first one deals with the morphology of the words and the second one classifies the e-mails by using the roots of the words extracted by the morphological analysis. Two ANN structures, single layer perceptron and multi-layer perceptron, are considered and the inputs to the networks are determined using binary model and probabilistic model. Similarly, for Bayesian classification, three different approaches are employed: binary model, probabilistic model, and advanced probabilistic model. In the experiments, a total of 750 e-mails (410 spam and 340 normal) were used and a success rate of about 90% was achieved.     

On Learning Conditional Random Fields for Stereo

Until recently, the lack of ground truth data has hindered the application of discriminative structured prediction techniques to the stereo problem. In this paper we use ground truth data sets that we have recently constructed to explore different model structures and parameter learning techniques. To estimate parameters in Markov random fields (MRFs) via maximum likelihood one usually needs to perform approximate probabilistic inference. Conditional random fields (CRFs) are discriminative versions of traditional MRFs. We explore a number of novel CRF model structures including a CRF for stereo matching with an explicit occlusion model. CRFs require expensive inference steps for each iteration of optimization and inference is particularly slow when there are many discrete states. We explore belief propagation, variational message passing and graph cuts as inference methods during learning and compare with learning via pseudolikelihood. To accelerate approximate inference we have developed a new method called sparse variational message passing which can reduce inference time by an order of magnitude with negligible loss in quality. Learning using sparse variational message passing improves upon previous approaches using graph cuts and allows efficient learning over large data sets when energy functions violate the constraints imposed by graph cuts.     

利用标注者相关性的深度生成式众包学习

传统监督学习需要训练样本的真实标记信息,而在很多情况下,真实标记并不容易收集.与之对比,众包学习从多个可能犯错的非专家收集标注,通过某种融合方式估计样本的真实标记.注意到现有深度众包学习工作对标注者相关性建模不足,而非深度众包学习方面的工作表明,标注者相关性建模利用有助于改善学习效果.提出一种深度生成式众包学习方法,以...