Lazy Learning of Bayesian Rules

The naive Bayesian classifier provides a simple and effective approach to classifier learning, but its attribute independence assumption is often violated in the real world. A number of approaches have sought to alleviate this problem. A Bayesian tree learning algorithm builds a decision tree, and generates a local naive Bayesian classifier at each leaf. The tests leading to a leaf can alleviate attribute inter-dependencies for the local naive Bayesian classifier. However, Bayesian tree learning still suffers from the small disjunct problem of tree learning. While inferred Bayesian trees demonstrate low average prediction error rates, there is reason to believe that error rates will be higher for those leaves with few training examples. This paper proposes the application of lazy learning techniques to Bayesian tree induction and presents the resulting lazy Bayesian rule learning algorithm, called LBR. This algorithm can be justified by a variant of Bayes theorem which supports a weaker conditional attribute independence assumption than is required by naive Bayes. For each test example, it builds a most appropriate rule with a local naive Bayesian classifier as its consequent. It is demonstrated that the computational requirements of LBR are reasonable in a wide cross-section of natural domains. Experiments with these domains show that, on average, this new algorithm obtains lower error rates significantly more often than the reverse in comparison to a naive Bayesian classifier, C4.5, a Bayesian tree learning algorithm, a constructive Bayesian classifier that eliminates attributes and constructs new attributes using Cartesian products of existing nominal attributes, and a lazy decision tree learning algorithm. It also outperforms, although the result is not statistically significant, a selective naive Bayesian classifier.     

Automated extraction of fragments of Bayesian networks from textual sources

Mining large amounts of unstructured data for extracting meaningful, accurate, and actionable information, is at the core of a variety of research disciplines including computer science, mathematical and statistical modelling, as well as knowledge engineering. In particular, the ability to model complex scenarios based on unstructured datasets is an important step towards an integrated and accurate knowledge extraction approach. This would provide a significant insight in any decision making process driven by Big Data analysis activities. However, there are multiple challenges that need to be fully addressed in order to achieve this, especially when large and unstructured data sets are considered.In this article we propose and analyse a novel method to extract and build fragments of Bayesian networks (BNs) from unstructured large data sources. The results of our analysis show the potential of our approach, and highlight its accuracy and efficiency. More specifically, when compared with existing approaches, our method addresses specific challenges posed by the automated extraction of BNs with extensive applications to unstructured and highly dynamic data sources.The aim of this work is to advance the current state-of-the-art approaches to the automated extraction of BNs from unstructured datasets, which provide a versatile and powerful modelling framework to facilitate knowledge discovery in complex decision scenarios.     

Understanding the scalability of Bayesian network inference using clique tree growth curves

One of the main approaches to performing computation in Bayesian networks (BNs) is clique tree clustering and propagation. The clique tree approach consists of propagation in a clique tree compiled from a BN, and while it was introduced in the 1980s, there is still a lack of understanding of how clique tree computation time depends on variations in BN size and structure. In this article, we improve this understanding by developing an approach to characterizing clique tree growth as a function of parameters that can be computed in polynomial time from BNs, specifically: (i) the ratio of the number of a BN's non-root nodes to the number of root nodes, and (ii) the expected number of moral edges in their moral graphs. Analytically, we partition the set of cliques in a clique tree into different sets, and introduce a growth curve for the total size of each set. For the special case of bipartite BNs, there are two sets and two growth curves, a mixed clique growth curve and a root clique growth curve. In experiments, where random bipartite BNs generated using the BPART algorithm are studied, we systematically increase the out-degree of the root nodes in bipartite Bayesian networks, by increasing the number of leaf nodes. Surprisingly, root clique growth is well-approximated by Gompertz growth curves, an S-shaped family of curves that has previously been used to describe growth processes in biology, medicine, and neuroscience. We believe that this research improves the understanding of the scaling behavior of clique tree clustering for a certain class of Bayesian networks; presents an aid for trade-off studies of clique tree clustering using growth curves; and ultimately provides a foundation for benchmarking and developing improved BN inference and machine learning algorithms.     

Predicting football results using Bayesian nets and other machine learning techniques

Bayesian networks (BNs) provide a means for representing, displaying, and making available in a usable form the knowledge of experts in a given field. In this paper, we look at the performance of an expert constructed BN compared with other machine learning (ML) techniques for predicting the outcome (win, lose, or draw) of matches played by Tottenham Hotspur Football Club. The period under study was 1995–1997 – the expert BN was constructed at the start of that period, based almost exclusively on subjective judgement. Our objective was to determine retrospectively the comparative accuracy of the expert BN compared to some alternative ML models that were built using data from the two-year period. The additional ML techniques considered were: MC4, a decision tree learner; Naive Bayesian learner; Data Driven Bayesian (a BN whose structure and node probability tables are learnt entirely from data); and a K-nearest neighbour learner. The results show that the expert BN is generally superior to the other techniques for this domain in predictive accuracy. The results are even more impressive for BNs given that, in a number of key respects, the study assumptions place them at a disadvantage. For example, we have assumed that the BN prediction is ‘incorrect’ if a BN predicts more than one outcome as equally most likely (whereas, in fact, such a prediction would prove valuable to somebody who could place an ‘each way’ bet on the outcome). Although the expert BN has now long been irrelevant (since it contains variables relating to key players who have retired or left the club) the results here tend to confirm the excellent potential of BNs when they are built by a reliable domain expert. The ability to provide accurate predictions without requiring much learning data are an obvious bonus in any domain where data are scarce. Moreover, the BN was relatively simple for the expert to build and its structure could be used again in this and similar types of problems.     

决策树分类算法中C4.5算法的研究与改进

C4.5算法是用于生成决策树的一种经典算法,虽然其有很强的噪声处理能力,但当属性值缺失率高时,分类准确率会明显下降,而且该算法在构建决策树时,需要多次扫描、排序数据集、以及频繁调用对数,针对以上缺点,本文提出一种改进的分类算法.采用一种基于朴素贝叶斯定理方法,来处理空缺属性值,提高分类准确率.通过优化精简计算公式,在计算过程中,改进后的计算公式使用四则混合运算代替原来的对数运算,减少构建决策树的运行时间.为了验证该算法的性能,通过对UCI数据库中5个数据集进行实验,实验结果表明,改进后的算法极大的提高了运行效率.     

Bayesian learning of Bayesian networks with informative priors

This paper presents and evaluates an approach to Bayesian model averaging where the models are Bayesian nets (BNs). A comprehensive study of the literature on structural priors for BNs is conducted. A number of prior distributions are defined using stochastic logic programs and the MCMC Metropolis-Hastings algorithm is used to (approximately) sample from the posterior. We use proposals which are tightly coupled to the priors which give rise to cheaply computable acceptance probabilities. Experiments using data generated from known BNs have been conducted to evaluate the method. The experiments used 6 different BNs and varied: the structural prior, the parameter prior, the Metropolis-Hasting proposal and the data size. Each experiment was repeated three times with different random seeds to test the robustness of the MCMC-produced results. Our results show that with effective priors (i) robust results are produced and (ii) informative priors improve results significantly.     

Learning Bayesian networks for clustering by means of constructive induction

The purpose of this paper is to present and evaluate a heuristic algorithm for learning Bayesian networks for clustering. Our approach is based upon improving the Naive-Bayes model by means of constructive induction. A key idea in this approach is to treat expected data as real data. This allows us to complete the database and to take advantage of factorable closed forms for the marginal likelihood. In order to get such an advantage, we search for parameter values using the EM algorithm or another alternative approach that we have developed: a hybridization of the Bound and Collapse method and the EM algorithm, which results in a method that exhibits a faster convergence rate and a more effective behaviour than the EM algorithm. Also, we consider the possibility of interleaving runnings of these two methods after each structural change. We evaluate our approach on synthetic and real-world databases.     

An artificial bee colony algorithm for learning Bayesian networks

One basic approach to learn Bayesian networks (BNs) from data is to apply a search procedure to explore the set of candidate networks for the database in light of a scoring metric, where the most popular stochastic methods are based on some meta-heuristic mechanisms, such as Genetic Algorithm, Evolutionary Programming and Ant Colony Optimization. In this paper, we have developed a new algorithm for learning BNs which employs a recently introduced meta-heuristic: artificial bee colony (ABC). All the phases necessary to tackle our learning problem using this meta-heuristic are described, and some experimental results to compare the performance of our ABC-based algorithm with other algorithms are given in the paper.     

贝叶斯决策树在客户流失预测中的应用

针对电信企业客户流失问题,提出采用贝叶斯决策树算法的预测模型,将贝叶斯分类的先验信息方法与决策树分类的信息熵增益方法相结合,应用到电信行业客户流失分析中,分别将移动公司的客户数据以及UCI数据纳入到模型中得出相应的结果。加入贝叶斯节点弥补决策树不能处理缺失值以及二义性数据的缺点。检验结果表明,基于贝叶斯推理的决策树算法在牺牲了较小的训练时间与分类时间的情况下,得到了比仅基于决策树算法更高的覆盖率与命中率。     

Portfolios in Stochastic Local Search: Efficiently Computing Most Probable Explanations in Bayesian Networks

Portfolio methods support the combination of different algorithms and heuristics, including stochastic local search (SLS) heuristics, and have been identified as a promising approach to solve computationally hard problems. While successful in experiments, theoretical foundations and analytical results for portfolio-based SLS heuristics are less developed. This article aims to improve the understanding of the role of portfolios of heuristics in SLS. We emphasize the problem of computing most probable explanations (MPEs) in Bayesian networks (BNs). Algorithmically, we discuss a portfolio-based SLS algorithm for MPE computation, Stochastic Greedy Search (SGS). SGS supports the integration of different initialization operators (or initialization heuristics) and different search operators (greedy and noisy heuristics), thereby enabling new analytical and experimental results. Analytically, we introduce a novel Markov chain model tailored to portfolio-based SLS algorithms including SGS, thereby enabling us to analytically form expected hitting time results that explain empirical run time results. For a specific BN, we show the benefit of using a homogenous initialization portfolio. To further illustrate the portfolio approach, we consider novel additive search heuristics for handling determinism in the form of zero entries in conditional probability tables in BNs. Our additive approach adds rather than multiplies probabilities when computing the utility of an explanation. We motivate the additive measure by studying the dramatic impact of zero entries in conditional probability tables on the number of zero-probability explanations, which again complicates the search process. We consider the relationship between MAXSAT and MPE, and show that additive utility (or gain) is a generalization, to the probabilistic setting, of MAXSAT utility (or gain) used in the celebrated GSAT and WalkSAT algorithms and their descendants. Utilizing our Markov chain framework, we show that expected hitting time is a rational function—i.e. a ratio of two polynomials—of the probability of applying an additive search operator. Experimentally, we report on synthetically generated BNs as well as BNs from applications, and compare SGS’s performance to that of Hugin, which performs BN inference by compilation to and propagation in clique trees. On synthetic networks, SGS speeds up computation by approximately two orders of magnitude compared to Hugin. In application networks, our approach is highly competitive in Bayesian networks with a high degree of determinism. In addition to showing that stochastic local search can be competitive with clique tree clustering, our empirical results provide an improved understanding of the circumstances under which portfolio-based SLS outperforms clique tree clustering and vice versa.     

Potential of Bayesian networks for adaptive management in water recycling

Water recycling is an important solution to increasing water scarcity. However, universal wastewater treatment standards deter uptake of recycling schemes. Lack of data also impedes fit-for-purpose water recycling and water managers are challenged by decision making under uncertain conditions. Bayesian networks (BNs) are increasingly recognised as a valuable tool for decision making under uncertainty. In this study BNs are used to model health risks associated with wastewater irrigation of a public open space. Three BNs based on quantitative microbial risk assessment model risk reduction potential along a treatment chain and at the site of reuse. The BNs simulate multiple exposure profiles and scenarios, providing conditional probability of infection or illness, comparable with health-based targets. Study findings highlight the significant impact of post treatment risk mitigation, despite challenging conditions. BNs provide a transparent, defensible evidence base for mapping and quantifying risk pathways, comparing decision options and predicting outcomes of management policies.     

Learning Bayesian network parameters under incomplete data with domain knowledge

Bayesian networks (BNs) have gained increasing attention in recent years. One key issue in Bayesian networks is parameter learning. When training data is incomplete or sparse or when multiple hidden nodes exist, learning parameters in Bayesian networks becomes extremely difficult. Under these circumstances, the learning algorithms are required to operate in a high-dimensional search space and they could easily get trapped among copious local maxima. This paper presents a learning algorithm to incorporate domain knowledge into the learning to regularize the otherwise ill-posed problem, to limit the search space, and to avoid local optima. Unlike the conventional approaches that typically exploit the quantitative domain knowledge such as prior probability distribution, our method systematically incorporates qualitative constraints on some of the parameters into the learning process. Specifically, the problem is formulated as a constrained optimization problem, where an objective function is defined as a combination of the likelihood function and penalty functions constructed from the qualitative domain knowledge. Then, a gradient-descent procedure is systematically integrated with the E-step and M-step of the EM algorithm, to estimate the parameters iteratively until it converges. The experiments with both synthetic data and real data for facial action recognition show our algorithm improves the accuracy of the learned BN parameters significantly over the conventional EM algorithm.     

Collective Mining of Bayesian Networks from Distributed Heterogeneous Data

We present a collective approach to learning a Bayesian network from distributed heterogeneous data. In this approach, we first learn a local Bayesian network at each site using the local data. Then each site identifies the observations that are most likely to be evidence of coupling between local and non-local variables and transmits a subset of these observations to a central site. Another Bayesian network is learnt at the central site using the data transmitted from the local site. The local and central Bayesian networks are combined to obtain a collective Bayesian network, which models the entire data. Experimental results and theoretical justification that demonstrate the feasibility of our approach are presented.16 November 2001     

贝叶斯网等价类学习算法

贝叶斯网用一种紧凑的形式表示联合概率分布,具有完备的语义和坚实的理论基础,目前已成为人工智能领域处理不确定性问题的最佳方法之一。贝叶斯网学习是其关键问题,传统学习方法存在如下不足：（1）随节点数增多非法结构以指数级增加,影响学习效率;（2）在等价结构之间进行打分搜索,影响收敛速度;（3）假设每个结构具有相同的先验概率,造成等价类中包含结构越多则先验概率越高。本文提出一种学习马尔科夫等价类算法,该算法基于骨架空间进行状态转换,利用从骨架空间到等价类空间的映 映射关系实现学习贝叶斯网等价类。实验数据证明,该方法可有效缩小搜索空间规模,相对于在有向图空间搜索的算法加快了算法的收敛速度,提高了执行效率。     

Bayesian Model Learning Based on Predictive Entropy

Bayesian paradigm has been widely acknowledged as a coherent approach to learning putative probability model structures from a finite class of candidate models. Bayesian learning is based on measuring the predictive ability of a model in terms of the corresponding marginal data distribution, which equals the expectation of the likelihood with respect to a prior distribution for model parameters. The main controversy related to this learning method stems from the necessity of specifying proper prior distributions for all unknown parameters of a model, which ensures a complete determination of the marginal data distribution. Even for commonly used models, subjective priors may be difficult to specify precisely, and therefore, several automated learning procedures have been suggested in the literature. Here we introduce a novel Bayesian learning method based on the predictive entropy of a probability model, that can combine both subjective and objective probabilistic assessment of uncertain quantities in putative models. It is shown that our approach can avoid some of the limitations of the earlier suggested objective Bayesian methods.     

Participatory Bayesian modelling for sustainable and efficient river restoration projects: Feedback from the case study of the Gave de Pau River,Hautes‐Pyrénées,France

Through the diversity of criteria and stakes, the uncertain nature of the entailed phenomena and the multi‐scale aspects to be taken into account, a river restoration project can be considered as a complex problem. Integrative approaches and modelling tools are thus needed to help river managers make predictions on the evolution of hydromorphological, socio‐economic, safety and ecological issues. Such approach can provide valuable information for handling long‐term management plans that consider the interaction and the balance of stakeholders interests and river system functioning. In this paper, we present a probabilistic participatory modelling (PM) method that assesses the effects of given restoration actions, knowing the hydromorphological modifications that they may induce on the safety, ecological and socio‐economic aspects with the help of local stakeholders through several workshops. To support this strategy, we used Bayesian networks (BNs) as modelling tools as their causal graphs can combine multidimensional knowledge and data from diverse natures. We introduce the causal graphs elaborated with the help of the stakeholders and convert it into BNs that can assist restoration decisions by considering the available decision and utility functions to provide guidance to decision‐makers. This was applied to the “Lac des Gaves” reach in the Hautes‐Pyrénées, France, a reach that has gone through severe sediment extractions for over 50 years. Each network represents possible restoration decisions linked to one of the observed issues. The paper demonstrates how BNs used as a decision support system (DSS) can help to assess the influence of given management strategies on the river system with the consideration of stakeholders’ knowledge and integration in all the modelling process.     

Unravelling infectious disease eco-epidemiology using Bayesian networks and scenario analysis: A case study of leptospirosis in Fiji

Regression models are the standard approaches used in infectious disease epidemiology, but have limited ability to represent causality or complexity. We explore Bayesian networks (BNs) as an alternative approach for modelling infectious disease transmission, using leptospirosis as an example. Data were obtained from a leptospirosis study in Fiji in 2013. We compared the performance of naïve versus expert-structured BNs for modelling the relative importance of animal species in disease transmission in different ethnic groups and residential settings. For BNs of animal exposures at the individual/household level, R² for predicted versus observed infection rates were 0.59 for naïve and 0.75–0.93 for structured models of ethnic groups; and 0.54 for naïve and 0.93–1.00 for structured models of residential settings. BNs provide a promising approach for modelling infectious disease transmission under complex scenarios. The relative importance of animal species varied between subgroups, with important implications for more targeted public health control strategies.