Construction and Methods of Learning of Bayesian Networks

Methods of learning Bayesian networks from databases, basic concepts of Bayesian networks, basic methods of learning, methods of learning parameters, and the structures of a network and hidden parameters are considered. Basic definitions and key concepts with illustrative examples are presented. __________ Translated from Kibernetika i Sistemnyi Analiz, No. 4, pp. 133–147, July–August 2005.     

Improving Bayesian Network Structure Learning with Mutual Information-Based Node Ordering in the K2 Algorithm

Structure learning of Bayesian networks is a well-researched but computationally hard task. We present an algorithm that integrates an information-theory-based approach and a scoring-function-based approach for learning structures of Bayesian networks. Our algorithm also makes use of basic Bayesian network concepts like d-separation and condition independence. We show that the proposed algorithm is capable of handling networks with a large number of variables. We present the applicability of the proposed algorithm on four standard network data sets and also compare its performance and computational efficiency with other standard structure-learning methods. The experimental results show that our method can efficiently and accurately identify complex network structures from data.     

Bayesian network modelling through qualitative patterns

In designing a Bayesian network for an actual problem, developers need to bridge the gap between the mathematical abstractions offered by the Bayesian-network formalism and the features of the problem to be modelled. Qualitative probabilistic networks (QPNs) have been put forward as qualitative analogues to Bayesian networks, and allow modelling interactions in terms of qualitative signs. They thus have the advantage that developers can abstract from the numerical detail, and therefore the gap may not be as wide as for their quantitative counterparts. A notion that has been suggested in the literature to facilitate Bayesian-network development is causal independence. It allows exploiting compact representations of probabilistic interactions among variables in a network. In the paper, we deploy both causal independence and QPNs in developing and analysing a collection of qualitative, causal interaction patterns, called QC patterns. These are endowed with a fixed qualitative semantics, and are intended to offer developers a high-level starting point when developing Bayesian networks.     

Representing Probabilistic Rules with Networks of Gaussian Basis Functions

There is great interest in understanding the intrinsic knowledge neural networks have acquired during training. Most work in this direction is focussed on the multi-layer perceptron architecture. The topic of this paper is networks of Gaussian basis functions which are used extensively as learning systems in neural computation. We show that networks of Gaussian basis functions can be generated from simple probabilistic rules. Also, if appropriate learning rules are used, probabilistic rules can be extracted from trained networks. We present methods for the reduction of network complexity with the goal of obtaining concise and meaningful rules. We show how prior knowledge can be refined or supplemented using data by employing either a Bayesian approach, by a weighted combination of knowledge bases, or by generating artificial training data representing the prior knowledge. We validate our approach using a standard statistical data set.     

新的贝叶斯网络结构学习方法

贝叶斯网络是一种将贝叶斯概率方法和有向无环图的网络拓扑结构有机结合的表示模型,它描述了数据项及数据项之间的非线性依赖关系.报告了贝叶斯网络研究的现状,并针对传统算法需要主观规定网络中结点顺序的缺点,提出了一个新的可以在无约束条件下,根据观测得到的训练样本集的概率关系,自动完成学习贝叶斯网络结构的新方法.     

Bayes网络学习的MCMC方法

基于Bayes统计理论, 提出了一种从数据样本中学习Bayes网络的Markov链Monte Carlo(MCMC)方法. 首先通过先验概率和数据样本的结合得到未归一化的后验概率, 然后使用此后验概率指导随机搜索算法寻找“好”的网络结构模型. 通过对Alarm网络的学习表明了本算法具有较好的性能.     

Associative learning in random environments using neural networks

Associative learning is investigated using neural networks and concepts based on learning automata. The behavior of a single decision-maker containing a neural network is studied in a random environment using reinforcement learning. The objective is to determine the optimal action corresponding to a particular state. Since decisions have to be made throughout the context space based on a countable number of experiments, generalization is inevitable. Many different approaches can be followed to generate the desired discriminant function. Three different methods which use neural networks are discussed and compared. In the most general method, the output of the network determines the probability with which one of the actions is to be chosen. The weights of the network are updated on the basis of the actions and the response of the environment. The extension of similar concepts to decentralized decision-making in a context space is also introduced. Simulation results are included. Modifications in the implementations of the most general method to make it practically viable are also presented. All the methods suggested are feasible and the choice of a specific method depends on the accuracy desired as well as on the available computational power.     

Neural associative memory with optimal Bayesian learning

Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.     

贝叶斯网络结构稀疏学习研究进展*

贝叶斯网络结构稀疏化学习因其既能简化结构又能保留原始网络中的重要信息,已经成为当前贝叶斯网络的研究热点.文中首先讨论贝叶斯网络结构稀疏学习的必要性、贝叶斯网络稀疏性的定义,并在此基础上介绍现有的贝叶斯网络结构稀疏学习研究思路.然后,回顾一般的贝叶斯网络结构学习方法,并分析它们在高维背景下存在的问题,进而发现基于评分的方法通常适合于贝叶斯网络结构的稀疏学习,因此重点介绍贝叶斯网络结构稀疏学习的目标函数和优化求解算法.最后,探讨未来贝叶斯网络结构稀疏学习的一些研究方向.     

Restricted Bayesian classification networks

Bayesian networks are graphical models that describe dependency relationships between variables, and are powerful tools for studying probability classifiers. At present, the causal Bayesian network learning method is used in constructing Bayesian network classifiers while the contribution of attribute to class is over-looked. In this paper, a Bayesian network specifically for classification-restricted Bayesian classification networks is proposed. Combining dependency analysis between variables, classification accuracy evaluation criteria and a search algorithm, a learning method for restricted Bayesian classification networks is presented. Experiments and analysis are done using data sets from UCI machine learning repository. The results show that the restricted Bayesian classification network is more accurate than other well-known classifiers.     

A Bayesian Method for the Induction of Probabilistic Networks from Data

This paper presents a Bayesian method for constructing probabilistic networks from databases. In particular, we focus on constructing Bayesian belief networks. Potential applications include computer-assisted hypothesis testing, automated scientific discovery, and automated construction of probabilistic expert systems. We extend the basic method to handle missing data and hidden (latent) variables. We show how to perform probabilistic inference by averaging over the inferences of multiple belief networks. Results are presented of a preliminary evaluation of an algorithm for constructing a belief network from a database of cases. Finally, we relate the methods in this paper to previous work, and we discuss open problems.     

Stochastic combinational networks

A new model is defined for combinational networks of probabilistic logic gates which differs from the earlier models of von Neumann and McCulloch in the way in which timing is handled. This differences makes the new model compatible with ordinary deterministic switching theory in that the outputs of any network containing feedforward connections, but not feedback loops, depend only on the network's inputs after a short, fixed time delay, not on any preceding inputs. A general method of analyzing arbitrary combinational networks within the new model is developed using a formalism based on the algebra of stochastic matrices. Methods of simplifying network analysis under certain special circumstances are also given. Probabilistic generalizations of several concepts from ordinary deterministic switching theory are developed in some detail. These include the complement of a function, the dual of a function, a fixed variable, a dummy variable, a symmetric function, and an associative function. De Morgan's law and the principle of duality are also shown to have probabilistic generalizations. Several methods are next developed for synthesizing one-output combinational networks within the new model. Included are generalizations of several basic methods from deterministic switching theory such as the and-or, or-and, and Shannon expansion methods. Several methods of multiple-output network synthesis are also developed, including a set of cascade networks as well as generalizations of the various one-output network synthesis methods. Finally, the concept of a sequential network is introduced, and the relationships between combinational networks, sequential networks, and stochastic sequential machines are developed in some detail.     

Mathematical Foundations of Probabilistic Estimation in Fuzzy Bayesian Belief Networks

Specific problems arising in the creation of Bayesian networks for processing fuzzy information are considered. Conditions are obtained that ensure the probabilistic correctness of a priori data and results of network computing. For the case of nondeterministic states of network vertices, a fuzzy linear interpolation procedure is proposed. The presented results make it possible to perform correct probabilistic Bayesian estimation over fuzzy networks of any configuration.     

A method of learning implication networks from empirical data:algorithm and Monte-Carlo simulation-based validation

The paper describes an algorithmic means for inducing implication networks from empirical data samples. The induced network enables efficient inferences about the values of network nodes if certain observations are made. This implication induction method is approximate in nature as probabilistic network requirements are relaxed in the construction of dependence relationships based on statistical testing. In order to examine the effectiveness and validity of the induction method, several Monte Carlo simulations were conducted, where theoretical Bayesian networks were used to generate empirical data samples-some of which were used to induce implication relations, whereas others were used to verify the results of evidential reasoning with the induced networks. The values in the implication networks were predicted by applying a modified version of the Dempster-Shafer belief updating scheme. The results of predictions were, furthermore, compared to the ones generated by Pearl's (1986) stochastic simulation method, a probabilistic reasoning method that operates directly on the theoretical Bayesian networks. The comparisons consistently show that the results of predictions based on the induced networks would be comparable to those generated by Pearl's method, when reasoning in a variety of uncertain knowledge domains-those that were simulated using the presumed theoretical probabilistic networks of different topologies     

Method of probabilistic inference from learning data in Bayesian networks

Bayesian networks (BN) are a powerful tool for various data-mining systems. The available methods of probabilistic inference from learning data have shortcomings such as high computation complexity and cumulative error. This is due to a partial loss of information in transition from empiric information to conditional probability tables. The paper presents a new simple and exact algorithm for probabilistic inference in BN from learning data. __________ Translated from Kibernetika i Sistemnyi Analiz, No. 3, pp. 93–99, May–June 2007.     

混合贝叶斯网络隐藏变量学习研究

目前,具有已知结构的隐藏变量学习主要针对具有离散变量的贝叶斯网和具有连续变量的高斯网．该文给出了具有连续和离散变量的混合贝叶斯网络隐藏变量学习方法．该方法不需要离散化连续变量,依据专业知识或贝叶斯网络道德图中Cliques的维数发现隐藏变量的位置,基于依赖结构（星形结构或先验结构）和Gibbs抽样确定隐藏变量的值,结合扩展的MDL标准和统计方法发现隐藏变量的最优维数．实验结果表明,这种方法能够有效地进行具有已知结构的混合贝叶斯网络隐藏变量学习．     

机器学习赋能的软件自适应性综述

软件系统自适应机制提供了应对动态变化的环境和不确定的需求的技术方案.在已有的软件系统自适应性的相关研究中，有一类工作将软件系统自适应性转换为回归、分类、聚类、决策等问题，并利用强化学习、神经网络/深度学习、贝叶斯决策理论和概率图模型、规则学习等进行问题建模，并以此构造软件系统自适应机制.本文通过系统化的文献调研，综述了机器学习赋能的软件系统自适应性的工作.首先介绍基本概念，然后从不同视角对当前工作进行分类；按被控系统、监测和控制过程、学习算法、学习赋能方式等方面进行分析，并讨论不同机器学习方法赋能的软件系统自适应性的切入点及其优势和不足；最后对未来研究进行展望.     

Bayesian dual neural networks for recommendation

Most traditional collaborative filtering (CF) methods only use the user-item rating matrix to make recommendations, which usually suffer from cold-start and sparsity problems. To address these problems, on the one hand, some CF methods are proposed to incorporate auxiliary information such as user/item profiles; on the other hand, deep neural networks, which have powerful ability in learning effective representations, have achieved great success in recommender systems. However, these neural network based recommendation methods rarely consider the uncertainty of weights in the network and only obtain point estimates of the weights. Therefore, they maybe lack of calibrated probabilistic predictions and make overly confident decisions. To this end, we propose a new Bayesian dual neural network framework, named BDNet, to incorporate auxiliary information for recommendation. Specifically, we design two neural networks, one is to learn a common low dimensional space for users and items from the rating matrix, and another one is to project the attributes of users and items into another shared latent space. After that, the outputs of these two neural networks are combined to produce the final prediction. Furthermore, we introduce the uncertainty to all weights which are represented by probability distributions in our neural networks to make calibrated probabilistic predictions. Extensive experiments on real-world data sets are conducted to demonstrate the superiority of our model over various kinds of competitors.     

概率图模型推理方法的研究进展

近年来概率图模型已成为不确定性推理的研究热点,在人工智能、机器学习与计算机视觉等领域有广阔的应用前景.根据网络结构与查询问题类型的不同,系统地综述了概率图模型的推理算法.首先讨论了贝叶斯网络与马尔可夫网络中解决概率查询问题的精确推理算法与近似推理算法,其中主要介绍精确推理中的VE算法、递归约束算法和团树算法,以及近似推理中的变分近似推理和抽样近似推理算法,并给出了解决MAP查询问题的常用推理算法;然后分别针对混合网络的连续与混合情况阐述其推理算法,并分析了暂态网络的精确推理、近似推理以及混合情况下的推理;最后指出了概率图模型推理方法未来的研究方向.     

基于贝叶斯方法的分类预测

介绍了基本的贝叶斯分类模型和贝叶斯信念网络模型,对网络模型的学习进行了讨论。并从实际出发,提出了几种可以简化模型结构、降低学习复杂性的可行方法,简要说明了这些方法在网络模型中的应用。对贝叶斯分类模型的准确性及其主要特点进行了分析。