The Bayesian evidence scheme for regularizing probability-density estimating neural networks

Training probability-density estimating neural networks with the expectation-maximization (EM) algorithm aims to maximize the likelihood of the training set and therefore leads to overfitting for sparse data. In this article, a regularization method for mixture models with generalized linear kernel centers is proposed, which adopts the Bayesian evidence approach and optimizes the hyperparameters of the prior by type II maximum likelihood. This includes a marginalization over the parameters, which is done by Laplace approximation and requires the derivation of the Hessian of the log-likelihood function. The incorporation of this approach into the standard training scheme leads to a modified form of the EM algorithm, which includes a regularization term and adapts the hyperparameters on-line after each EM cycle. The article presents applications of this scheme to classification problems, the prediction of stochastic time series, and latent space models.     

A method for explaining Bayesian networks for legal evidence with scenarios

In a criminal trial, a judge or jury needs to reason about what happened based on the available evidence, often including statistical evidence. While a probabilistic approach is suitable for analysing the statistical evidence, a judge or jury may be more inclined to use a narrative or argumentative approach when considering the case as a whole. In this paper we propose a combination of two approaches, combining Bayesian networks with scenarios. Whereas a Bayesian network is a popular tool for analysing parts of a case, constructing and understanding a network for an entire case is not straightforward. We propose an explanation method for understanding a Bayesian network in terms of scenarios. This method builds on a previously proposed construction method, which we slightly adapt with the use of scenario schemes for the purpose of explaining. The resulting structure is explained in terms of scenarios, scenario quality and evidential support. A probabilistic interpretation of scenario quality is provided using the concept of scenario schemes. Finally, the method is evaluated by means of a case study.     

Building Bayesian networks for legal evidence with narratives: a case study evaluation

In a criminal trial, evidence is used to draw conclusions about what happened concerning a supposed crime. Traditionally, the three main approaches to modeling reasoning with evidence are argumentative, narrative and probabilistic approaches. Integrating these three approaches could arguably enhance the communication between an expert and a judge or jury. In previous work, techniques were proposed to represent narratives in a Bayesian network and to use narratives as a basis for systematizing the construction of a Bayesian network for a legal case. In this paper, these techniques are combined to form a design method for constructing a Bayesian network based on narratives. This design method is evaluated by means of an extensive case study concerning the notorious Dutch case of the Anjum murders.     

A Bayesian framework for sensory adaptation

Adaptation allows biological sensory systems to adjust to variations in the environment and thus to deal better with them. In this article, we propose a general framework of sensory adaptation. The underlying principle of this framework is the setting of internal parameters of the system such that certain prespecified tasks can be performed optimally. Because sensorial inputs vary probabilistically with time and biological mechanisms have noise, the tasks could be performed incorrectly. We postulate that the goal of adaptation is to minimize the number of task errors. This minimization requires prior knowledge of the environment and of the limitations of the mechanisms processing the information. Because these processes are probabilistic, we formulate the minimization with a Bayesian approach. Application of this Bayesian framework to the retina is successful in accounting for a host of experimental findings.     

A framework for the analysis of dynamic processes based on Bayesian networks and case-based reasoning

Bayesian networks are knowledge representation schemes that can capture probabilistic relationships among variables and perform probabilistic inference. Arrival of new evidence propagates through the network until all variables are updated. At the end of propagation, the network becomes a static snapshot representing the state of the domain for that particular time. This weakness in capturing temporal semantics has limited the use of Bayesian networks to domains in which time dependency is not a critical factor. This paper describes a framework that combines Bayesian networks and case-based reasoning to create a knowledge representation scheme capable of dealing with time-varying processes. Static Bayesian network topologies are learned from previously available raw data and from sets of constraints describing significant events. These constraints are defined as sets of variables assuming significant values. As new data are gathered, dynamic changes to the topology of a Bayesian network are assimilated using techniques that combine single-value decomposition and minimum distance length. The new topologies are capable of forecasting the occurrences of significant events given specific conditions and monitoring changes over time. Since environment problems are good examples of temporal variations, the problem of forecasting ozone levels in Mexico City was used to test this framework.     

Bayesian networks for student model engineering

Bayesian networks are graphical modeling tools that have been proven very powerful in a variety of application contexts. The purpose of this paper is to provide education practitioners with the background and examples needed to understand Bayesian networks and use them to design and implement student models. The student model is the key component of any adaptive tutoring system, as it stores all the information about the student (for example, knowledge, interest, learning styles, etc.) so the tutoring system can use this information to provide personalized instruction. Basic and advanced concepts and techniques are introduced and applied in the context of typical student modeling problems. A repertoire of models of varying complexity is discussed. To illustrate the proposed methodology a Bayesian Student Model for the Simplex algorithm is developed.     

On compatible priors for Bayesian networks

Given a Bayesian network of discrete random variables with a hyper-Dirichlet prior, a method is proposed for assigning Dirichlet priors to the conditional probabilities of structurally different networks. It defines a distance measure between priors which is to be minimized for the assignment process. Intuitively one would expect that if two models priors are to qualify as being `close' in some sense, then their posteriors should also be nearby after an observation. However one does not know in advance what will be observed next. Thus we are led to propose an expectation of Kullback-Leibler distances over all possible next observations to define a measure of distance between priors. In conjunction with the additional assumptions of global and local independence of the parameters, a number of theorems emerge which are usually taken as reasonable assumptions in the Bayesian network literature. A simple example is given to illustrate the technique     

Bayesian wavelet networks for nonparametric regression

Radial wavelet networks have been proposed previously as a method for nonparametric regression. We analyze their performance within a Bayesian framework. We derive probability distributions over both the dimension of the networks and the network coefficients by placing a prior on the degrees of freedom of the model. This process bypasses the need to test or select a finite number of networks during the modeling process. Predictions are formed by mixing over many models of varying dimension and parameterization. We show that the complexity of the models adapts to the complexity of the data and produces good results on a number of benchmark test series.     

Learning Bayesian networks for discrete data

Bayesian networks have received much attention in the recent literature. In this article, we propose an approach to learn Bayesian networks using the stochastic approximation Monte Carlo (SAMC) algorithm. Our approach has two nice features. Firstly, it possesses the self-adjusting mechanism and thus avoids essentially the local-trap problem suffered by conventional MCMC simulation-based approaches in learning Bayesian networks. Secondly, it falls into the class of dynamic importance sampling algorithms; the network features can be inferred by dynamically weighted averaging the samples generated in the learning process, and the resulting estimates can have much lower variation than the single model-based estimates. The numerical results indicate that our approach can mix much faster over the space of Bayesian networks than the conventional MCMC simulation-based approaches.     

Inductive transfer for learning Bayesian networks

In several domains it is common to have data from different, but closely related problems. For instance, in manufacturing, many products follow the same industrial process but with different conditions; or in industrial diagnosis, where there is equipment with similar specifications. In these cases it is common to have plenty of data for some scenarios but very little for others. In order to learn accurate models for rare cases, it is desirable to use data and knowledge from similar cases; a technique known as transfer learning. In this paper we propose an inductive transfer learning method for Bayesian networks, that considers both structure and parameter learning. For structure learning we use conditional independence tests, by combining measures from the target task with those obtained from one or more auxiliary tasks, using a novel weighted sum of the conditional independence measures. For parameter learning, we propose two variants of the linear pool for probability aggregation, combining the probability estimates from the target task with those from the auxiliary tasks. To validate our approach, we used three Bayesian networks models that are commonly used for evaluating learning techniques, and generated variants of each model by changing the structure as well as the parameters. We then learned one of the variants with a small dataset and combined it with information from the other variants. The experimental results show a significant improvement in terms of structure and parameters when we transfer knowledge from similar tasks. We also evaluated the method with real-world data from a manufacturing process considering several products, obtaining an improvement in terms of log-likelihood between the data and the model when we do transfer learning from related products.     

New skeleton-based approaches for Bayesian structure learning of Bayesian networks

Automatically learning the graph structure of a single Bayesian network (BN) which accurately represents the underlying multivariate probability distribution of a collection of random variables is a challenging task. But obtaining a Bayesian solution to this problem based on computing the posterior probability of the presence of any edge or any directed path between two variables or any other structural feature is a much more involved problem, since it requires averaging over all the possible graph structures. For the former problem, recent advances have shown that search + score approaches find much more accurate structures if the search is constrained by a previously inferred skeleton (i.e. a relaxed structure with undirected edges which can be inferred using local search based methods). Based on similar ideas, we propose two novel skeleton-based approaches to approximate a Bayesian solution to the BN learning problem: a new stochastic search which tries to find directed acyclic graph (DAG) structures with a non-negligible score; and a new Markov chain Monte Carlo method over the DAG space. These two approaches are based on the same idea. In a first step, both employ a previously given skeleton and build a Bayesian solution constrained by this skeleton. In a second step, using the preliminary solution, they try to obtain a new Bayesian approximation but this time in an unconstrained graph space, which is the final outcome of the methods. As shown in the experimental evaluation, this new approach strongly boosts the performance of these two standard techniques proving that the idea of employing a skeleton to constrain the model space is also a successful strategy for performing Bayesian structure learning of BNs.     

A Bayesian network-based framework for semantic image understanding

Current research in content-based semantic image understanding is largely confined to exemplar-based approaches built on low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features is expected to raise semantic image understanding to a new level. Belief networks, or Bayesian networks (BN), have proven to be an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditional independence relationships. In this paper, we present a general-purpose knowledge integration framework that employs BN in integrating both low-level and semantic features. The efficacy of this framework is demonstrated via three applications involving semantic understanding of pictorial images. The first application aims at detecting main photographic subjects in an image, the second aims at selecting the most appealing image in an event, and the third aims at classifying images into indoor or outdoor scenes. With these diverse examples, we demonstrate that effective inference engines can be built within this powerful and flexible framework according to specific domain knowledge and available training data to solve inherently uncertain vision problems.     

Using Bayesian networks for monitoring computer users

The structure and methods of a knowledgebase are discussed. This knowledgebase is considered to be a component part of personal expert systems designed for monitoring computer users and computer environments with a view to detecting a harmful effect of computers on the state of health of computer users and making recommendations concerning the neutralization of this effect. Such expert systems are based on Bayesian networks that have a simple topology and represent random relations between harmful conditions of a computer environment and the state of help of users.Translated from Kibernetika i Sistemnyi Analiz, No. 6, pp. 3–15, November–December 2004.This revised version was published online in April 2005 with a corrected cover date.     

Bayesian MLP neural networks for image analysis

We demonstrate the advantages of using Bayesian multi-layer perceptron (MLP) neural networks for image analysis. The Bayesian approach provides consistent way to do inference by combining the evidence from the data to prior knowledge from the problem. A practical problem with MLPs is to select the correct complexity for the model, i.e., the right number of hidden units or correct regularization parameters. The Bayesian approach offers efficient tools for avoiding overfitting even with very complex models, and facilitates estimation of the confidence intervals of the results. In this contribution we review the Bayesian methods for MLPs and present comparison results from two case studies. In the first case, MLPs were used to solve the inverse problem in electrical impedance tomography. The Bayesian MLP provided consistently better results than other methods. In the second case, the goal was to locate trunks of trees in forest scenes. With Bayesian MLP it was possible to use large number of potentially useful features and prior for determining the relevance of the features automatically.     

Bayesian methods for elucidating genetic regulatory networks

Bayesian network methods are useful for elucidating genetic regulatory networks because they can represent more than pair-wise relationships between variables, are resistant to overfitting, and remain robust in the face of noisy data.     

Bayesian networks for imputation in classification problems

Missing values are an important problem in data mining. In order to tackle this problem in classification tasks, we propose two imputation methods based on Bayesian networks. These methods are evaluated in the context of both prediction and classification tasks. We compare the obtained results with those achieved by classical imputation methods (Expectation–Maximization, Data Augmentation, Decision Trees, and Mean/Mode). Our simulations were performed by means of four datasets (Congressional Voting Records, Mushroom, Wisconsin Breast Cancer and Adult), which are benchmarks for data mining methods. Missing values were simulated in these datasets by means of the elimination of some known values. Thus, it is possible to assess the prediction capability of an imputation method, comparing the original values with the imputed ones. In addition, we propose a methodology to estimate the bias inserted by imputation methods in classification tasks. In this sense, we use four classifiers (One Rule, Naïve Bayes, J4.8 Decision Tree and PART) to evaluate the employed imputation methods in classification scenarios. Computing times consumed to perform imputations are also reported. Simulation results in terms of prediction, classification, and computing times allow us performing several analyses, leading to interesting conclusions. Bayesian networks have shown to be competitive with classical imputation methods.