Being Bayesian About Network Structure. A Bayesian Approach to Structure Discovery in Bayesian Networks

In many multivariate domains, we are interested in analyzing the dependency structure of the underlying distribution, e.g., whether two variables are in direct interaction. We can represent dependency structures using Bayesian network models. To analyze a given data set, Bayesian model selection attempts to find the most likely (MAP) model, and uses its structure to answer these questions. However, when the amount of available data is modest, there might be many models that have non-negligible posterior. Thus, we want compute the Bayesian posterior of a feature, i.e., the total posterior probability of all models that contain it. In this paper, we propose a new approach for this task. We first show how to efficiently compute a sum over the exponential number of networks that are consistent with a fixed order over network variables. This allows us to compute, for a given order, both the marginal probability of the data and the posterior of a feature. We then use this result as the basis for an algorithm that approximates the Bayesian posterior of a feature. Our approach uses a Markov Chain Monte Carlo (MCMC) method, but over orders rather than over network structures. The space of orders is smaller and more regular than the space of structures, and has much a smoother posterior landscape. We present empirical results on synthetic and real-life datasets that compare our approach to full model averaging (when possible), to MCMC over network structures, and to a non-Bayesian bootstrap approach.     

Parameterization of a process-based tree-growth model: Comparison of optimization, MCMC and Particle Filtering algorithms

Finely tuned process-based tree-growth models are of considerable help in understanding the variations of biomass increments measured in the dendrochronological series. Using site and species parameters, as well as daily climate variables, the MAIDEN model computes the water balance at ecosystem level and the daily increment of carbon storage in the stem through photosynthesis processes to reproduce the structure of the tree-ring series. In this paper, we use three techniques to calibrate this model with Pinus halepensis data sampled in the Mediterranean part of France: a standard optimization (PEST), Monte Carlo Markov Chains (MCMC) and Particle Filtering (PF). Contrary to PEST, which tries to find an optimum fit (giving the lowest error between observations and simulations), the principle of MCMC and PF is to walk, from a priori distributions, in the parameter space according to particular statistical rules to compute each parameter distribution. The PEST and MCMC calibrations of our dendrochronological series lead to rather similar adjustments between simulations and observations. PF and MCMC calibrations give different parameter distributions, showing how complementary are these methods, with a better fit for MCMC. Yet, independent validations over 11 independent meteorological years show a higher efficiency of the recent PF method over the others.     

Classification with Bayesian MARS

We present a new method for classification using a Bayesian version of the Multivariate Adaptive Regression Spline (MARS) model of J.H. Friedman (Annals of Statistics, 19, 1–141, 1991). Special attention is paid to the use of Markov chain Monte Carlo (MCMC) simulation to gain inference under the model. In particular we discuss three important developments in MCMC methodology. First, we describe the reversible jump MCMC algorithm of P.J. Green (Biometrika, 82, 711–732, 1995) which allows inference on a varying dimensional, possibly uncountable, model space. This allows us to consider MARS models of differing numbers and positions of splines. Secondly, we discuss marginalisation which is used to reduce the effective dimension of the parameter space under consideration. Thirdly, we describe the use of latent variables to improve the MCMC computation. Our methods are generic and can be applied to any basis function model including, wavelets, artificial neural nets and radial basis functions. We present examples to show that the Bayesian MARS classifier is competitive with other approaches on a number of benchmark data sets.     

Convergence of concurrent Markov chain Monte-Carlo algorithms

We examine the parallel execution of a class of stochastic algorithms called Markov chain Monte-Carlo (MCMC) algorithms. We focus on MCMC algorithms in the context of image processing, using Markov random field models. Our parallelisation approach is based on several, concurrently running, instances of the same stochastic algorithm that deal with the whole data set. Firstly we show that the speed-up of the parallel algorithm is limited because of the statistical properties of the MCMC algorithm. We examine coupled MCMC as a remedy for this problem. Secondly, we exploit the parallel execution to monitor the convergence of the stochastic algorithms in a statistically reliable manner. This new convergence measure for MCMC algorithms performs well, and is an improvement on known convergence measures. We also link our findings with recent work in the statistical theory of MCMC.     

Variational Bayesian methods for spatial data analysis

With scientific data available at geocoded locations, investigators are increasingly turning to spatial process models for carrying out statistical inference. However, fitting spatial models often involves expensive matrix decompositions, whose computational complexity increases in cubic order with the number of spatial locations. This situation is aggravated in Bayesian settings where such computations are required once at every iteration of the Markov chain Monte Carlo (MCMC) algorithms. In this paper, we describe the use of Variational Bayesian (VB) methods as an alternative to MCMC to approximate the posterior distributions of complex spatial models. Variational methods, which have been used extensively in Bayesian machine learning for several years, provide a lower bound on the marginal likelihood, which can be computed efficiently. We provide results for the variational updates in several models especially emphasizing their use in multivariate spatial analysis. We demonstrate estimation and model comparisons from VB methods by using simulated data as well as environmental data sets and compare them with inference from MCMC.     

Computational Issues in the Sequential Probit Model: A Monte Carlo Study

We discuss computational issues in the sequential probit model that have limited its use in applied research. We estimate parameters of the model by the method of simulated maximum likelihood (SML) and by Bayesian MCMC algorithms. We provide Monte Carlo evidence on the relative performance of both estimators and find that the SML procedure computes standard errors of the estimated correlation coefficients that are less reliable. Given the numerical difficulties associated with the estimation procedures, we advise the applied researcher to use both the stochastic optimization algorithm in the Simulated Maximum Likelihood approach and the Bayesian MCMC algorithm to check the compatibility of the results. JEL Classifications: C11, C15, C35, C63     

Classification of newborn EEG maturity with Bayesian averaging over decision trees

EEG experts can assess a newborn’s brain maturity by visual analysis of age-related patterns in sleep EEG. It is highly desirable to make the results of assessment most accurate and reliable. However, the expert analysis is limited in capability to provide the estimate of uncertainty in assessments. Bayesian inference has been shown providing the most accurate estimates of uncertainty by using Markov Chain Monte Carlo (MCMC) integration over the posterior distribution. The use of MCMC enables to approximate the desired distribution by sampling the areas of interests in which the density of distribution is high. In practice, the posterior distribution can be multimodal, and so that the existing MCMC techniques cannot provide the proportional sampling from the areas of interest. The lack of prior information makes MCMC integration more difficult when a model parameter space is large and cannot be explored in detail within a reasonable time. In particular, the lack of information about EEG feature importance can affect the results of Bayesian assessment of EEG maturity. In this paper we explore how the posterior information about EEG feature importance can be used to reduce a negative influence of disproportional sampling on the results of Bayesian assessment. We found that the MCMC integration tends to oversample the areas in which a model parameter space includes one or more features, the importance of which counted in terms of their posterior use is low. Using this finding, we proposed to cure the results of MCMC integration and then described the results of testing the proposed method on a set of sleep EEG recordings.     

Bayesian inference and prediction of the inverse Weibull distribution for Type-II censored data

This paper describes the Bayesian inference and prediction of the inverse Weibull distribution for Type-II censored data. First we consider the Bayesian inference of the unknown parameter under a squared error loss function. Although we have discussed mainly the squared error loss function, any other loss function can easily be considered. A Gibbs sampling procedure is used to draw Markov Chain Monte Carlo (MCMC) samples, and they have in turn, been used to compute the Bayes estimates and also to construct the corresponding credible intervals with the help of an importance sampling technique. We have performed a simulation study in order to compare the proposed Bayes estimators with the maximum likelihood estimators. We further consider one-sample and two-sample Bayes prediction problems based on the observed sample and provide appropriate predictive intervals with a given coverage probability. A real life data set is used to illustrate the results derived. Some open problems are indicated for further research.     

Using a parallelized MCMC algorithm in R to identify appropriate likelihood functions for SWAT

Markov Chain Monte Carlo (MCMC) algorithms allow the analysis of parameter uncertainty. This analysis can inform the choice of appropriate likelihood functions, thereby advancing hydrologic modeling with improved parameter and quantity estimates and more reliable assessment of uncertainty. For long-running models, the Differential Evolution Adaptive Metropolis (DREAM) algorithm offers spectacular reductions in time required for MCMC analysis. This is partly due to multiple parameter sets being evaluated simultaneously. The ability to use this feature is hindered in models that have a large number of input files, such as SWAT. A conceptually simple, robust method for applying DREAM to SWAT in R is provided. The general approach is transferrable to any executable that reads input files. We provide this approach to reduce barriers to the use of MCMC algorithms and to promote the development of appropriate likelihood functions.     

Bayesian learning of finite generalized inverted Dirichlet mixtures: Application to object classification and forgery detection

The advent of mixture models has opened the possibility of flexible models which are practical to work with. A common assumption is that practitioners typically expect that data are generated from a Gaussian mixture. The inverted Dirichlet mixture has been shown to be a better alternative to the Gaussian mixture and to be of significant value in a variety of applications involving positive data. The inverted Dirichlet is, however, usually undesirable, since it forces an assumption of positive correlation. Our focus here is to develop a Bayesian alternative to both the Gaussian and the inverted Dirichlet mixtures when dealing with positive data. The alternative that we propose is based on the generalized inverted Dirichlet distribution which offers high flexibility and ease of use, as we show in this paper. Moreover, it has a more general covariance structure than the inverted Dirichlet. The proposed mixture model is subjected to a fully Bayesian analysis based on Markov Chain Monte Carlo (MCMC) simulation methods namely Gibbs sampling and Metropolis–Hastings used to compute the posterior distribution of the parameters, and on Bayesian information criterion (BIC) used for model selection. The adoption of this purely Bayesian learning choice is motivated by the fact that Bayesian inference allows to deal with uncertainty in a unified and consistent manner. We evaluate our approach on the basis of two challenging applications concerning object classification and forgery detection.     

A multi-objective Markov Chain Monte Carlo cellular automata model: Simulating multi-density urban expansion in NYC

Cellular automata (CA) models have increasingly been used to simulate land use/cover changes (LUCC). Metaheuristic optimization algorithms such as particle swarm optimization (PSO) and genetic algorithm (GA) have been recently introduced into CA frameworks to generate more accurate simulations. Although Markov Chain Monte Carlo (MCMC) is simpler than PSO and GA, it is rarely used to calibrate CA models. In this article, we introduce a novel multi-chain multi-objective MCMC (mc-MO-MCMC) CA model to simulate LUCC. Unlike the classical MCMC, the proposed mc-MO-MCMC is a multiple chains method that imports crossover operation from classical evolutionary optimization algorithms. In each new chain, after the initial one, the crossover operator generates the initial solution. The selection of solutions to be crossed over are made according to their fitness score. In this paper, we chose the example of New York City (USA) to apply our model to simulate three conflicting objectives of changes from non-urban to low-, medium- or high-density urban between 2001 and 2016 using USA National Land Cover Database (NLCD). Elevation, slope, Euclidean distance to highways and local roads, population volume and average household income are used as LUCC causative factors. Furthermore, to demonstrate the efficiency of our proposed model, we compare it with the multi-objective genetic algorithm (MO-GA) and standard single-chain multi-objective MCMC (sc-MO-MCMC). Our results demonstrate that mc-MO-MCMC produces accurate simulations of land use dynamics featured by faster convergence to the Pareto frontier comparing to MO-GA and sc-MO-MCMC. The proposed multi-objective cellular automata model should efficiently help to simulate a trade-off among multiple and, possibly, conflicting land use change dynamics at once.     

Generating Design Suggestions under Tight Constraints with Gradient‐based Probabilistic Programming

We present a system for generating suggestions from highly‐constrained, continuous design spaces. We formulate suggestion as sampling from a probability distribution; constraints are represented as factors that concentrate probability mass around sub‐manifolds of the design space. These sampling problems are intractable using typical random walk MCMC techniques, so we adopt Hamiltonian Monte Carlo (HMC), a gradient‐based MCMC method. We implement HMC in a high‐performance probabilistic programming language, and we evaluate its ability to efficiently generate suggestions for two different, highly‐constrained example applications: vector art coloring and designing stable stacking structures.     

Dynamic Bayesian beta models

We develop a dynamic Bayesian beta model for modeling and forecasting single time series of rates or proportions. This work is related to a class of dynamic generalized linear models (DGLMs), although, for convenience, we use non-conjugate priors. The proposed methodology is based on approximate analysis relying on Bayesian linear estimation, nonlinear system of equations solution and Gaussian quadrature. Intentionally we avoid MCMC strategy, keeping the desired sequential nature of the Bayesian analysis. Applications to both real and simulated data are provided.     

A theoretical and real world evaluation of two Bayesian techniques for the calibration of variety parameters in a sugarcane crop model

Process based agricultural systems models allow researchers to investigate the interactions between variety, environment and management. The ‘Sugar’ module in the Agricultural Productions Systems sIMulator (APSIM-Sugar) currently includes definitions for 14 sugarcane varieties, most of which are no longer commercially grown. This study evaluated the use of two Bayesian approaches to calibrate sugarcane varieties in APSIM-Sugar: Generalized Likelihood Uncertainty Estimation (GLUE) and Markov Chain Monte Carlo (MCMC). Both GLUE and MCMC calibrations were able to accurately simulate green biomass and sucrose yield in both a theoretical and real world evaluation. In the theoretical evaluation GLUE and MCMC parameter estimates accurately reflected differences between two pre-defined sugarcane varieties. We found that the MCMC approach can be used to calibrate varieties in APSIM-Sugar based on yield data. With appropriate variety definitions, APSIM-Sugar could be used for early risk assessment of adopting new varieties.     

WFS + branch and bound = stable models

Though the semantics of nonmonotonic logic programming has been studied extensively, relatively little work has been done on operational aspects of these semantics. In this paper, we develop techniques to compute the well-founded model of a logic program. We describe a prototype implementation and show, based on experimental results, that our technique is more efficient than the standard alternating fixpoint computation. Subsequently, we develop techniques to compute the set of all stable models of a deductive database. These techniques first compute the well-founded semantics and then use an intelligent branch and bound strategy to compute the stable models. We report on our implementation, as well as on experiments that we have conducted on the efficiency of our approach     

Efficient Markov chain Monte Carlo methods for decoding neural spike trains

Stimulus reconstruction or decoding methods provide an important tool for understanding how sensory and motor information is represented in neural activity. We discuss Bayesian decoding methods based on an encoding generalized linear model (GLM) that accurately describes how stimuli are transformed into the spike trains of a group of neurons. The form of the GLM likelihood ensures that the posterior distribution over the stimuli that caused an observed set of spike trains is log concave so long as the prior is. This allows the maximum a posteriori (MAP) stimulus estimate to be obtained using efficient optimization algorithms. Unfortunately, the MAP estimate can have a relatively large average error when the posterior is highly nongaussian. Here we compare several Markov chain Monte Carlo (MCMC) algorithms that allow for the calculation of general Bayesian estimators involving posterior expectations (conditional on model parameters). An efficient version of the hybrid Monte Carlo (HMC) algorithm was significantly superior to other MCMC methods for gaussian priors. When the prior distribution has sharp edges and corners, on the other hand, the "hit-and-run" algorithm performed better than other MCMC methods. Using these algorithms, we show that for this latter class of priors, the posterior mean estimate can have a considerably lower average error than MAP, whereas for gaussian priors, the two estimators have roughly equal efficiency. We also address the application of MCMC methods for extracting nonmarginal properties of the posterior distribution. For example, by using MCMC to calculate the mutual information between the stimulus and response, we verify the validity of a computationally efficient Laplace approximation to this quantity for gaussian priors in a wide range of model parameters; this makes direct model-based computation of the mutual information tractable even in the case of large observed neural populations, where methods based on binning the spike train fail. Finally, we consider the effect of uncertainty in the GLM parameters on the posterior estimators.     

An application of reversible-jump Markov chain Monte Carlo to spike classification of multi-unit extracellular recordings

Multi-electrode recordings in neural tissue contain the action potential waveforms of many closely spaced neurons. While we can observe the action potential waveforms, we cannot observe which neuron is the source for which waveform nor how many source neurons are being recorded. Current spike-sorting algorithms solve this problem by assuming a fixed number of source neurons and assigning the action potentials given this fixed number. We model the spike waveforms as an anisotropic Gaussian mixture model and present, as an alternative, a reversible-jump Markov chain Monte Carlo (MCMC) algorithm to simultaneously estimate the number of source neurons and to assign each action potential to a source. We derive this MCMC algorithm and illustrate its application using simulated three-dimensional data and real four-dimensional feature vectors extracted from tetrode recordings of rat entorhinal cortex neurons. In the analysis of the simulated data our algorithm finds the correct number of mixture components (sources) and classifies the action potential waveforms with minimal error. In the analysis of real data, our algorithm identifies clusters closely resembling those previously identified by a user-dependent graphical clustering procedure. Our findings suggest that a reversible-jump MCMC algorithm could offer a new strategy for designing automated spike-sorting algorithms.     

Static equilibrium of hyperelastic thin shell: symbolic and numerical computation

We here examine the natural shapes of an hyperelastic thin shell called a Carpentier's joint, when the terminal position is known. More specifically we study a rectangular strip that is a flexible thin shell with a constant curvature in its width and a null curvature in its length, at its unconstrained state. We use the theory of large displacement and small strain for hyperelastic material. We first consider an appropriate parameterization of the joint. Then we compute the Green-St Venant strain tensor with a symbolic computation system and we generate the numerical code to compute the elastic energy. In particular, we make strong use of symbolic elements to resolve some problems with zero division. Numerical minimization of this energy is used to find the shape and a couple of simulation are presented.