Efficient Markov chain Monte Carlo methods for decoding neural spike trains

Stimulus reconstruction or decoding methods provide an important tool for understanding how sensory and motor information is represented in neural activity. We discuss Bayesian decoding methods based on an encoding generalized linear model (GLM) that accurately describes how stimuli are transformed into the spike trains of a group of neurons. The form of the GLM likelihood ensures that the posterior distribution over the stimuli that caused an observed set of spike trains is log concave so long as the prior is. This allows the maximum a posteriori (MAP) stimulus estimate to be obtained using efficient optimization algorithms. Unfortunately, the MAP estimate can have a relatively large average error when the posterior is highly nongaussian. Here we compare several Markov chain Monte Carlo (MCMC) algorithms that allow for the calculation of general Bayesian estimators involving posterior expectations (conditional on model parameters). An efficient version of the hybrid Monte Carlo (HMC) algorithm was significantly superior to other MCMC methods for gaussian priors. When the prior distribution has sharp edges and corners, on the other hand, the "hit-and-run" algorithm performed better than other MCMC methods. Using these algorithms, we show that for this latter class of priors, the posterior mean estimate can have a considerably lower average error than MAP, whereas for gaussian priors, the two estimators have roughly equal efficiency. We also address the application of MCMC methods for extracting nonmarginal properties of the posterior distribution. For example, by using MCMC to calculate the mutual information between the stimulus and response, we verify the validity of a computationally efficient Laplace approximation to this quantity for gaussian priors in a wide range of model parameters; this makes direct model-based computation of the mutual information tractable even in the case of large observed neural populations, where methods based on binning the spike train fail. Finally, we consider the effect of uncertainty in the GLM parameters on the posterior estimators.     

Objective Bayesian analysis of accelerated competing failure models under Type-I censoring

This paper discusses the Bayesian inference of accelerated life tests (ALT) in the presence of competing failure causes. The time to failure due to a specific cause is described by a Weibull distribution. A two-stage approach is utilized to obtain the estimates of parameters in the model. We use the Bayesian method to estimate the parameters of the distribution of component lifetimes in the first stage, in which two noninformative priors (Jeffreys prior and reference prior) are derived in the case of ALT, and based on these two priors we present the Gibbs sampling procedures to obtain the posterior estimates of the parameters. Besides, to overcome the problem of improper posterior densities under some conditions, we modify the likelihood function to make the posterior densities proper. In the second stage, parameters in the accelerating function are obtained by least squares approach. A numerical example is given to show the effectiveness of the method and a real data from Nelson (1990) is analyzed.     

Heterogeneous data analysis using a mixture of Laplace models with conjugate priors

The development of flexible parametric classes of probability models in Bayesian analysis is a very popular approach. This study is designed for heterogeneous population for a two-component mixture of the Laplace probability distribution. When a process initially starts, the researcher expects that the failure components will be very high but after some improvement/inspection it is assumed that the failure components will decrease sufficiently. That is why in such situation the Laplace model is more suitable as compared to the normal distribution due to its fatter tails behaviour. We considered the derivation of the posterior distribution for censored data assuming different conjugate informative priors. Various kinds of loss functions are used to derive these Bayes estimators and their posterior risks. A method of elicitation of hyperparameter is discussed based on a prior predictive approach. The results are also compared with the non-informative priors. To examine the performance of these estimators we have evaluated their properties for different sample sizes, censoring rates and proportions of the component of the mixture through the simulation study. To highlight the practical significance we have included an illustrative application example based on real-life mixture data.     

Objective priors from maximum entropy in data classification

Lack of knowledge of the prior distribution in classification problems that operate on small data sets may make the application of Bayes’ rule questionable. Uniform or arbitrary priors may provide classification answers that, even in simple examples, may end up contradicting our common sense about the problem. Entropic priors (EPs), via application of the maximum entropy (ME) principle, seem to provide good objective answers in practical cases leading to more conservative Bayesian inferences. EP are derived and applied to classification tasks when only the likelihood functions are available. In this paper, when inference is based only on one sample, we review the use of the EP also in comparison to priors that are obtained from maximization of the mutual information between observations and classes. This last criterion coincides with the maximization of the KL divergence between posteriors and priors that for large sample sets leads to the well-known reference (or Bernardo’s) priors. Our comparison on single samples considers both approaches in prospective and clarifies differences and potentials. A combinatorial justification for EP, inspired by Wallis’ combinatorial argument for entropy definition, is also included.The application of the EP to sequences (multiple samples) that may be affected by excessive domination of the class with the maximum entropy is also considered with a solution that guarantees posterior consistency. An explicit iterative algorithm is proposed for EP determination solely from knowledge of the likelihood functions. Simulations that compare EP with uniform priors on short sequences are also included.     

Reference analysis for Birnbaum-Saunders distribution

In this paper we consider the Bayesian estimators for the unknown parameters of the Birnbaum-Saunders distribution under the reference prior. The Bayesian estimators cannot be obtained in closed forms. An approximate Bayesian approach is proposed using the idea of Lindley and Gibbs sampling procedure is also used to obtain the Bayesian estimators. These results are compared using Monte Carlo simulations with the maximum likelihood method and another approximate Bayesian approach Laplace’s approximation. Two real data sets are analyzed for illustrative purposes.     

Bayesian Model Learning Based on Predictive Entropy

Bayesian paradigm has been widely acknowledged as a coherent approach to learning putative probability model structures from a finite class of candidate models. Bayesian learning is based on measuring the predictive ability of a model in terms of the corresponding marginal data distribution, which equals the expectation of the likelihood with respect to a prior distribution for model parameters. The main controversy related to this learning method stems from the necessity of specifying proper prior distributions for all unknown parameters of a model, which ensures a complete determination of the marginal data distribution. Even for commonly used models, subjective priors may be difficult to specify precisely, and therefore, several automated learning procedures have been suggested in the literature. Here we introduce a novel Bayesian learning method based on the predictive entropy of a probability model, that can combine both subjective and objective probabilistic assessment of uncertain quantities in putative models. It is shown that our approach can avoid some of the limitations of the earlier suggested objective Bayesian methods.     

Bayesian combination of sparse and non-sparse priors in image super resolution

In this paper the application of image prior combinations to the Bayesian Super Resolution (SR) image registration and reconstruction problem is studied. Two sparse image priors, a Total Variation (TV) prior and a prior based on the ?1 norm of horizontal and vertical first-order differences (f.o.d.), are combined with a non-sparse Simultaneous Auto Regressive (SAR) prior. Since, for a given observation model, each prior produces a different posterior distribution of the underlying High Resolution (HR) image, the use of variational approximation will produce as many posterior approximations as priors we want to combine. A unique approximation is obtained here by finding the distribution on the HR image given the observations that minimizes a linear convex combination of Kullback–Leibler (KL) divergences. We find this distribution in closed form. The estimated HR images are compared with the ones obtained by other SR reconstruction methods.     

Bayesian learning of Bayesian networks with informative priors

This paper presents and evaluates an approach to Bayesian model averaging where the models are Bayesian nets (BNs). A comprehensive study of the literature on structural priors for BNs is conducted. A number of prior distributions are defined using stochastic logic programs and the MCMC Metropolis-Hastings algorithm is used to (approximately) sample from the posterior. We use proposals which are tightly coupled to the priors which give rise to cheaply computable acceptance probabilities. Experiments using data generated from known BNs have been conducted to evaluate the method. The experiments used 6 different BNs and varied: the structural prior, the parameter prior, the Metropolis-Hasting proposal and the data size. Each experiment was repeated three times with different random seeds to test the robustness of the MCMC-produced results. Our results show that with effective priors (i) robust results are produced and (ii) informative priors improve results significantly.     

Generalized Dirichlet priors for Naïve Bayesian classifiers with multinomial models in document classification

The generalized Dirichlet distribution has been shown to be a more appropriate prior than the Dirichlet distribution for naïve Bayesian classifiers. When the dimension of a generalized Dirichlet random vector is large, the computational effort for calculating the expected value of a random variable can be high. In document classification, the number of distinct words that is the dimension of a prior for naïve Bayesian classifiers is generally more than ten thousand. Generalized Dirichlet priors can therefore be inapplicable for document classification from the viewpoint of computational efficiency. In this paper, some properties of the generalized Dirichlet distribution are established to accelerate the calculation of the expected values of random variables. Those properties are then used to construct noninformative generalized Dirichlet priors for naïve Bayesian classifiers with multinomial models. Our experimental results on two document sets show that generalized Dirichlet priors can achieve a significantly higher prediction accuracy and that the computational efficiency of naïve Bayesian classifiers is preserved.     

Generalized exponential distribution: Bayesian estimations

Recently two-parameter generalized exponential distribution has been introduced by the authors. In this paper we consider the Bayes estimators of the unknown parameters under the assumptions of gamma priors on both the shape and scale parameters. The Bayes estimators cannot be obtained in explicit forms. Approximate Bayes estimators are computed using the idea of Lindley. We also propose Gibbs sampling procedure to generate samples from the posterior distributions and in turn computing the Bayes estimators. The approximate Bayes estimators obtained under the assumptions of non-informative priors, are compared with the maximum likelihood estimators using Monte Carlo simulations. One real data set has been analyzed for illustrative purposes.     

Bayesian estimation of beta mixture models with variational inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation to the prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed form) Bayesian approach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles of the VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate the distribution of the parameters in BMM. In a fully Bayesian model where all of the parameters of the BMM are considered as variables and assigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also, the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numerical calculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximization algorithm. The good performance of this approach is verified by experiments with both synthetic and real data.     

A variational Bayesian strategy for solving the DOA estimation problem in sparse array

This paper reformulates the problem of direction-of-arrival (DOA) estimation for sparse array from a variational Bayesian perspective. In this context, we propose a hierarchical prior for the signal coefficients that amounts marginally to a sparsity-inducing penalty in maximum a posterior (MAP) estimation. Further, the specific hierarchy gives rise to a variational inference technique which operates in latent variable space iteratively. Our hierarchical formulation of the prior allow users to model the sparsity of the unknown signal with a high degree, and the corresponding Bayesian algorithm leads to sparse estimators reflecting posterior information beyond the mode. We provide experimental results with synthetic signals and compare with state-of-the-art DOA estimation algorithm, in order to demonstrate the superior performance of the proposed approach.     

Finite mixture of gamma distributions: A conjugate prior

A finite mixture of gamma distributions [Finite mixture of certain distributions. Comm. Statist. Theory Methods 31(12), 2123-2137] is used as a conjugate prior, which gives a nice form of posterior distribution. This class of conjugate priors offers a more flexible class of priors than the class of gamma prior distributions. The usefulness of a mixture gamma-type prior and the posterior of uncertain parameters λ for the Poisson distribution are illustrated by using Markov Chain Monte Carlo (MCMC), Gibbs sampling approach, on hierarchical models. Using the generalized hypergeometric function, the method to approximate maximum likelihood estimators for the parameters of Agarwal and Al-Saleh [Generalized gamma type distribution and its hazard rate function. Comm. Statist. Theory Methods 30(2), 309-318] generalized gamma-type distribution is also suggested.     

Minimum Mean-Square Error Estimation of Discrete Fourier Coefficients With Generalized Gamma Priors

This paper considers techniques for single-channel speech enhancement based on the discrete Fourier transform (DFT). Specifically, we derive minimum mean-square error (MMSE) estimators of speech DFT coefficient magnitudes as well as of complex-valued DFT coefficients based on two classes of generalized gamma distributions, under an additive Gaussian noise assumption. The resulting generalized DFT magnitude estimator has as a special case the existing scheme based on a Rayleigh speech prior, while the complex DFT estimators generalize existing schemes based on Gaussian, Laplacian, and Gamma speech priors. Extensive simulation experiments with speech signals degraded by various additive noise sources verify that significant improvements are possible with the more recent estimators based on super-Gaussian priors. The increase in perceptual evaluation of speech quality (PESQ) over the noisy signals is about 0.5 points for street noise and about 1 point for white noise, nearly independent of input signal-to-noise ratio (SNR). The assumptions made for deriving the complex DFT estimators are less accurate than those for the magnitude estimators, leading to a higher maximum achievable speech quality with the magnitude estimators.     

Comparison of Bayesian survival analysis and Cox regression analysis in simulated and breast cancer data sets

We aimed to compare the performance of Cox regression analysis (CRA) and Bayesian survival analysis (BSA) by using simulations and breast cancer data.Simulation study was carried out with two different algorithms that were informative and noninformative priors. Moreover, in a real data set application, breast cancer data set related to disease-free survival (DFS) that was obtained from 423 breast cancer patients diagnosed between 1998 and 2007 was used.In the simulation application, it was observed that BSA with noninformative priors and CRA methods showed similar performances in point of convergence to simulation parameter. In the informative priors’ simulation application, BSA with proper informative prior showed a good performance with too little bias. It was found out that the bias of BSA increased while priors were becoming distant from reliability in all sample sizes. In addition, BSA obtained predictions with more little bias and standard error than the CRA in both small and big samples in the light of proper priors.In the breast cancer data set, age, tumor size, hormonal therapy, and axillary nodal status were found statistically significant prognostic factors for DFS in stepwise CRA and BSA with informative and noninformative priors. Furthermore, standard errors of predictions in BSA with informative priors were observed slightly.As a result, BSA showed better performance than CRA, when subjective data analysis was performed by considering expert opinions and historical knowledge about parameters. Consequently, BSA should be preferred in existence of reliable informative priors, in the contrast cases, CRA should be preferred.     

Inference for grouped data with a truncated skew-Laplace distribution

The skew-Laplace distribution has been used for modelling particle size with point observations. In reality, the observations are truncated and grouped (rounded). This must be formally taken into account for accurate modelling, and it is shown how this leads to convenient closed-form expressions for the likelihood in this model. In a Bayesian framework, “noninformative” benchmark priors, which only require the choice of a single scalar prior hyperparameter, are specified. Conditions for the existence of the posterior distribution are derived when rounding and various forms of truncation are considered. The main application focus is on modelling microbiological data obtained with flow cytometry. However, the model is also applied to data often used to illustrate other skewed distributions, and it is shown that our modelling compares favourably with the popular skew-Student models. Further examples with simulated data illustrate the wide applicability of the model.     

Bayesian nonlinear model selection and neural networks: a conjugateprior approach

In order to select the best predictive neural-network architecture in a set of several candidate networks, we propose a general Bayesian nonlinear regression model comparison procedure, based on the maximization of an expected utility criterion. This criterion selects the model under which the training set achieves the highest level of internal consistency, through the predictive probability distribution of each model. The density of this distribution is computed as the model posterior predictive density and is asymptotically approximated from the assumed Gaussian likelihood of the data set and the related conjugate prior density of the parameters. The use of such a conjugate prior allows the analytic calculation of the parameter posterior and predictive posterior densities, in an empirical Bayes-like approach. This Bayesian selection procedure allows us to compare general nonlinear regression models and in particular feedforward neural networks, in addition to embedded models as usual with asymptotic comparison tests.     

Default Bayesian model determination methods for generalised linear mixed models

A default strategy for fully Bayesian model determination for generalised linear mixed models (GLMMs) is considered which addresses the two key issues of default prior specification and computation. In particular, the concept of unit-information priors is extended to the parameters of a GLMM. A combination of Markov chain Monte Carlo (MCMC) and Laplace approximations is used to compute approximations to the posterior model probabilities to find a subset of models with high posterior model probability. Bridge sampling is then used on the models in this subset to approximate the posterior model probabilities more accurately. The strategy is applied to four examples.     

Learning Bayesian Networks: The Combination of Knowledge and Statistical Data

We describe a Bayesian approach for learning Bayesian networks from a combination of prior knowledge and statistical data. First and foremost, we develop a methodology for assessing informative priors needed for learning. Our approach is derived from a set of assumptions made previously as well as the assumption of likelihood equivalence, which says that data should not help to discriminate network structures that represent the same assertions of conditional independence. We show that likelihood equivalence when combined with previously made assumptions implies that the user's priors for network parameters can be encoded in a single Bayesian network for the next case to be seen—a prior network—and a single measure of confidence for that network. Second, using these priors, we show how to compute the relative posterior probabilities of network structures given data. Third, we describe search methods for identifying network structures with high posterior probabilities. We describe polynomial algorithms for finding the highest-scoring network structures in the special case where every node has at most k = 1 parent. For the general case (k > 1), which is NP-hard, we review heuristic search algorithms including local search, iterative local search, and simulated annealing. Finally, we describe a methodology for evaluating Bayesian-network learning algorithms, and apply this approach to a comparison of various approaches.     

Blind source separation with time series variational Bayes expectation maximization algorithm

This paper presents a variational Bayes expectation maximization algorithm for time series based on Attias? variational Bayesian theory. The proposed algorithm is applied in the blind source separation (BSS) problem to estimate both the source signals and the mixing matrix for the optimal model structure. The distribution of the mixing matrix is assumed to be a matrix Gaussian distribution due to the correlation of its elements and the inverse covariance of the sensor noise is assumed to be Wishart distributed for the correlation between sensor noises. The mixture of Gaussian model is used to approximate the distribution of each independent source. The rules to update the posterior hyperparameters and the posterior of the model structure are obtained. The optimal model structure is selected as the one with largest posterior. The source signals and mixing matrix are estimated by applying LMS and MAP estimators to the posterior distributions of the hidden variables and the model parameters respectively for the optimal structure. The proposed algorithm is tested with synthetic data. The results show that: (1) the logarithm posterior of the model structure increases with the accuracy of the posterior mixing matrix; (2) the accuracies of the prior mixing matrix, the estimated mixing matrix, and the estimated source signals increase with the logarithm posterior of the model structure. This algorithm is applied to Magnetoencephalograph data to localize the source of the equivalent current dipoles.