Bayesian estimation of beta mixture models with variational inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation to the prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed form) Bayesian approach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles of the VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate the distribution of the parameters in BMM. In a fully Bayesian model where all of the parameters of the BMM are considered as variables and assigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also, the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numerical calculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximization algorithm. The good performance of this approach is verified by experiments with both synthetic and real data.     

融合专家经验和寿命试验数据的产品可靠性评估

专家经验是可靠性工程中常见的一类可靠性数据,通过将其与产品的寿命试验数据融合,可以扩充可靠性信息,为产品可靠性的评估提供新的思路.对此,利用Bayes理论,考虑不同类型和不同形式的专家经验,通过验前矩拟合的方法将其转化为产品寿命分布参数的验前分布.进一步,根据寿命试验数据确定似然函数,推断分布参数的验后分布,可求得数据融合后产品的可靠度和剩余寿命等可靠性评估结果.通过蓄电池算例分析,表明所提出方法的应用及其有效性.     

Bayesian mixture of AR models for time series clustering

In this paper, we propose a Bayesian framework for estimation of parameters of a mixture of autoregressive models for time series clustering. The proposed approach is based on variational principles and provides a tractable approximation to the true posterior density that minimizes Kullback–Liebler (KL) divergence with respect to prior distribution. This method simultaneously addresses the model complexity and parameter estimation problems. The proposed approach is applied both on simulated and real-world time series datasets. It is found to be useful in exploring and finding the true number of underlying clusters, starting from an arbitrarily large number of clusters.     

A Bayesian analysis of moving average processes with time-varying parameters

A new Bayesian method is proposed for estimation and forecasting with Gaussian moving average (MA) processes with time-varying parameters. The focus is placed on MA models of order one, but a general result is given for an MA process of an arbitrary known order. A multiplicative model for the evolution of the squares of the parameters is introduced following Bayesian conjugacy through beta and truncated gamma distributions and a discount factor. Two new distributions are proposed providing the prior and posterior distributions of the parameters of the model and the one-step forecast distribution of the process. Several well-known distributional results are extended by replacing the gamma distribution with the truncated gamma distribution. The proposed methodology is illustrated with two examples consisting of simulated data and of aluminium spot prices of the London metal exchange.     

Estimating spiking irregularities under changing environments

We considered a gamma distribution of interspike intervals as a statistical model for neuronal spike generation. A gamma distribution is a natural extension of the Poisson process taking the effect of a refractory period into account. The model is specified by two parameters: a time-dependent firing rate and a shape parameter that characterizes spiking irregularities of individual neurons. Because the environment changes over time, observed data are generated from a model with a time-dependent firing rate, which is an unknown function. A statistical model with an unknown function is called a semiparametric model and is generally very difficult to solve. We used a novel method of estimating functions in information geometry to estimate the shape parameter without estimating the unknown function. We obtained an optimal estimating function analytically for the shape parameter independent of the functional form of the firing rate. This estimation is efficient without Fisher information loss and better than maximum likelihood estimation. We suggest a measure of spiking irregularity based on the estimating function, which may be useful for characterizing individual neurons in changing environments.     

QUADRATIC TSALLIS ENTROPY BIAS AND GENERALIZED MAXIMUM ENTROPY MODELS

In density estimation task, Maximum Entropy (Maxent) model can effectively use reliable prior information via nonparametric constraints, that is, linear constraints without empirical parameters. However, reliable prior information is often insufficient, and parametric constraints becomes necessary but poses considerable implementation complexity. Improper setting of parametric constraints can result in overfitting or underfitting. To alleviate this problem, a generalization of Maxent, under Tsallis entropy framework, is proposed. The proposed method introduces a convex quadratic constraint for the correction of (expected) quadratic Tsallis Entropy Bias (TEB). Specifically, we demonstrate that the expected quadratic Tsallis entropy of sampling distributions is smaller than that of the underlying real distribution with regard to frequentist, Bayesian prior, and Bayesian posterior framework, respectively. This expected entropy reduction is exactly the (expected) TEB, which can be expressed by the closed‐form formula and acts as a consistent and unbiased correction with an appropriate convergence rate. TEB indicates that the entropy of a specific sampling distribution should be increased accordingly. This entails a quantitative reinterpretation of the Maxent principle. By compensating TEB and meanwhile forcing the resulting distribution to be close to the sampling distribution, our generalized quadratic Tsallis Entropy Bias Compensation (TEBC) Maxent can be expected to alleviate the overfitting and underfitting. We also present a connection between TEB and Lidstone estimator. As a result, TEB–Lidstone estimator is developed by analytically identifying the rate of probability correction in Lidstone. Extensive empirical evaluation shows promising performance of both TEBC Maxent and TEB‐Lidstone in comparison with various state‐of‐the‐art density estimation methods.     

Estimation of the parameters in regularized simultaneous super-resolution

We describe a method for automatic determination of the regularization parameters for the class of simultaneous super-resolution (SR) algorithms. This method, proposed in (Zibetti et al., 2008c), is based on the joint maximum a posteriori (JMAP) estimation technique, which is a fast alternative to estimate the parameters. However, the classical JMAP technique can be unstable and may generate multiple local minima. In order to stabilize the JMAP estimation, while achieving a cost function with a unique global solution, we derive an improved solution by modeling the JMAP hyperparameters with a gamma prior distribution. In this work, experimental results are provided to illustrate the effectiveness of the proposed method for automatic determination of the regularization parameters for the simultaneous SR. Moreover, we contrast the proposed method to a reference method with known fixed parameters as well as to other parameter selection methods based on the L-curve. These results validate the proposed method as a very attractive alternative for estimating the regularization parameters.     

Reversible jump MCMC for nonparametric drift estimation for diffusion processes

In the context of nonparametric Bayesian estimation a Markov chain Monte Carlo algorithm is devised and implemented to sample from the posterior distribution of the drift function of a continuously or discretely observed one-dimensional diffusion. The drift is modeled by a scaled linear combination of basis functions with a Gaussian prior on the coefficients. The scaling parameter is equipped with a partially conjugate prior. The number of basis functions in the drift is equipped with a prior distribution as well. For continuous data, a reversible jump Markov chain algorithm enables the exploration of the posterior over models of varying dimension. Subsequently, it is explained how data-augmentation can be used to extend the algorithm to deal with diffusions observed discretely in time. Some examples illustrate that the method can give satisfactory results. In these examples a comparison is made with another existing method as well.     

A non-stationary image prior combination in super-resolution

A new Bayesian Super-Resolution (SR) image registration and reconstruction method is proposed. The new method utilizes a prior distribution based on a general combination of spatially adaptive, or non-stationary, image filters, which includes an adaptive local strength parameter able to preserve both image edges and textures. With the application of variational techniques, the proposed method allows for the automatic estimation of all problem unknowns. An experimental comparison between state of the art methods and the proposed SR approach has been performed on both synthetic and real images.     

Bayesian R-estimates in two-sample location models

Implementation of nonparametric rank-based procedures in the Bayesian context is discussed primarily for the two-sample location models. The information in the data is summarized via the (possibly asymptotic) distribution of some rank-based quantity, which is used as a pseudo-likelihood. The complete posterior distribution (or the posterior distribution up to a normalizing constant) of the parameter of interest given the rank-based quantity can be obtained by assuming a prior distribution for the parameter. Statistical inference then proceeds based on this posterior distribution. Posterior estimation and testing are considered, along with a discussion of a normal approximation to the posterior distribution.     

Particle filtering with analytically guided sampling

Particle filtering (PF) is a popular nonlinear estimation technique and has been widely used in a variety of applications such as target tracking. Within the PF framework, one critical design choice is the selection of the proposal distribution from which particles are drawn. In this paper, we advocate using as proposal distribution a Gaussian-mixture-based approximation of the posterior probability density function (pdf) after taking into account the most recent measurement. The novelty of our approach is that the parameters of each Gaussian used in the mixture are determined analytically to match the modes of the underlying unknown posterior pdf. As a result, particles are sampled along the most probable regions of the state space, hence reducing the probability of particle depletion. Based on the analytically determined proposal distribution, we introduce a novel PF, termed analytically guided sampling-based PF, which is validated in range-only and bearing-only target tracking.     

小数据集条件下基于双重约束的BN参数学习

针对小数据集条件下的贝叶斯网络（Bayesian network,BN）参数学习问题,提出了一种基于双重约束的贝叶斯网络参数学习方法. 首先,对网络中的参数进行分析并将网络中的参数划分为: 父节点组合状态相同而子节点状态不同的参数和父节点组合状态不同而子节点状态相同的参数;然后,针对第一类参数提出了一种新的基于Beta分布拟合的贝叶斯估计方法,而针对第二类参数利用已有的保序回归估计方法进行学习,进而实现了对网络中参数的双重约束学习;最后,通过仿真实例说明了基于双重约束的参数学习方法对小数据集条件下贝叶斯网络参数学习精度提高的有效性.     

高斯渐进贝叶斯滤波器

渐进贝叶斯方法将先验分布到后验分布的演化描述为一阶动态系统,通过在伪时间上连续地引入观测信息实现后验状态估计.该方法的一般形式解,即动态系统的时间导数,是难以得到的.本文提出一种高斯型渐进贝叶斯滤波器.首先在线性高斯条件下推导了时间导数的解析解;然后证明了在该条件下,由该解析解确定的一阶动态系统与常量状态估计的Kalman-Bucy滤波器是一致的,且由此导出的高斯渐进贝叶斯滤波器与卡尔曼滤波器是一致的.最后利用一阶Taylor展开推导了滤波器在非线性高斯条件下的近似解表达式,并采用Monte Carlo方法给出了具体实现方法.通过若干仿真算例表明,新滤波器具有较高的精度,且在一定精度条件下的时间复杂度低于一般粒子滤波器.     

Efficient parameter learning of Bayesian network classifiers

Recent advances have demonstrated substantial benefits from learning with both generative and discriminative parameters. On the one hand, generative approaches address the estimation of the parameters of the joint distribution—\(\mathrm{P}(y,\mathbf{x})\), which for most network types is very computationally efficient (a notable exception to this are Markov networks) and on the other hand, discriminative approaches address the estimation of the parameters of the posterior distribution—and, are more effective for classification, since they fit \(\mathrm{P}(y|\mathbf{x})\) directly. However, discriminative approaches are less computationally efficient as the normalization factor in the conditional log-likelihood precludes the derivation of closed-form estimation of parameters. This paper introduces a new discriminative parameter learning method for Bayesian network classifiers that combines in an elegant fashion parameters learned using both generative and discriminative methods. The proposed method is discriminative in nature, but uses estimates of generative probabilities to speed-up the optimization process. A second contribution is to propose a simple framework to characterize the parameter learning task for Bayesian network classifiers. We conduct an extensive set of experiments on 72 standard datasets and demonstrate that our proposed discriminative parameterization provides an efficient alternative to other state-of-the-art parameterizations.     

基于高斯混合聚类的切换系统的辨识

针对具有未知切换规则与未知子系统数量的切换系统的辨识问题,提出一种两阶段辨识方法,包括模式检测与参数辨识.在模式检测阶段,首先建立高斯混合模型表示采样数据的分布,并通过轮盘法选择合适的初始模型参数.其次,计算采样数据属于每个子系统的后验概率,通过极大似然估计算法迭代更新模型参数,使高斯混合模型最大化地拟合采样数据的分布.在此基础上,通过贝叶斯信息准则确定子系统的数量,并根据最大后验概率准则估计切换规则.在参数辨识阶段,通过递推增广最小二乘法估计每个子系统的参数向量.最后,通过仿真结果验证了所提方法的有效性.     

Bayesian estimation and variable selection for single index models

We develop a fully Bayesian method to analyze the single index models, including variable selection, the index vector estimation and the link function fitting with free-knot splines. The proposed method is implemented by means of the reversible jump Markov chain Monte Carlo technique. We treat the marginal posterior of all the unknown quantities except the spline coefficients and error variance as the target distribution to reduce the dimension of the parameters and to obtain a rapid algorithm. We design a new random walk Metropolis sampler to sample from the conditional posterior distribution of the index vector. The proposed method is verified by simulation studies, and is applied to analyze two real data sets.     

Predictive performance of Dirichlet process shrinkage methods in linear regression

An obvious Bayesian nonparametric generalization of ridge regression assumes that coefficients are exchangeable, from a prior distribution of unknown form, which is given a Dirichlet process prior with a normal base measure. The purpose of this paper is to explore predictive performance of this generalization, which does not seem to have received any detailed attention, despite related applications of the Dirichlet process for shrinkage estimation in multivariate normal means, analysis of randomized block experiments and nonparametric extensions of random effects models in longitudinal data analysis. We consider issues of prior specification and computation, as well as applications in penalized spline smoothing. With a normal base measure in the Dirichlet process and letting the precision parameter approach infinity the procedure is equivalent to ridge regression, whereas for finite values of the precision parameter the discreteness of the Dirichlet process means that some predictors can be estimated as having the same coefficient. Estimating the precision parameter from the data gives a flexible method for shrinkage estimation of mean parameters which can work well when ridge regression does, but also adapts well to sparse situations. We compare our approach with ridge regression, the lasso and the recently proposed elastic net in simulation studies and also consider applications to penalized spline smoothing.     

Inference on exponential families with mixture of prior distributions

A Bayesian analysis of the natural exponential families with quadratic variance function when there are several sources of prior information is considered. The belief of each source is expressed as a conjugate prior distribution. Then, a mixture of them is considered to represent a consensus of the sources. A unified framework considering unknown weights is presented. Firstly, a general procedure based on Kullback-Leibler (K-L) distance to obtain the weights is proposed. The main advantage is that the weights can be analytically calculated. In addition, expressions that allow a direct implementation for these families are shown. Secondly, the experts’ prior beliefs are calibrated with respect to the combined posterior belief by using K-L distances. A straightforward Monte Carlo-based approach to estimate these distances is proposed. Finally, two illustrative examples are presented to show the ease of application of the proposed technique, as well as its usefulness in a Bayesian framework.     

Objective Bayesian analysis of accelerated competing failure models under Type-I censoring

This paper discusses the Bayesian inference of accelerated life tests (ALT) in the presence of competing failure causes. The time to failure due to a specific cause is described by a Weibull distribution. A two-stage approach is utilized to obtain the estimates of parameters in the model. We use the Bayesian method to estimate the parameters of the distribution of component lifetimes in the first stage, in which two noninformative priors (Jeffreys prior and reference prior) are derived in the case of ALT, and based on these two priors we present the Gibbs sampling procedures to obtain the posterior estimates of the parameters. Besides, to overcome the problem of improper posterior densities under some conditions, we modify the likelihood function to make the posterior densities proper. In the second stage, parameters in the accelerating function are obtained by least squares approach. A numerical example is given to show the effectiveness of the method and a real data from Nelson (1990) is analyzed.     

Bayesian estimation for homogeneous and inhomogeneous Gaussianrandom fields

This paper investigates Bayesian estimation for Gaussian Markov random fields. In particular, a new class of compound model is proposed which describes the observed intensities using an inhomogeneous model and the degree of spatial variation described by a second random field. The coupled Markov random fields are used as prior distributions, and combined with Gaussian noise models to produce posterior distributions on which estimation is based. All model parameters are estimated, in a fully Bayesian setting, using the Metropolis-Hasting algorithm. The full posterior estimation procedures are illustrated and compared using various artificial examples. For these examples the inhomogeneous model performs very favorably when compared to the homogeneous model, allowing differential degrees of smoothing and varying local textures