Predictive performance of Dirichlet process shrinkage methods in linear regression

An obvious Bayesian nonparametric generalization of ridge regression assumes that coefficients are exchangeable, from a prior distribution of unknown form, which is given a Dirichlet process prior with a normal base measure. The purpose of this paper is to explore predictive performance of this generalization, which does not seem to have received any detailed attention, despite related applications of the Dirichlet process for shrinkage estimation in multivariate normal means, analysis of randomized block experiments and nonparametric extensions of random effects models in longitudinal data analysis. We consider issues of prior specification and computation, as well as applications in penalized spline smoothing. With a normal base measure in the Dirichlet process and letting the precision parameter approach infinity the procedure is equivalent to ridge regression, whereas for finite values of the precision parameter the discreteness of the Dirichlet process means that some predictors can be estimated as having the same coefficient. Estimating the precision parameter from the data gives a flexible method for shrinkage estimation of mean parameters which can work well when ridge regression does, but also adapts well to sparse situations. We compare our approach with ridge regression, the lasso and the recently proposed elastic net in simulation studies and also consider applications to penalized spline smoothing.     

Least-informative Bayesian prior distributions for finite samplesbased on information theory

A procedure is presented, based on Shannon information theory, for producing least-informative prior distributions for Bayesian estimation and identification. This approach relies on constructing an optimal mixture distribution and applies in small sample sizes (unlike certain approaches based on asymptotic theory). The procedure is illustrated in a small-scale numerical study and is contrasted with an approach based on maximum entropy     

A mathematical formulation of uncertain information

This paper introduces a mathematical model of uncertain information. Each body of uncertain information is an information quadruplet, consisting of a code space, a message space, an interpretation function, and an evidence space. Each information quadruplet contains prior information as well as possible new evidence which may appear later. The definitions of basic probability and belief function are based on the prior information. Given new evidence, Bayes' rule is used to update the prior information. This paper also introduces an idea of independent information and its combination. A combination formula is derived for combining independent information. Both the conventional Bayesian approach and Dempster-Shafer's approach belong to this mathematical model. A Bayesian prior probability measure is the prior information of a special information quadruplet; Bayesian conditioning is the combination of special independent information. A Dempster's belief function is the belief function of a different information quadruplet; the Dempster combination rule is the combination rule of independent quadruplets. This paper is a mathematical study of handling uncertainty and shows that both the conventional Bayesian approach and Dempster-Shafer's approach originate from the same mathematical theory.This work was supported in part by the National Science Foundation under grant number IRI-8505735 and a summer research grant of Ball State University.     

Bayesian estimation of beta mixture models with variational inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation to the prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed form) Bayesian approach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles of the VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate the distribution of the parameters in BMM. In a fully Bayesian model where all of the parameters of the BMM are considered as variables and assigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also, the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numerical calculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximization algorithm. The good performance of this approach is verified by experiments with both synthetic and real data.     

Bayesian mixture of AR models for time series clustering

In this paper, we propose a Bayesian framework for estimation of parameters of a mixture of autoregressive models for time series clustering. The proposed approach is based on variational principles and provides a tractable approximation to the true posterior density that minimizes Kullback–Liebler (KL) divergence with respect to prior distribution. This method simultaneously addresses the model complexity and parameter estimation problems. The proposed approach is applied both on simulated and real-world time series datasets. It is found to be useful in exploring and finding the true number of underlying clusters, starting from an arbitrarily large number of clusters.     

A note on bimodality in the log-likelihood function for penalized spline mixed models

For a smoothing spline or general penalized spline model, the smoothing parameter can be estimated using residual maximum likelihood (REML) methods by expressing the spline in the form of a mixed model. The possibility of bimodality in the profile log-likelihood function for the smoothing parameter of these penalized spline mixed models is demonstrated. A canonical transformation into independent observations is used to provide efficient evaluation of the log-likelihood function and gives insight into the incompatibilities between the model and data that cause bimodality. This transformation can also be used to assess the influence of different frequency components in the data on the estimated smoothing parameter. It is demonstrated that, where bimodality occurs in the log-likelihood, Bayesian penalized spline models may show poor mixing in MCMC chains and be sensitive to the choice of prior distributions for variance components.     

Fuzzy local linearization and local basis function expansion innonlinear system modeling

Fuzzy local linearization is compared with local basis function expansion for modeling unknown nonlinear processes. First-order Takagi-Sugeno fuzzy model and the analysis of variance (ANOVA) decomposition are combined for the fuzzy local linearization of nonlinear systems, in which B-splines are used as membership functions of the fuzzy sets for input space partition. A modified algorithm for adaptive spline modeling of observation data (MASMOD) is developed for determining the number of necessary B-splines and their knot positions to achieve parsimonious models. This paper illustrates that fuzzy local linearization models have several advantages over local basis function expansion based models in nonlinear system modeling.     

Random effects in promotion time cure rate models

In this paper, a survival model with long-term survivors and random effects, based on the promotion time cure rate model formulation for models with a surviving fraction is investigated. We present Bayesian and classical estimation approaches. The Bayesian approach is implemented using a Markov chain Monte Carlo (MCMC) based on the Metropolis-Hastings algorithms. For the second one, we use restricted maximum likelihood (REML) estimators. A simulation study is performed to evaluate the accuracy of the applied techniques for the estimates and their standard deviations. An example on an oropharynx cancer study is used to illustrate the model and the estimation approaches considered in the study.     

On the estimation of transfer functions,regularizations and Gaussian processes—Revisited

Intrigued by some recent results on impulse response estimation by kernel and nonparametric techniques, we revisit the old problem of transfer function estimation from input–output measurements. We formulate a classical regularization approach, focused on finite impulse response (FIR) models, and find that regularization is necessary to cope with the high variance problem. This basic, regularized least squares approach is then a focal point for interpreting other techniques, like Bayesian inference and Gaussian process regression. The main issue is how to determine a suitable regularization matrix (Bayesian prior or kernel). Several regularization matrices are provided and numerically evaluated on a data bank of test systems and data sets. Our findings based on the data bank are as follows. The classical regularization approach with carefully chosen regularization matrices shows slightly better accuracy and clearly better robustness in estimating the impulse response than the standard approach–the prediction error method/maximum likelihood (PEM/ML) approach. If the goal is to estimate a model of given order as well as possible, a low order model is often better estimated by the PEM/ML approach, and a higher order model is often better estimated by model reduction on a high order regularized FIR model estimated with careful regularization. Moreover, an optimal regularization matrix that minimizes the mean square error matrix is derived and studied. The importance of this result lies in that it gives the theoretical upper bound on the accuracy that can be achieved for this classical regularization approach.     

Document Ink Bleed-Through Removal with Two Hidden Markov Random Fields and a Single Observation Field

We present a new method for blind document bleed-through removal based on separate Markov Random Field (MRF) regularization for the recto and for the verso side, where separate priors are derived from the full graph. The segmentation algorithm is based on Bayesian Maximum a Posteriori (MAP) estimation. The advantages of this separate approach are the adaptation of the prior to the contents creation process (e.g., superimposing two handwritten pages), and the improvement of the estimation of the recto pixels through an estimation of the verso pixels covered by recto pixels; moreover, the formulation as a binary labeling problem with two hidden labels per pixels naturally leads to an efficient optimization method based on the minimum cut/maximum flow in a graph. The proposed method is evaluated on scanned document images from the 18th century, showing an improvement of character recognition results compared to other restoration methods.     

Bayesian estimation based on vague lifetime data

In Classical Bayesian approach, estimation of lifetime data usually is dealing with precise information. However, in real world, some informations about an underlying system might be imprecise and represented in the form of vague quantities. In these situations, we need to generalize classical methods to vague environment for studying and analyzing the systems of interest. In this paper, we propose the Bayesian estimation of failure rate and mean time to failure based on vague set theory in the case of complete and censored data sets. To employ the Bayesian approach, model parameters are assumed to be vague random variables with vague prior distributions. This approach will be used to induce the vague Bayes estimate of failure rate and mean time to failure by introducing and applying a theorem called “Resolution Identity” for vague sets. In order to evaluate the membership degrees of vague Bayesian estimate for these quantities, a computational procedure is investigated. In the proposed method, the original problem is transformed into a nonlinear programming problem which is then divided into eight subproblems to simplifying computations.     

Determination of first and second derivatives of progress curves in the case of unknown experimental error

The first and second derivatives of progress curves are obtained from the cubic spline function. The new approach is based on a development of the splining quality test which was used for estimating the precision of the splining. The proposed method is used on a computer with a FORTRAN 77 program. The method may also be applied for an approximate estimation of experimental error.     

Bayesian estimation of Dirichlet mixture model with variational inference

In statistical modeling, parameter estimation is an essential and challengeable task. Estimation of the parameters in the Dirichlet mixture model (DMM) is analytically intractable, due to the integral expressions of the gamma function and its corresponding derivatives. We introduce a Bayesian estimation strategy to estimate the posterior distribution of the parameters in DMM. By assuming the gamma distribution as the prior to each parameter, we approximate both the prior and the posterior distribution of the parameters with a product of several mutually independent gamma distributions. The extended factorized approximation method is applied to introduce a single lower-bound to the variational objective function and an analytically tractable estimation solution is derived. Moreover, there is only one function that is maximized during iterations and, therefore, the convergence of the proposed algorithm is theoretically guaranteed. With synthesized data, the proposed method shows the advantages over the EM-based method and the previously proposed Bayesian estimation method. With two important multimedia signal processing applications, the good performance of the proposed Bayesian estimation method is demonstrated.     

Semiparametric regression for assessing agreement using tolerance bands

This article describes a Bayesian semiparametric approach for assessing agreement between two methods for measuring a continuous variable using tolerance bands. A tolerance band quantifies the extent of agreement in methods as a function of a covariate by estimating the range of their differences in a specified large proportion of population. The mean function of differences is modelled using a penalized spline through its mixed model representation. The covariance matrix of the errors may also depend on a covariate. The Bayesian approach is straightforward to implement using the Markov chain Monte Carlo methodology. It provides an alternative to the rather ad hoc frequentist likelihood-based approaches that do not work well in general. Simulation for two commonly used models and their special cases suggests that the proposed Bayesian method has reasonably good frequentist coverage. Two real data sets are used for illustration, and the Bayesian and the frequentist inferences are compared.     

Bayesian nonparametric quantile regression using splines

A new technique based on Bayesian quantile regression that models the dependence of a quantile of one variable on the values of another using a natural cubic spline is presented. Inference is based on the posterior density of the spline and an associated smoothing parameter and is performed by means of a Markov chain Monte Carlo algorithm. Examples of the application of the new technique to two real environmental data sets and to simulated data for which polynomial modelling is inappropriate are given. An aid for making a good choice of proposal density in the Metropolis-Hastings algorithm is discussed. The new nonparametric methodology provides more flexible modelling than the currently used Bayesian parametric quantile regression approach.     

A unified wavelet-based modelling framework for non-linear system identification: the WANARX model structure

A new unified modelling framework based on the superposition of additive submodels, functional components, and wavelet decompositions is proposed for non-linear system identification. A non-linear model, which is often represented using a multivariate non-linear function, is initially decomposed into a number of functional components via the well-known analysis of variance (ANOVA) expression, which can be viewed as a special form of the NARX (non-linear autoregressive with exogenous inputs) model for representing dynamic input–output systems. By expanding each functional component using wavelet decompositions including the regular lattice frame decomposition, wavelet series and multiresolution wavelet decompositions, the multivariate non-linear model can then be converted into a linear-in-the-parameters problem, which can be solved using least-squares type methods. An efficient model structure determination approach based upon a forward orthogonal least squares (OLS) algorithm, which involves a stepwise orthogonalization of the regressors and a forward selection of the relevant model terms based on the error reduction ratio (ERR), is employed to solve the linear-in-the-parameters problem in the present study. The new modelling structure is referred to as a wavelet-based ANOVA decomposition of the NARX model or simply WANARX model, and can be applied to represent high-order and high dimensional non-linear systems.     

Averaging, maximum penalized likelihood and Bayesian estimation forimproving Gaussian mixture probability density estimates

We apply the idea of averaging ensembles of estimators to probability density estimation. In particular, we use Gaussian mixture models which are important components in many neural-network applications. We investigate the performance of averaging using three data sets. For comparison, we employ two traditional regularization approaches, i.e., a maximum penalized likelihood approach and a Bayesian approach. In the maximum penalized likelihood approach we use penalty functions derived from conjugate Bayesian priors such that an expectation maximization (EM) algorithm can be used for training. In all experiments, the maximum penalized likelihood approach and averaging improved performance considerably if compared to a maximum likelihood approach. In two of the experiments, the maximum penalized likelihood approach outperformed averaging. In one experiment averaging was clearly superior. Our conclusion is that maximum penalized likelihood gives good results if the penalty term in the cost function is appropriate for the particular problem. If this is not the case, averaging is superior since it shows greater robustness by not relying on any particular prior assumption. The Bayesian approach worked very well on a low-dimensional toy problem but failed to give good performance in higher dimensional problems.     

Bayesian Model Learning Based on Predictive Entropy

Bayesian paradigm has been widely acknowledged as a coherent approach to learning putative probability model structures from a finite class of candidate models. Bayesian learning is based on measuring the predictive ability of a model in terms of the corresponding marginal data distribution, which equals the expectation of the likelihood with respect to a prior distribution for model parameters. The main controversy related to this learning method stems from the necessity of specifying proper prior distributions for all unknown parameters of a model, which ensures a complete determination of the marginal data distribution. Even for commonly used models, subjective priors may be difficult to specify precisely, and therefore, several automated learning procedures have been suggested in the literature. Here we introduce a novel Bayesian learning method based on the predictive entropy of a probability model, that can combine both subjective and objective probabilistic assessment of uncertain quantities in putative models. It is shown that our approach can avoid some of the limitations of the earlier suggested objective Bayesian methods.