Variational EM method for blur estimation using the spike-and-slab image prior

Most state-of-the-art blind image deconvolution methods rely on the Bayesian paradigm to model the deblurring problem and estimate both the blur kernel and latent image. It is customary to model the image in the filter space, where it is supposed to be sparse, and utilize convenient priors to account for this sparsity. In this paper, we propose the use of the spike-and-slab prior together with an efficient variational Expectation Maximization (EM) inference scheme to estimate the blur in the image. The spike-and-slab prior, which constitutes the gold standard in sparse machine learning, selectively shrinks irrelevant variables while mildly regularizing the relevant ones. The proposed variational Expectation Maximization algorithm is more efficient than usual Markov Chain Monte Carlo (MCMC) inference and, also, proves to be more accurate than the standard mean-field variational approximation. Additionally, all the prior model parameters are estimated by the proposed scheme. After blur estimation, a non-blind restoration method is used to obtain the actual estimation of the sharp image. We investigate the behavior of the prior in the experimental section together with a series of experiments with synthetically generated and real blurred images that validate the method's performance in comparison with state-of-the-art blind deconvolution techniques.     

A Survey of Model Reduction by Balanced Truncation and Some New Results

Balanced truncation is one of the most common model reduction schemes. In this note, we present a survey of balancing related model reduction methods and their corresponding error norms, and also introduce some new results. Five balancing methods are studied: (1) Lyapunov balancing, (2) stochastic balancing, (3) bounded real balancing, (4) positive real balancing and (5) frequency weighted balancing. For positive real balancing, we introduce a multiplicative-type error bound. Moreover, for a certain subclass of positive real systems, a modified positive-real balancing scheme with an absolute error bound is proposed. We also develop a new frequency-weighted balanced reduction method with a simple bound on the error system based on the frequency domain representations of the system gramians. Two numerical examples are illustrated to verify the efficiency of the proposed methods.     

A Spatially Constrained Generative Model and an EM Algorithm for Image Segmentation

In this paper, we present a novel spatially constrained generative model and an expectation-maximization (EM) algorithm for model-based image segmentation. The generative model assumes that the unobserved class labels of neighboring pixels in the image are generated by prior distributions with similar parameters, where similarity is defined by entropic quantities relating to the neighboring priors. In order to estimate model parameters from observations, we derive a spatially constrained EM algorithm that iteratively maximizes a lower bound on the data log-likelihood, where the penalty term is data-dependent. Our algorithm is very easy to implement and is similar to the standard EM algorithm for Gaussian mixtures with the main difference that the labels posteriors are "smoothed" over pixels between each E- and M-step by a standard image filter. Experiments on synthetic and real images show that our algorithm achieves competitive segmentation results compared to other Markov-based methods, and is in general faster     

深度生成模型综述

通过学习可观测数据的概率密度而随机生成样本的生成模型在近年来受到人们的广泛关注,网络结构中包含多个隐藏层的深度生成式模型以更出色的生成能力成为研究热点,深度生成模型在计算机视觉、密度估计、自然语言和语音识别、半监督学习等领域得到成功应用,并给无监督学习提供了良好的范式.本文根据深度生成模型处理似然函数的不同方法将模型分...     

Accuracy Analysis of the Frisch Scheme for Identifying Errors-in-Variables Systems

Several estimation methods have been proposed for identifying errors-in-variables systems, where both input and output measurements are corrupted by noise. One of the promising approaches is the so-called Frisch scheme. This paper provides an accuracy analysis of the Frisch scheme applied to system identification. The estimates of the system parameters and the noise variances are shown to be asymptotically Gaussian distributed. An explicit expression for the covariance matrix of the asymptotic distribution is given as well. Numerical simulations support the theoretical results. A comparison with the Cramer-Rao lower bound is also given in the examples, and it is shown that the Frisch scheme gives a performance close to the Cramer-Rao bound for large signal-to-noise ratios (SNRs).     

An alternative view of variational Bayes and asymptotic approximations of free energy

Bayesian learning, widely used in many applied data-modeling problems, is often accomplished with approximation schemes because it requires intractable computation of the posterior distributions. In this study, we focus on two approximation methods, variational Bayes and local variational approximation. We show that the variational Bayes approach for statistical models with latent variables can be viewed as a special case of local variational approximation, where the log-sum-exp function is used to form the lower bound of the log-likelihood. The minimum variational free energy, the objective function of variational Bayes, is analyzed and related to the asymptotic theory of Bayesian learning. This analysis additionally implies a relationship between the generalization performance of the variational Bayes approach and the minimum variational free energy.     

Variational approximations in Bayesian model selection for finite mixture distributions

Variational methods, which have become popular in the neural computing/machine learning literature, are applied to the Bayesian analysis of mixtures of Gaussian distributions. It is also shown how the deviance information criterion, (DIC), can be extended to these types of model by exploiting the use of variational approximations. The use of variational methods for model selection and the calculation of a DIC are illustrated with real and simulated data. The variational approach allows the simultaneous estimation of the component parameters and the model complexity. It is found that initial selection of a large number of components results in superfluous components being eliminated as the method converges to a solution. This corresponds to an automatic choice of model complexity. The appropriateness of this is reflected in the DIC values.     

Bayesian estimation of beta mixture models with variational inference

Bayesian estimation of the parameters in beta mixture models (BMM) is analytically intractable. The numerical solutions to simulate the posterior distribution are available, but incur high computational cost. In this paper, we introduce an approximation to the prior/posterior distribution of the parameters in the beta distribution and propose an analytically tractable (closed form) Bayesian approach to the parameter estimation. The approach is based on the variational inference (VI) framework. Following the principles of the VI framework and utilizing the relative convexity bound, the extended factorized approximation method is applied to approximate the distribution of the parameters in BMM. In a fully Bayesian model where all of the parameters of the BMM are considered as variables and assigned proper distributions, our approach can asymptotically find the optimal estimate of the parameters posterior distribution. Also, the model complexity can be determined based on the data. The closed-form solution is proposed so that no iterative numerical calculation is required. Meanwhile, our approach avoids the drawback of overfitting in the conventional expectation maximization algorithm. The good performance of this approach is verified by experiments with both synthetic and real data.     

Online variational inference for state-space models with point-process observations

We present a variational Bayesian (VB) approach for the state and parameter inference of a state-space model with point-process observations, a physiologically plausible model for signal processing of spike data. We also give the derivation of a variational smoother, as well as an efficient online filtering algorithm, which can also be used to track changes in physiological parameters. The methods are assessed on simulated data, and results are compared to expectation-maximization, as well as Monte Carlo estimation techniques, in order to evaluate the accuracy of the proposed approach. The VB filter is further assessed on a data set of taste-response neural cells, showing that the proposed approach can effectively capture dynamical changes in neural responses in real time.     

On inequalities between mutual information and variation

We continue studying the relationship between mutual information and variational distance started in Pinsker’s paper [1], where an upper bound for the mutual information via variational distance was obtained. We present a simple lower bound, which in some cases is optimal or asymptotically optimal. A uniform upper bound for the mutual information via variational distance is also derived for random variables with a finite number of values. For such random variables, the asymptotic behaviour of the maximum of mutual information is also investigated in the cases where the variational distance tends either to zero or to its maximum value.     

Asymptotically efficient allocation rules for the multiarmed bandit problem with multiple plays-Part II: Markovian rewards

At each instant of time we are required to sample a fixed numberm geq 1out ofNMarkov chains whose stationary transition probability matrices belong to a family suitably parameterized by a real numbertheta. The objective is to maximize the long run expected value of the samples. The learning loss of a sampling scheme corresponding to a parameters configurationC = (theta_{1}, ..., theta_{N})is quantified by the regretR_{n}(C). This is the difference between the maximum expected reward that could be achieved ifCwere known and the expected reward actually achieved. We provide a lower bound for the regret associated with any uniformly good scheme, and construct a sampling scheme which attains the lower bound for everyC. The lower bound is given explicitly in terms of the Kullback-Liebler number between pairs of transition probabilities.     

基于随机变分推理贝叶斯神经网络的发电机轴承故障诊断

近年来,许多基于深度学习的方法被用于故障诊断领域,并且取得了良好的效果,但是发电机故障样本数据难以获取,在数据量较少的情况下,基于深度学习的方法存在过拟合现象,导致模型泛化能力差、诊断精度不高.为了解决这一问题,提出一种基于随机变分推理贝叶斯神经网络的故障诊断方法.该方法以贝叶斯推理与随机变分推理为基础,可以根据少量数据得到较为可靠的模型,获得网络各层参数的概率分布,有效解决过拟合的问题.采用证据下限(evidence lower bound, ELBO)派生类函数TraceGraph ELBO进行随机变分推理,解决派生类函数Trace ELBO诊断精度较低的问题.将所提方法应用于发电机轴承的故障诊断,并与其他方法对比,结果表明,所提方法在故障样本数据量较少的情况下具有较高的诊断性能.     

Variational Bayesian methods for spatial data analysis

With scientific data available at geocoded locations, investigators are increasingly turning to spatial process models for carrying out statistical inference. However, fitting spatial models often involves expensive matrix decompositions, whose computational complexity increases in cubic order with the number of spatial locations. This situation is aggravated in Bayesian settings where such computations are required once at every iteration of the Markov chain Monte Carlo (MCMC) algorithms. In this paper, we describe the use of Variational Bayesian (VB) methods as an alternative to MCMC to approximate the posterior distributions of complex spatial models. Variational methods, which have been used extensively in Bayesian machine learning for several years, provide a lower bound on the marginal likelihood, which can be computed efficiently. We provide results for the variational updates in several models especially emphasizing their use in multivariate spatial analysis. We demonstrate estimation and model comparisons from VB methods by using simulated data as well as environmental data sets and compare them with inference from MCMC.     

Solution of the generalized exponential cosine screened Coulomb potential

Solutions of Schrödinger's equation are presented for two-particle system interacting through generalized exponential cosine screened Coulomb potential. Solutions are computed for several values of screening parameters. In the present context, Ritz variation method is used with hydrogenic wave function as a trial wave function. The bound energies are derived from an energy equation which contains one unknown variational parameter. To calculate the variational parameter numerically fixed-point iteration scheme is used. The calculated energy eigenvalues for exponential cosine screened Coulomb potential agree excellently with the available other theoretical results. Under screening, all energy levels are shifted away from their unscreened values toward the continuum. The radial wave functions, radial probability distribution functions are presented for different screening parameters.     

Statistical Mechanics of Online Learning of Drifting Concepts: A Variational Approach

We review the application of statistical mechanics methods to the study of online learning of a drifting concept in the limit of large systems. The model where a feed-forward network learns from examples generated by a time dependent teacher of the same architecture is analyzed. The best possible generalization ability is determined exactly, through the use of a variational method. The constructive variational method also suggests a learning algorithm. It depends, however, on some unavailable quantities, such as the present performance of the student. The construction of estimators for these quantities permits the implementation of a very effective, highly adaptive algorithm. Several other algorithms are also studied for comparison with the optimal bound and the adaptive algorithm, for different types of time evolution of the rule.     

A MIXTURE MODEL CLASSIFIER AND ITS APPLICATION ON THE BIOMEDICAL TIME SERIES

This article presents a methodology based on the mixture model to classify the real biomedical time series. The mixture model is shown to be an efficient probabilistic density estimation scheme aimed at approximating the posterior probability distribution of a certain class of data. The approximation is conducted by employing a weighted mixture of a finite number of Gaussian kernels whose parameters and mixing coefficients are estimated iteratively through a maximum likelihood method. A database of the real electrocardiogram (ECG) time series of out-of-hospital cardiac arrest patients suffering ventricular fibrillation (VF) with known defibrillation outcomes was adopted to evaluate the performance of this model and confirm its efficiency compared with other classification methods.     

Over-Parameterized Variational Optical Flow

A novel optical flow estimation process based on a spatio-temporal model with varying coefficients multiplying a set of basis functions at each pixel is introduced. Previous optical flow estimation methodologies did not use such an over parameterized representation of the flow field as the problem is ill-posed even without introducing any additional parameters: Neighborhood based methods of the Lucas–Kanade type determine the flow at each pixel by constraining the flow to be described by a few parameters in small neighborhoods. Modern variational methods represent the optic flow directly via the flow field components at each pixel. The benefit of over-parametrization becomes evident in the smoothness term, which instead of directly penalizing for changes in the optic flow, accumulates a cost of deviating from the assumed optic flow model. Our proposed method is very general and the classical variational optical flow techniques are special cases of it, when used in conjunction with constant basis functions. Experimental results with the novel flow estimation process yield significant improvements with respect to the best results published so far.     

Computation of the real structured singular value via pole migration

The paper introduces a new computationally efficient algorithm to determine a lower bound on the real structured singular value μ. The algorithm is based on a pole migration approach where an optimization solver is used to compute a lower bound on real μ independent of a frequency sweep. A distinguishing feature of this algorithm from other frequency independent one‐shot tests is that multiple localized optima (if they exist) are identified and returned from the search. This is achieved by using a number of alternative methods to generate different initial conditions from which the optimization solver can initiate its search from. The pole migration algorithm presented has also been extended to determine lower bounds for complex parametric uncertainties as well as full complex blocks. However, the results presented are for strictly real and repeated parametric uncertainty problems as this class of problem is the focus of this paper and are in general the most difficult to solve. Copyright © 2014 John Wiley & Sons, Ltd.     

Robust adaptive control for a class of MIMO nonlinear systems with guaranteed error bounds

The design of stabilizing controllers for multiple-input-multiple-output (MIMO) nonlinear plants with unknown nonlinearities is a challenging problem. The high dimensionality coupled with the inability to identify the nonlinearities online or offline accurately motivates the design of stabilizing controllers based on approximations or on approximate estimates of the plant nonlinearities that are simple enough to be generated in real time. The price paid in such case, could be lack of theoretical guarantees for global stability, and nonzero tracking or regulation error at steady state. In this paper, a nonlinear robust adaptive control algorithm is designed and analyzed for a class of MIMO nonlinear systems with unknown nonlinearities. The proposed control scheme provides a general approach to bypass the stabilizability problem where the estimated plant becomes uncontrollable without any restrictive assumptions. The controller is continuous and guarantees closed-loop semi-global stability and convergence of the tracking error to a small residual set. The size of the tracking error at steady state can be specified a priori and guaranteed by choosing certain design parameters. A procedure for choosing these parameters is presented. The properties of the proposed control algorithm are demonstrated using simulations.