A New Solution Path Algorithm in Support Vector Regression

In this paper, regularization path algorithms were proposed as a novel approach to the model selection problem by exploring the path of possibly all solutions with respect to some regularization hyperparameter in an efficient way. This approach was later extended to a support vector regression (SVR) model called epsiv -SVR. However, the method requires that the error parameter epsiv be set a priori. This is only possible if the desired accuracy of the approximation can be specified in advance. In this paper, we analyze the solution space for epsiv-SVR and propose a new solution path algorithm, called epsiv-path algorithm, which traces the solution path with respect to the hyperparameter epsiv rather than lambda. Although both two solution path algorithms possess the desirable piecewise linearity property, our epsiv-path algorithm overcomes some limitations of the original lambda-path algorithm and has more advantages. It is thus more appealing for practical use.     

Solution path for manifold regularized semisupervised classification

Traditional learning algorithms use only labeled data for training. However, labeled examples are often difficult or time consuming to obtain since they require substantial human labeling efforts. On the other hand, unlabeled data are often relatively easy to collect. Semisupervised learning addresses this problem by using large quantities of unlabeled data with labeled data to build better learning algorithms. In this paper, we use the manifold regularization approach to formulate the semisupervised learning problem where a regularization framework which balances a tradeoff between loss and penalty is established. We investigate different implementations of the loss function and identify the methods which have the least computational expense. The regularization hyperparameter, which determines the balance between loss and penalty, is crucial to model selection. Accordingly, we derive an algorithm that can fit the entire path of solutions for every value of the hyperparameter. Its computational complexity after preprocessing is quadratic only in the number of labeled examples rather than the total number of labeled and unlabeled examples.     

Regularization Methods for Blind Deconvolution and Blind Source Separation Problems

This paper is devoted to blind deconvolution and blind separation problems. Blind deconvolution is the identification of a point spread function and an input signal from an observation of their convolution. Blind source separation is the recovery of a vector of input signals from a vector of observed signals, which are mixed by a linear (unknown) operator. We show that both problems are paradigms of nonlinear ill-posed problems. Consequently, regularization techniques have to be used for stable numerical reconstructions. In this paper we develop a rigorous convergence analysis for regularization techniques for the solution of blind deconvolution and blind separation problems. Convergence of regularized point spread functions and signals to a solution is established and a convergence rate result in dependence of the noise level is presented. Moreover, we prove convergence of the alternating minimization algorithm for the numerical solution of regularized blind deconvolution problems and present some numerical examples. Moreover, we show that many neural network approaches for blind inversion can be considered in the framework of regularization theory. Date received: August 17, 1999. Date revised: September 1, 2000.     

On the estimation of transfer functions,regularizations and Gaussian processes—Revisited

Intrigued by some recent results on impulse response estimation by kernel and nonparametric techniques, we revisit the old problem of transfer function estimation from input–output measurements. We formulate a classical regularization approach, focused on finite impulse response (FIR) models, and find that regularization is necessary to cope with the high variance problem. This basic, regularized least squares approach is then a focal point for interpreting other techniques, like Bayesian inference and Gaussian process regression. The main issue is how to determine a suitable regularization matrix (Bayesian prior or kernel). Several regularization matrices are provided and numerically evaluated on a data bank of test systems and data sets. Our findings based on the data bank are as follows. The classical regularization approach with carefully chosen regularization matrices shows slightly better accuracy and clearly better robustness in estimating the impulse response than the standard approach–the prediction error method/maximum likelihood (PEM/ML) approach. If the goal is to estimate a model of given order as well as possible, a low order model is often better estimated by the PEM/ML approach, and a higher order model is often better estimated by model reduction on a high order regularized FIR model estimated with careful regularization. Moreover, an optimal regularization matrix that minimizes the mean square error matrix is derived and studied. The importance of this result lies in that it gives the theoretical upper bound on the accuracy that can be achieved for this classical regularization approach.     

Approximation-theoretic methods for nonlinear deconvolution and inversion

Nonlinear deconvolution and nonlinear inversion are cast as inverse problems in generalized Fock spaces. Generalized Fock spaces, introduced by de Figueiredo and Dwyer in [1], are reproducing kernel Hilbert spaces (RKHSs) of input-output maps represented by Volterra series equipped with an appropriately weighted inner product, the choice of the weights in the inner product depending on the particular problem under consideration. The solution to the nonlinear deconvolution problem presented here is the same as the one obtained previously for the nonlinear system identification problem [1–4]. However, the present solution to the nonlinear inversion problem consists of a new approach, whereby the unknown samples of the input are obtained from the given samples of the output by means of an efficient sequential algorithm. The algorithm is based on a framework which interpolates the input samples by an appropriate spline, and its sequential nature is elicited by the use of a truncated function basis to represent the spline.     

Bayesian regularization in the problem of point-by-point function approximation using an orthogonalized basis

An algorithm of approximation of a multidimensional point-by-point scalar function is considered. The solution is sought as a series in a set of basis functions. The approximation is regularized by the introduction of a stabilizing function in the Gaussian form; the parameter of regularization is sought by using the Bayesian approach. The proposed algorithm is inexpensive in terms of computations. Unlike other Bayesian models of approximation, it has a unique analytical solution for the regularization parameters.     

Regularization networks for inverse problems: A state-space approach

Linear inverse problems with discrete data are equivalent to the estimation of the continuous-time input of a linear dynamical system from samples of its output. The solution obtained by means of regularization theory has the structure of a neural network similar to classical RBF networks. However, the basis functions depend in a nontrivial way on the specific linear operator to be inverted and the adopted regularization strategy. By resorting to the Bayesian interpretation of regularization, we show that such networks can be implemented rigorously and efficiently whenever the linear operator admits a state-space representation. An analytic expression is provided for the basis functions as well as for the entries of the matrix of the linear system used to compute the weights. The results are illustrated through a deconvolution problem where the spontaneous secretory rate of luteinizing hormone (LH) of the hypophisis is reconstructed from measurements of plasma LH concentrations.     

一种基于L1范数正则化的回声状态网络

针对回声状态网络存在的病态解以及模型规模控制问题,本文提出一种基于L1范数正则化的改进回声状态网络.该方法通过在目标函数中添加L1范数惩罚项,提高模型求解的数值稳定性,同时借助于L1范数正则化的特征选择能力,控制网络的复杂程度,防止出现过拟合.对于L1范数正则化的求解,采用最小角回归算法计算正则化路径,通过贝叶斯信息准则进行模型选择,避免估计正则化参数.将模型应用于人造数据和实际数据的时间序列预测中,仿真结果证明了本文方法的有效性和实用性.     

Blind image deconvolution using machine learning for three-dimensional microscopy

In this work, we propose a novel method for the regularization of blind deconvolution algorithms. The proposed method employs example-based machine learning techniques for modeling the space of point spread functions. During an iterative blind deconvolution process, a prior term attracts the point spread function estimates to the learned point spread function space. We demonstrate the usage of this regularizer within a Bayesian blind deconvolution framework and also integrate into the latter a method for noise reduction, thus creating a complete blind deconvolution method. The application of the proposed algorithm is demonstrated on synthetic and real-world three-dimensional images acquired by a wide-field fluorescence microscope, where the need for blind deconvolution algorithms is indispensable, yielding excellent results.     

Speech deconvolution as an inverse problem

In this paper, the problem of speech deconvolution is solved. This problem is encountered in limited-bandwidth speech communication systems such as telephone systems. Three solutions are presented for this problem. In the first solution, a Linear Minimum Mean Square Error (LMMSE) approach is used. The necessary assumptions required to reduce the computational complexity of the LMMSE solution are presented. In the second solution, an inverse filter deconvolution approach is presented. Finally, the regularization theory is used to solve this problem. The common thread between all these solutions is that they treat the speech deconvolution problem as an inverse problem considering the speech degradation model. Simulation results reveal the superiority of these solutions for solving the speech deconvolution problem.     

Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg–Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.      

Regularized variational Bayesian learning of echo state networks with delay&sum readout

In this work, a variational Bayesian framework for efficient training of echo state networks (ESNs) with automatic regularization and delay&sum (D&S) readout adaptation is proposed. The algorithm uses a classical batch learning of ESNs. By treating the network echo states as fixed basis functions parameterized with delay parameters, we propose a variational Bayesian ESN training scheme. The variational approach allows for a seamless combination of sparse Bayesian learning ideas and a variational Bayesian space-alternating generalized expectation-maximization (VB-SAGE) algorithm for estimating parameters of superimposed signals. While the former method realizes automatic regularization of ESNs, which also determines which echo states and input signals are relevant for "explaining" the desired signal, the latter method provides a basis for joint estimation of D&S readout parameters. The proposed training algorithm can naturally be extended to ESNs with fixed filter neurons. It also generalizes the recently proposed expectation-maximization-based D&S readout adaptation method. The proposed algorithm was tested on synthetic data prediction tasks as well as on dynamic handwritten character recognition.     

Identifiability of the stochastic semi-blind deconvolution problem for a class of time-invariant linear systems

Semi-blind deconvolution is the process of estimating the unknown input of a linear system, starting from output data, when the kernel of the system contains unknown parameters. In this paper, identifiability issues related to such a problem are investigated. In particular, we consider time-invariant linear models whose impulse response is given by a sum of exponentials and assume that smoothness is the sole available a priori information on the unknown signal. We state the semi-blind deconvolution problem in a Bayesian setting where prior knowledge on the smoothness of the unknown function is mathematically formalized by describing the system input as a Brownian motion. This leads to a Tychonov-type estimator containing unknown smoothness and system parameters which we estimate by maximizing their marginal likelihood/posterior. The mathematical structure of this estimator is studied in the ideal situation of output data noiseless with their number tending to infinity. Simulated case studies are used to illustrate the practical implications of the theoretical findings in system modeling. Finally, we show how semi-blind deconvolution can be improved by proposing a new prior for signals that are initially highly nonstationary but then become, as time progresses, more regular.     

基于双层反卷积的宽场荧光显微图像盲复原*

针对宽场荧光显微图像盲复原中的不适定性和细节模糊问题,提出了基于双层反卷积的宽场荧光显微图像盲复原算法,该算法通过双层反卷积,结合图像金字塔,实现了由粗略到细致的图像复原。为抑制不适定性,外层反卷积采用全变分模型,对复原图像和光学传递函数进行正则化约束。在内层反卷积中,通过残差图像进一步复原出图像细节。实验结果表明,该算法能在有效抑制伪影和噪声的同时,复原出宽场荧光显微图像的细节。与近几年图像盲复原算法相比,该算法所需的计算时间短,复原出的宽场荧光显微图像不仅有更好的视觉效果,而且客观上有较高的峰值信噪比和图像熵。     

基于替代函数及贝叶斯框架的1范数ELM算法

针对极端学习机 (Extreme learning machine, ELM)算法的不适定问题和模型规模控制问题,本文提出基于1范数正则项的改进型ELM算法. 通过在二次损失函数基础上引入1范数正则项以控制模型规模,改善ELM的泛化能力.此外,为简化1范数 正则化方法的求解过程,利用边际优化方法,构建适当的替代函数,以便于采用贝叶斯方法代替计算复杂的 交叉检验方法,并实现正则化参数的自适应估计.仿真结果表明,本文所提算法能够有效简化模型结构,并 保持较高的预测精度.     

Averaging, maximum penalized likelihood and Bayesian estimation forimproving Gaussian mixture probability density estimates

We apply the idea of averaging ensembles of estimators to probability density estimation. In particular, we use Gaussian mixture models which are important components in many neural-network applications. We investigate the performance of averaging using three data sets. For comparison, we employ two traditional regularization approaches, i.e., a maximum penalized likelihood approach and a Bayesian approach. In the maximum penalized likelihood approach we use penalty functions derived from conjugate Bayesian priors such that an expectation maximization (EM) algorithm can be used for training. In all experiments, the maximum penalized likelihood approach and averaging improved performance considerably if compared to a maximum likelihood approach. In two of the experiments, the maximum penalized likelihood approach outperformed averaging. In one experiment averaging was clearly superior. Our conclusion is that maximum penalized likelihood gives good results if the penalty term in the cost function is appropriate for the particular problem. If this is not the case, averaging is superior since it shows greater robustness by not relying on any particular prior assumption. The Bayesian approach worked very well on a low-dimensional toy problem but failed to give good performance in higher dimensional problems.     

正则化路径上的支持向量机模型组合方法

模型组合是提高支持向量机泛化性的重要方法,但存在计算效率较低的问题。提出一种基于正则化路径上贝叶斯模型平均的支持向量机模型组合方法,在提高支持向量机泛化性的同时,具有较高的计算效率。基于正则化路径算法建立初始模型集,引入对支持向量机的概率解释。模型的先验可看做是一个高斯过程,模型的后验概率通过贝叶斯公式求得,使用贝叶斯模型平均对模型进行组合。在标准数据集上,实验比较了所提出的模型组合方法与交叉验证及广义近似交叉验证（GACV）方法的性能,验证所提出的模型组合方法的有效性。     

Bayesian sparse solutions to linear inverse problems with non-stationary noise with Student-t priors

Bayesian approach has become a commonly used method for inverse problems arising in signal and image processing. One of the main advantages of the Bayesian approach is the possibility to propose unsupervised methods where the likelihood and prior model parameters can be estimated jointly with the main unknowns. In this paper, we propose to consider linear inverse problems in which the noise may be non-stationary and where we are looking for a sparse solution. To consider both of these requirements, we propose to use Student-t prior model both for the noise of the forward model and the unknown signal or image. The main interest of the Student-t prior model is its Infinite Gaussian Scale Mixture (IGSM) property. Using the resulted hierarchical prior models we obtain a joint posterior probability distribution of the unknowns of interest (input signal or image) and their associated hidden variables. To be able to propose practical methods, we use either a Joint Maximum A Posteriori (JMAP) estimator or an appropriate Variational Bayesian Approximation (VBA) technique to compute the Posterior Mean (PM) values. The proposed method is applied in many inverse problems such as deconvolution, image restoration and computed tomography. In this paper, we show only some results in signal deconvolution and in periodic components estimation of some biological signals related to circadian clock dynamics for cancer studies.     

A Cumulant Expansion Technique for Simultaneous Markov Random Field Image Restoration and Hyperparameter Estimation

We investigate hyperparameter estimation for incomplete data in Markov random Field image restoration. Assuming linear dependence of energies with respect to hyperparameters framework, we use a cumulant expansion technique widely known in Statistical Physics and Signal Processing. New insight is given on Maximum Likelihood estimation of hyperparameters of the prior, regularization and contour probability distribution functions (pdfs) for an explicit joint boundary-pixel process aimed to preserve discontinuities. In particular the case where the prior regularization potential is an homogeneous function of pixels is fully analyzed. A Generalized Stochastic Gradient (GSG) algorithm with a fast sampling technique is devised aiming to achieve simultaneous hyperparameter estimation and pixel restoration. Image restoration performances of Posterior Mean performed during GSG convergence and of Simulated Annealing performed after GSG convergence are compared experimentally. Results and perspectives are given.     

Nonlinear time series forecasting with Bayesian neural networks

The Bayesian learning provides a natural way to model the nonlinear structure as the artificial neural networks due to their capability to cope with the model complexity. In this paper, an evolutionary Monte Carlo (MC) algorithm is proposed to train the Bayesian neural networks (BNNs) for the time series forecasting. This approach called as Genetic MC is based on Gaussian approximation with recursive hyperparameter. Genetic MC integrates MC simulations with the genetic algorithms and the fuzzy membership functions. In the implementations, Genetic MC is compared with the traditional neural networks and time series techniques in terms of their forecasting performances over the weekly sales of a Finance Magazine.