The copula echo state network

Echo state networks (ESNs) constitute a novel approach to recurrent neural network (RNN) training, with an RNN (the reservoir) being generated randomly, and only a readout being trained using a simple, computationally efficient algorithm. ESNs have greatly facilitated the practical application of RNNs, outperforming classical approaches on a number of benchmark tasks. This paper studies the formulation of a class of copula-based semiparametric models for sequential data modeling, characterized by nonparametric marginal distributions modeled by postulating suitable echo state networks, and parametric copula functions that help capture all the scale-free temporal dependence of the modeled processes. We provide a simple algorithm for the data-driven estimation of the marginal distribution and the copula parameters of our model under the maximum-likelihood framework. We exhibit the merits of our approach by considering a number of applications; as we show, our method offers a significant enhancement in the dynamical data modeling capabilities of ESNs, without significant compromises in the algorithm's computational efficiency.     

基于变分贝叶斯推断的字典学习算法

传统的字典学习算法在对训练图像进行学习时收敛速率慢,当图像受到噪声干扰时学习效果变差.对此,提出一种基于变分推断的字典学习算法.首先设定模型中各参数的共轭稀疏先验分布;然后基于贝叶斯网络求出所有参数的联合概率密度函数;最后利用变分贝叶斯推断原理计算出各参数的最优边缘分布,训练出自适应学习字典.利用该字典进行图像去噪实验以及压缩感知重构实验,仿真结果表明,所提出的算法可显著提高字典学习效率,对测试图像的去噪效果和重构精度有很大改善.     

Heuristic Dynamic Programming Using Echo State Network For Multivariable Tracking Control Of Wastewater Treatment Process

This paper proposes a heuristic dynamic programming (HDP) scheme to simultaneously control the dissolved oxygen concentration and the nitrate level in wastewater treatment processes (WWTP). Unlike traditional HDP schemes, the optimal control values are calculated in an analytical way by the proposed HDP controller. It can reduce the learning burden of the HDP controller to a great extent. The system model and the evaluation index J are approximated by two echo state networks (ESNs). Gradient‐based learning algorithms are employed to train both ESNs online, and the convergence of the training algorithm is investigated based on Lyapunov theory. The performance of the proposed ESN‐based HDP (E‐HDP) controller is tested and evaluated on a WWTP benchmark. Experimental results demonstrate that the proposed approach can achieve effective performance.     

Bayesian feature and model selection for Gaussian mixture models

We present a Bayesian method for mixture model training that simultaneously treats the feature selection and the model selection problem. The method is based on the integration of a mixture model formulation that takes into account the saliency of the features and a Bayesian approach to mixture learning that can be used to estimate the number of mixture components. The proposed learning algorithm follows the variational framework and can simultaneously optimize over the number of components, the saliency of the features, and the parameters of the mixture model. Experimental results using high-dimensional artificial and real data illustrate the effectiveness of the method.     

Recursive Bayesian Recurrent Neural Networks for Time-Series Modeling

This paper develops a probabilistic approach to recursive second-order training of recurrent neural networks (RNNs) for improved time-series modeling. A general recursive Bayesian Levenberg–Marquardt algorithm is derived to sequentially update the weights and the covariance (Hessian) matrix. The main strengths of the approach are a principled handling of the regularization hyperparameters that leads to better generalization, and stable numerical performance. The framework involves the adaptation of a noise hyperparameter and local weight prior hyperparameters, which represent the noise in the data and the uncertainties in the model parameters. Experimental investigations using artificial and real-world data sets show that RNNs equipped with the proposed approach outperform standard real-time recurrent learning and extended Kalman training algorithms for recurrent networks, as well as other contemporary nonlinear neural models, on time-series modeling.      

Regularized Negative Correlation Learning for Neural Network Ensembles

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when $lambda=1$) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter $lambda$ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set.      

Variational learning for Gaussian mixture models.

This paper proposes a joint maximum likelihood and Bayesian methodology for estimating Gaussian mixture models. In Bayesian inference, the distributions of parameters are modeled, characterized by hyperparameters. In the case of Gaussian mixtures, the distributions of parameters are considered as Gaussian for the mean, Wishart for the covariance, and Dirichlet for the mixing probability. The learning task consists of estimating the hyperparameters characterizing these distributions. The integration in the parameter space is decoupled using an unsupervised variational methodology entitled variational expectation-maximization (VEM). This paper introduces a hyperparameter initialization procedure for the training algorithm. In the first stage, distributions of parameters resulting from successive runs of the expectation-maximization algorithm are formed. Afterward, maximum-likelihood estimators are applied to find appropriate initial values for the hyperparameters. The proposed initialization provides faster convergence, more accurate hyperparameter estimates, and better generalization for the VEM training algorithm. The proposed methodology is applied in blind signal detection and in color image segmentation.     

基于增量式学习的正则化回声状态网络

针对回声状态网络(ESN)的结构设计问题,提出增量式正则化回声状态网络(IRESN).该网络由相互独立的子储备池模块构成,首先,子储备池根据奇异值分解方法生成,且可以保证每个子储备池权值矩阵的奇异值都小于1;其次,利用问题复杂度或者残差,将网络中逐一添加子储备池,直至满足预设的终止条件,在生成IRESN的过程中,回声状...     

Variational Bayesian inference based robust multiple measurement sparse signal recovery

This work concentrates on not only probing into a novel Bayesian probabilistic model to formulate a general type of robust multiple measurement vectors sparse signal recovery problem with impulsive noise, but also developing an improved variational Bayesian method to recover the original joint row sparse signals. In the design of the model, two three-level hierarchical Bayesian estimation procedures are designed to characterize impulsive noise and joint row sparse source signals by means of Gaussian scale mixtures and multivariate generalized t distribution. Those hidden variables, included in signal and measurement models are estimated based on a variational Bayesian framework, in which multiple kinds of probability distributions are adopted to express their features. In the design of the algorithm, the proposed algorithm is a full Bayesian inference approach related to variational Bayesian estimation. It is robust to impulsive noise, since the posterior distribution estimation can be effectively approached through estimating unknown parameters. Extensive simulation results show that the proposed algorithm significantly outperforms the compared robust sparse signal recovery approaches under different kinds of impulsive noises.     

小数据集条件下基于双重约束的BN参数学习

针对小数据集条件下的贝叶斯网络（Bayesian network,BN）参数学习问题,提出了一种基于双重约束的贝叶斯网络参数学习方法. 首先,对网络中的参数进行分析并将网络中的参数划分为: 父节点组合状态相同而子节点状态不同的参数和父节点组合状态不同而子节点状态相同的参数;然后,针对第一类参数提出了一种新的基于Beta分布拟合的贝叶斯估计方法,而针对第二类参数利用已有的保序回归估计方法进行学习,进而实现了对网络中参数的双重约束学习;最后,通过仿真实例说明了基于双重约束的参数学习方法对小数据集条件下贝叶斯网络参数学习精度提高的有效性.     

Bayesian regularization in the problem of point-by-point function approximation using an orthogonalized basis

An algorithm of approximation of a multidimensional point-by-point scalar function is considered. The solution is sought as a series in a set of basis functions. The approximation is regularized by the introduction of a stabilizing function in the Gaussian form; the parameter of regularization is sought by using the Bayesian approach. The proposed algorithm is inexpensive in terms of computations. Unlike other Bayesian models of approximation, it has a unique analytical solution for the regularization parameters.     

回响状态网络输出连接权重的一个稳定训练方法

鉴于在回响状态网络(ESN)的应用中常使用Wiener-Hopf方程学习输出连接权重,但该方法难以保证自治ESN的稳定性,首先分析了导致该稳定性丧失的原因,提出并证明了自治ESN具备Lyapunov稳定性的一个充分条件;然后将输出连接权重学习问题转化为一个非线性约束的最优化问题,并采用粒子群优化算法求解.仿真结果表明,所提方法既能确保ESN获取高精度的预测输出,又能保ESN的Lyapunov稳定性.     

A nonparametric Bayesian approach toward robot learning by demonstration

In the past years, many authors have considered application of machine learning methodologies to effect robot learning by demonstration. Gaussian mixture regression (GMR) is one of the most successful methodologies used for this purpose. A major limitation of GMR models concerns automatic selection of the proper number of model states, i.e., the number of model component densities. Existing methods, including likelihood- or entropy-based criteria, usually tend to yield noisy model size estimates while imposing heavy computational requirements. Recently, Dirichlet process (infinite) mixture models have emerged in the cornerstone of nonparametric Bayesian statistics as promising candidates for clustering applications where the number of clusters is unknown a priori. Under this motivation, to resolve the aforementioned issues of GMR-based methods for robot learning by demonstration, in this paper we introduce a nonparametric Bayesian formulation for the GMR model, the Dirichlet process GMR model. We derive an efficient variational Bayesian inference algorithm for the proposed model, and we experimentally investigate its efficacy as a robot learning by demonstration methodology, considering a number of demanding robot learning by demonstration scenarios.     

一种有效的储备池在线稀疏学习算法

为克服传统储备池方法缺乏良好在线学习算法的问题, 同时考虑到储备池本身存在的不适定问题, 本文提出一种储备池在线稀疏学习算法, 对储备池目标函数施加L1正则化约束,并采用截断梯度算法在线近似求解.所提算法在对储备池输出权值进行在线调整的同时, 可对储备池输出权值的稀疏性进行有效控制, 有效保证了网络的泛化性能.理论分析和仿真实例证明所提算法的有效性.     

基于扩散方法的分布式随机变分推断算法

分布式网络上的聚类、估计或推断具有广泛的应用,因此引起了许多关注.针对己有的分布式变分贝叶斯(Vari-ational Bayesian,VB)算法效率低,可扩展性差的问题,本文借用扩散方法提出了一种新的分布式随机变分推断(Stochastic variational inference,SVI)算法,其中我们选择自然...     

一种基于L1范数正则化的回声状态网络

针对回声状态网络存在的病态解以及模型规模控制问题,本文提出一种基于L1范数正则化的改进回声状态网络.该方法通过在目标函数中添加L1范数惩罚项,提高模型求解的数值稳定性,同时借助于L1范数正则化的特征选择能力,控制网络的复杂程度,防止出现过拟合.对于L1范数正则化的求解,采用最小角回归算法计算正则化路径,通过贝叶斯信息准则进行模型选择,避免估计正则化参数.将模型应用于人造数据和实际数据的时间序列预测中,仿真结果证明了本文方法的有效性和实用性.     

Active learning SVM with regularization path for image classification

In classification problems, many different active learning techniques are often adopted to find the most informative samples for labeling in order to save human labors. Among them, active learning support vector machine (SVM) is one of the most representative approaches, in which model parameter is usually set as a fixed default value during the whole learning process. Note that model parameter is closely related to the training set. Hence dynamic parameter is desirable to make a satisfactory learning performance. To target this issue, we proposed a novel algorithm, called active learning SVM with regularization path, which can fit the entire solution path of SVM for every value of model parameters. In this algorithm, we first traced the entire solution path of the current classifier to find a series of candidate model parameters, and then used unlabeled samples to select the best model parameter. Besides, in the initial phase of training, we constructed a training sample sets by using an improved K-medoids cluster algorithm. Experimental results conducted from real-world data sets showed the effectiveness of the proposed algorithm for image classification problems.     

Variational Bayesian sparse additive matrix factorization

Principal component analysis (PCA) approximates a data matrix with a low-rank one by imposing sparsity on its singular values. Its robust variant can cope with spiky noise by introducing an element-wise sparse term. In this paper, we extend such sparse matrix learning methods, and propose a novel framework called sparse additive matrix factorization (SAMF). SAMF systematically induces various types of sparsity by a Bayesian regularization effect, called model-induced regularization. Although group LASSO also allows us to design arbitrary types of sparsity on a matrix, SAMF, which is based on the Bayesian framework, provides inference without any requirement for manual parameter tuning. We propose an efficient iterative algorithm called the mean update (MU) for the variational Bayesian approximation to SAMF, which gives the global optimal solution for a large subset of parameters in each step. We demonstrate the usefulness of our method on benchmark datasets and a foreground/background video separation problem.     

Recursive Noise Adaptive Kalman Filtering by Variational Bayesian Approximations

This article considers the application of variational Bayesian methods to joint recursive estimation of the dynamic state and the time-varying measurement noise parameters in linear state space models. The proposed adaptive Kalman filtering method is based on forming a separable variational approximation to the joint posterior distribution of states and noise parameters on each time step separately. The result is a recursive algorithm, where on each step the state is estimated with Kalman filter and the sufficient statistics of the noise variances are estimated with a fixed-point iteration. The performance of the algorithm is demonstrated with simulated data.      

基于K-SVD超声渡越时间获取方法研究

在超声回波检测信号中,反映污垢特征的冲击信号非常微弱,容易被噪声淹没。针对信号稀疏分解中常用匹配追踪分解不够准确的问题,提出基于K-SVD奇异值分解的超声渡越时间获取方法,利用K-SVD训练得到超声回波信号的过完备字典,结合正交匹配追踪进行局部搜索适配原子,以提高信号稀疏分解的速度和准确度。基于Comsol Multipysics仿真软件建立充液污垢管道三维有限元模型,研究了超声回波传播特性规律。将K-SVD算法应用于超声回波仿真信号和换热污垢管道回波检测信号的处理,并与原始小波训练字典进行对比。结果表明：改进的K-SVD字典学习算法能够在提高信号稀疏分解的同时,获得较好的降噪结果和污垢特征信息提取,对超声检测信号的处理具有实际意义。