Gaussian kernel optimization: Complex problem and a simple solution

The Gaussian kernel function implicitly defines the feature space of an algorithm and plays an essential role in the application of kernel methods. The parameter of Gaussian kernel function is a scalar that has significant influences on final results. However, until now, it is still unclear how to choose an optimal kernel parameter. In this paper, we propose a novel data-driven method to optimize the Gaussian kernel parameter, which only depends on the original dataset distribution and yields a simple solution to this complex problem. The proposed method is task irrelevant and can be used in any Gaussian kernel-based approach, including supervised and unsupervised machine learning. Simulation experiments demonstrate the efficacy of the obtained results. A user-friendly online calculator is implemented at: www.csbio.sjtu.edu.cn/bioinf/kernel/ for public use.     

一种确定高斯核模型参数的新方法

支持向量机中核函数及其参数的选择非常重要，该文提出了一种利用支持向量之间的距离求取高斯核函数参数的有效方法。该方法充分利用了支持向量机方法的最优判别函数仅仅与支持向量有关，并且支持向量为高斯核中心的特点。实验结果表明，该方法较好地反映了图像特征的本质，解决了高斯核函数参数在实际使用中不易确定的问题。     

基于高斯分布监督学习样本的Fisher核构造方法

结合实际应用背景, 针对各类样本服从高斯分布的监督学习情形, 提出了构造Fisher核的新方法. 由于利用了样本中的类别信息, 该方法用极大似然估计代替EM算法估计GMM参数, 有效降低了Fisher核构造的时间复杂度. 结合核Fisher分类法, 上述方法在标准人脸库上的仿真实验结果显示, 用所提方法所构造的Fisher核不仅时间复杂度低, 且识别率也优于传统的高斯核与多项式核. 本文的研究有利于将Fisher 核的应用从语音识别领域拓展到图像识别等领域.     

An approximate inference with Gaussian process to latent functions from uncertain data

Most formulations of supervised learning are often based on the assumption that only the outputs data are uncertain. However, this assumption might be too strong for some learning tasks. This paper investigates the use of Gaussian processes to infer latent functions from a set of uncertain input-output examples. By assuming Gaussian distributions with known variances over the inputs-outputs and using the expectation of the covariance function, it is possible to analytically compute the expected covariance matrix of the data to obtain a posterior distribution over functions. The method is evaluated on a synthetic problem and on a more realistic one, which consist in learning the dynamics of a cart-pole balancing task. The results indicate an improvement of the mean squared error and the likelihood of the posterior Gaussian process when the data uncertainty is significant.     

Gaussian kernel optimization for pattern classification

This paper presents a novel algorithm to optimize the Gaussian kernel for pattern classification tasks, where it is desirable to have well-separated samples in the kernel feature space. We propose to optimize the Gaussian kernel parameters by maximizing a classical class separability criterion, and the problem is solved through a quasi-Newton algorithm by making use of a recently proposed decomposition of the objective criterion. The proposed method is evaluated on five data sets with two kernel-based learning algorithms. The experimental results indicate that it achieves the best overall classification performance, compared with three competing solutions. In particular, the proposed method provides a valuable kernel optimization solution in the severe small sample size scenario.     

基于高斯矩的NoisyICA研究

独立分量分析(ICA)作为有效的盲源分离技术(BSS)是信号处理领域的热点。实际信号或多或少的都含有噪声，如果信噪比低于某值将得不到良好的分离效果。该文定义不同参量的高斯函数的期望为随机向量的高斯矩，证明随机向量的高斯矩可作为无偏估计的单值对照函数应用于带高斯噪声的ICA模型。由此利用最大化基于高斯矩的对照函数．得到FastICA改进算法-noisvICA．     

基于K-近邻点云去噪算法的研究与改进

针对三维扫描获取的带噪声和离群点的点云数据,提出了改进的去噪算法。通过K-近邻搜索建立散乱点云之间的拓扑关系,进而采用高斯影响函数作为核函数来估计当前测点对周围邻近点的影响力,从而限制噪声并剔除离群点。重点讨论了密度熵的概念以及如何优化高斯核函数的参数,解决了窗宽尺寸在使用中不易确定的问题。仿真实验证明,该算法能够很容易地检测出离群点,同时也避免了将模型上的点误判为离群点的问题。     

基于自适应密度估计的系统参数辨识

针对噪声分布未知的ARMAX系统,提出了一种自适应非参数噪声密度估计方法,由估计误差动态调整高斯核函数的全局带宽和局部带宽,实现了未知噪声分布密度的自适应估计;通过极小化似然函数,给出了基于噪声密度估计的参数辨识迭代算法,分析了算法的收敛性并给出了算法收敛的充分条件.仿真结果表明本文提出的算法在系统噪声未知时具有较强的抗噪能力和良好的收敛性.     

基于高斯矩的噪声独立分量分析

独立分量分析(ICA)作为有效的盲源分离技术(BSS)是信号处理领域的热点。但是许多ICA算法都是建立在无噪模型基础上实现盲源分离的,忽略了噪声对分离信号的影响。而实际信号或多或少的都含有噪声,如果信噪比低于某值将得不到良好的分离效果。该文定义不同参量的高斯函数的期望为随机向量的高斯矩,证明随机向量的高斯矩可作为无偏估计的单值对照函数应用于带高斯噪声的noisyICA模型。由此利用最大化基于高斯矩的对照函数,得到FastICA改进算法—noisyICA,并通过模拟实验证明了算法的可行性和健壮性。     

基于EM算法的图像小波系数统计研究

基于小波分析的贝叶斯(Bayes)图像处理方法常常需要获得图像小波波系数的先验概率分布密度,该文提出,利用混合高斯模型对正交小波域中自然图像的父子小波系数的联合分布密度进行建模,运用非完备数据的极大似然估计算法——期望极大(EM)算法,对该模型的参数进行估计并且给出了联合分布密度函数的模型分量数与迭代次数的确定过程。最后,在后验均值(PM)方法下,把该联合分布密度模型运用于图像去噪研究;仿真结果表明该方法能够获得较好的效果。     

一种基于各向异性高斯核核惩罚的PCA特征提取算法

提出了一种基于各向异性高斯核核惩罚的主成分分析的特征提取算法.该算法不同于传统的核主成分分析算法.在非线性数据降维中,传统的核主成分分析算法忽略了原始数据的无量纲化.此外,传统的核函数在各维度上主要由一个相同的核宽参数控制,该方法无法准确反映各维度不同特征的重要性,从而导致降维过程中准确率低下.为了解决上述问题,首先针对现原始数据的无量纲化问题,提出了一种均值化算法,使得原始数据的总方差贡献率有明显的提高.其次,引入了各向异性高斯核函数,该核函数每个维度拥有不同的核宽参数,各核宽参数能够准确地反映所在维度数据特征的重要性.再次,基于各向异性高斯核函数建立了核主成分分析的特征惩罚目标函数,以便用较少的特征表示原始数据,并反映每个主成分信息的重要性.最后,为了寻求最佳特征,引入梯度下降算法来更新特征惩罚目标函数中的核宽度和控制特征提取算法的迭代过程.为了验证所提出算法的有效性,各算法在UCI公开数据集上和KDDCUP99数据集上进行了比较.实验结果表明,所提基于各向异性高斯核核惩罚的主成分分析的特征提取算法比传统的主成分分析算法在9种公开的UCI公开数据集上准确率平均提高了4.49%.在KDDCUP99数据集上,所提基于各向异性高斯核核惩罚的主成分分析的特征提取算法比传统的主成分分析算法准确率提高了8%.     

高斯小波支持向量机的研究

证明了偶数阶高斯小波函数满足支持向量机的平移不变核函数条件．应用小波核函数建立了相应的高斯小波支持向量机,并且使用云遗传算法对支持向量机及其核函数的参数进行优化．用该算法与常用的高斯核和Morlet小波核支持向量机进行对比实验．通过对非线性函数的逼近和电力系统短期负荷的预测,验证了该算法的有效性和优越性,表明其具有一定的实用价值．     

快速核密度估计定理和大规模图论松弛聚类方法

首先证明了快速核密度估计 (Fast kernel density estimate, FKDE) 定理: 基于抽样子集的高斯核密度估计(KDE)与原数据集的KDE间的误差与抽样容量和核参数相关, 而与总样本容量无关. 接着本文揭示了基于高斯核形式的图论松弛聚类(Graph-based relaxed clustering, GRC)算法的目标表达式可分解成“Parzen窗加权和 + 平方熵”的形式, 即此时GRC可视作一个核密度估计问题, 这样基于KDE近似策略, 本文提出了大规模图论松弛聚类方法(Scaling up GRC by KDE approximation, SUGRC-KDEA). 较之先前的工作, 这一方法的优势在于为GRC作用于大规模数据集提供了更简单和易于实现的方案.     

Classification of incomplete feature vectors by radial basis function networks

The paper describes the use of radial basis function neural networks with Gaussian basis functions to classify incomplete feature vectors. The method uses the fact that any marginal distribution of a Gaussian distribution can be determined from the mean vector and covariance matrix of the joint distribution.     

Bounded generalized Gaussian mixture model

The generalized Gaussian mixture model (GGMM) provides a flexible and suitable tool for many computer vision and pattern recognition problems. However, generalized Gaussian distribution is unbounded. In many applications, the observed data are digitalized and have bounded support. A new bounded generalized Gaussian mixture model (BGGMM), which includes the Gaussian mixture model (GMM), Laplace mixture model (LMM), and GGMM as special cases, is presented in this paper. We propose an extension of the generalized Gaussian distribution in this paper. This new distribution has a flexibility to fit different shapes of observed data such as non-Gaussian and bounded support data. In order to estimate the model parameters, we propose an alternate approach to minimize the higher bound on the data negative log-likelihood function. We quantify the performance of the BGGMM with simulations and real data.     

Parameter estimation in exponential models by linear and nonlinear fitting methods

Estimation of unknown parameters in exponential models by linear and nonlinear fitting methods is discussed. Based on the extreme value theorem and Taylor series expansion, it is proved theoretically that the parameters estimated by the linear fitting method alone cannot minimize the sum of the squared residual errors in the measurement data when measurement noise is involved in the data. Numerical simulation is performed to compare the performance of the linear and nonlinear fitting methods. Simulation results show that the linear method can obtain only a suboptimal estimate of the unknown parameters and that the nonlinear method gives more accurate results. Application of the fitting methods is demonstrated where the water spectral attenuation coefficient is estimated from underwater images and imaging distances, which supports the improvement in the accuracy of parameter estimation by the nonlinear fitting method.     

基于人工蜂群优化高斯过程的运动想象脑电信号分类

针对传统的高斯过程采用共轭梯度法确定超参数时对初值有较强依赖性且易陷入局部最优的问题,提出了一种基于人工蜂群优化的高斯过程分类方法,用于脑电信号的模式识别.首先,构建高斯过程模型,选择合适的核函数且确定待优化的参数.然后,选取识别错误率的倒数为适应度函数,使用人工蜂群算法搜索寻找出限定范围内可以取得最优准确率的超参数.最后,采用参数优化后的高斯过程分类器对样本分类.分别采用2008年竞赛数据集BCI Competition Ⅳ Data Set 1和2005年数据集BCI Competition Ⅲ Data Set Ⅳa对所提方法进行验证,并与支持向量机(SVM)、人工蜂群优化的支持向量机(ABC-SVM)、高斯过程分类(GPC)方法进行比较,实验结果表明了所提方法的有效性.     

基于状态-动作图测地高斯基的策略迭代强化学习

在策略迭代强化学习中,基函数构造是影响动作值函数逼近精度的一个重要因素.为了给动作值函数逼近提供合适的基函数,提出一种基于状态-动作图测地高斯基的策略迭代强化学习方法.首先,根据离策略方法建立马尔可夫决策过程的状态-动作图论描述;然后,在状态-动作图上定义测地高斯核函数,利用基于近似线性相关的核稀疏方法自动选择测地高斯...     

Semi-supervised clustering with metric learning: An adaptive kernel method

Most existing representative works in semi-supervised clustering do not sufficiently solve the violation problem of pairwise constraints. On the other hand, traditional kernel methods for semi-supervised clustering not only face the problem of manually tuning the kernel parameters due to the fact that no sufficient supervision is provided, but also lack a measure that achieves better effectiveness of clustering. In this paper, we propose an adaptive Semi-supervised Clustering Kernel Method based on Metric learning (SCKMM) to mitigate the above problems. Specifically, we first construct an objective function from pairwise constraints to automatically estimate the parameter of the Gaussian kernel. Then, we use pairwise constraint-based K-means approach to solve the violation issue of constraints and to cluster the data. Furthermore, we introduce metric learning into nonlinear semi-supervised clustering to improve separability of the data for clustering. Finally, we perform clustering and metric learning simultaneously. Experimental results on a number of real-world data sets validate the effectiveness of the proposed method.     

Asymmetric Gaussian Process multi-view learning for visual classification

Methods of multi-view learning attain outstanding performance in different fields compared with the single-view based strategies. In this paper, the Gaussian Process Latent Variable Model (GPVLM), which is a generative and non-parametric model, is exploited to represent multiple views in a common subspace. Specifically, there exists a shared latent variable across various views that is assumed to be transformed to observations by using distinctive Gaussian Process projections. However, this assumption is only a generative strategy, being intractable to simply estimate the fused variable at the testing step. In order to tackle this problem, another projection from observed data to the shared variable is simultaneously learned by enjoying the view-shared and view-specific kernel parameters under the Gaussian Process structure. Furthermore, to achieve the classification task, label information is also introduced to be the generation from the latent variable through a Gaussian Process transformation. Extensive experimental results on multi-view datasets demonstrate the superiority and effectiveness of our model in comparison to state-of-the-art algorithms.