Learning bounds for kernel regression using effective data dimensionality

Kernel methods can embed finite-dimensional data into infinite-dimensional feature spaces. In spite of the large underlying feature dimensionality, kernel methods can achieve good generalization ability. This observation is often wrongly interpreted, and it has been used to argue that kernel learning can magically avoid the "curse-of-dimensionality" phenomenon encountered in statistical estimation problems. This letter shows that although using kernel representation, one can embed data into an infinite-dimensional feature space; the effective dimensionality of this embedding, which determines the learning complexity of the underlying kernel machine, is usually small. In particular, we introduce an algebraic definition of a scale-sensitive effective dimension associated with a kernel representation. Based on this quantity, we derive upper bounds on the generalization performance of some kernel regression methods. Moreover, we show that the resulting convergent rates are optimal under various circumstances.     

基于随机素描方法的在线核回归

在线核回归学习中,每当一个新的样本到来,训练器都需要计算核矩阵的逆矩阵,这个过程的计算复杂度至少为关于回合数的平方级别.提出将素描方法应用于假设的更新,给出一个基于素描方法的更高效的在线核回归算法.首先,将损失函数设定为平方损失,应用Nystr?m近似方法来近似核,并借鉴跟导方法(FTL)的思想,提出一个新的梯度下降算...     

基于核学习方法的短时交通流量预测

基于核学习的强大非线性映射性能,针对短时交通流量预测,提出一类基于核学习方法的预测模型。核递推最小二乘(KRLS)基于近似线性依赖(approximate linear dependence,ALD) 技术可降低计算复杂度及存储量,是一种在线核学习方法,适用于较大规模数据集的学习;核偏最小二乘(KPLS)方法将输入变量投影在潜在变量上,利用输入与输出变量之间的协方差信息提取潜在特征;核极限学习机(KELM)方法用核函数表示未知的隐含层非线性特征映射,通过正则化最小二乘算法计算网络的输出权值,能以极快的学习速度获得良好的推广性。为验证所提方法的有效性,将KELM、KPLS、ALD-KRLS用于不同实测交通流数据中,在同等条件下,与现有方法进行比较。实验结果表明,不同核学习方法的预测精度和训练速度均有提高,体现了核学习方法在短时交通流量预测中的应用潜力。     

Bridging deep and multiple kernel learning: A review

Kernel methods and deep learning are two of the most currently remarkable machine learning techniques that have achieved great success in many applications. Kernel methods are powerful tools to capture nonlinear patterns behind data. They implicitly learn high (even infinite) dimensional nonlinear features in the reproducing kernel Hilbert space (RKHS) while making the computation tractable by leveraging the kernel trick. It is commonly agreed that the success of kernel methods is very much dependent on the choice of kernel. Multiple kernel learning (MKL) is one possible scheme that performs kernel combination and selection for a variety of learning tasks, such as classification, clustering, and dimensionality reduction. Deep learning models project input data through several layers of nonlinearity and learn different levels of abstraction. The composition of multiple layers of nonlinear functions can approximate a rich set of naturally occurring input-output dependencies. To bridge kernel methods and deep learning, deep kernel learning has been proven to be an effective method to learn complex feature representations by combining the nonparametric flexibility of kernel methods with the structural properties of deep learning. This article presents a comprehensive overview of the state-of-the-art approaches that bridge the MKL and deep learning techniques. Specifically, we systematically review the typical hybrid models, training techniques, and their theoretical and practical benefits, followed by remaining challenges and future directions. We hope that our perspectives and discussions serve as valuable references for new practitioners and theoreticians seeking to innovate in the applications of the approaches incorporating the advantages of both paradigms and exploring new synergies.     

Wavelet kernel learning

This paper addresses the problem of optimal feature extraction from a wavelet representation. Our work aims at building features by selecting wavelet coefficients resulting from signal or image decomposition on an adapted wavelet basis. For this purpose, we jointly learn in a kernelized large-margin context the wavelet shape as well as the appropriate scale and translation of the wavelets, hence the name “wavelet kernel learning”. This problem is posed as a multiple kernel learning problem, where the number of kernels can be very large. For solving such a problem, we introduce a novel multiple kernel learning algorithm based on active constraints methods. We furthermore propose some variants of this algorithm that can produce approximate solutions more efficiently. Empirical analysis show that our active constraint MKL algorithm achieves state-of-the art efficiency. When used for wavelet kernel learning, our experimental results show that the approaches we propose are competitive with respect to the state-of-the-art on brain–computer interface and Brodatz texture datasets.     

Stochastic subset selection for learning with kernel machines

Kernel machines have gained much popularity in applications of machine learning. Support vector machines (SVMs) are a subset of kernel machines and generalize well for classification, regression, and anomaly detection tasks. The training procedure for traditional SVMs involves solving a quadratic programming (QP) problem. The QP problem scales super linearly in computational effort with the number of training samples and is often used for the offline batch processing of data. Kernel machines operate by retaining a subset of observed data during training. The data vectors contained within this subset are referred to as support vectors (SVs). The work presented in this paper introduces a subset selection method for the use of kernel machines in online, changing environments. Our algorithm works by using a stochastic indexing technique when selecting a subset of SVs when computing the kernel expansion. The work described here is novel because it separates the selection of kernel basis functions from the training algorithm used. The subset selection algorithm presented here can be used in conjunction with any online training technique. It is important for online kernel machines to be computationally efficient due to the real-time requirements of online environments. Our algorithm is an important contribution because it scales linearly with the number of training samples and is compatible with current training techniques. Our algorithm outperforms standard techniques in terms of computational efficiency and provides increased recognition accuracy in our experiments. We provide results from experiments using both simulated and real-world data sets to verify our algorithm.     

核学习自适应预测控制器的在线更新方法

针对非线性过程控制器的设计问题,将基于稀疏核学习的一种具有解析形式的自适应预测控制算法与选择性递推核学习相结合.该在线核学习模型可以通过递推算法进行节点增长和删减的有效更新.因此,所提出的控制器复杂度可控,且能学习过程的时变等特性,从而获得更好的性能.通过一非线性时变过程的仿真研究,验证了所提出的核学习控制器较传统的PID和无在线更新的核学习控制器等具有更好的自适应能力和鲁棒性.     

核参数判别选择方法在核主元分析中的应用

针对核主元分析(KPCA)中高斯核参数β的经验选取问题,提出了核主元分析的核参数判别选择方法。依据训练样本的类标签计算类内、类间核窗宽,在以上核窗宽中经判别选择方法确定核参数。根据判别选择核参数所确定的核矩阵,能够准确描述训练空间的结构特征。用主成分分析(PCA)对特征空间进行分解,提取主成分以实现降维和特征提取。判别核窗宽方法在分类密集区域选择较小窗宽,在分类稀疏区域选择较大窗宽。将判别核主成分分析(Dis-KPCA)应用到数据模拟实例和田纳西过程(TEP),通过与KPCA、PCA方法比较,实验结果表明,Dis-KPCA方法有效地对样本数据降维且将三个类别数据100%分开,因此,所提方法的降维精度更高。     

Multiple kernel learning with gaussianity measures

Kernel methods are known to be effective for nonlinear multivariate analysis. One of the main issues in the practical use of kernel methods is the selection of kernel. There have been a lot of studies on kernel selection and kernel learning. Multiple kernel learning (MKL) is one of the promising kernel optimization approaches. Kernel methods are applied to various classifiers including Fisher discriminant analysis (FDA). FDA gives the Bayes optimal classification axis if the data distribution of each class in the feature space is a gaussian with a shared covariance structure. Based on this fact, an MKL framework based on the notion of gaussianity is proposed. As a concrete implementation, an empirical characteristic function is adopted to measure gaussianity in the feature space associated with a convex combination of kernel functions, and two MKL algorithms are derived. From experimental results on some data sets, we show that the proposed kernel learning followed by FDA offers strong classification power.     

基于贪心核特征提取方法的中期峰值负荷预测

针对中期电力负荷预测, 提出基于贪心核主元回归(GKPCR)、贪心核岭回归(GKRR) 的特征提取建模方法. 通过对核矩阵的稀疏逼近, GKPCR和GKRR两种贪心核特征提取方法旨在寻找特征空间中数据的低维表示, 计算需求低, 适用于大数据集的在线学习. 将所提出的方法应用于不同地区的电力负荷中期峰值预测, 并与现有预测方法进行了比较. 实验结果表明, 在同等条件下, 所提出的方法能有效地改进预测精度, 而且性能更好, 显示了其有效性和应用潜力.

     

基于Seed集的半监督核聚类

提出了一种新的半监督核聚类算法——SKK-均值算法。算法利用一定数量的标记样本构成seed集,作为监督信息来初始化K-均值算法的聚类中心,引导聚类过程并约束数据划分;同时还采用了核方法把输入数据映射到高维特征空间,并用核函数来实现样本之间的距离计算。在UCI数据集上进行了数值实验,并与K-均值算法和核-K-均值算法进行了比较。     

基于GSA与DE优化混合核ELM的网络异常检测模型

为了增强网络入侵检测模型的准确率与泛化性,提出一种基于引力搜索算法（GSA）与差分进化（DE）算法优化混合核极限学习机（ELM）的网络入侵检测模型。该模型针对采用单个核函数的ELM模型存在的泛化能力弱、学习能力差的问题,结合多项式核函数和径向基函数的优点,构建混合核ELM模型（HKELM）,将GSA和DE相结合优化HKELM模型参数,从而提高其在异常检测过程中的全局和局部优化能力,在此基础上利用核主成分分析算法进行入侵检测数据的数据降维和特征抽取,构建网络入侵检测模型KPCA-GSADE-HKELM。在KDD99数据集上的实验结果表明,与KDDwinner、CSVAC、CPSO-SVM、Dendron等模型进行对比,KPCA-GSADE-HKELM模型具有更高的检测精度和更快的检测速度。     

Fast kernel Fisher discriminant analysis via approximating the kernel principal component analysis

Kernel Fisher discriminant analysis (KFDA) extracts a nonlinear feature from a sample by calculating as many kernel functions as the training samples. Thus, its computational efficiency is inversely proportional to the size of the training sample set. In this paper we propose a more approach to efficient nonlinear feature extraction, FKFDA (fast KFDA). This FKFDA consists of two parts. First, we select a portion of training samples based on two criteria produced by approximating the kernel principal component analysis (AKPCA) in the kernel feature space. Then, referring to the selected training samples as nodes, we formulate FKFDA to improve the efficiency of nonlinear feature extraction. In FKFDA, the discriminant vectors are expressed as linear combinations of nodes in the kernel feature space, and the extraction of a feature from a sample only requires calculating as many kernel functions as the nodes. Therefore, the proposed FKFDA has a much faster feature extraction procedure compared with the naive kernel-based methods. Experimental results on face recognition and benchmark datasets classification suggest that the proposed FKFDA can generate well classified features.     

A sparse kernel algorithm for online time series data prediction

Kernel based methods have been widely applied for signal analysis and processing. In this paper, we propose a sparse kernel based algorithm for online time series prediction. In classical kernel methods, the kernel function number is very large which makes them of a high computational cost and only applicable for off-line or batch learning. In online learning settings, the learning system is updated when each training sample is obtained and it requires a higher computational speed. To make the kernel methods suitable for online learning, we propose a sparsification method based on the Hessian matrix of the system loss function to continuously examine the significance of the new training sample in order to select a sparse dictionary (support vector set). The Hessian matrix is equivalent to the correlation matrix of sample inputs in the kernel weight updating using the recursive least square (RLS) algorithm. This makes the algorithm able to be easily implemented with an affordable computational cost for real-time applications. Experimental results show the ability of the proposed algorithm for both real-world and artificial time series data forecasting and prediction.     

Accelerated max-margin multiple kernel learning

Kernel machines such as Support Vector Machines (SVM) have exhibited successful performance in pattern classification problems mainly due to their exploitation of potentially nonlinear affinity structures of data through the kernel functions. Hence, selecting an appropriate kernel function, equivalently learning the kernel parameters accurately, has a crucial impact on the classification performance of the kernel machines. In this paper we consider the problem of learning a kernel matrix in a binary classification setup, where the hypothesis kernel family is represented as a convex hull of fixed basis kernels. While many existing approaches involve computationally intensive quadratic or semi-definite optimization, we propose novel kernel learning algorithms based on large margin estimation of Parzen window classifiers. The optimization is cast as instances of linear programming. This significantly reduces the complexity of the kernel learning compared to existing methods, while our large margin based formulation provides tight upper bounds on the generalization error. We empirically demonstrate that the new kernel learning methods maintain or improve the accuracy of the existing classification algorithms while significantly reducing the learning time on many real datasets in both supervised and semi-supervised settings.     

核函数的选择研究综述

核方法是解决非线性模式分析问题的一种有效方法,是当前机器学习领域的一个研究热点.核函数是影响核方法性能的关键因素,以支持向量机作为核函数的载体,从核函数的构造、核函数中参数的选择、多核学习3个角度对核函数的选择的研究现状及其进展情况进行了系统地概述,并指出根据特定应用领域选择核函数、设计有效的核函数度量标准和拓宽核函数选择的研究范围是其中3个值得进一步研究的方向.     

Interval type-2 fuzzy kernel based support vector machine algorithm for scene classification of humanoid robot

This paper proposed an Interval Type-2 Fuzzy Kernel based Support Vector Machine (IT2FK-SVM) for scene classification of humanoid robot. Type-2 fuzzy sets have been shown to be a more promising method to manifest the uncertainties. Kernel design is a key component for many kernel-based methods. By integrating the kernel design with type-2 fuzzy sets, a systematic design methodology of IT2FK-SVM classification for scene images is presented to improve robustness and selectivity in the humanoid robot vision, which involves feature extraction, dimensionality reduction and classifier learning. Firstly, scene images are represented as high dimensional vector extracted from intensity, edge and orientation feature maps by biological-vision feature extraction method. Furthermore, a novel three-domain Fuzzy Kernel-based Principal Component Analysis (3DFK-PCA) method is proposed to select the prominent variables from the high-dimensional scene image representation. Finally, an IT2FM SVM classifier is developed for the comprehensive learning of scene images in complex environment. Different noisy, different view angle, and variations in lighting condition can be taken as the uncertainties in scene images. Compare to the traditional SVM classifier with RBF kernel, MLP kernel, and the Weighted Kernel (WK), respectively, the proposed method performs much better than conventional WK method due to its integration of IT2FK, and WK method performs better than the single kernel methods (SVM classifier with RBF kernel or MLP kernel). IT2FK-SVM is able to deal with uncertainties when scene images are corrupted by various noises and captured by different view angles. The proposed IT2FK-SVM method yields over $92~\% $ classification rates for all cases. Moreover, it even achieves $98~\% $ classification rate on the newly built dataset with common light case.     

Model-based transductive learning of the kernel matrix

This paper addresses the problem of transductive learning of the kernel matrix from a probabilistic perspective. We define the kernel matrix as a Wishart process prior and construct a hierarchical generative model for kernel matrix learning. Specifically, we consider the target kernel matrix as a random matrix following the Wishart distribution with a positive definite parameter matrix and a degree of freedom. This parameter matrix, in turn, has the inverted Wishart distribution (with a positive definite hyperparameter matrix) as its conjugate prior and the degree of freedom is equal to the dimensionality of the feature space induced by the target kernel. Resorting to a missing data problem, we devise an expectation-maximization (EM) algorithm to infer the missing data, parameter matrix and feature dimensionality in a maximum a posteriori (MAP) manner. Using different settings for the target kernel and hyperparameter matrices, our model can be applied to different types of learning problems. In particular, we consider its application in a semi-supervised learning setting and present two classification methods. Classification experiments are reported on some benchmark data sets with encouraging results. In addition, we also devise the EM algorithm for kernel matrix completion. Editor: Philip M. Long     

基于流形距离的半监督判别分析

大量无类别标签的数据具有对分类有用的信息,有效地利用这些信息来提高分类精确度,是半监督分类研究的主要内容.提出了一种基于流形距离的半监督判别分析(semi-supervised discriminant analysis based on manifold distance,简称SSDA)算法,通过定义的流形距离,能够选择位于流形上的数据点的同类近邻点、异类近邻点以及全局近邻点,并依据流形距离定义数据点与其各近邻点之间的相似度,利用这种相似度度量构造算法的目标函数.通过在ORL,YALE人脸数据库上的实验表明,与现有算法相比,数据集通过该算法降维后,能够使基于距离的识别算法具有更高的分类精确度.同时,为了解决非线性降维问题,提出了Kernel SSDA,同样通过实验验证了算法的有效性.     

基于RGB-D图像核描述子的物体识别方法

针对传统的颜色-深度（RGB-D）图像物体识别的方法所存在的图像特征学习不全面、特征编码鲁棒性不够等问题,提出了基于核描述子局部约束线性编码（KD-LLC）的RGB-D图像物体识别方法。首先,在图像块间匹配核函数基础上,应用核主成分分析法提取RGB-D图像的3D形状、尺寸、边缘、颜色等多个互补性核描述子;然后,分别对它们进行LLC编码及空间池化处理以形成相应的图像编码向量;最后,把这些图像编码向量融合成具有鲁棒性、区分性的图像表示。基于RGB-D数据集的仿真实验结果表明,作为一种基于人工设计特征的RGB-D图像物体识别方法,由于所提算法综合利用深度图像和RGB图像的多方面特征,而且对传统深度核描述子的采样点选取和紧凑基向量的计算这两方面进行了改进,使得物体类别识别率达到86.8%,实体识别率达到92.7%,比其他同类方法具有更高的识别准确率。