Learning low-rank kernel matrices for constrained clustering

Constrained clustering methods (that usually use must-link and/or cannot-link constraints) have been received much attention in the last decade. Recently, kernel adaptation or kernel learning has been considered as a powerful approach for constrained clustering. However, these methods usually either allow only special forms of kernels or learn non-parametric kernel matrices and scale very poorly. Therefore, they either learn a metric that has low flexibility or are applicable only on small data sets due to their high computational complexity. In this paper, we propose a more efficient non-linear metric learning method that learns a low-rank kernel matrix from must-link and cannot-link constraints and the topological structure of data. We formulate the proposed method as a trace ratio optimization problem and learn appropriate distance metrics through finding optimal low-rank kernel matrices. We solve the proposed optimization problem much more efficiently than SDP solvers. Additionally, we show that the spectral clustering methods can be considered as a special form of low-rank kernel learning methods. Extensive experiments have demonstrated the superiority of the proposed method compared to recently introduced kernel learning methods.     

马氏距离多核支持向量机学习模型

支持向量机是统计机器学习中的一种重要方法,被广泛地应用于模式识别、回归分析等问题。但一般支持向量机未考虑样本的总体分布,降低了支持向量机的泛化能力。针对该问题,提出一种马氏距离支持向量机学习模型,考虑总体样本的分布,并将该模型扩展到多核学习模型。通过数学方法将欧式距离核矩阵转化为马氏距离核矩阵,降低模型的实现难度。实验结果证明,该模型不仅保持了欧式距离多核学习模型的原有性质,且具有更好的分类精确度。     

混杂数据的多核几何平均度量学习

在机器学习和模式识别任务中,选择一种合适的距离度量方法是至关重要的.度量学习主要利用判别性信息学习一个马氏距离或相似性度量.然而,大多数现有的度量学习方法都是针对数值型数据的,对于一些有结构的数据（比如符号型数据）,用传统的距离度量来度量两个对象之间的相似性是不合理的;其次,大多数度量学习方法会受到维度的困扰,高维度使得训练时间长,模型的可扩展性差.提出了一种基于几何平均的混杂数据度量学习方法.采用不同的核函数将数值型数据和符号型数据分别映射到可再生核希尔伯特空间,从而避免了特征的高维度带来的负面影响.同时,提出了一个基于几何平均的多核度量学习模型,将混杂数据的度量学习问题转化为求黎曼流形上两个点的中心点问题.在UCI数据集上的实验结果表明,针对混杂数据的多核度量学习方法与现有的度量学习方法相比,在准确性方面展现出更优异的性能.     

A Method for Metric Learning with Multiple-Kernel Embedding

Distance metric learning is rather important for measuring the similarity (/dissimilarity) of two instances in many pattern recognition algorithms. Although many linear Mahalanobis metric learning methods can be extended to their kernelized versions for dealing with the nonlinear structure data, choosing the proper kernel and determining the kernel parameters are still tough problems. Furthermore, the single kernel embedded metric is not suited for the problems with multi-view feature representations. In this paper, we address the problem of metric learning with multiple kernels embedding. By analyzing the existing formulations of metric learning with multiple-kernel embedding, we propose a new framework to learn multi-metrics as well as the corresponding weights jointly, the objective function can be shown to be convex and it can be converted to be a multiple kernel learning-support vector machine problem, which can be solved by existing methods. The experiments on single-view and multi-view data show the effectiveness of our method.     

Accelerated max-margin multiple kernel learning

Kernel machines such as Support Vector Machines (SVM) have exhibited successful performance in pattern classification problems mainly due to their exploitation of potentially nonlinear affinity structures of data through the kernel functions. Hence, selecting an appropriate kernel function, equivalently learning the kernel parameters accurately, has a crucial impact on the classification performance of the kernel machines. In this paper we consider the problem of learning a kernel matrix in a binary classification setup, where the hypothesis kernel family is represented as a convex hull of fixed basis kernels. While many existing approaches involve computationally intensive quadratic or semi-definite optimization, we propose novel kernel learning algorithms based on large margin estimation of Parzen window classifiers. The optimization is cast as instances of linear programming. This significantly reduces the complexity of the kernel learning compared to existing methods, while our large margin based formulation provides tight upper bounds on the generalization error. We empirically demonstrate that the new kernel learning methods maintain or improve the accuracy of the existing classification algorithms while significantly reducing the learning time on many real datasets in both supervised and semi-supervised settings.     

Document Classification via Nonlinear Metric Learning

Learning a proper distance metric is an important problem in document classification, because the similarities of samples in many problems are usually measured by distance metric. In this paper, we address the nonlinear metric leaning problem with applying in the document classification. First, we propose a new representation about nonlinear metric by using a linear combination of some basic kernels. Second, we give a linear metric learning method by a triplet constraint and k-nearest neighbors, and then we develop it to a nonlinear method based on multiple kernel by above nonlinear metric. Further, the corresponding problem can be rewritten as an unconstrained optimization problem on positive definite matrices groups. At last, to ensure the learned distance matrix must be a positive definite matrix, we provide an improved intrinsic steepest descent algorithm with adaptive step-size to solve this unconstrained optimization. The experimental results show that our proposed method is effective on some document classification problems.     

Low-Rank Bilinear Classification: Efficient Convex Optimization and Extensions

In pattern classification, it is needed to efficiently treat not only feature vectors but also feature matrices defined as two-way data, while preserving the two-way structure such as spatio-temporal relationships. The classifier for the feature matrix is generally formulated in a bilinear form composed of row and column weights which jointly result in a matrix weight. The rank of the matrix should be low from the viewpoint of generalization performance and computational cost. For that purpose, we propose a low-rank bilinear classifier based on the efficient convex optimization. In the proposed method, the classifier is optimized by minimizing the trace norm of the classifier (matrix) to reduce the rank without any hard constraint on it. We formulate the optimization problem in a tractable convex form and provide the procedure to solve it efficiently with the global optimum. In addition, we propose two novel extensions of the bilinear classifier in terms of multiple kernel learning and cross-modal learning. Through kernelizing the bilinear method, we naturally induce a novel multiple kernel learning. The method integrates both the inter kernels between heterogeneous reproducing kernel Hilbert spaces (RKHSs) and the ordinary kernels within respective RKHSs into a new discriminative kernel in a unified manner using the bilinear model. Besides, for cross-modal learning, we consider to map into the common space the multi-modal features which are subsequently classified in that space. We show that the projection and the classification are jointly represented by the bilinear model, and then propose the method to optimize both of them simultaneously in the bilinear framework. In the experiments on various visual classification tasks, the proposed methods exhibit favorable performances compared to the other methods.     

Some sets of orthogonal polynomial kernel functions

Kernel methods provide high performance in a variety of machine learning tasks. However, the success of kernel methods is heavily dependent on the selection of the right kernel function and proper setting of its parameters. Several sets of kernel functions based on orthogonal polynomials have been proposed recently. Besides their good performance in the error rate, these kernel functions have only one parameter chosen from a small set of integers, and it facilitates kernel selection greatly. Two sets of orthogonal polynomial kernel functions, namely the triangularly modified Chebyshev kernels and the triangularly modified Legendre kernels, are proposed in this study. Furthermore, we compare the construction methods of some orthogonal polynomial kernels and highlight the similarities and differences among them. Experiments on 32 data sets are performed for better illustration and comparison of these kernel functions in classification and regression scenarios. In general, there is difference among these orthogonal polynomial kernels in terms of accuracy, and most orthogonal polynomial kernels can match the commonly used kernels, such as the polynomial kernel, the Gaussian kernel and the wavelet kernel. Compared with these universal kernels, the orthogonal polynomial kernels each have a unique easily optimized parameter, and they store statistically significantly less support vectors in support vector classification. New presented kernels can obtain better generalization performance both for classification tasks and regression tasks.     

Alignment based kernel learning with a continuous set of base kernels

The success of kernel-based learning methods depends on the choice of kernel. Recently, kernel learning methods have been proposed that use data to select the most appropriate kernel, usually by combining a set of base kernels. We introduce a new algorithm for kernel learning that combines a continuous set of base kernels, without the common step of discretizing the space of base kernels. We demonstrate that our new method achieves state-of-the-art performance across a variety of real-world datasets. Furthermore, we explicitly demonstrate the importance of combining the right dictionary of kernels, which is problematic for methods that combine a finite set of base kernels chosen a priori. Our method is not the first approach to work with continuously parameterized kernels. We adopt a two-stage kernel learning approach. We also show that our method requires substantially less computation than previous such approaches, and so is more amenable to multi-dimensional parameterizations of base kernels, which we demonstrate.     

Model-based transductive learning of the kernel matrix

This paper addresses the problem of transductive learning of the kernel matrix from a probabilistic perspective. We define the kernel matrix as a Wishart process prior and construct a hierarchical generative model for kernel matrix learning. Specifically, we consider the target kernel matrix as a random matrix following the Wishart distribution with a positive definite parameter matrix and a degree of freedom. This parameter matrix, in turn, has the inverted Wishart distribution (with a positive definite hyperparameter matrix) as its conjugate prior and the degree of freedom is equal to the dimensionality of the feature space induced by the target kernel. Resorting to a missing data problem, we devise an expectation-maximization (EM) algorithm to infer the missing data, parameter matrix and feature dimensionality in a maximum a posteriori (MAP) manner. Using different settings for the target kernel and hyperparameter matrices, our model can be applied to different types of learning problems. In particular, we consider its application in a semi-supervised learning setting and present two classification methods. Classification experiments are reported on some benchmark data sets with encouraging results. In addition, we also devise the EM algorithm for kernel matrix completion. Editor: Philip M. Long     

A Kernel Approach for Semisupervised Metric Learning

While distance function learning for supervised learning tasks has a long history, extending it to learning tasks with weaker supervisory information has only been studied recently. In particular, some methods have been proposed for semisupervised metric learning based on pairwise similarity or dissimilarity information. In this paper, we propose a kernel approach for semisupervised metric learning and present in detail two special cases of this kernel approach. The metric learning problem is thus formulated as an optimization problem for kernel learning. An attractive property of the optimization problem is that it is convex and, hence, has no local optima. While a closed-form solution exists for the first special case, the second case is solved using an iterative majorization procedure to estimate the optimal solution asymptotically. Experimental results based on both synthetic and real-world data show that this new kernel approach is promising for nonlinear metric learning     

Multiple graph kernel learning based on GMDH-type neural network

Multiple kernel learning (MKL), as a principled classification method, selects and combines base kernels to increase the categorization accuracy of Support Vector Machines (SVMs). The group method of data handling neural network (GMDH-NN) has been applied in many fields of optimization, data mining, and pattern recognition. It can automatically seek interrelatedness in data, select an optimal structure for the model or network, and enhance the accuracy of existing algorithms. We can utilize the advantages of the GMDH-NN to build a multiple graph kernel learning (MGKL) method and enhance the categorization performance of graph kernel SVMs. In this paper, we first define a unitized symmetric regularity criterion (USRC) to improve the symmetric regularity criterion of GMDH-NN. Second, a novel structure for the initial model of the GMDH-NN is defined, which uses the posterior probability output of graph kernel SVMs. We then use a hybrid graph kernel in the $H^1$-space for MGKL in combination with the GMDH-NN. This way, we can obtain a pool of optimal graph kernels with different kernel parameters. Our experiments on standard graph datasets show that this new MGKL method is highly effective.     

权重马氏距离高斯核在谱分割中的应用

为了使经典谱分割的Nystrm采样快速算法得到更清晰的结果,将权重马氏距离高斯核应用于其中,相对于常用的马氏距离高斯核,得到了更好的分割效果。结果表明,使用权重马氏距离高斯核更能准确的反映两个向量的相似度,从而实现准确的分割。     

Wavelet kernel learning

This paper addresses the problem of optimal feature extraction from a wavelet representation. Our work aims at building features by selecting wavelet coefficients resulting from signal or image decomposition on an adapted wavelet basis. For this purpose, we jointly learn in a kernelized large-margin context the wavelet shape as well as the appropriate scale and translation of the wavelets, hence the name “wavelet kernel learning”. This problem is posed as a multiple kernel learning problem, where the number of kernels can be very large. For solving such a problem, we introduce a novel multiple kernel learning algorithm based on active constraints methods. We furthermore propose some variants of this algorithm that can produce approximate solutions more efficiently. Empirical analysis show that our active constraint MKL algorithm achieves state-of-the art efficiency. When used for wavelet kernel learning, our experimental results show that the approaches we propose are competitive with respect to the state-of-the-art on brain–computer interface and Brodatz texture datasets.     

基于最大间隔理论的组合距离学习算法

从已知数据集中学习距离度量在许多机器学习应用中都起着重要作用。传统的距离学习方法通常假定目标距离函数为马氏距离的形式,这使得学习出的距离度量在应用上具有局限性。提出了一种新的距离学习方法,将目标距离函数表示为若干候选距离的线性组合,依据最大间隔理论利用数据集的边信息学习得到组合距离中各距离分量的权值,从而得到新的距离度量。通过该距离度量在模糊C均值聚类算法中的表现来对其进行评价。在UCI数据集上,与其他已有的距离学习算法的对比实验结果证明了该文算法的有效性。     

独立分量分析结合马氏距离的非监督损伤识别方法

为了解决结构损伤识别中监督学习方法在实际中难以获得损伤样本的限制,提出基于独立分量分析ICA( IndependentComponent Analysis)结合马氏距离判断结构损伤的方法.首先采用ICA方法提取统计独立源信号和混合矩阵,将混合矩阵作为特征参数输入至马氏距离判别函数,然后根据结构健康状态的马氏距离设计门限值,该门限值与检测信号的马氏距离的比较结果作为损伤判断的依据.在冲击载荷作用下,对钢框架结构模型进行了振动实验,结果表明:ICA方法提取的混合矩阵是一种有效的损伤特征参数,基于ICA和马氏距离的非监督学习方法能够正确识别结构损伤,从而为结构健康监测提供了一种行之有效的损伤识别方法.     

A kernel learning framework for domain adaptation learning

Domain adaptation learning(DAL) methods have shown promising results by utilizing labeled samples from the source(or auxiliary) domain(s) to learn a robust classifier for the target domain which has a few or even no labeled samples.However,there exist several key issues which need to be addressed in the state-of-theart DAL methods such as sufficient and effective distribution discrepancy metric learning,effective kernel space learning,and multiple source domains transfer learning,etc.Aiming at the mentioned-above issues,in this paper,we propose a unified kernel learning framework for domain adaptation learning and its effective extension based on multiple kernel learning(MKL) schema,regularized by the proposed new minimum distribution distance metric criterion which minimizes both the distribution mean discrepancy and the distribution scatter discrepancy between source and target domains,into which many existing kernel methods(like support vector machine(SVM),v-SVM,and least-square SVM) can be readily incorporated.Our framework,referred to as kernel learning for domain adaptation learning(KLDAL),simultaneously learns an optimal kernel space and a robust classifier by minimizing both the structural risk functional and the distribution discrepancy between different domains.Moreover,we extend the framework KLDAL to multiple kernel learning framework referred to as MKLDAL.Under the KLDAL or MKLDAL framework,we also propose three effective formulations called KLDAL-SVM or MKLDAL-SVM with respect to SVM and its variant μ-KLDALSVM or μ-MKLDALSVM with respect to v-SVM,and KLDAL-LSSVM or MKLDAL-LSSVM with respect to the least-square SVM,respectively.Comprehensive experiments on real-world data sets verify the outperformed or comparable effectiveness of the proposed frameworks.     

Invariant kernel functions for pattern analysis and machine learning

In many learning problems prior knowledge about pattern variations can be formalized and beneficially incorporated into the analysis system. The corresponding notion of invariance is commonly used in conceptionally different ways. We propose a more distinguishing treatment in particular in the active field of kernel methods for machine learning and pattern analysis. Additionally, the fundamental relation of invariant kernels and traditional invariant pattern analysis by means of invariant representations will be clarified. After addressing these conceptional questions, we focus on practical aspects and present two generic approaches for constructing invariant kernels. The first approach is based on a technique called invariant integration. The second approach builds on invariant distances. In principle, our approaches support general transformations in particular covering discrete and non-group or even an infinite number of pattern-transformations. Additionally, both enable a smooth interpolation between invariant and non-invariant pattern analysis, i.e. they are a covering general framework. The wide applicability and various possible benefits of invariant kernels are demonstrated in different kernel methods. Editor: Phil Long.     

基于Boosting框架的非稀疏多核学习方法

针对传统的分类器集成的每次迭代通常是将单个最优个体分类器集成到强分类器中,而其它可能有辅助作用的个体分类器被简单抛弃的问题,提出了一种基于Boosting框架的非稀疏多核学习方法MKL-Boost,利用了分类器集成学习的思想,每次迭代时,首先从训练集中选取一个训练子集,然后利用正则化非稀疏多核学习方法训练最优个体分类器,求得的个体分类器考虑了M个基本核的最优非稀疏线性凸组合,通过对核组合系数施加LP范数约束,一些好的核得以保留,从而保留了更多的有用特征信息,差的核将会被去掉,保证了有选择性的核融合,然后将基于核组合的最优个体分类器集成到强分类器中。提出的算法既具有Boosting集成学习的优点,同时具有正则化非稀疏多核学习的优点,实验表明,相对于其它Boosting算法,MKL-Boost可以在较少的迭代次数内获得较高的分类精度。     

基于中心核对齐的多核单类支持向量机

多核学习(MKL)方法在分类及回归任务中均取得了优于单核学习方法的性能,但传统的MKL方法均用于处理两类或多类分类问题.为了使MKL方法适用于处理单类分类(OCC)问题,提出了基于中心核对齐(CKA)的单类支持向量机(OCSVM).首先利用CKA计算每个核矩阵的权重,然后将所得权重用作线性组合系数,进而将不同类型的核函...