A Method for Metric Learning with Multiple-Kernel Embedding

Distance metric learning is rather important for measuring the similarity (/dissimilarity) of two instances in many pattern recognition algorithms. Although many linear Mahalanobis metric learning methods can be extended to their kernelized versions for dealing with the nonlinear structure data, choosing the proper kernel and determining the kernel parameters are still tough problems. Furthermore, the single kernel embedded metric is not suited for the problems with multi-view feature representations. In this paper, we address the problem of metric learning with multiple kernels embedding. By analyzing the existing formulations of metric learning with multiple-kernel embedding, we propose a new framework to learn multi-metrics as well as the corresponding weights jointly, the objective function can be shown to be convex and it can be converted to be a multiple kernel learning-support vector machine problem, which can be solved by existing methods. The experiments on single-view and multi-view data show the effectiveness of our method.     

Distance metric learning-based kernel gram matrix learning for pattern analysis tasks in kernel feature space

Approaches to distance metric learning (DML) for Mahalanobis distance metric involve estimating a parametric matrix that is associated with a linear transformation. For complex pattern analysis tasks, it is necessary to consider the approaches to DML that involve estimating a parametric matrix that is associated with a nonlinear transformation. One such approach involves performing the DML of Mahalanobis distance in the feature space of a Mercer kernel. In this approach, the problem of estimation of a parametric matrix of Mahalanobis distance is formulated as a problem of learning an optimal kernel gram matrix from the kernel gram matrix of a base kernel by minimizing the logdet divergence between the kernel gram matrices. We propose to use the optimal kernel gram matrices learnt from the kernel gram matrix of the base kernels in pattern analysis tasks such as clustering, multi-class pattern classification and nonlinear principal component analysis. We consider the commonly used kernels such as linear kernel, polynomial kernel, radial basis function kernel and exponential kernel as well as hyper-ellipsoidal kernels as the base kernels for optimal kernel learning. We study the performance of the DML-based class-specific kernels for multi-class pattern classification using support vector machines. Results of our experimental studies on benchmark datasets demonstrate the effectiveness of the DML-based kernels for different pattern analysis tasks.     

Wavelet kernel learning

This paper addresses the problem of optimal feature extraction from a wavelet representation. Our work aims at building features by selecting wavelet coefficients resulting from signal or image decomposition on an adapted wavelet basis. For this purpose, we jointly learn in a kernelized large-margin context the wavelet shape as well as the appropriate scale and translation of the wavelets, hence the name “wavelet kernel learning”. This problem is posed as a multiple kernel learning problem, where the number of kernels can be very large. For solving such a problem, we introduce a novel multiple kernel learning algorithm based on active constraints methods. We furthermore propose some variants of this algorithm that can produce approximate solutions more efficiently. Empirical analysis show that our active constraint MKL algorithm achieves state-of-the art efficiency. When used for wavelet kernel learning, our experimental results show that the approaches we propose are competitive with respect to the state-of-the-art on brain–computer interface and Brodatz texture datasets.     

用于文本分类的多核SVM算法研究

根据文本分类通常包含多异类数据源的特点，提出了多核SVM学习算法。该算法将分类核矩阵的二次组合重新表述成半无限规划，并说明其可以通过重复利用SVM来实现有效求解。实验结果表明，提出的算法可以用于数百个核的结合或者是数十万个样本的结合，对于多异类数据源的文本分类具有较高的查全率和查准率。     

Image category learning and classification via optimal linear combination of multiple partially matching kernels

Multiple kernel learning (MKL) aims at simultaneously optimizing kernel weights while training the support vector machine (SVM) to get satisfactory classification or regression results. Recent publications and developments based on SVM have shown that by using MKL one can enhance interpretability of the decision function and improve classifier performance, which motivates researchers to explore the use of homogeneous model obtained as linear combination of various types of kernels. In this paper, we show that MKL problems can be solved efficiently by modified projection gradient method and applied for image categorization and object detection. The kernel is defined as a linear combination of feature histogram function that can measure the degree of similarity of partial correspondence between feature sets for discriminative classification, which allows recognition robust to within-class variation, pose changes, and articulation. We evaluate our proposed framework on the ETH-80 dataset for several multi-level image encodings for supervised and unsupervised object recognition and report competitive results.     

基于Boosting框架的非稀疏多核学习方法

针对传统的分类器集成的每次迭代通常是将单个最优个体分类器集成到强分类器中,而其它可能有辅助作用的个体分类器被简单抛弃的问题,提出了一种基于Boosting框架的非稀疏多核学习方法MKL-Boost,利用了分类器集成学习的思想,每次迭代时,首先从训练集中选取一个训练子集,然后利用正则化非稀疏多核学习方法训练最优个体分类器,求得的个体分类器考虑了M个基本核的最优非稀疏线性凸组合,通过对核组合系数施加LP范数约束,一些好的核得以保留,从而保留了更多的有用特征信息,差的核将会被去掉,保证了有选择性的核融合,然后将基于核组合的最优个体分类器集成到强分类器中。提出的算法既具有Boosting集成学习的优点,同时具有正则化非稀疏多核学习的优点,实验表明,相对于其它Boosting算法,MKL-Boost可以在较少的迭代次数内获得较高的分类精度。     

Accelerated max-margin multiple kernel learning

Kernel machines such as Support Vector Machines (SVM) have exhibited successful performance in pattern classification problems mainly due to their exploitation of potentially nonlinear affinity structures of data through the kernel functions. Hence, selecting an appropriate kernel function, equivalently learning the kernel parameters accurately, has a crucial impact on the classification performance of the kernel machines. In this paper we consider the problem of learning a kernel matrix in a binary classification setup, where the hypothesis kernel family is represented as a convex hull of fixed basis kernels. While many existing approaches involve computationally intensive quadratic or semi-definite optimization, we propose novel kernel learning algorithms based on large margin estimation of Parzen window classifiers. The optimization is cast as instances of linear programming. This significantly reduces the complexity of the kernel learning compared to existing methods, while our large margin based formulation provides tight upper bounds on the generalization error. We empirically demonstrate that the new kernel learning methods maintain or improve the accuracy of the existing classification algorithms while significantly reducing the learning time on many real datasets in both supervised and semi-supervised settings.     

Improving Semantic Concept Detection Through Optimizing Ranking Function

In this paper, a kernel-based learning algorithm, kernel rank, is presented for improving the performance of semantic concept detection. By designing a classifier optimizing the receiver operating characteristic (ROC) curve using kernel rank, we provide a generic framework to optimize any differentiable ranking function using effective smoothing functions. kernel rank directly maximizes a 1-D quality measure of ROC, i.e., AUC (area under the ROC). It exploits the kernel density estimation to model the ranking score distributions and approximate the correct ranking count. The ranking metric is then derived and the learnable parameters are naturally embedded. To address the issues of computation and memory in learning, an efficient implementation is developed based on the gradient descent algorithm. We apply kernel rank with two types of kernel density functions to train the linear discriminant function and the Gaussian mixture model classifiers. From our experiments carried out on the development set for TREC Video Retrieval 2005, we conclude that (1) kernel rank is capable of training any differentiable classifier with various kernels; and (2) the learned ranking function performs better than traditional maximization likelihood or classification error minimization based algorithms in terms of AUC and average precision (AP).     

面向高光谱图像分类的内容引导卷积深度网络并行实现

目的 受限于卷积核形状固定,传统卷积神经网络（convolutional neural network,CNN）方法难以精确分类高光谱图像（hyperspectral image,HSI）中的跨类别边缘区域,导致地物边界模糊。内容引导CNN （content-guided CNN,CGCNN）能够根据地物形态自适应调整卷积核形状,具有地物边缘保持分类能力。但由于内容引导卷积属于非固定模板结构,不能直接调用现有深度学习加速库实现并行计算。针对该问题,本文设计了一种内容引导卷积的并行计算方法,并验证其加速及分类性能。方法 本文基于内容引导卷积等价于各向异性核加权和标准卷积的组合结构,通过利用深度学习库中的平铺、堆叠、网格和采样等底层函数构造索引矩阵来定义重采样方式,以将内容引导卷积分解为与空间位置无关的像素级独立计算过程,并在图形处理器（graphics processing unit,GPU）上并行执行。结果 经测试,本文提出的并行化内容引导卷积相比串行运算方式平均提速近700倍。在分类性能测试中,并行化CGCNN在合成数据集上表现出优异的细节保持分类能力,总精度平均高于对比方法7.10%;同时在两组真实数据集上亦取得最优分类结果,分别高于对比方法7.21%、2.70%。结论 通过将内容引导卷积分步拆解,能够将其转化为一系列并行计算过程,且能够在GPU上高效执行;并通过在多组数据集上的分类精度、参数敏感度和小样本学习等综合性能测试进一步表明,并行化CGCNN在具有优良分类性能的同时,亦具有对不同地物的边缘保持分类能力,能够获得更精细的分类结果。     

Multi-metric learning for multi-sensor fusion based classification

In this paper, we propose a multiple-metric learning algorithm to learn jointly a set of optimal homogenous/heterogeneous metrics in order to fuse the data collected from multiple sensors for joint classification. The learned metrics have the potential to perform better than the conventional Euclidean metric for classification. Moreover, in the case of heterogenous sensors, the learned multiple metrics can be quite different, which are adapted to each type of sensor. By learning the multiple metrics jointly within a single unified optimization framework, we can learn better metrics to fuse the multi-sensor data for a joint classification. Furthermore, we also exploit multi-metric learning in a kernel induced feature space to capture the non-linearity in the original feature space via kernel mapping.     

Multiple-kernel-learning-based extreme learning machine for classification design

The extreme learning machine (ELM) is a new method for using single hidden layer feed-forward networks with a much simpler training method. While conventional kernel-based classifiers are based on a single kernel, in reality, it is often desirable to base classifiers on combinations of multiple kernels. In this paper, we propose the issue of multiple-kernel learning (MKL) for ELM by formulating it as a semi-infinite linear programming. We further extend this idea by integrating with techniques of MKL. The kernel function in this ELM formulation no longer needs to be fixed, but can be automatically learned as a combination of multiple kernels. Two formulations of multiple-kernel classifiers are proposed. The first one is based on a convex combination of the given base kernels, while the second one uses a convex combination of the so-called equivalent kernels. Empirically, the second formulation is particularly competitive. Experiments on a large number of both toy and real-world data sets (including high-magnification sampling rate image data set) show that the resultant classifier is fast and accurate and can also be easily trained by simply changing linear program.     

Speaker recognition using pyramid match kernel based support vector machines

Gaussian mixture model (GMM) based approaches have been commonly used for speaker recognition tasks. Methods for estimation of parameters of GMMs include the expectation-maximization method which is a non-discriminative learning based method. Discriminative classifier based approaches to speaker recognition include support vector machine (SVM) based classifiers using dynamic kernels such as generalized linear discriminant sequence kernel, probabilistic sequence kernel, GMM supervector kernel, GMM-UBM mean interval kernel (GUMI) and intermediate matching kernel. Recently, the pyramid match kernel (PMK) using grids in the feature space as histogram bins and vocabulary-guided PMK (VGPMK) using clusters in the feature space as histogram bins have been proposed for recognition of objects in an image represented as a set of local feature vectors. In PMK, a set of feature vectors is mapped onto a multi-resolution histogram pyramid. The kernel is computed between a pair of examples by comparing the pyramids using a weighted histogram intersection function at each level of pyramid. We propose to use the PMK-based SVM classifier for speaker identification and verification from the speech signal of an utterance represented as a set of local feature vectors. The main issue in building the PMK-based SVM classifier is construction of a pyramid of histograms. We first propose to form hard clusters, using k-means clustering method, with increasing number of clusters at different levels of pyramid to design the codebook-based PMK (CBPMK). Then we propose the GMM-based PMK (GMMPMK) that uses soft clustering. We compare the performance of the GMM-based approaches, and the PMK and other dynamic kernel SVM-based approaches to speaker identification and verification. The 2002 and 2003 NIST speaker recognition corpora are used in evaluation of different approaches to speaker identification and verification. Results of our studies show that the dynamic kernel SVM-based approaches give a significantly better performance than the state-of-the-art GMM-based approaches. For speaker recognition task, the GMMPMK-based SVM gives a performance that is better than that of SVMs using many other dynamic kernels and comparable to that of SVMs using state-of-the-art dynamic kernel, GUMI kernel. The storage requirements of the GMMPMK-based SVMs are less than that of SVMs using any other dynamic kernel.     

基于极限学习机的船舶柴油机故障诊断

针对传统深度核极限学习机网络仅利用端层特征进行分类导致特征不全面，以及故障诊断分类器中核函数选择不恰当等问题，提出基于多层特征表达和多核极限学习机的船舶柴油机故障诊断方法。利用深度极限学习机网络提取故障数据的多层特征；将提取出的各层特征级联为一个具有多属性特征的故障数据特征向量；使用多核极限学习机分类器准确地实现柴油机的故障诊断。在标准分类数据集和船舶柴油机仿真故障数据集上的实验结果表明，与其他极限学习机算法相比，该方法能够有效提高故障诊断的准确率和稳定性，且具有较好的泛化性能，是柴油机故障诊断一个更为优秀实用的工具。     

Methods for the combination of kernel matrices within a support vector framework

The problem of combining different sources of information arises in several situations, for instance, the classification of data with asymmetric similarity matrices or the construction of an optimal classifier from a collection of kernels. Often, each source of information can be expressed as a similarity matrix. In this paper we propose a new class of methods in order to produce, for classification purposes, a single kernel matrix from a collection of kernel (similarity) matrices. Then, the constructed kernel matrix is used to train a Support Vector Machine (SVM). The key ideas within the kernel construction are twofold: the quantification, relative to the classification labels, of the difference of information among the similarities; and the extension of the concept of linear combination of similarity matrices to the concept of functional combination of similarity matrices. The proposed methods have been successfully evaluated and compared with other powerful classifiers and kernel combination techniques on a variety of artificial and real classification problems.     

A Kernel Approach to Multi-Task Learning with Task-Specific Kernels

Several kernel-based methods for multi-task learning have been proposed,which leverage relations among tasks as regularization to enhance the overall learning accuracies.These methods assume that the tasks share the same kernel,which could limit their applications because in practice different tasks may need different kernels.The main challenge of introducing multiple kernels into multiple tasks is that models from different reproducing kernel Hilbert spaces(RKHSs) are not comparable,making it difficult to exploit relations among tasks.This paper addresses the challenge by formalizing the problem in the square integrable space(SIS).Specially,it proposes a kernel-based method which makes use of a regularization term defined in SIS to represent task relations.We prove a new representer theorem for the proposed approach in SIS.We further derive a practical method for solving the learning problem and conduct consistency analysis of the method.We discuss the relationship between our method and an existing method.We also give an SVM(support vector machine)based implementation of our method for multi-label classification.Experiments on an artificial example and two real-world datasets show that the proposed method performs better than the existing method.     

Local structure preservation in Kernel space for feature selection

For many machine learning and data mining tasks in the information explosion environment, one is often confronted with very high dimensional heterogeneous data. Demands for new methods to select discrimination and valuable features that are beneficial to classification and cluster have increased. In this paper, we propose a novel feature selection method to jointly map original data from input space to kernel space and conduct both subspace learning (via locality preserving projection) and feature selection (via a sparsity constraint). Specifically, the nonlinear relationship between data is explored adequately through mapping data from original low-dimensional space to kernel space. Meanwhile, the subspace learning technique is leveraged to preserve available information of local structure in ambient space. Last, by restricting the sparsity of the coefficient matrix, the weight of some features is 0. As a result, we eliminate redundant and irrelevant features and thus make our method select informative and distinguishing features. By comparing our proposed method with some state-of-the-art methods, the experimental results demonstrate that the proposed method outperformed the comparisons in terms of clustering task.

     

Learning low-rank kernel matrices for constrained clustering

Constrained clustering methods (that usually use must-link and/or cannot-link constraints) have been received much attention in the last decade. Recently, kernel adaptation or kernel learning has been considered as a powerful approach for constrained clustering. However, these methods usually either allow only special forms of kernels or learn non-parametric kernel matrices and scale very poorly. Therefore, they either learn a metric that has low flexibility or are applicable only on small data sets due to their high computational complexity. In this paper, we propose a more efficient non-linear metric learning method that learns a low-rank kernel matrix from must-link and cannot-link constraints and the topological structure of data. We formulate the proposed method as a trace ratio optimization problem and learn appropriate distance metrics through finding optimal low-rank kernel matrices. We solve the proposed optimization problem much more efficiently than SDP solvers. Additionally, we show that the spectral clustering methods can be considered as a special form of low-rank kernel learning methods. Extensive experiments have demonstrated the superiority of the proposed method compared to recently introduced kernel learning methods.     

Efficient 2D viewpoint combination for human action recognition

The ability to recognize human actions using a single viewpoint is affected by phenomena such as self-occlusions or occlusions by other objects. Incorporating multiple cameras can help overcome these issues. However, the question remains how to efficiently use information from all viewpoints to increase performance. Researchers have reconstructed a 3D model from multiple views to reduce dependency on viewpoint, but this 3D approach is often computationally expensive. Moreover, the quality of each view influences the overall model and the reconstruction is limited to volumes where the views overlap. In this paper, we propose a novel method to efficiently combine 2D data from different viewpoints. Spatio-temporal features are extracted from each viewpoint and then used in a bag-of-words framework to form histograms. Two different sizes of codebook are exploited. The similarity between the obtained histograms is represented via the Histogram Intersection kernel as well as the RBF kernel with \(\chi ^2\) distance. Lastly, we combine all the basic kernels generated by selection of different viewpoints, feature types, codebook sizes and kernel types. The final kernel is a linear combination of basic kernels that are properly weighted based on an optimization process. For higher accuracy, the sets of kernel weights are computed separately for each binary SVM classifier. Our method not only combines the information from multiple viewpoints efficiently, but also improves the performance by mapping features into various kernel spaces. The efficiency of the proposed method is demonstrated by testing on two commonly used multi-view human action datasets. Moreover several experiments indicate the efficacy of each part of the method on the overall performance.     

Towards learning a semantic-consistent subspace for cross-modal retrieval

A great many of approaches have been developed for cross-modal retrieval, among which subspace learning based ones dominate the landscape. Concerning whether using the semantic label information or not, subspace learning based approaches can be categorized into two paradigms, unsupervised and supervised. However, for multi-label cross-modal retrieval, supervised approaches just simply exploit multi-label information towards a discriminative subspace, without considering the correlations between multiple labels shared by multi-modalities, which often leads to an unsatisfactory retrieval performance. To address this issue, in this paper we propose a general framework, which jointly incorporates semantic correlations into subspace learning for multi-label cross-modal retrieval. By introducing the HSIC-based regularization term, the correlation information among multiple labels can be not only leveraged but also the consistency between the modality similarity from each modality is well preserved. Besides, based on the semantic-consistency projection, the semantic gap between the low-level feature space of each modality and the shared high-level semantic space can be balanced by a mid-level consistent one, where multi-label cross-modal retrieval can be performed effectively and efficiently. To solve the optimization problem, an effective iterative algorithm is designed, along with its convergence analysis theoretically and experimentally. Experimental results on real-world datasets have shown the superiority of the proposed method over several existing cross-modal subspace learning methods.

     

基于图卷积的异质网络节点分类方法

图表示学习已成为图深度学习领域的一个研究热点. 大多数图神经网络存在过平滑现象,这类方法重点关注图节点特征,对图的结构特征关注度不高. 为了提升对图结构特征的表征能力,提出了一种基于图核同构网络的图分类方法,即KerGIN. 该方法首先通过图同构网络（graph isomorphism network,GIN）对图进行节点特征编码,并使用图核方法对图进行结构编码,进一步利用Nyström方法降低图核矩阵的维度. 其次借助MLP将图核矩阵与图特征矩阵对齐,通过注意力机制将图的特征编码和结构编码进行自适应加权融合,进而得到图的最终特征表示,提升了图结构特征信息的表达能力. 最后在7个公开的图分类数据集上对模型进行了实验评估：与现有图表示模型相比,KerGIN模型能够在图分类准确度上有较大幅度提升,它可以增强GIN对图结构特征信息的表达能力.