Accelerated max-margin multiple kernel learning

Kernel machines such as Support Vector Machines (SVM) have exhibited successful performance in pattern classification problems mainly due to their exploitation of potentially nonlinear affinity structures of data through the kernel functions. Hence, selecting an appropriate kernel function, equivalently learning the kernel parameters accurately, has a crucial impact on the classification performance of the kernel machines. In this paper we consider the problem of learning a kernel matrix in a binary classification setup, where the hypothesis kernel family is represented as a convex hull of fixed basis kernels. While many existing approaches involve computationally intensive quadratic or semi-definite optimization, we propose novel kernel learning algorithms based on large margin estimation of Parzen window classifiers. The optimization is cast as instances of linear programming. This significantly reduces the complexity of the kernel learning compared to existing methods, while our large margin based formulation provides tight upper bounds on the generalization error. We empirically demonstrate that the new kernel learning methods maintain or improve the accuracy of the existing classification algorithms while significantly reducing the learning time on many real datasets in both supervised and semi-supervised settings.     

Domain transfer multiple kernel learning

Cross-domain learning methods have shown promising results by leveraging labeled patterns from the auxiliary domain to learn a robust classifier for the target domain which has only a limited number of labeled samples. To cope with the considerable change between feature distributions of different domains, we propose a new cross-domain kernel learning framework into which many existing kernel methods can be readily incorporated. Our framework, referred to as Domain Transfer Multiple Kernel Learning (DTMKL), simultaneously learns a kernel function and a robust classifier by minimizing both the structural risk functional and the distribution mismatch between the labeled and unlabeled samples from the auxiliary and target domains. Under the DTMKL framework, we also propose two novel methods by using SVM and prelearned classifiers, respectively. Comprehensive experiments on three domain adaptation data sets (i.e., TRECVID, 20 Newsgroups, and email spam data sets) demonstrate that DTMKL-based methods outperform existing cross-domain learning and multiple kernel learning methods.     

Deep multilayer multiple kernel learning

Multiple kernel learning (MKL) approach has been proposed for kernel methods and has shown high performance for solving some real-world applications. It consists on learning the optimal kernel from one layer of multiple predefined kernels. Unfortunately, this approach is not rich enough to solve relatively complex problems. With the emergence and the success of the deep learning concept, multilayer of multiple kernel learning (MLMKL) methods were inspired by the idea of deep architecture. They are introduced in order to improve the conventional MKL methods. Such architectures tend to learn deep kernel machines by exploring the combinations of multiple kernels in a multilayer structure. However, existing MLMKL methods often have trouble with the optimization of the network for two or more layers. Additionally, they do not always outperform the simplest method of combining multiple kernels (i.e., MKL). In order to improve the effectiveness of MKL approaches, we introduce, in this paper, a novel backpropagation MLMKL framework. Specifically, we propose to optimize the network over an adaptive backpropagation algorithm. We use the gradient ascent method instead of dual objective function, or the estimation of the leave-one-out error. We test our proposed method through a large set of experiments on a variety of benchmark data sets. We have successfully optimized the system over many layers. Empirical results over an extensive set of experiments show that our algorithm achieves high performance compared to the traditional MKL approach and existing MLMKL methods.     

Non-uniform multiple kernel learning with cluster-based gating functions

Recently, multiple kernel learning (MKL) has gained increasing attention due to its empirical superiority over traditional single kernel based methods. However, most of state-of-the-art MKL methods are “uniform” in the sense that the relative weights of kernels keep fixed among all data.Here we propose a “non-uniform” MKL method with a data-dependent gating mechanism, i.e., adaptively determine the kernel weights for the samples. We utilize a soft clustering algorithm and then tune the weight for each cluster under the graph embedding (GE) framework. The idea of exploiting cluster structures is based on the observation that data from the same cluster tend to perform consistently, which thus increases the resistance to noises and results in more reliable estimate. Moreover, it is computationally simple to handle out-of-sample data, whose implicit RKHS representations are modulated by the posterior to each cluster.Quantitative studies between the proposed method and some representative MKL methods are conducted on both synthetic and widely used public data sets. The experimental results well validate its superiorities.     

Double-fold localized multiple matrix learning machine with Universum

Matrix learning, multiple-view learning, Universum learning, and local learning are four hot spots of present research. Matrix learning aims to design feasible machines to process matrix patterns directly. Multiple-view learning takes pattern information from multiple aspects, i.e., multiple-view information into account. Universum learning can reflect priori knowledge about application domain and improve classification performances. A good local learning approach is important to the finding of local structures and pattern information. Our previous proposed learning machine, double-fold localized multiple matrix learning machine is a one with multiple-view information, local structures, and matrix learning. But this machine does not take Universum learning into account. Thus, this paper proposes a double-fold localized multiple matrix learning machine with Universum (Uni-DLMMLM) so as to improve the performance of a learning machine. Experimental results have validated that Uni-DLMMLM (1) makes full use of the domain knowledge of whole data distribution as well as inherits the advantages of matrix learning; (2) combines Universum learning with matrix learning so as to capture more global knowledge; (3) has a good ability to process different kinds of data sets; (4) has a superior classification performance and leads to a low empirical generation risk bound.     

A primal method for multiple kernel learning

The canonical support vector machines (SVMs) are based on a single kernel, recent publications have shown that using multiple kernels instead of a single one can enhance interpretability of the decision function and promote classification accuracy. However, most of existing approaches mainly reformulate the multiple kernel learning as a saddle point optimization problem which concentrates on solving the dual. In this paper, we show that the multiple kernel learning (MKL) problem can be reformulated as a BiConvex optimization and can also be solved in the primal. While the saddle point method still lacks convergence results, our proposed method exhibits strong optimization convergence properties. To solve the MKL problem, a two-stage algorithm that optimizes canonical SVMs and kernel weights alternately is proposed. Since standard Newton and gradient methods are too time-consuming, we employ the truncated-Newton method to optimize the canonical SVMs. The Hessian matrix need not be stored explicitly, and the Newton direction can be computed using several Preconditioned Conjugate Gradient steps on the Hessian operator equation, the algorithm is shown more efficient than the current primal approaches in this MKL setting. Furthermore, we use the Nesterov’s optimal gradient method to optimize the kernel weights. One remarkable advantage of solving in the primal is that it achieves much faster convergence rate than solving in the dual and does not require a two-stage algorithm even for the single kernel LapSVM. Introducing the Laplacian regularizer, we also extend our primal method to semi-supervised scenario. Extensive experiments on some UCI benchmarks have shown that the proposed algorithm converges rapidly and achieves competitive accuracy.     

Human action recognition with salient trajectories and multiple kernel learning

Action recognition in videos plays an important role in the field of computer vision and multimedia, and there exist lots of challenges due to the complexity of spatial and temporal information. Trajectory-based approach has shown to be efficient recently, and a new framework and algorithm of trajectory space information based multiple kernel learning (TSI-MKL) is exploited in this paper. First, dense trajectories are extracted as raw features, and three saliency maps are computed corresponding to color, space, and optical flow on frames at the same time. Secondly, a new method combining above saliency maps is proposed to filter the achieved trajectories, by which a set of salient trajectories only containing foreground motion regions is obtained. Afterwards, a novel two-layer clustering is developed to cluster the obtained trajectories into several semantic groups and the ultimate video representation is generated by encoding each group. Finally, representations of different semantic groups are fed into the proposed kernel function of a multiple kernel classifier. Experiments are conducted on three popular video action datasets and the results demonstrate that our presented approach performs competitively compared with the state-of-the-art.     

MultiK-MHKS: a novel multiple kernel learning algorithm

In this paper, we develop a new effective multiple kernel learning algorithm. First, map the input data into m different feature spaces by m empirical kernels, where each generatedfeature space is takenas one viewof the input space. Then through the borrowing the motivating argument from Canonical Correlation Analysis (CCA)that can maximally correlate the m views in the transformed coordinates, we introduce a special term called Inter-Function Similarity Loss R IFSL into the existing regularization framework so as to guarantee the agreement of multi-view outputs. In implementation, we select the Modification of Ho-Kashyap algorithm with Squared approximation of the misclassification errors (MHKS) as the incorporated paradigm, and the experimental results on benchmark data sets demonstrate the feasibility and effectiveness of the proposed algorithm named MultiK-MHKS.     

Regularizing multiple kernel learning using response surface methodology

In recent years, several methods have been proposed to combine multiple kernels using a weighted linear sum of kernels. These different kernels may be using information coming from multiple sources or may correspond to using different notions of similarity on the same source. We note that such methods, in addition to the usual ones of the canonical support vector machine formulation, introduce new regularization parameters that affect the solution quality and, in this work, we propose to optimize them using response surface methodology on cross-validation data. On several bioinformatics and digit recognition benchmark data sets, we compare multiple kernel learning and our proposed regularized variant in terms of accuracy, support vector count, and the number of kernels selected. We see that our proposed variant achieves statistically similar or higher accuracy results by using fewer kernel functions and/or support vectors through suitable regularization; it also allows better knowledge extraction because unnecessary kernels are pruned and the favored kernels reflect the properties of the problem at hand.     

Interpolation with multiple norm constraints

This paper considers the problem of finding matrix-valued rational functions that satisfy two-sided residue interpolation conditions subject to norm constraints on their components. It is shown that this problem can be reduced to a finite-dimensional convex optimization problem. As an application, we show that under suitable assumptions on the plant, multiple objective ?₂ and ?_∞ control problems admit finite-dimensional optimal solutions and that such solutions can be computed using finite-dimensional convex programs.     

Human activity recognition using multi-features and multiple kernel learning

This paper presents two sets of features, shape representation and kinematic structure, for human activity recognition using a sequence of RGB-D images. The shape features are extracted using the depth information in the frequency domain via spherical harmonics representation. The other features include the motion of the 3D joint positions (i.e. the end points of the distal limb segments) in the human body. Both sets of features are fused using the Multiple Kernel Learning (MKL) technique at the kernel level for human activity recognition. Our experiments on three publicly available datasets demonstrate that the proposed features are robust for human activity recognition and particularly when there are similarities among the actions.     

Analytic center cutting plane method for multiple kernel learning

Multiple Kernel Learning (MKL) is a popular generalization of kernel methods which allows the practitioner to optimize over convex combinations of kernels. We observe that many recent MKL solutions can be cast in the framework of oracle based optimization, and show that they vary in terms of query point generation. The popularity of such methods is because the oracle can fortuitously be implemented as a support vector machine. Motivated by the success of centering approaches in interior point methods, we propose a new approach to optimize the MKL objective based on the analytic center cutting plane method (accpm). Our experimental results show that accpm outperforms state of the art in terms of rate of convergence and robustness. Further analysis sheds some light as to why MKL may not always improve classification accuracy over naive solutions.     

结合罚函数与序列二次规划的[lp]范数优化方法

结合罚函数法与序列二次规划（SQP）方法研究了[lp]范数优化的求解算法。分析了基于SQP方法的[lp]范数优化算法,探讨了初值选取对算法收敛性的影响;针对SQP方法受迭代初值的限制,引入罚函数优化方法对迭代初值作预估计,使其进入可行域,采用SQP方法求解计算。实验结果表明,结合罚函数与SQP方法的[lp]范数优化算法对稀疏信号有较优的重构效果。     

多核学习中基于复合梯度映射的学习算法研究

现有的多核学习算法大多假设训练样本分类完全正确,将其应用到受扰分类样本上时,由于分类存在差错,因此往往只能实现次优性能.为了解决这一问题,首先将受扰分类多核学习问题建模为随机规划问题,并得到一种极小极大表达式;然后提出基于复合梯度映射的一阶学习算法对问题进行求解.理论分析表明,该算法的收敛速度为O(1/T),大大快于传统算法的收敛速度O(1/√T).最后,基于五个UCI数据集的实验结果也验证了本文观点和优化算法的有效性.     

结合加权核范数与全变分的图像二级去噪

为提升图像去噪后的视觉感受,提出一种加权核范数最小化（WNNM）结合全变分（TV）的二级图像降噪方法。首先对含噪图像进行TV基础去噪,其次用噪声图像与基础去噪结果图做差分运算,并对差分后的结果自适应维纳滤波,然后将滤波后图像与基础TV降噪图像叠加,利用块匹配做相似补丁收集,最后运用加权核范数最小化进行二次去噪,得到最终降噪图像。通过与原WNNM、三维块匹配去噪（BM3D）、漏斗自相似非局部去噪（FNLM）方法对比,该方法不仅对平滑区域有较优的降噪效果,同时处理了漏斗自相似非局部去噪与BM3D在高噪声情况下带来花斑与假条纹状况,并且使结构纹理信息最大化相似。     

基于多核学习的多特征融合图像分类研究

图像分类任务是计算机视觉中的一个重要研究方向。组合多种特征在一定程度上能够使得图像分类准确度得到提高。然而,如何组合多种图像特征是一个悬而未决的难题。提出了一种基于多类多核学习的多特征融合算法,并应用到图像分类任务。算法在有效地利用多核学习自动选取对当前任务有价值特征的优势的同时,避免了在多核学习中将多类问题分解为多个二分问题。在图像特征表示方面,使用字典自学习方法。实验结果表明,提出的算法能够有效地提高图像分类的准确度。     

Multiple kernel learning with gaussianity measures

Kernel methods are known to be effective for nonlinear multivariate analysis. One of the main issues in the practical use of kernel methods is the selection of kernel. There have been a lot of studies on kernel selection and kernel learning. Multiple kernel learning (MKL) is one of the promising kernel optimization approaches. Kernel methods are applied to various classifiers including Fisher discriminant analysis (FDA). FDA gives the Bayes optimal classification axis if the data distribution of each class in the feature space is a gaussian with a shared covariance structure. Based on this fact, an MKL framework based on the notion of gaussianity is proposed. As a concrete implementation, an empirical characteristic function is adopted to measure gaussianity in the feature space associated with a convex combination of kernel functions, and two MKL algorithms are derived. From experimental results on some data sets, we show that the proposed kernel learning followed by FDA offers strong classification power.     

Wavelet kernel learning

This paper addresses the problem of optimal feature extraction from a wavelet representation. Our work aims at building features by selecting wavelet coefficients resulting from signal or image decomposition on an adapted wavelet basis. For this purpose, we jointly learn in a kernelized large-margin context the wavelet shape as well as the appropriate scale and translation of the wavelets, hence the name “wavelet kernel learning”. This problem is posed as a multiple kernel learning problem, where the number of kernels can be very large. For solving such a problem, we introduce a novel multiple kernel learning algorithm based on active constraints methods. We furthermore propose some variants of this algorithm that can produce approximate solutions more efficiently. Empirical analysis show that our active constraint MKL algorithm achieves state-of-the art efficiency. When used for wavelet kernel learning, our experimental results show that the approaches we propose are competitive with respect to the state-of-the-art on brain–computer interface and Brodatz texture datasets.     

基于多核学习卷积神经网络的稀疏数据推荐

针对当前稀疏数据推荐准确率低的问题,提出一种基于多核学习卷积神经网络的稀疏数据推荐算法.将项目的辅助信息送入卷积神经网络学习特征,将向量在可再生核希尔伯特空间组合,利用多核学习技术增强卷积神经网络的特征学习能力;基于学习的卷积特征集初始化非负矩阵模型,通过非负矩阵模型实现对缺失评分的预测.实验结果表明,该算法有效提高了稀疏数据集的推荐性能,验证了多核学习卷积神经网络的有效性.