Multiple Empirical Kernel Learning with dynamic pairwise constraints

Unlike the traditional Multiple Kernel Learning (MKL) with the implicit kernels, Multiple Empirical Kernel Learning (MEKL) explicitly maps the original data space into multiple feature spaces via different empirical kernels. MEKL has been demonstrated to bring good classification performance and to be much easier in processing and analyzing the adaptability of kernels for the input space. In this paper, we incorporate the dynamic pairwise constraints into MEKL to propose a novel Multiple Empirical Kernel Learning with dynamic Pairwise Constraints method (MEKLPC). It is known that the pairwise constraint provides the relationship between two samples, which tells whether these samples belong to the same class or not. In the present work, we boost the original pairwise constraints and design the dynamic pairwise constraints which can pay more attention onto the boundary samples and thus to make the decision hyperplane more reasonable and accurate. Thus, the proposed MEKLPC not only inherits the advantages of the MEKL, but also owns multiple folds of prior information. Firstly, MEKLPC gets the side-information and boosts the classification performance significantly in each feature space. Here, the side-information is the dynamic pairwise constraints which are constructed by the samples near the decision boundary, i.e. the boundary samples. Secondly, in each mapped feature space, MEKLPC still measures the empirical risk and generalization risk. Lastly, different feature spaces mapped by multiple empirical kernels can agree to their outputs for the same input sample as much as possible. To the best of our knowledge, it is the first time to introduce the dynamic pairwise constraints into the MEKL framework in the present work. The experiments on a number of real-world data sets demonstrate the feasibility and effectiveness of MEKLPC.     

结合类标签关联度的有序核判别回归学习

有序离散类标号通常由原始连续标号按一定规则映射得到,因此它们彼此间是存在关联信息的,现有有序回归方法对此类关联信息的考虑仍然较少。首先提出一类有序标号间关联度的量化表示,进而将其与典型有序回归方法(Kernel discriminant learning for ordinal regression, KDLOR)相结合,设计出了一种结合类标号关联度的有序核判别回归学习方法(Kernel discriminant learning for ordinal regression using label membership,LM KDLOR),最后通过在多个标准有序回归数据集上的对比实验验证了所提方法的有效性。     

Classifying molecular sequences using a linkage graph with their pairwise similarities

This paper presents a method for classifying a large and mixed set of uncharacterized sequences provided by genome projects. As the measure of sequence similarity, we use similarity score computed by a method based on the dynamic programming (DP), such as the Smith-Waterman local alignment algorithm. Although comparison by DP based method is very sensitive, when given sequences include a family of sequences that are much diverged in evolutionary process, similarity among some of them may be hidden behind spurious similarity of some unrelated sequences. Also the distance derived from the similarity score may not be metric (i.e., triangle inequality may not hold) when some sequences have multi-domain structure. To cope with these problems, we introduce a new graph structure called p-quasi complete graph for describing a family of sequences with a confidence measure. We prove that a restricted version of the p-quasi complete graph problem (given a positive integer k, whether a graph contains a 0.5-quasi complete subgraph of which size k or not) is NP-complete. Thus we present an approximation algorithm for classifying a set of sequences using p-quasi complete subgraphs. The effectiveness of our method is demonstrated by the result of classifying over 4000 protein sequences on the Escherichia coli genome that was completely determined recently.     

Domain-independent feature extraction for multi-classification using multi-objective genetic programming

We propose three model-free feature extraction approaches for solving the multiple class classification problem; we use multi-objective genetic programming (MOGP) to derive (near-)optimal feature extraction stages as a precursor to classification with a simple and fast-to-train classifier. Statistically-founded comparisons are made between our three proposed approaches and seven conventional classifiers over seven datasets from the UCI Machine Learning database. We also make comparisons with other reported evolutionary computation techniques. On almost all the benchmark datasets, the MOGP approaches give better or identical performance to the best of the conventional methods. Of our proposed MOGP-based algorithms, we conclude that hierarchical feature extraction performs best on multi-classification problems.     

A Fast Dual Algorithm for Kernel Logistic Regression

This paper gives a new iterative algorithm for kernel logistic regression. It is based on the solution of a dual problem using ideas similar to those of the Sequential Minimal Optimization algorithm for Support Vector Machines. Asymptotic convergence of the algorithm is proved. Computational experiments show that the algorithm is robust and fast. The algorithmic ideas can also be used to give a fast dual algorithm for solving the optimization problem arising in the inner loop of Gaussian Process classifiers. Editor: Shai Ben-David     

Maximum Penalized Likelihood Kernel Regression for Fast Adaptation

This paper proposes a nonlinear generalization of the popular maximum-likelihood linear regression (MLLR) adaptation algorithm using kernel methods. The proposed method, called maximum penalized likelihood kernel regression adaptation (MPLKR), applies kernel regression with appropriate regularization to determine the affine model transform in a kernel-induced high-dimensional feature space. Although this is not the first attempt of applying kernel methods to conventional linear adaptation algorithms, unlike most of other kernelized adaptation methods such as kernel eigenvoice or kernel eigen-MLLR, MPLKR has the advantage that it is a convex optimization and its solution is always guaranteed to be globally optimal. In fact, the adapted Gaussian means can be obtained analytically by simply solving a system of linear equations. From the Bayesian perspective, MPLKR can also be considered as the kernel version of maximum a posteriori linear regression (MAPLR) adaptation. Supervised and unsupervised speaker adaptation using MPLKR were evaluated on the Resource Management and Wall Street Journal 5K tasks, respectively, achieving a word error rate reduction of 23.6% and 15.5% respectively over the speaker-independently model.     

机器学习中的核覆盖算法

基于统计学习理论的支持向量机（SVM）方法在样本空间或特征空间构造最优分类超平面解决了分类器的构造问题,但其本质是二分类的,且核函数中的参数难以确定,计算复杂性高．构造性学习算法根据训练样本构造性地设计分类网络,运行效率高,便于处理多分类问题,但存在所得的分界面零乱、测试计算量大的缺点．该文将SVM中的核函数法与构造性学习的覆盖算法相融合,给出一种新的核覆盖算法．新算法克服了以上两种模型的缺点,具有运算速度快、精度高、鲁棒性强的优点．其次．文中给出风险误差上界与覆盖个数的关系．最后给出实验模     

Message estimation for universal steganalysis using multi-classification support vector machine

To prevent misusing of the steganography from the terrorists, effective steganalysis schemes which discriminate the stego-images from suspicious images are necessary. Some steganalysis methods can accurately estimate the length of embedded messages but they are only useful in the pre-defined condition. Active steganalysis methods are powerful in length estimation such as regular singular (RS) and sample pairs analysis (SPA) steganalysis schemes, but they would become invalid in frequency domain. Passive steganalysis methods may discriminate stego-images from suspicious images in spatial and frequency domains such as Lyu and Fraid's steganalysis scheme, but they could not estimate the length of hidden messages. Although length estimation has been discussed in the active steganalysis methods for a while, it is a novel study in passive steganalysis method. We improve the Lyu and Fraid's universal steganalysis scheme and design an efficient length estimation policy in passive steganalysis methods. Experimental results demonstrate the efficiency and practicability of the proposed universal steganalysis scheme.     

Efficient image signatures and similarities using tensor products of local descriptors

In this paper, we introduce a novel image signature effective in both image retrieval and image classification. Our approach is based on the aggregation of tensor products of discriminant local features, named VLATs (vector of locally aggregated tensors). We also introduce techniques for the packing and the fast comparison of VLATs. We present connections between VLAT and methods like kernel on bags and Fisher vectors. Finally, we show the ability of our method to be effective for two different retrieval problems, thanks to experiments carried out on similarity search and classification datasets.     

核回归方法的散点拟合曲面重构

散点曲面重构是计算机图形学中的一个基本问题,针对这个问题提出了一种全新的基于核回归方法的散点曲面重构方法,使用二维信号处理方法中非参数滤波等成熟手段进行曲面重构.这种方法可以生成任意阶数连续的曲面,在理论上保证了生成曲面的连续性,可以自定义网格的拓扑,在曲率大或者感兴趣的局部能够自适应调整网格点的密度,生成的结果方便LOD建模,数据的拟合精度也可以通过调整滤波参数控制,算法自适应调整滤波器的方向,使结果曲面可以更好保持尖锐特征.同时在构造过程中避免了传统的细分曲面方法中迭代、Delaunay剖分和点云数据中重采样等时间开销大的过程,提高了效率.对于采样不均、噪声较大的数据,该算法的鲁棒性很好.实验表明这种曲面建模方法能够散点重构出精度较高的连续曲面,在效率上有很大提高,在只需要估计曲面和其一阶导数时,利用Nadaraya-Watson快速算法可以使算法时间复杂度降为O(N),远低于其他曲面重构平滑方法.同时算法可以对曲面的局部点云密度、网格顶点法矢等信息做有效的估计.重构出的曲面对类似数字高程模型(DEM)的数据可以保证以上的优点.但如果散点数据不能被投影到2维平面上,曲面重构就需要包括基网格生成、重构面片缝合等过程.缝合边缘的连续性也不能在理论上得到保证.     

A Kernel Approach for Semisupervised Metric Learning

While distance function learning for supervised learning tasks has a long history, extending it to learning tasks with weaker supervisory information has only been studied recently. In particular, some methods have been proposed for semisupervised metric learning based on pairwise similarity or dissimilarity information. In this paper, we propose a kernel approach for semisupervised metric learning and present in detail two special cases of this kernel approach. The metric learning problem is thus formulated as an optimization problem for kernel learning. An attractive property of the optimization problem is that it is convex and, hence, has no local optima. While a closed-form solution exists for the first special case, the second case is solved using an iterative majorization procedure to estimate the optimal solution asymptotically. Experimental results based on both synthetic and real-world data show that this new kernel approach is promising for nonlinear metric learning     

A Primal Framework for Indefinite Kernel Learning

Kernel methods have been widely applied in machine learning to solve complex nonlinear problems. Kernel selection is one of the key issues in kernel methods, since it is vital for improving generalization performance. Traditionally, the selection of kernel is restricted to be positive definite which makes their applicability partially limited. Actually, in many real applications such as gene identification and object recognition, indefinite kernels frequently emerge and can achieve better performance. However, compared to positive definite ones, indefinite kernels are more complicated due to the non-convexity of the subsequent optimization problems, which leads to the incapability of most existing kernel algorithms. Some indefinite kernel methods have been proposed based on the dual of support vector machine (SVM), which mostly emphasize on how to transform the non-convex optimization to be convex by using positive definite kernels to approximate indefinite ones. In fact, the duality gap in SVM usually exists in the case of indefinite kernels and therefore these algorithms do not indeed solve the indefinite kernel problems themselves. In this paper, we present a novel framework for indefinite kernel learning derived directly from the primal of SVM, which establishes several new models not only for single indefinite kernel but also extends to multiple indefinite kernel scenarios. Several algorithms are developed to handle the non-convex optimization problems in these models. We further provide a constructive approach for kernel selection in the algorithms by using the theory of similarity functions. Experiments on real world datasets demonstrate the superiority of our models.     

Kernel discriminant transformation for image set-based face recognition

This study presents a novel kernel discriminant transformation (KDT) algorithm for face recognition based on image sets. As each image set is represented by a kernel subspace, we formulate a KDT matrix that maximizes the similarities of within-kernel subspaces, and simultaneously minimizes those of between-kernel subspaces. Although the KDT matrix cannot be computed explicitly in a high-dimensional feature space, we propose an iterative kernel discriminant transformation algorithm to solve the matrix in an implicit way. Another perspective of similarity measure, namely canonical difference, is also addressed for matching each pair of the kernel subspaces, and employed to simplify the formulation. The proposed face recognition system is demonstrated to outperform existing still-image-based as well as image set-based face recognition methods using the Yale Face database B, Labeled Faces in the Wild and a self-compiled database.     

Local Kernel Regression Score for Selecting Features of High-Dimensional Data

In general, irrelevant features of high-dimensional data will degrade the performance of an inference system, e.g., a clustering algorithm or a classifier. In this paper, we therefore present a Local Kernel Regression (LKR) scoring approach to evaluate the relevancy of features based on their capabilities of keeping the local configuration in a small patch of data. Accordingly, a score index featuring applicability to both of supervised learning and unsupervised learning is developed to identify the relevant features within the framework of local kernel regression. Experimental results show the efficacy of the proposed approach in comparison with the existing methods.     

Kernel PCA for Feature Extraction and De-Noising in Nonlinear Regression

In this paper, we propose the application of the Kernel Principal Component Analysis (PCA) technique for feature selection in a high-dimensional feature space, where input variables are mapped by a Gaussian kernel. The extracted features are employed in the regression problems of chaotic Mackey–Glass time-series prediction in a noisy environment and estimating human signal detection performance from brain event-related potentials elicited by task relevant signals. We compared results obtained using either Kernel PCA or linear PCA as data preprocessing steps. On the human signal detection task, we report the superiority of Kernel PCA feature extraction over linear PCA. Similar to linear PCA, we demonstrate de-noising of the original data by the appropriate selection of various nonlinear principal components. The theoretical relation and experimental comparison of Kernel Principal Components Regression, Kernel Ridge Regression and ε-insensitive Support Vector Regression is also provided.     

一种新的支持向量回归核函数构建方法

首先，讨论了支持向量回归（support vector regression，SVR）的基本原理．然后，从信息几何的角度分析了核函数的几何结构，通过共形变换（conformal transformation）构建与数据依赖(data-dependent)的核函数，使得特征空间在支持向量附近的体积元缩小，以改善SVR的机器性能．实验结果表明了方法的有效性．     

Joint regression and learning from pairwise rankings for personalized image aesthetic assessment

Computational Visual Media - Recent image aesthetic assessment methods have achieved remarkable progress due to the emergence of deep convolutional neural networks (CNNs). However, these methods...     

基于池的无监督线性回归主动学习

在许多现实的机器学习应用场景中, 获取大量未标注的数据是很容易的, 但标注过程需要花费大量的时间和经济成本. 因此, 在这种情况下, 需要选择一些最有价值的样本进行标注, 从而只利用较少的标注数据就能训练出较好的机器学习模型. 目前, 主动学习(Active learning)已广泛应用于解决这种场景下的问题. 但是, 大多数现有的主动学习方法都是基于有监督场景: 能够从少量带标签的样本中训练初始模型, 基于模型查询新的样本, 然后迭代更新模型. 无监督情况下的主动学习却很少有人考虑, 即在不知道任何标签信息的情况下最佳地选择要标注的初始训练样本. 这种场景下, 主动学习问题变得更加困难, 因为无法利用任何标签信息. 针对这一场景, 本文研究了基于池的无监督线性回归问题, 提出了一种新的主动学习方法, 该方法同时考虑了信息性、代表性和多样性这三个标准. 本文在3个不同的线性回归模型(岭回归、LASSO (Least absolute shrinkage and selection operator)和线性支持向量回归)和来自不同应用领域的12个数据集上进行了广泛的实验, 验证了其有效性.     

On Incremental Learning for Gradient Boosting Decision Trees

Neural Processing Letters - Boosting algorithms, as a class of ensemble learning methods, have become very popular in data classification, owing to their strong theoretical guarantees and...