Feature space interpretation of SVMs with indefinite kernels

Kernel methods are becoming increasingly popular for various kinds of machine learning tasks, the most famous being the support vector machine (SVM) for classification. The SVM is well understood when using conditionally positive definite (cpd) kernel functions. However, in practice, non-cpd kernels arise and demand application in SVM. The procedure of "plugging" these indefinite kernels in SVM often yields good empirical classification results. However, they are hard to interpret due to missing geometrical and theoretical understanding. In this paper, we provide a step toward the comprehension of SVM classifiers in these situations. We give a geometric interpretation of SVM with indefinite kernel functions. We show that such SVM are optimal hyperplane classifiers not by margin maximization, but by minimization of distances between convex hulls in pseudo-Euclidean spaces. By this, we obtain a sound framework and motivation for indefinite SVM. This interpretation is the basis for further theoretical analysis, e.g., investigating uniqueness, and for the derivation of practical guidelines like characterizing the suitability of indefinite SVM.     

Enhanced FMAM based on empirical kernel map

The existing morphological auto-associative memory models based on the morphological operations, typically including morphological auto-associative memories (auto-MAM) proposed by Ritter et al. and our fuzzy morphological auto-associative memories (auto-FMAM), have many attractive advantages such as unlimited storage capacity, one-shot recall speed and good noise-tolerance to single erosive or dilative noise. However, they suffer from the extreme vulnerability to noise of mixing erosion and dilation, resulting in great degradation on recall performance. To overcome this shortcoming, we focus on FMAM and propose an enhanced FMAM (EFMAM) based on the empirical kernel map. Although it is simple, EFMAM can significantly improve the auto-FMAM with respect to the recognition accuracy under hybrid-noise and computational effort. Experiments conducted on the thumbnail-sized faces (28/spl times/23 and 14/spl times/11) scaled from the ORL database show the average accuracies of 92%, 90%, and 88% with 40 classes under 10%, 20%, and 30% randomly generated hybrid-noises, respectively, which are far higher than the auto-FMAM (67%, 46%, 31%) under the same noise levels.     

Optimizing the kernel in the empirical feature space

In this paper, we present a method of kernel optimization by maximizing a measure of class separability in the empirical feature space, an Euclidean space in which the training data are embedded in such a way that the geometrical structure of the data in the feature space is preserved. Employing a data-dependent kernel, we derive an effective kernel optimization algorithm that maximizes the class separability of the data in the empirical feature space. It is shown that there exists a close relationship between the class separability measure introduced here and the alignment measure defined recently by Cristianini. Extensive simulations are carried out which show that the optimized kernel is more adaptive to the input data, and leads to a substantial, sometimes significant, improvement in the performance of various data classification algorithms.     

Feature space perspectives for learning the kernel

In this paper, we continue our study of learning an optimal kernel in a prescribed convex set of kernels (Micchelli & Pontil, 2005) . We present a reformulation of this problem within a feature space environment. This leads us to study regularization in the dual space of all continuous functions on a compact domain with values in a Hilbert space with a mix norm. We also relate this problem in a special case to regularization. Editors: Olivier Bousquet and Andre Elisseeff This work was supported by NSF Grant ITR-0312113, EPSRC Grant GR/T18707/01 and by the IST Programme of the European Community, under the PASCAL Network of Excellence IST-2002-506778.     

Feature space locality constraint for kernel based nonlinear discriminant analysis

Subspace learning is an important approach in pattern recognition. Nonlinear discriminant analysis (NDA), due to its capability of describing nonlinear manifold structure of samples, is considered to be more powerful to undertake classification tasks in image related problems. In kernel based NDA representation, there are three spaces involved, i.e., original data space, implicitly mapped high dimension feature space and the target low dimension subspace. Existing methods mainly focus on the information in original data space to find the most discriminant low dimension subspace. The implicit high dimension feature space plays a role that connects the original space and the target subspace to realize the nonlinear dimension reduction, but the sample geometric structure information in feature space is not involved. In this work, we try to utilize and explore this information. Specifically, the locality information of samples in feature space is modeled and integrated into the traditional kernel based NDA methods. In this way, both the sample distributions in original data space and the mapped high dimension feature space are modeled and more information is expected to be explored to improve the discriminative ability of the subspace. Two algorithms, named FSLC-KDA and FSLC-KSR, are presented. Extensive experiments on ORL, Extended-YaleB, PIE, Multi-PIE and FRGC databases validate the efficacy of the proposed method.     

Feature selection in MLPs and SVMs based on maximum output information

This paper presents feature selection algorithms for multilayer perceptrons (MLPs) and multiclass support vector machines (SVMs), using mutual information between class labels and classifier outputs, as an objective function. This objective function involves inexpensive computation of information measures only on discrete variables; provides immunity to prior class probabilities; and brackets the probability of error of the classifier. The maximum output information (MOI) algorithms employ this function for feature subset selection by greedy elimination and directed search. The output of the MOI algorithms is a feature subset of user-defined size and an associated trained classifier (MLP/SVM). These algorithms compare favorably with a number of other methods in terms of performance on various artificial and real-world data sets.     

Shape statistics in kernel space for variational image segmentation

We present a variational integration of nonlinear shape statistics into a Mumford-Shah based segmentation process. The nonlinear statistics are derived from a set of training silhouettes by a novel method of density estimation which can be considered as an extension of kernel PCA to a probabilistic framework.We assume that the training data forms a Gaussian distribution after a nonlinear mapping to a higher-dimensional feature space. Due to the strong nonlinearity, the corresponding density estimate in the original space is highly non-Gaussian.Applications of the nonlinear shape statistics in segmentation and tracking of 2D and 3D objects demonstrate that the segmentation process can incorporate knowledge on a large variety of complex real-world shapes. It makes the segmentation process robust against misleading information due to noise, clutter and occlusion.     

Distance metric learning-based kernel gram matrix learning for pattern analysis tasks in kernel feature space

Approaches to distance metric learning (DML) for Mahalanobis distance metric involve estimating a parametric matrix that is associated with a linear transformation. For complex pattern analysis tasks, it is necessary to consider the approaches to DML that involve estimating a parametric matrix that is associated with a nonlinear transformation. One such approach involves performing the DML of Mahalanobis distance in the feature space of a Mercer kernel. In this approach, the problem of estimation of a parametric matrix of Mahalanobis distance is formulated as a problem of learning an optimal kernel gram matrix from the kernel gram matrix of a base kernel by minimizing the logdet divergence between the kernel gram matrices. We propose to use the optimal kernel gram matrices learnt from the kernel gram matrix of the base kernels in pattern analysis tasks such as clustering, multi-class pattern classification and nonlinear principal component analysis. We consider the commonly used kernels such as linear kernel, polynomial kernel, radial basis function kernel and exponential kernel as well as hyper-ellipsoidal kernels as the base kernels for optimal kernel learning. We study the performance of the DML-based class-specific kernels for multi-class pattern classification using support vector machines. Results of our experimental studies on benchmark datasets demonstrate the effectiveness of the DML-based kernels for different pattern analysis tasks.     

Input space versus feature space in kernel-based methods

This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data     

Sparse online kernelized actor-critic Learning in reproducing kernel Hilbert space

In this paper, we develop a novel non-parametric online actor-critic reinforcement learning (RL) algorithm to solve optimal regulation problems for a class of continuous-time affine nonlinear dynamical systems. To deal with the value function approximation (VFA) with inherent nonlinear and unknown structure, a reproducing kernel Hilbert space (RKHS)-based kernelized method is designed through online sparsification, where the dictionary size is fixed and consists of updated elements. In addition, the linear independence check condition, i.e., an online criteria, is designed to determine whether the online data should be inserted into the dictionary. The RHKS-based kernelized VFA has a variable structure in accordance with the online data collection, which is different from classical parametric VFA methods with a fixed structure. Furthermore, we develop a sparse online kernelized actor-critic learning RL method to learn the unknown optimal value function and the optimal control policy in an adaptive fashion. The convergence of the presented kernelized actor-critic learning method to the optimum is provided. The boundedness of the closed-loop signals during the online learning phase can be guaranteed. Finally, a simulation example is conducted to demonstrate the effectiveness of the presented kernelized actor-critic learning algorithm.

     

Nonlinear feature extraction and classification of multivariate data in kernel feature space

Batch processes have played an essential role in the production of high value-added product of chemical, pharmaceutical, food, bio-chemical, and semi-conductor industries. For productivity and quality improvement, several multivariate statistical techniques such as principal component analysis (PCA) and Fisher discriminant analysis (FDA) have been developed to solve a fault diagnosis problem of batch processes. Fisher discriminant analysis, as a traditional statistical technique for feature extraction and classification, has been shown to be a good linear technique for fault diagnosis and outperform PCA based diagnosis methods. This paper proposes a more efficient nonlinear diagnosis method for batch processes using a kernel version of Fisher discriminant analysis (KFDA). A case study on two batch processes has been conducted. In addition, the diagnosis performance of the proposed method was compared with that of an existing diagnosis method based on linear FDA. The diagnosis results showed that the proposed KFDA based diagnosis method outperforms the linear FDA based method.     

Complete discriminant evaluation and feature extraction in kernel space for face recognition

This work proposes a method to decompose the kernel within-class eigenspace into two subspaces: a reliable subspace spanned mainly by the facial variation and an unreliable subspace due to limited number of training samples. A weighting function is proposed to circumvent undue scaling of eigenvectors corresponding to the unreliable small and zero eigenvalues. Eigenfeatures are then extracted by the discriminant evaluation in the whole kernel space. These efforts facilitate a discriminative and stable low-dimensional feature representation of the face image. Experimental results on FERET, ORL and GT databases show that our approach consistently outperforms other kernel based face recognition methods.

3D head model retrieval in kernel feature space using HSOM

In this paper, we propose a novel 3D head model retrieval framework. Specifically, to facilitate better classification and retrieval, the original 3D head model representations are embedded into another kernel feature space in which kernel principal component analysis (kernel PCA) is then performed to search for the optimal basis representation. Based on the extracted nonlinear features, a hierarchical indexing structure for 3D model retrieval is constructed using the hierarchical self organizing map (HSOM). The proposed indexing structure clusters the database into a hierarchy so that head models are partitioned by coarse features initially and then by finer scale features at lower levels. The main motivation of adopting this approach is that subspace technique like kernel PCA provides an elegant mechanism to describe the 3D head models on multiple resolutions based on the choices for reconstruction error and the orthogonal property of the produced eigenvectors. To further enhance the performance, a fuzzy metric between the query and the feature vector associated with each node on the SOMs is adopted instead of the usual Euclidean metric. Only nodes that possess high fuzzy measure values will be considered further for retrieval. In this way, the fuzzy measure approach is able to pick up potential relevant models even though they may be distributed across a number of neighbouring nodes. In addition to model categorization, the topology-preserving property of HSOM also facilitates the exploration of the model database with the possibility for further knowledge discovery. The effectiveness of the proposed approach is verified by a set of simulation examples on a 3D head model database.     

Robust semi-supervised discriminant embedding method with soft label in kernel space

Neural Computing and Applications - Considering some problems of local linear embedding methods in semi-supervised scenarios, a robust scheme for generating soft labels is designed and a...     

Feature space and metric measures for fusing multisensor images

A multivalued wavelet transform (MWT) is proposed to fuse multisensor images in feature space. First, feature space is constructed using image‐derived features, and then the MWT is introduced. The multisensor images are then fused in the MWT domain using a voting and electing fuser based on the cross‐feature scale guideline and the posterior probability of the MWT coefficient. The performance of the MWT is estimated using metric measures regarding various aspects of image quality. A fusion experiment using Thematic Mapper (TM) multispectral and SPOT panchromatic images of south China demonstrates that MWT outperforms smoothing filter‐based intensity modulation (SFM) in terms of the fidelity to spectral properties and the injection of salient information. The experimental results confirm that the MWT is a superior fusion method for enhancing spatial quality of multispectral images with their spectral properties reliably preserved.     

Image quality assessment method based on nonlinear feature extraction in kernel space

To match human perception, extracting perceptual features effectively plays an important role in image quality assessment. In contrast to most existing methods that use linear transformations or models to represent images, we employ a complex mathematical expression of high dimensionality to reveal the statistical characteristics of the images. Furthermore, by introducing kernel methods to transform the linear problem into a nonlinear one, a full-reference image quality assessment method is proposed based on high-dimensional nonlinear feature extraction. Experiments on the LIVE, TID2008, and CSIQ databases demonstrate that nonlinear features offer competitive performance for image inherent quality representation and the proposed method achieves a promising performance that is consistent with human subjective evaluation.     

核空间结合样本中心角度的大规模支持向量机

为了提高支持向量机在大规模数据集处理时的精度,提出了基于核空间和样本中心角度的支持向量机算法.在核特征空间下,求得原训练集的两类中心点和两个中心点的超法平面,并获取原训练集样本到超法平面距离和到两中心点中点的比值,用比值最小的n个样本点替代训练集.给出的数学模型显示,该算法不需要计算核空间,比现有的同类缩减策略保留了更多的支持向量数目.结合实例对算法进行了仿真实验,实验结果表明,与同类算法相比,该算法在基本没有降低训练速度的情况下获得了更准确的训练精度.     

Efficient solution of a class of partial integro-differential equation in reproducing kernel space

In [Y.l. Wang, T. Chaolu, Z. Chen, Using reproducing kernel for solving a class of singular weakly nonlinear boundary value problems, Int. J. Comput. Math. 87(2) (2010), pp. 367–380], we present three algorithms to solve a class of ordinary differential equations boundary value problems in reproducing kernel space. It is worth noting that our methods can get the solution of partial integro-differential equation. In this note, we use method 2 [M. Dehghan, Solution of a partial integro-differential equation arising from viscoelasticity, Int. J. Comput. Math. 83(1) (2006), pp. 123–129] to solve a class of partial integro-differential equation in reproducing kernel space. Numerical example shows our method is effective and has high accuracy.     

Design and comparative study of online kernel methods identification of nonlinear system in RKHS space

This paper proposes the design and a comparative study of two proposed online kernel methods identification in the reproducing kernel Hilbert space and other two kernel method existing in the literature. The two proposed methods, titled SVD-KPCA, online RKPCA. The two other techniques named Sliding Window Kernel Recursive Least Square and the Kernel Recursive Least Square. The considered performances are the Normalized Means Square Error, the consumed time and the numerical complexity. All methods are evaluated by handling a chemical process known as the Continuous Stirred Tank Reactor and Wiener-Hammerstein benchmark.