Learning bounds for kernel regression using effective data dimensionality

Kernel methods can embed finite-dimensional data into infinite-dimensional feature spaces. In spite of the large underlying feature dimensionality, kernel methods can achieve good generalization ability. This observation is often wrongly interpreted, and it has been used to argue that kernel learning can magically avoid the "curse-of-dimensionality" phenomenon encountered in statistical estimation problems. This letter shows that although using kernel representation, one can embed data into an infinite-dimensional feature space; the effective dimensionality of this embedding, which determines the learning complexity of the underlying kernel machine, is usually small. In particular, we introduce an algebraic definition of a scale-sensitive effective dimension associated with a kernel representation. Based on this quantity, we derive upper bounds on the generalization performance of some kernel regression methods. Moreover, we show that the resulting convergent rates are optimal under various circumstances.     

基于前向查找和均值漂移的点模型鲁棒降噪算法

针对点模型提出了基于前向查找和均值漂移两种鲁棒统计方法的滤波算法。前向查找算法根据残差图自动检测离群点,并将输入的点云数据划分为多个不带离群点的最优局部降噪邻域。对局部邻域进行加权协方差分析,估计出该邻域的最小二乘拟合平面。在局部邻域内估计采样点的核密度函数并通过均值漂移算法计算它的局部最大值点,核密度函数的局部最大值点确定了点云数据的聚类中心并能准确逼近采样点曲面,将每一个采样点漂移到密度函数的局部最大值点,使点云曲面收敛为一个稳定的三维数字模型。实验结果表明,本文的算法是鲁棒的,能在有效剔除点模型表面噪声的同时较好地保持模型表面的尖锐特征。     

Robust estimating equations and bias correction of correlation parameters for longitudinal data

The estimation of correlation parameters has received attention for both its own interest and improvement of the estimation efficiency of mean parameters by the generalized estimating equations (GEE) approach. Many of the well-established methods for the estimation of correlation parameters can be constructed under the GEE framework which is, however, sensitive to outliers. In this paper, we consider two ways of constructing robust estimating equations for achieving robust estimation of the correlation parameters. Furthermore, the estimators of the correlation parameters from the robustified GEE may be still biased as the expectation of the estimating equation is biased from zero when the underlying distribution is not symmetric. Therefore, bias-corrected robust estimators of correlation parameters are proposed. The performance of the proposed methods are investigated by simulation. The results show that the proposed robust and bias-corrected robust estimators can reduce the bias successfully. Two real data sets are analyzed for illustration.     

Domain transfer multiple kernel learning

Cross-domain learning methods have shown promising results by leveraging labeled patterns from the auxiliary domain to learn a robust classifier for the target domain which has only a limited number of labeled samples. To cope with the considerable change between feature distributions of different domains, we propose a new cross-domain kernel learning framework into which many existing kernel methods can be readily incorporated. Our framework, referred to as Domain Transfer Multiple Kernel Learning (DTMKL), simultaneously learns a kernel function and a robust classifier by minimizing both the structural risk functional and the distribution mismatch between the labeled and unlabeled samples from the auxiliary and target domains. Under the DTMKL framework, we also propose two novel methods by using SVM and prelearned classifiers, respectively. Comprehensive experiments on three domain adaptation data sets (i.e., TRECVID, 20 Newsgroups, and email spam data sets) demonstrate that DTMKL-based methods outperform existing cross-domain learning and multiple kernel learning methods.     

Efficient kernel density estimation using the fast gauss transform with applications to color modeling and tracking

Many vision algorithms depend on the estimation of a probability density function from observations. Kernel density estimation techniques are quite general and powerful methods for this problem, but have a significant disadvantage in that they are computationally intensive. In this paper, we explore the use of kernel density estimation with the fast Gauss transform (FGT) for problems in vision. The FGT allows the summation of a mixture of ill Gaussians at N evaluation points in O(M+N) time, as opposed to O(MN) time for a naive evaluation and can be used to considerably speed up kernel density estimation. We present applications of the technique to problems from image segmentation and tracking and show that the algorithm allows application of advanced statistical techniques to solve practical vision problems in real-time with today's computers.     

Statistical analysis of kernel-based least-squares density-ratio estimation

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), feature selection (mutual information), and conditional probability estimation. Several methods of directly estimating the density ratio have recently been developed, e.g., moment matching estimation, maximum-likelihood density-ratio estimation, and least-squares density-ratio fitting. In this paper, we propose a kernelized variant of the least-squares method for density-ratio estimation, which is called kernel unconstrained least-squares importance fitting (KuLSIF). We investigate its fundamental statistical properties including a non-parametric convergence rate, an analytic-form solution, and a leave-one-out cross-validation score. We further study its relation to other kernel-based density-ratio estimators. In experiments, we numerically compare various kernel-based density-ratio estimation methods, and show that KuLSIF compares favorably with other approaches.     

Decision tree ensembles based on kernel features

A classifier ensemble is a set of classifiers whose individual decisions are combined to classify new examples. Classifiers, which can represent complex decision boundaries are accurate. Kernel functions can also represent complex decision boundaries. In this paper, we study the usefulness of kernel features for decision tree ensembles as they can improve the representational power of individual classifiers. We first propose decision tree ensembles based on kernel features and found that the performance of these ensembles is strongly dependent on the kernel parameters; the selected kernel and the dimension of the kernel feature space. To overcome this problem, we present another approach to create ensembles that combines the existing ensemble methods with the kernel machine philosophy. In this approach, kernel features are created and concatenated with the original features. The classifiers of an ensemble are trained on these extended feature spaces. Experimental results suggest that the approach is quite robust to the selection of parameters. Experiments also show that different ensemble methods (Random Subspace, Bagging, Adaboost.M1 and Random Forests) can be improved by using this approach.     

A Primal Framework for Indefinite Kernel Learning

Kernel methods have been widely applied in machine learning to solve complex nonlinear problems. Kernel selection is one of the key issues in kernel methods, since it is vital for improving generalization performance. Traditionally, the selection of kernel is restricted to be positive definite which makes their applicability partially limited. Actually, in many real applications such as gene identification and object recognition, indefinite kernels frequently emerge and can achieve better performance. However, compared to positive definite ones, indefinite kernels are more complicated due to the non-convexity of the subsequent optimization problems, which leads to the incapability of most existing kernel algorithms. Some indefinite kernel methods have been proposed based on the dual of support vector machine (SVM), which mostly emphasize on how to transform the non-convex optimization to be convex by using positive definite kernels to approximate indefinite ones. In fact, the duality gap in SVM usually exists in the case of indefinite kernels and therefore these algorithms do not indeed solve the indefinite kernel problems themselves. In this paper, we present a novel framework for indefinite kernel learning derived directly from the primal of SVM, which establishes several new models not only for single indefinite kernel but also extends to multiple indefinite kernel scenarios. Several algorithms are developed to handle the non-convex optimization problems in these models. We further provide a constructive approach for kernel selection in the algorithms by using the theory of similarity functions. Experiments on real world datasets demonstrate the superiority of our models.     

Nonlinear nonnegative matrix factorization based on Mercer kernel construction

Generalizations ofnonnegative matrix factorization (NMF) in kernel feature space, such as projected gradient kernel NMF (PGKNMF) and polynomial Kernel NMF (PNMF), have been developed for face and facial expression recognition recently. However, these existing kernel NMF approaches cannot guarantee the nonnegativity of bases in kernel feature space and thus are essentially semi-NMF methods. In this paper, we show that nonlinear semi-NMF cannot extract the localized components which offer important information in object recognition. Therefore, nonlinear NMF rather than semi-NMF is needed to be developed for extracting localized component as well as learning the nonlinear structure. In order to address the nonlinear problem of NMF and the semi-nonnegative problem of the existing kernel NMF methods, we develop the nonlinear NMF based on a self-constructed Mercer kernel which preserves the nonnegative constraints on both bases and coefficients in kernel feature space. Experimental results in face and expressing recognition show that the proposed approach outperforms the existing state-of-the-art kernel methods, such as KPCA, GDA, PNMF and PGKNMF.     

Bayesian predictive kernel discriminant analysis

Discriminant analysis using Kernel Density Estimator (KDE) is a common tool for classification, but depends on the choice of the bandwidth or smoothing parameter of kernel. In this paper, we introduce a Bayesian Predictive Kernel Discriminant Analysis (BPKDA) eliminating this dependence by integrating the KDE with respect to an appropriate prior probability distribution for the bandwidth. Keypoints of the method are: (1) the formulation of the classification rule in terms of mixture predictive densities obtained by integrating kernel; (2) use of Independent Components Analysis (ICA) to choose a transform matrix so that transformed components are as independent as possible; and (3) nonparametric estimation of the predictive density by KDE for each independent component. Results on benchmark data sets and simulations show that the performance of BPKDA is competitive with, and in some cases significantly better than, Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA) and Naives Bayes discriminant Analysis with normal distribution (NNBDA).     

Clustering by kernel density

Kernel methods have been used for various supervised learning tasks. In this paper, we present a new clustering method based on kernel density. The method does not make any assumption on the number of clusters or on their shapes. The method is simple, robust, and behaves equally or better than other methods on problems known as difficult.     

Generalized Core Vector Machines

Kernel methods, such as the support vector machine (SVM), are often formulated as quadratic programming (QP) problems. However, given$m$training patterns, a naive implementation of the QP solver takes$O(m^3)$training time and at least$O(m^2)$space. Hence, scaling up these QPs is a major stumbling block in applying kernel methods on very large data sets, and a replacement of the naive method for finding the QP solutions is highly desirable. Recently, by using approximation algorithms for the minimum enclosing ball (MEB) problem, we proposed the core vector machine (CVM) algorithm that is much faster and can handle much larger data sets than existing SVM implementations. However, the CVM can only be used with certain kernel functions and kernel methods. For example, the very popular support vector regression (SVR) cannot be used with the CVM. In this paper, we introduce the center-constrained MEB problem and subsequently extend the CVM algorithm. The generalized CVM algorithm can now be used with any linear/nonlinear kernel and can also be applied to kernel methods such as SVR and the ranking SVM. Moreover, like the original CVM, its asymptotic time complexity is again linear in$m$and its space complexity is independent of$m$. Experiments show that the generalized CVM has comparable performance with state-of-the-art SVM and SVR implementations, but is faster and produces fewer support vectors on very large data sets.     

基于RGB-D图像核描述子的物体识别方法

针对传统的颜色-深度（RGB-D）图像物体识别的方法所存在的图像特征学习不全面、特征编码鲁棒性不够等问题,提出了基于核描述子局部约束线性编码（KD-LLC）的RGB-D图像物体识别方法。首先,在图像块间匹配核函数基础上,应用核主成分分析法提取RGB-D图像的3D形状、尺寸、边缘、颜色等多个互补性核描述子;然后,分别对它们进行LLC编码及空间池化处理以形成相应的图像编码向量;最后,把这些图像编码向量融合成具有鲁棒性、区分性的图像表示。基于RGB-D数据集的仿真实验结果表明,作为一种基于人工设计特征的RGB-D图像物体识别方法,由于所提算法综合利用深度图像和RGB图像的多方面特征,而且对传统深度核描述子的采样点选取和紧凑基向量的计算这两方面进行了改进,使得物体类别识别率达到86.8%,实体识别率达到92.7%,比其他同类方法具有更高的识别准确率。     

Robust regression with asymmetric heavy-tail noise distributions

In the presence of a heavy-tail noise distribution, regression becomes much more difficult. Traditional robust regression methods assume that the noise distribution is symmetric, and they downweight the influence of so-called outliers. When the noise distribution is asymmetric, these methods yield biased regression estimators. Motivated by data-mining problems for the insurance industry, we propose a new approach to robust regression tailored to deal with asymmetric noise distribution. The main idea is to learn most of the parameters of the model using conditional quantile estimators (which are biased but robust estimators of the regression) and to learn a few remaining parameters to combine and correct these estimators, to minimize the average squared error in an unbiased way. Theoretical analysis and experiments show the clear advantages of the approach. Results are on artificial data as well as insurance data, using both linear and neural network predictors.     

基于核学习方法的短时交通流量预测

基于核学习的强大非线性映射性能,针对短时交通流量预测,提出一类基于核学习方法的预测模型。核递推最小二乘(KRLS)基于近似线性依赖(approximate linear dependence,ALD) 技术可降低计算复杂度及存储量,是一种在线核学习方法,适用于较大规模数据集的学习;核偏最小二乘(KPLS)方法将输入变量投影在潜在变量上,利用输入与输出变量之间的协方差信息提取潜在特征;核极限学习机(KELM)方法用核函数表示未知的隐含层非线性特征映射,通过正则化最小二乘算法计算网络的输出权值,能以极快的学习速度获得良好的推广性。为验证所提方法的有效性,将KELM、KPLS、ALD-KRLS用于不同实测交通流数据中,在同等条件下,与现有方法进行比较。实验结果表明,不同核学习方法的预测精度和训练速度均有提高,体现了核学习方法在短时交通流量预测中的应用潜力。     

Discriminant Kernel Learning Using Hybrid Regularization

Kernel discriminant analysis (KDA) is one of the state-of-the-art kernel-based methods for pattern classification and dimensionality reduction. It performs linear discriminant analysis in the feature space via kernel function. However, the performance of KDA greatly depends on the selection of the optimal kernel for the learning task of interest. In this paper, we propose a novel algorithm termed as elastic multiple kernel discriminant analysis (EMKDA) by using hybrid regularization for automatically learning kernels over a linear combination of pre-specified kernel functions. EMKDA makes use of a mixing norm regularization function to compromise the sparsity and non-sparsity of the kernel weights. A semi-infinite program based algorithm is then proposed to solve EMKDA. Extensive experiments on synthetic datasets, UCI benchmark datasets, digit and terrain database are conducted to show the effectiveness of the proposed methods.     

Local normalized linear summation kernel for fast and robust recognition

Kernel-based methods are effective for object detection and recognition. However, the computational cost when using kernel functions is high, except when using linear kernels. To realize fast and robust recognition, we apply normalized linear kernels to local regions of a recognition target, and the kernel outputs are integrated by summation. This kernel is referred to as a local normalized linear summation kernel. Here, we show that kernel-based methods that employ local normalized linear summation kernels can be computed by a linear kernel of local normalized features. Thus, the computational cost of the kernel is nearly the same as that of a linear kernel and much lower than that of radial basis function (RBF) and polynomial kernels. The effectiveness of the proposed method is evaluated in face detection and recognition problems, and we confirm that our kernel provides higher accuracy with lower computational cost than RBF and polynomial kernels. In addition, our kernel is also robust to partial occlusion and shadows on faces since it is based on the summation of local kernels.     

Kernel Bandwidth Estimation for Nonparametric Modeling

Kernel density estimation is a nonparametric procedure for probability density modeling, which has found several applications in various fields. The smoothness and modeling ability of the functional approximation are controlled by the kernel bandwidth. In this paper, we describe a Bayesian estimation method for finding the bandwidth from a given data set. The proposed bandwidth estimation method is applied in three different computational-intelligence methods that rely on kernel density estimation: 1) scale space; 2) mean shift; and 3) quantum clustering. The third method is a novel approach that relies on the principles of quantum mechanics. This method is based on the analogy between data samples and quantum particles and uses the Schrodinger potential as a cost function. The proposed methodology is used for blind-source separation of modulated signals and for terrain segmentation based on topography information.     

Fast cross-validation of high-breakdown resampling methods for PCA

Cross-validation (CV) is a very popular technique for model selection and model validation. The general procedure of leave-one-out CV (LOO-CV) is to exclude one observation from the data set, to construct the fit of the remaining observations and to evaluate that fit on the item that was left out. In classical procedures such as least-squares regression or kernel density estimation, easy formulas can be derived to compute this CV fit or the residuals of the removed observations. However, when high-breakdown resampling algorithms are used, it is no longer possible to derive such closed-form expressions. High-breakdown methods are developed to obtain estimates that can withstand the effects of outlying observations. Fast algorithms are presented for LOO-CV when using a high-breakdown method based on resampling, in the context of robust covariance estimation by means of the MCD estimator and robust principal component analysis. A robust PRESS curve is introduced as an exploratory tool to select the number of principal components. Simulation results and applications on real data show the accuracy and the gain in computation time of these fast CV algorithms.     

Cluster Kernels: Resource-Aware Kernel Density Estimators over Streaming Data

A variety of real-world applications heavily relies on an adequate analysis of transient data streams. Due to the rigid processing requirements of data streams, common analysis techniques as known from data mining are not directly applicable. A fundamental building block of many data mining and analysis approaches is density estimation. It provides a well-defined estimation of a continuous data distribution, a fact, which makes its adaptation to data streams desirable. A convenient method for density estimation utilizes kernels. The computational complexity of kernel density estimation, however, renders its application to data streams impossible. In this paper, we tackle this problem and propose our Cluster Kernel approach which provides continuously computed kernel density estimators over streaming data. Not only do Cluster Kernels meet the rigid processing requirements of data streams, they also allocate only a constant amount of memory, even with the opportunity to adapt it dynamically to changing system resources. For this purpose, we develop an intelligent merge scheme for Cluster Kernels and utilize continuously collected local statistics to resample already processed data. We focus on Cluster Kernels for one-dimensional data streams, but also address the multi-dimensional case. We validate the efficacy of Cluster Kernels for a variety of real-world data streams in an extensive experimental study.