基于密度估计的逻辑回归模型

介绍了一种基于密度的逻辑回归（Density-based logistic regression,DLR）分类模型以解决逻辑回归中非线性分类的问题. 其主要思想是根据Nadarays-Watson密度估计将训练数据映射到特定的特征空间,然后组建优化模型优化特征权重以及Nadarays-Watson 密度估计算法的宽度. 其主要优点在于:它不仅优于标准的逻辑回归,而且优于基于径向基函数（Radial basis function,RBF）内核的核逻辑回归（Kernel logistic regression,KLR）. 特别是与核逻辑回归分析和支持向量机（Support vector machine,SVM）相比,该方法不仅达到更好的分类精度,而且有更好的时间效率. 该方法的另一个显著优点是,它可以很自然地扩展到数值类型和分类型混合的数据集中. 除此之外,该方法和逻辑回归（Logistic regression,LR）一样,有同样的模型可解释的优点,这恰恰是其他如核逻辑回归分析和支持向量机所不具备的.     

Financial distress prediction using support vector machines: Ensemble vs. individual

Financial distress prediction (FDP) is of great importance to both inner and outside parts of companies. Though lots of literatures have given comprehensive analysis on single classifier FDP method, ensemble method for FDP just emerged in recent years and needs to be further studied. Support vector machine (SVM) shows promising performance in FDP when compared with other single classifier methods. The contribution of this paper is to propose a new FDP method based on SVM ensemble, whose candidate single classifiers are trained by SVM algorithms with different kernel functions on different feature subsets of one initial dataset. SVM kernels such as linear, polynomial, RBF and sigmoid, and the filter feature selection/extraction methods of stepwise multi discriminant analysis (MDA), stepwise logistic regression (logit), and principal component analysis (PCA) are applied. The algorithm for selecting SVM ensemble's base classifiers from candidate ones is designed by considering both individual performance and diversity analysis. Weighted majority voting based on base classifiers’ cross validation accuracy on training dataset is used as the combination mechanism. Experimental results indicate that SVM ensemble is significantly superior to individual SVM classifier when the number of base classifiers in SVM ensemble is properly set. Besides, it also shows that RBF SVM based on features selected by stepwise MDA is a good choice for FDP when individual SVM classifier is applied.     

Facilitating the applications of support vector machine by using a new kernel

In the last few years, the applications of support vector machine (SVM) have substantially increased due to the high generalization performance and modeling of non-linear relationships. However, whether SVM behaves well largely depends on its adopted kernel function. The most commonly used kernels include linear, polynomial inner product functions and the Radial Basis Function (RBF), etc. Since the nature of the data is usually unknown, it is very difficult to make, on beforehand, a proper choice from the mentioned kernels. Usually, more than one kernel are applied to select the one which gives the best prediction performance but with a very time-consuming optimization procedure. This paper presents a kernel function based on Lorentzian function which is well-known in the field of statistics. The presented kernel can properly deal with a large variety of mapping problems due to its flexibility to vary. The applicability, suitability, performance and robustness of the presented kernel are investigated on bi-spiral benchmark data set as well as seven data sets from the UCI benchmark repository. The experiment results demonstrate that the presented kernel is robust and has stronger mapping ability comparing with the standard kernel functions, and it can obtain better generalization performance. In general, the proposed kernel can be served as a generic alternative for the common linear, polynomial and RBF kernels.     

Multi-layer radial basis function neural network based on multi-scale kernel learning

The conversion functions in the hidden layer of radial basis function neural networks (RBFNN) are Gaussian functions. The Gaussian functions are local to the kernel centers. In most of the existing research, the spatial local response of the sample is inaccurately calculated because the kernels have the same shape as a hypersphere, and the kernel parameters in the network are determined by experience. The influence of the fine structure in the local space is not considered during feature extraction. In addition, it is difficult to obtain a better feature extraction ability with less computational complexity. Therefore, this paper develops a multi-scale RBF kernel learning algorithm and proposes a new multi-layer RBF neural network model. For the samples of each class, the expectation maximization (EM) algorithm is used to obtain multi-layer nested sub-distribution models with different local response ranges, which are called multi-scale kernels in the network. The prior information of each sub-distribution is used as the connection weight between the multi-scale kernels. Finally, feature extraction is implemented using multi-layer kernel subspace embedding. The multi-scale kernel learning model can efficiently and accurately describe the fine structure of the samples and is fault tolerant to setting the number of kernels to a certain extent. Considering the prior probability of each kernel as the weight makes the feature extraction process satisfy the Bayes rule, which can enhance the interpretability of feature extraction in the network. This paper also theoretically proves that the proposed neural network is a generalized version of the original RBFNN. The experimental results show that the proposed method has better performance compared with some state-of-the-art algorithms.     

基于混合核函数的SVM在文本自动分类的应用

核函数是SVM的关键技术,核函数的选择将影响着学习机器的学习能力和泛化能力。不同的核函数确定了不同的非线性变换和特征空间,选取不同核函数训练SVM就会得到不同的分类效果。本文提出了一种混合的核函数[1]Kmix=λKpoly+(1-λ)Krbf,从而兼并二项式核函数及径向基核函数的优势。实验证明选用混合核函数的支持向量机,与普通核函数构造的支持向量机的评估效果进行比较,混合核函数支持向量机具有较高的分类精度。     

支持向量机在柴油机尾气分析中的核模型选择

研究支持向量机(SVM)中常用核函数及其参数对分类效果的影响.在此基础上,联系柴油机尾气数据,应用交又确认法(CV)得到在该数据集下拥有不同常用核函数的SVM的最优参数,及在最优参数下SVM的三个性能指标,即对训练集的交叉确认准确率、对测试集的分类准确率和寻优时间.对比各性能指标,结果表明:对于柴油机尾气数据,径向基核...     

Mixed KPCA结合纹理特征的SVM盐碱土信息提取

核函数是核主成分分析（Kernel Principal Component Analysis,KPCA）的核心,目前使用的核函数都是单一核函数。尝试通过将光谱角径向基核函数（Spectral Angle Radial Basis Function,SA-RBF）与RBF组合形成混合核函数。在研究中,利用基于该混合核函数的KPCA进行特征提取,将其光谱特征波段和纹理特征相结合用于盐碱土的SVM分类,将分类结果与其他SVM分类进行比较,结果表明：该方法优于其他SVM方法,能有效提取玛纳斯河流域绿洲区的盐碱土专题信息,分类精度是89.000%,kappa系数是0.876。     

应用于垃圾邮件过滤的词序列核

针对支持向量机(SVM)中常用核函数由于忽略文本结构而导致大量语义信息丢失的现象,提出一种类别相关度量的词序列核(WSK),并将其应用于垃圾邮件过滤。首先提取邮件文本特征并计算特征的类别相关度量,然后利用词序列核作为核函数训练支持向量机,训练过程中利用类别相关度量计算词的衰减系数,最后对邮件进行分类。实验结果表明,与常用核函数和字符串核相比,改进的词序列核分类准确率更高,提高了垃圾邮件过滤的准确率。     

A novel hybrid KPCA and SVM with GA model for intrusion detection

A novel support vector machine (SVM) model combining kernel principal component analysis (KPCA) with genetic algorithm (GA) is proposed for intrusion detection. In the proposed model, a multi-layer SVM classifier is adopted to estimate whether the action is an attack, KPCA is used as a preprocessor of SVM to reduce the dimension of feature vectors and shorten training time. In order to reduce the noise caused by feature differences and improve the performance of SVM, an improved kernel function (N-RBF) is proposed by embedding the mean value and the mean square difference values of feature attributes in RBF kernel function. GA is employed to optimize the punishment factor C, kernel parameters σ and the tube size ɛ of SVM. By comparison with other detection algorithms, the experimental results show that the proposed model performs higher predictive accuracy, faster convergence speed and better generalization.     

Benchmarking Least Squares Support Vector Machine Classifiers

In Support Vector Machines (SVMs), the solution of the classification problem is characterized by a (convex) quadratic programming (QP) problem. In a modified version of SVMs, called Least Squares SVM classifiers (LS-SVMs), a least squares cost function is proposed so as to obtain a linear set of equations in the dual space. While the SVM classifier has a large margin interpretation, the LS-SVM formulation is related in this paper to a ridge regression approach for classification with binary targets and to Fisher's linear discriminant analysis in the feature space. Multiclass categorization problems are represented by a set of binary classifiers using different output coding schemes. While regularization is used to control the effective number of parameters of the LS-SVM classifier, the sparseness property of SVMs is lost due to the choice of the 2-norm. Sparseness can be imposed in a second stage by gradually pruning the support value spectrum and optimizing the hyperparameters during the sparse approximation procedure. In this paper, twenty public domain benchmark datasets are used to evaluate the test set performance of LS-SVM classifiers with linear, polynomial and radial basis function (RBF) kernels. Both the SVM and LS-SVM classifier with RBF kernel in combination with standard cross-validation procedures for hyperparameter selection achieve comparable test set performances. These SVM and LS-SVM performances are consistently very good when compared to a variety of methods described in the literature including decision tree based algorithms, statistical algorithms and instance based learning methods. We show on ten UCI datasets that the LS-SVM sparse approximation procedure can be successfully applied.     

使用超椭球参数化坐标的支持向量机

基于n维超椭球面坐标变换公式,构造一类核函数--n维超椭球坐标变换核.由于是同维映射,且增大了类间距离,这类核函数在一定程度上改善了支持向量机的性能.与其他核函数(如高斯核)相比,将所构造的核函数用于支持向量机,仅产生了很少的支持向量,因而大大加快了学习速度,改善了泛化性能.数值实验结果表明了所构造的核函数的有效性和正确性.     

基于混合核支持向量机的金融时间序列分析

核函数是支持向量机（SVM）的重要部分,它直接影响到SVM的各项性能。当前SVM在金融时间序列分析中,基本上采用高斯径向核函数（RBF）,其次才是多项式核函数。然而,每种核函数都有它的优势和不足,整合两个或多个核函数对于学习能力和泛化能力的提高是一个有效的途径。采用高斯径向核函数与多项式核函数的混合核函数运用于金融时间序列预测中,且与其单个核函数的支持向量机的实验结果进行了比较。结果表明,混合核函数具有更好的性能。     

An enhanced Support Vector Machine classification framework by using Euclidean distance function for text document categorization

This paper presents the implementation of a new text document classification framework that uses the Support Vector Machine (SVM) approach in the training phase and the Euclidean distance function in the classification phase, coined as Euclidean-SVM. The SVM constructs a classifier by generating a decision surface, namely the optimal separating hyper-plane, to partition different categories of data points in the vector space. The concept of the optimal separating hyper-plane can be generalized for the non-linearly separable cases by introducing kernel functions to map the data points from the input space into a high dimensional feature space so that they could be separated by a linear hyper-plane. This characteristic causes the implementation of different kernel functions to have a high impact on the classification accuracy of the SVM. Other than the kernel functions, the value of soft margin parameter, C is another critical component in determining the performance of the SVM classifier. Hence, one of the critical problems of the conventional SVM classification framework is the necessity of determining the appropriate kernel function and the appropriate value of parameter C for different datasets of varying characteristics, in order to guarantee high accuracy of the classifier. In this paper, we introduce a distance measurement technique, using the Euclidean distance function to replace the optimal separating hyper-plane as the classification decision making function in the SVM. In our approach, the support vectors for each category are identified from the training data points during training phase using the SVM. In the classification phase, when a new data point is mapped into the original vector space, the average distances between the new data point and the support vectors from different categories are measured using the Euclidean distance function. The classification decision is made based on the category of support vectors which has the lowest average distance with the new data point, and this makes the classification decision irrespective of the efficacy of hyper-plane formed by applying the particular kernel function and soft margin parameter. We tested our proposed framework using several text datasets. The experimental results show that this approach makes the accuracy of the Euclidean-SVM text classifier to have a low impact on the implementation of kernel functions and soft margin parameter C.     

Support vector learning for fuzzy rule-based classification systems

To design a fuzzy rule-based classification system (fuzzy classifier) with good generalization ability in a high dimensional feature space has been an active research topic for a long time. As a powerful machine learning approach for pattern recognition problems, the support vector machine (SVM) is known to have good generalization ability. More importantly, an SVM can work very well on a high- (or even infinite) dimensional feature space. This paper investigates the connection between fuzzy classifiers and kernel machines, establishes a link between fuzzy rules and kernels, and proposes a learning algorithm for fuzzy classifiers. We first show that a fuzzy classifier implicitly defines a translation invariant kernel under the assumption that all membership functions associated with the same input variable are generated from location transformation of a reference function. Fuzzy inference on the IF-part of a fuzzy rule can be viewed as evaluating the kernel function. The kernel function is then proven to be a Mercer kernel if the reference functions meet a certain spectral requirement. The corresponding fuzzy classifier is named positive definite fuzzy classifier (PDFC). A PDFC can be built from the given training samples based on a support vector learning approach with the IF-part fuzzy rules given by the support vectors. Since the learning process minimizes an upper bound on the expected risk (expected prediction error) instead of the empirical risk (training error), the resulting PDFC usually has good generalization. Moreover, because of the sparsity properties of the SVMs, the number of fuzzy rules is irrelevant to the dimension of input space. In this sense, we avoid the "curse of dimensionality." Finally, PDFCs with different reference functions are constructed using the support vector learning approach. The performance of the PDFCs is illustrated by extensive experimental results. Comparisons with other methods are also provided.     

疲劳驾驶面部表情识别算法

针对疲劳驾驶的六种表情 ,提出几何规范化结合 Gabor滤波提取表情特征 ,使用支持向量机对疲劳驾驶的面部表情分类识别的系统。首先对视频图像预处理进行几何规范化 ,利用二维 Gabor核函数构造最优滤波器 48个,获取 48个面部表情特征点 ,最后利用支持向量机进行面部表情分类识别。实验结果表明径向基函数的 SVM性能最好。     

基于混合核函数FOA-LSSVM的预测模型

支持向量机（SVM）的核函数类型和超参数对预测的精度有重要影响。由于局部核函数学习能力强、泛化性能弱,而全局核函数泛化性能强、学习能力弱的矛盾,通过综合两类核函数各自优点构造了基于全局多项式核和高斯核的混合核函数,并引入果蝇优化算法（FOA）对最小二乘支持向量机（LSSVM）参数进行全局寻优,提出了混合核函数FOA-LSSVM 预测模型。结果表明,该模型较传统方法在电力负荷预测精度上有了明显提高,预测结果科学可靠,在预测中具有良好的实际应用价值。     

多尺度核方法的自适应序列学习及应用

多尺度核方法是当前核机器学习领域的一个热点。通常多尺度核的学习在多核处理时存在诸如多核平均组合、迭代学习时间长、经验选择合成系数等弊端。文中基于核目标度量规则,提出一种多尺度核方法的自适应序列学习算法,实现多核加权系数的自动快速求取。实验表明,该方法在回归精度、分类正确率方面比单核支持向量机方法结果更优,函数拟合与分类稳定性更强,证明该算法具有普遍适用性。     

Support vector machines for histogram-based image classification.

Traditional classification approaches generalize poorly on image classification tasks, because of the high dimensionality of the feature space. This paper shows that support vector machines (SVM) can generalize well on difficult image classification problems where the only features are high dimensional histograms. Heavy-tailed RBF kernels of the form K(x, y)=e(-rho)Sigma(i)(|xia-yia|b ) with a =/<1 and b=/<2 are evaluated on the classification of images extracted from the Corel stock photo collection and shown to far outperform traditional polynomial or Gaussian radial basis function (RBF) kernels. Moreover, we observed that a simple remapping of the input x(i)-->x(i)(a) improves the performance of linear SVM to such an extend that it makes them, for this problem, a valid alternative to RBF kernels.     

Local normalized linear summation kernel for fast and robust recognition

Kernel-based methods are effective for object detection and recognition. However, the computational cost when using kernel functions is high, except when using linear kernels. To realize fast and robust recognition, we apply normalized linear kernels to local regions of a recognition target, and the kernel outputs are integrated by summation. This kernel is referred to as a local normalized linear summation kernel. Here, we show that kernel-based methods that employ local normalized linear summation kernels can be computed by a linear kernel of local normalized features. Thus, the computational cost of the kernel is nearly the same as that of a linear kernel and much lower than that of radial basis function (RBF) and polynomial kernels. The effectiveness of the proposed method is evaluated in face detection and recognition problems, and we confirm that our kernel provides higher accuracy with lower computational cost than RBF and polynomial kernels. In addition, our kernel is also robust to partial occlusion and shadows on faces since it is based on the summation of local kernels.     

一种支持向量机的混合核函数

核函数是支持向量机的核心,不同的核函数将产生不同的分类效果.由于普通核函数各有其利弊,为了得到学习能力和泛化能力较强的核函数,根据核函数的基本性质,两个核函数之和仍然是核函数,将局部核函数和全局核函数线性组合构成新的核函数--混合核函数.该核函数吸取了局部核函数和全局核函数的优点.利用混合核函数进行流程企业供应链预测实验,仿真结果验证了该核函数的有效性和正确性.