Lagrangian relaxation for SVM feature selection

We discuss a Lagrangian-relaxation-based heuristics for dealing with feature selection in the Support Vector Machine (SVM) framework for binary classification. In particular we embed into our objective function a weighted combination of the L₁ and L₀ norm of the normal to the separating hyperplane. We come out with a Mixed Binary Linear Programming problem which is suitable for a Lagrangian relaxation approach.Based on a property of the optimal multiplier setting, we apply a consolidated nonsmooth optimization ascent algorithm to solve the resulting Lagrangian dual. In the proposed approach we get, at every ascent step, both a lower bound on the optimal solution as well as a feasible solution at low computational cost.We present the results of our numerical experiments on some benchmark datasets.     

改进的支持向量机工作集选择算法

SVM-light使用可行方向法来完成工作集选取,但仅考虑了目标函数的一次信息导致收敛速度不高.通过对LibSVM中所使用二次信息来选择工作集方法的研究,对SVM-light的工作集选择的过程进行改进,在其工作集选择算法中加入了二次信息.几个公共数据集的实验中证明了改进的SVM-light比原有SVM-light有着更快的收敛速度.     

Optimal SVM parameter selection for non-separable and unbalanced datasets

This article presents a study of three validation metrics used for the selection of optimal parameters of a support vector machine (SVM) classifier in the case of non-separable and unbalanced datasets. This situation is often encountered when the data is obtained experimentally or clinically. The three metrics selected in this work are the area under the ROC curve (AUC), accuracy, and balanced accuracy. These validation metrics are tested using computational data only, which enables the creation of fully separable sets of data. This way, non-separable datasets, representative of a real-world problem, can be created by projection onto a lower dimensional sub-space. The knowledge of the separable dataset, unknown in real-world problems, provides a reference to compare the three validation metrics using a quantity referred to as the “weighted likelihood”. As an application example, the study investigates a classification model for hip fracture prediction. The data is obtained from a parameterized finite element model of a femur. The performance of the various validation metrics is studied for several levels of separability, ratios of unbalance, and training set sizes.     

Automatic model selection for the optimization of SVM kernels

This approach aims to optimize the kernel parameters and to efficiently reduce the number of support vectors, so that the generalization error can be reduced drastically. The proposed methodology suggests the use of a new model selection criterion based on the estimation of the probability of error of the SVM classifier. For comparison, we considered two more model selection criteria: GACV (‘Generalized Approximate Cross-Validation’) and VC (‘Vapnik-Chernovenkis’) dimension. These criteria are algebraic estimates of upper bounds of the expected error. For the former, we also propose a new minimization scheme. The experiments conducted on a bi-class problem show that we can adequately choose the SVM hyper-parameters using the empirical error criterion. Moreover, it turns out that the criterion produces a less complex model with fewer support vectors. For multi-class data, the optimization strategy is adapted to the one-against-one data partitioning. The approach is then evaluated on images of handwritten digits from the USPS database.     

Neighborhood based sample and feature selection for SVM classification learning

Support vector machines (SVMs) are a class of popular classification algorithms for their high generalization ability. However, it is time-consuming to train SVMs with a large set of learning samples. Improving learning efficiency is one of most important research tasks on SVMs. It is known that although there are many candidate training samples in some learning tasks, only the samples near decision boundary which are called support vectors have impact on the optimal classification hyper-planes. Finding these samples and training SVMs with them will greatly decrease training time and space complexity. Based on the observation, we introduce neighborhood based rough set model to search boundary samples. Using the model, we firstly divide sample spaces into three subsets: positive region, boundary and noise. Furthermore, we partition the input features into four subsets: strongly relevant features, weakly relevant and indispensable features, weakly relevant and superfluous features, and irrelevant features. Then we train SVMs only with the boundary samples in the relevant and indispensable feature subspaces, thus feature and sample selection is simultaneously conducted with the proposed model. A set of experimental results show the model can select very few features and samples for training; in the mean time the classification performances are preserved or even improved.     

支持向量机在多因子选股的预测优化

使用财务数据构建一个多因子选股模型,在支持向量机分类上进行预测优化。选股上使用排序法对数据进行预处理,再使用支持向量机对股票收益进行分类预测,最后使用数据到分离超平面的距离进行排序,优化支持向量机的分类预测。实证中,从中证500成分股中选出股票组合,在2016年四季度到2018年一季度获得累计收益88. 96%。择时策略的均线策略和通道突破策略均能有效降低波动率和回撤。还使用高频数据来降低均线策略的滞后性,波动率又得到进一步降低。本模型利用支持向量机性质提高预测精度,结合技术分析优化了策略的收益,为多因子选股和交易提供了新的研究视角。     

A Convergent Hybrid Decomposition Algorithm Model for SVM Training

Training of support vector machines (SVMs) requires to solve a linearly constrained convex quadratic problem. In real applications, the number of training data may be very huge and the Hessian matrix cannot be stored. In order to take into account this issue, a common strategy consists in using decomposition algorithms which at each iteration operate only on a small subset of variables, usually referred to as the working set. Training time can be significantly reduced by using a caching technique that allocates some memory space to store the columns of the Hessian matrix corresponding to the variables recently updated. The convergence properties of a decomposition method can be guaranteed by means of a suitable selection of the working set and this can limit the possibility of exploiting the information stored in the cache. We propose a general hybrid algorithm model which combines the capability of producing a globally convergent sequence of points with a flexible use of the information in the cache. As an example of a specific realization of the general hybrid model, we describe an algorithm based on a particular strategy for exploiting the information deriving from a caching technique. We report the results of computational experiments performed by simple implementations of this algorithm. The numerical results point out the potentiality of the approach.      

Multi-objective uniform design as a SVM model selection tool for face recognition

The primary difficulty of support vector machine (SVM) model selection is heavy computational cost, thus it is difficult for current model selection methods to be applied in face recognition. Model selection via uniform design can effectively alleviate the computational cost, but its drawback is that it adopts a single objective criterion which can not always guarantee the generalization capacity. The sensitivity and specificity as multi-objective criteria have been proved of better performance and can provide a means for obtaining more realistic models. This paper first proposes a multi-objective uniform design (MOUD) search method as a SVM model selection tool, and then applies this optimized SVM classifier to face recognition. Because of replacing single objective criterion with multi-objective criteria and adopting uniform design to seek experimental points that uniformly scatter on whole experimental domain, MOUD can reduce the computational cost and improve the classification ability simultaneously. The experiments are executed on UCI benchmark, and on Yale and CAS-PEAL-R1 face databases. The experimental results show that the proposed method outperforms other model search methods significantly, especially for face recognition.     

结合特征选择的二叉树SVM多分类算法

为解决现有二叉树SVM多分类算法采用固定的特征集和结构存在分类精度较低的问题,提出了一种结合特征选择的二又树SVM多类分类算法,采用自上而下分裂的方式构造整个二又树结构,首先计算各节点的所有可能分割,并以分离度和相似度作为依据为各分割选择有效的分类特征子集,再以相应的特征子集计算各分割的类间距,最后选择类间距最大的分割生成子节点,实验结果表明,该算法分类精度较高且计算复杂度低.     

基于一种改进粒子群算法的SVM参数选取

支持向量机作为一个新兴的数学建模工具已经被广泛地应用到很多工业控制领域中,其良好的泛化能力和预测精度在很大程度上受到其参数选取的影响.根据智能群体进化模式改进粒子群优化算法.利用模糊C均值聚类算法分类粒子群体,并用子群体最优点取代速度更新公式中的个体历史最优点,并利用该算法搜索支持向量机的最优参数组合.对比仿真实验表明:所提优化算法是支持向量机参数选取的有效算法,在非线性函数估计中体现出优良的性能.     

Feature selection via sensitivity analysis of SVM probabilistic outputs

Feature selection is an important aspect of solving data-mining and machine-learning problems. This paper proposes a feature-selection method for the Support Vector Machine (SVM) learning. Like most feature-selection methods, the proposed method ranks all features in decreasing order of importance so that more relevant features can be identified. It uses a novel criterion based on the probabilistic outputs of SVM. This criterion, termed Feature-based Sensitivity of Posterior Probabilities (FSPP), evaluates the importance of a specific feature by computing the aggregate value, over the feature space, of the absolute difference of the probabilistic outputs of SVM with and without the feature. The exact form of this criterion is not easily computable and approximation is needed. Four approximations, FSPP1-FSPP4, are proposed for this purpose. The first two approximations evaluate the criterion by randomly permuting the values of the feature among samples of the training data. They differ in their choices of the mapping function from standard SVM output to its probabilistic output: FSPP1 uses a simple threshold function while FSPP2 uses a sigmoid function. The second two directly approximate the criterion but differ in the smoothness assumptions of criterion with respect to the features. The performance of these approximations, used in an overall feature-selection scheme, is then evaluated on various artificial problems and real-world problems, including datasets from the recent Neural Information Processing Systems (NIPS) feature selection competition. FSPP1-3 show good performance consistently with FSPP2 being the best overall by a slight margin. The performance of FSPP2 is competitive with some of the best performing feature-selection methods in the literature on the datasets that we have tested. Its associated computations are modest and hence it is suitable as a feature-selection method for SVM applications. Editor: Risto Miikkulainen.     

事件检测支持向量机模型与神经网络模型比较

针对交通领域中的事件检测(无事件模式和有事件模式)模式识别问题,描述了支持向量机(SVM)的基本方法,建立了基于线性(linearfunction)、多项式(polynomialfunction)和径向基(radialbasisfunction)3种核函数的事件检测SVM模型,并与PNN、MLF模型进行了理论比较。采用I-880线圈数据集和事件数据集建立并验证SVM、PNN和MLF模型,结果发现:无论对于向北、向南或混合方向的事件检测,SVM模型的检测率(DR)、误报率(FAR)和平均检测时间(MTTD)指标均比MLF模型好;PNN模型的DR比SVM(P)模型的高,但FAR和MTTD指标不比SVM(P)模型好;在3个SVM模型中,SVM(P)检测效果最好,SVM(L)最差。SVM算法与神经网络算法相比具有避免局部最小,实现全局最优化,更好的泛化效果的优点,是高速公路事件检测的一种很有潜力的算法。     

基于支持向量机的TSK模糊模型辨识与控制

研究非线性系统TSK模糊模型的辨识与控制,利用TSK模型,可以将线性控制理论应用于非线性系统控制。基于支持向量机和递推最小二乘法,辨识出TSK模糊模型,并且通过遗传算法优化隶属度函数参数,最小化辨识误差。针对TSK模型进行控制,控制器包括两个部分：权重最大子系统反馈控制及其监督控制,监督控制保证了系统的稳定性。辨识和控制仿真结果证明了算法的有效性。     

Applying 1-norm SVM with squared loss to gene selection for cancer classification

Gene selection methods available have high computational complexity. This paper applies an 1-norm support vector machine with the squared loss (1-norm SVMSL) to implement fast gene selection for cancer classification. The 1-norm SVMSL, a variant of the 1-norm support vector machine (1-norm SVM) has been proposed. Basically, the 1-norm SVMSL can perform gene selection and classification at the same. However, to improve classification performance, we only use the 1-norm SVMSL as a gene selector, and adopt a subsequent classifier to classify the selected genes. We perform extensive experiments on four DNA microarray data sets. Experimental results indicate that the 1-norm SVMSL has a very fast gene selection speed compared with other methods. For example, the 1-norm SVMSL is almost an order of magnitude faster than the 1-norm SVM, and at least four orders of magnitude faster than SVM-RFE (recursive feature elimination), a state-of-the-art method.     

结合多特征的支持向量机中文组织机构名识别模型

以支持向量机（SVM）为基本框架,提出一种结合多特征的支持向量机中文组织机构名识别模型。考虑中文组织机构名的特点,抽取局部特征与全局特征,并将特征向量转化为二进制表示,在此基础上建立训练集。基于1998年《人民日报》语料的实验结果表明,该混合模型对中文组织机构名的识别是有效的。同时基于不同测试数据的实验结果表明．该模型对不同测试数据源具有一致性。     

A primal method for multiple kernel learning

The canonical support vector machines (SVMs) are based on a single kernel, recent publications have shown that using multiple kernels instead of a single one can enhance interpretability of the decision function and promote classification accuracy. However, most of existing approaches mainly reformulate the multiple kernel learning as a saddle point optimization problem which concentrates on solving the dual. In this paper, we show that the multiple kernel learning (MKL) problem can be reformulated as a BiConvex optimization and can also be solved in the primal. While the saddle point method still lacks convergence results, our proposed method exhibits strong optimization convergence properties. To solve the MKL problem, a two-stage algorithm that optimizes canonical SVMs and kernel weights alternately is proposed. Since standard Newton and gradient methods are too time-consuming, we employ the truncated-Newton method to optimize the canonical SVMs. The Hessian matrix need not be stored explicitly, and the Newton direction can be computed using several Preconditioned Conjugate Gradient steps on the Hessian operator equation, the algorithm is shown more efficient than the current primal approaches in this MKL setting. Furthermore, we use the Nesterov’s optimal gradient method to optimize the kernel weights. One remarkable advantage of solving in the primal is that it achieves much faster convergence rate than solving in the dual and does not require a two-stage algorithm even for the single kernel LapSVM. Introducing the Laplacian regularizer, we also extend our primal method to semi-supervised scenario. Extensive experiments on some UCI benchmarks have shown that the proposed algorithm converges rapidly and achieves competitive accuracy.     

基于SVM的丙烯酰胺均相聚合预测

为了提高丙烯酰胺均相聚合预测的精度,建立了基于支持向量机的丙烯酰胺均相聚合预测模型,并采用此模型对实测数据进行了预测。与神经网络的预测结果相比,建立的新型聚合预测模型具有更好的预测精度。     

SVM feature selection for classification of SPECT images of Alzheimer's disease using spatial information

Alzheimer's disease is the most frequent type of dementia for elderly patients. Due to aging populations, the occurrence of this disease will increase in the next years. Early diagnosis is crucial to be able to develop more powerful treatments. Brain perfusion changes can be a marker for Alzheimer's disease. In this article, we study the use of SPECT perfusion imaging for the diagnosis of Alzheimer's disease differentiating between images from healthy subjects and images from Alzheimer's disease patients. Our classification approach is based on a linear programming formulation similar to the 1-norm support vector machines. In contrast with other linear hyperplane-based methods that perform simultaneous feature selection and classification, our proposed formulation incorporates proximity information about the features and generates a classifier that does not just select the most relevant voxels but the most relevant “areas” for classification resulting in more robust classifiers that are better suitable for interpretation. This approach is compared with the classical Fisher linear discriminant (FLD) classifier as well as with statistical parametric mapping (SPM). We tested our method on data from four European institutions. Our method achieved sensitivity of 84.4% at 90.9% specificity, this is considerable better the human experts. Our method also outperformed the FLD and SPM techniques. We conclude that our approach has the potential to be a useful help for clinicians. Glenn Fung received a B.S. degree in pure mathematics from Universidad Lisandro Alvarado in Barquisimeto, Venezuela, then earned an M.S. in applied mathematics from Universidad Simon Bolivar, Caracas, Venezuela, where later he worked as an assistant professor for 2 years. Later, he earned an M.S. degree and a Ph.D. degree in computer sciences from the University of Wisconsin-Madison. His main interests are optimization approaches to machine learning and data mining, with emphasis in support vector machines. In the summer of 2003, he joined the computer aided diagnosis group at Siemens, Medical Solutions in Malvern, PA, where he has been applying machine learning techniques to solve challenging problems that arise in the medical domain. His recent papers are available at . Jonathan Stoeckel received a B.E. degree from Xi’an Jiao Tong University, Xi'an, China, in 1993 and an M.E. degree from Shanghai Jiao Tong University, Shanghai, China, in 1996. From 1997 to 1998, he did research work in the Data Mining Group at the School of Computing and Information Technology, Griffith University, Brisbane, Australia. He is currently a Ph.D. student at the Department of Computer Science, Dartmouth College, USA. His research interests include data mining, multimedia, database and software engineering.     

Model selection by bootstrap penalization for classification

We consider the binary classification problem. Given an i.i.d. sample drawn from the distribution of an χ×{0,1}?valued random pair, we propose to estimate the so-called Bayes classifier by minimizing the sum of the empirical classification error and a penalty term based on Efron’s or i.i.d. weighted bootstrap samples of the data. We obtain exponential inequalities for such bootstrap type penalties, which allow us to derive non-asymptotic properties for the corresponding estimators. In particular, we prove that these estimators achieve the global minimax risk over sets of functions built from Vapnik-Chervonenkis classes. The obtained results generalize Koltchinskii (2001) and Bartlett et al.’s (2002) ones for Rademacher penalties that can thus be seen as special examples of bootstrap type penalties. To illustrate this, we carry out an experimental study in which we compare the different methods for an intervals model selection problem.     

The curvature primal sketch

In this paper we introduce a novel representation of the significant changes in curvature along the bounding contour of planar shape. We call the representation the Curvature Primal Sketch because of the close analogy to the primal sketch representation advocated by Marr for describing significant intensity changes. We define a set of primitive parameterized curvature discontinuities, and derive expressions for their convolutions with the first and second derivatives of a Gaussian. We describe an implemented algorithm that computes the Curvature Primal Sketch by matching the multiscale convolutions of a shape, and illustrate its performance on a set of tool shapes. Several applications of the representation are sketched.