A novel fuzzy compensation multi-class support vector machine

This paper presents a novel fuzzy compensation multi-class support vector machine (FCM-SVM) to improve the outlier and noise sensitivity problem of traditional support vector machine (SVM) for multi-class data classification. The basic idea is to give the dual effects to penalty term through treating every data point as both positive and negative classes, but with different memberships. We fuzzify penalty term, compensate weight to classification, reconstruct the optimization problem and its restrictions, reconstruct {Lagrangian} formula, and present the theoretic deduction. By this way the new fuzzy compensation multi-class support vector machine is expected to have more generalization ability while preserving the merit of insensitive to outliers. Experimental results on benchmark data set and real data set show that the proposed method reduces the effect of noise data and yields higher classification rate than traditional multi-class SVM does.     

多标签代价敏感分类集成学习算法

尽管多标签分类问题可以转换成一般多分类问题解决,但多标签代价敏感分类问题却很难转换成多类代价敏感分类问题.通过对多分类代价敏感学习算法扩展为多标签代价敏感学习算法时遇到的一些问题进行分析,提出了一种多标签代价敏感分类集成学习算法.算法的平均错分代价为误检标签代价和漏检标签代价之和,算法的流程类似于自适应提升（Adaptive boosting,AdaBoost）算法,其可以自动学习多个弱分类器来组合成强分类器,强分类器的平均错分代价将随着弱分类器增加而逐渐降低.详细分析了多标签代价敏感分类集成学习算法和多类代价敏感AdaBoost算法的区别,包括输出标签的依据和错分代价的含义.不同于通常的多类代价敏感分类问题,多标签代价敏感分类问题的错分代价要受到一定的限制,详细分析并给出了具体的限制条件.简化该算法得到了一种多标签AdaBoost算法和一种多类代价敏感AdaBoost算法.理论分析和实验结果均表明提出的多标签代价敏感分类集成学习算法是有效的,该算法能实现平均错分代价的最小化.特别地,对于不同类错分代价相差较大的多分类问题,该算法的效果明显好于已有的多类代价敏感AdaBoost算法.     

A simplified multi-class support vector machine with reduced dual optimization

Support vector machine (SVM) was initially designed for binary classification. To extend SVM to the multi-class scenario, a number of classification models were proposed such as the one by Crammer and Singer (2001). However, the number of variables in Crammer and Singer’s dual problem is the product of the number of samples (l) by the number of classes (k), which produces a large computational complexity. This paper presents a simplified multi-class SVM (SimMSVM) that reduces the size of the resulting dual problem from l × k to l by introducing a relaxed classification error bound. The experimental results demonstrate that the proposed SimMSVM approach can greatly speed-up the training process, while maintaining a competitive classification accuracy.     

基于SVM的多类别轨迹的分类方法

轨迹分析是解决视觉监控系统中异常检测问题的重要途径.文章将对轨迹进行采样得到的坐标点集作为特征向量,利用SVM训练分类器,并采用一对一算法实现多类别轨迹的分类.实验结果表明,该方法能够满足SVM中核函数对于输入数据的要求,并实现对多类别轨迹的有效分类.     

A closed-form reduction of multi-class cost-sensitive learning to weighted multi-class learning

In cost-sensitive learning, misclassification costs can vary for different classes. This paper investigates an approach reducing a multi-class cost-sensitive learning to a standard classification task based on the data space expansion technique developed by Abe et al., which coincides with Elkan's reduction with respect to binary classification tasks. Using this proposed reduction approach, a cost-sensitive learning problem can be solved by considering a standard 0/1 loss classification problem on a new distribution determined by the cost matrix. We also propose a new weighting mechanism to solve the reduced standard classification problem, based on a theorem stating that the empirical loss on independently identically distributed samples from the new distribution is essentially the same as the loss on the expanded weighted training set. Experimental results on several synthetic and benchmark datasets show that our weighting approach is more effective than existing representative approaches for cost-sensitive learning.     

A genetically optimized neural network model for multi-class classification

Multi-class classification is one of the major challenges in real world application. Classification algorithms are generally binary in nature and must be extended for multi-class problems. Therefore, in this paper, we proposed an enhanced Genetically Optimized Neural Network (GONN) algorithm, for solving multi-class classification problems. We used a multi-tree GONN representation which integrates multiple GONN trees; each individual is a single GONN classifier. Thus enhanced classifier is an integrated version of individual GONN classifiers for all classes. The integrated version of classifiers is evolved genetically to optimize its architecture for multi-class classification. To demonstrate our results, we had taken seven datasets from UCI Machine Learning repository and compared the classification accuracy and training time of enhanced GONN with classical Koza’s model and classical Back propagation model. Our algorithm gives better classification accuracy of almost 5% and 8% than Koza’s model and Back propagation model respectively even for complex and real multi-class data in lesser amount of time. This enhanced GONN algorithm produces better results than popular classification algorithms like Genetic Algorithm, Support Vector Machine and Neural Network which makes it a good alternative to the well-known machine learning methods for solving multi-class classification problems. Even for datasets containing noise and complex features, the results produced by enhanced GONN is much better than other machine learning algorithms. The proposed enhanced GONN can be applied to expert and intelligent systems for effectively classifying large, complex and noisy real time multi-class data.     

一类直接构造的模糊多类支持向量分类器

针对直接多类分类方法,提出了一种新的基于直接构造多类SVM分类器的模糊多类支持向量机算法FCS-SVM。在算法中,重构了优化问题及其约束条件,以及Lagrange公式,并进行了推导。通过在标准数据集上的几个实验,对这些算法进行了比较分析。实验结果表明提出的算法可以得到比较理想的分类精度。     

Boosted multi-class semi-supervised learning for human action recognition

Human action recognition is a challenging task due to significant intra-class variations, occlusion, and background clutter. Most of the existing work use the action models based on statistic learning algorithms for classification. To achieve good performance on recognition, a large amount of the labeled samples are therefore required to train the sophisticated action models. However, collecting labeled samples is labor-intensive. To tackle this problem, we propose a boosted multi-class semi-supervised learning algorithm in which the co-EM algorithm is adopted to leverage the information from unlabeled data. Three key issues are addressed in this paper. Firstly, we formulate the action recognition in a multi-class semi-supervised learning problem to deal with the insufficient labeled data and high computational expense. Secondly, boosted co-EM is employed for the semi-supervised model construction. To overcome the high dimensional feature space, weighted multiple discriminant analysis (WMDA) is used to project the features into low dimensional subspaces in which the Gaussian mixture models (GMM) are trained and boosting scheme is used to integrate the subspace models. Thirdly, we present the upper bound of the training error in multi-class framework, which is able to guide the novel classifier construction. In theory, the proposed solution is proved to minimize this upper error bound. Experimental results have shown good performance on public datasets.     

Distance metric learning-based kernel gram matrix learning for pattern analysis tasks in kernel feature space

Approaches to distance metric learning (DML) for Mahalanobis distance metric involve estimating a parametric matrix that is associated with a linear transformation. For complex pattern analysis tasks, it is necessary to consider the approaches to DML that involve estimating a parametric matrix that is associated with a nonlinear transformation. One such approach involves performing the DML of Mahalanobis distance in the feature space of a Mercer kernel. In this approach, the problem of estimation of a parametric matrix of Mahalanobis distance is formulated as a problem of learning an optimal kernel gram matrix from the kernel gram matrix of a base kernel by minimizing the logdet divergence between the kernel gram matrices. We propose to use the optimal kernel gram matrices learnt from the kernel gram matrix of the base kernels in pattern analysis tasks such as clustering, multi-class pattern classification and nonlinear principal component analysis. We consider the commonly used kernels such as linear kernel, polynomial kernel, radial basis function kernel and exponential kernel as well as hyper-ellipsoidal kernels as the base kernels for optimal kernel learning. We study the performance of the DML-based class-specific kernels for multi-class pattern classification using support vector machines. Results of our experimental studies on benchmark datasets demonstrate the effectiveness of the DML-based kernels for different pattern analysis tasks.     

一种新的基于二叉树的SVM多类分类方法

介绍了几种常用的支持向量机多类分类方法,分析其存在的问题及缺点。提出了一种基于二叉树的支持向量机多类分类方法（BT SVM）,并将基于核的自组织映射引入进行聚类。结果表明,采用该方法进行多类分类比1 v r SVMs和1 v 1 SVMs具有更高的分类精度。     

结合主动反馈的图像多分类框架

为了解决图像语义分类中的训练数据不对称、小样本训练和噪声数据这3个难题,提出结合主动反馈的图像多分类框架。该框架将主动选择的策略应用到图像的多分类中,通过主动的选择出不确定的图片给用户手动标记,扩大训练图片集,提高分类的精度。为了验证该框架的有效性,提出一种有效的结合主动选择的图像多分类算法,即结合投票的DDAGSVM(decision directed acyclic graph support vector machine)算法。该算法提出了新的主动选择策略,即结合投票和旁移机制的主动选择策略。实验结果表明,该算法能有效应用到图像多分类中,比DDAGSVM和采用普通主动选择策略的DDAGSVM具有更高的分类的精度。     

多类支持向量机分类算法—DDAG①

随着支持向量机的发展,由最初的两类分类问题逐渐推广到多类分类问题,且其思想、算法多种多样,各有千秋。主要研究以当前比较流行的以多个二类分类器组合实现多类分类器的算法之一：DDAG。提出此算法在多类支持向量机应用分类中存在的优点和不足,并针对其不足,提出一种改进的算法思想。     

基于多类邻域三支决策模型的不平衡数据分类

不平衡数据分类是一种重要的数据分类问题。对于不平衡数据中规模较小的类,传统的分类算法的分类效果较差。对此,提出一种多类邻域三支决策模型的不平衡数据分类算法。首先,将传统的三支决策在混合数据和多个类的情形下进行推广,提出了混合数据的多类邻域三支决策模型;然后,在该模型中给出一种自适应代价函数的设定方法,并基于该方法提出了多类邻域三支决策模型的不平衡数据分类算法。仿真实验的结果表明,所提出的分类算法对于不平衡数据具有更好的分类性能。     

Recursive “concave–convex” Fisher Linear Discriminant with applications to face,handwritten digit and terrain recognition

In classification, previous studies have shown that an eigenvalue based technique can be cast as an related SVM-type problem and that by solving this SVM-type problem, the performance can be improved significantly. In this paper, we develop a recursive “concave–convex” Fisher Linear Discriminant (DR) (RPFLD) for dimension reduction technique of high-dimensional data to extract as many meaningful features as possible, which incorporates the fundamental idea behind Fisher Linear Discriminant and casts the Fisher Linear Discriminant as a “concave–convex” programming problem based on the hinge loss. The solution of our method follows from solving the related SVM-type optimization problems iteratively, which means the proposed method, can be viewed as the combination of multiple related SVM-type problems. The special formulation of our method provides convenience for constructing sparse multi-class Fisher Linear Discriminant directly. Due to use of a recursive procedure, the number of features available from RPFLD is independent of the number of classes, meaning that in contrast to the original Fisher Linear Discriminant the number of features available from our method has no upper bound. We evaluate our algorithm on the Yale, and ORL face image databases, handwritten digit database and Terrain image dataset. Experimental results show that RPFLD outperforms other Fisher Linear Discriminant algorithms.     

Classification of weld flaws with imbalanced class data

This paper presents research results of our investigation of the imbalanced data problem in the classification of different types of weld flaws, a multi-class classification problem. The one-against-all scheme is adopted to carry out multi-class classification and three algorithms including minimum distance, nearest neighbors, and fuzzy nearest neighbors are employed as the classifiers. The effectiveness of 22 data preprocessing methods for dealing with imbalanced data is evaluated in terms of eight evaluation criteria to determine whether any method would emerge to dominate the others. The test results indicate that: (1) nearest neighbor classifiers outperform the minimum distance classifier; (2) some data preprocessing methods do not improve any criterion and they vary from one classifier to another; (3) the combination of using the AHC_KM data preprocessing method with the 1-NN classifier is the best because they together produce the best performance in six of eight evaluation criteria; and (4) the most difficult weld flaw type to recognize is crack.     

Theory and Applications of Clifford Support Vector Machines

This paper introduces the Clifford Support Vector Machines (CSVM) as a generalization of the real- and complex-valued Support Vector Machines using the Clifford geometric algebra. In this framework we handle the design of kernels involving the Clifford or geometric product for linear and nonlinear classification and regression. The major advantage of our approach is that we redefine the optimization variables as multivectors. This allows us to have a multivector as output therefore we can represent multiple classes according to the dimension of the geometric algebra in which we work. We conduct comparisons between CSVM and the most used approaches to solve multi-class classification to show that our approach is more suitable for practical use on certain type of multi-class classification problems.     

Cost-Sensitive back-propagation neural networks with binarization techniques in addressing multi-class problems and non-competent classifiers

Multi-class classification problems can be addressed by using decomposition strategy. One of the most popular decomposition techniques is the One-vs-One (OVO) strategy, which consists of dividing multi-class classification problems into as many as possible pairs of easier-to-solve binary sub-problems. To discuss the presence of classes with different cost, in this paper, we examine the behavior of an ensemble of Cost-Sensitive Back-Propagation Neural Networks (CSBPNN) with OVO binarization techniques for multi-class problems. To implement this, the original multi-class cost-sensitive problem is decomposed into as many sub-problems as possible pairs of classes and each sub-problem is learnt in an independent manner using CSBPNN. Then a combination method is used to aggregate the binary cost-sensitive classifiers. To verify the synergy of the binarization technique and CSBPNN for multi-class cost-sensitive problems, we carry out a thorough experimental study. Specifically, we first develop the study to check the effectiveness of the OVO strategy for multi-class cost-sensitive learning problems. Then, we develop a comparison of several well-known aggregation strategies in our scenario. Finally, we explore whether further improvement can be achieved by using the management of non-competent classifiers. The experimental study is performed with three types of cost matrices and proper statistical analysis is employed to extract the meaningful findings.     

Using natural class hierarchies in multi-class visual classification

We address the problem of computationally efficient visual classification of objects, and propose a system for solving multi-class problems in domains that have inherent hierarchic structure, such as subclass-superclass-relationships based on visual similarity. Class relationships are used at runtime to select the computationally simplest feature space that allows classification at high level of confidence for each example view. Classification accuracies can then be further improved using rank-order voting over multiple views. Our experimental results show that our system compares favorably to previously published results using a demanding benchmark. The results support the hypothesis that class hierarchies based on visual similarities are feasible and useful in controlling the accuracy vs. speed tradeoffs in classification.     

Proximal gradient method for huberized support vector machine

The support vector machine (SVM) has been used in a wide variety of classification problems. The original SVM uses the hinge loss function, which is non-differentiable and makes the problem difficult to solve in particular for regularized SVMs, such as with \(\ell _1\)-regularization. This paper considers the Huberized SVM (HSVM), which uses a differentiable approximation of the hinge loss function. We first explore the use of the proximal gradient (PG) method to solving binary-class HSVM (B-HSVM) and then generalize it to multi-class HSVM (M-HSVM). Under strong convexity assumptions, we show that our algorithm converges linearly. In addition, we give a finite convergence result about the support of the solution, based on which we further accelerate the algorithm by a two-stage method. We present extensive numerical experiments on both synthetic and real datasets which demonstrate the superiority of our methods over some state-of-the-art methods for both binary- and multi-class SVMs.