Task decomposition and modular single-hidden-layer perceptron classifiers for multi-class learning problems

One of keys for multilayer perceptrons (MLPs) to solve the multi-class learning problems is how to make them get good convergence and generalization performances merely through learning small-scale subsets, i.e., a small part of the original larger-scale data sets. This paper first decomposes an n-class problem into n two-class problems, and then uses n class-modular MLPs to solve them one by one. A class-modular MLP is responsible for forming the decision boundaries of its represented class, and thus can be trained only by the samples from the represented class and some neighboring ones. When solving a two-class problem, an MLP has to face with such unfavorable situations as unbalanced training data, locally sparse and weak distribution regions, and open decision boundaries. One of solutions is that the samples from the minority classes or in the thin regions are virtually reinforced by suitable enlargement factors. And next, the effective range of an MLP is localized by a correction coefficient related to the distribution of its represented class. In brief, this paper focuses on the formation of economic learning subsets, the virtual balance of imbalanced training sets, and the localization of generalization regions of MLPs. The results for the letter and the extended handwritten digital recognitions show that the proposed methods are effective.     

A closed-form reduction of multi-class cost-sensitive learning to weighted multi-class learning

In cost-sensitive learning, misclassification costs can vary for different classes. This paper investigates an approach reducing a multi-class cost-sensitive learning to a standard classification task based on the data space expansion technique developed by Abe et al., which coincides with Elkan's reduction with respect to binary classification tasks. Using this proposed reduction approach, a cost-sensitive learning problem can be solved by considering a standard 0/1 loss classification problem on a new distribution determined by the cost matrix. We also propose a new weighting mechanism to solve the reduced standard classification problem, based on a theorem stating that the empirical loss on independently identically distributed samples from the new distribution is essentially the same as the loss on the expanded weighted training set. Experimental results on several synthetic and benchmark datasets show that our weighting approach is more effective than existing representative approaches for cost-sensitive learning.     

An active learning algorithm for multi-class classification

Pattern Analysis and Applications - Since the number of instances in the training set is very large, data annotating task consumes plenty of time and energy. Active learning algorithms can...     

实现兼类样本类增量学习的一种算法

针对兼类样本,提出一种类增量学习算法.利用超球支持向量机,对每类样本求得一个能包围该类尽可能多样本的最小超球,使各类样本之间通过超球隔开.增量学习时,对新增样本以及旧样本集中的支持向量和超球附近的非支持向量进行训练,使得算法在很小的空闻代价下实现兼类样本类增量学习.分类过程中,根据待分类样本到各超球球心的距离判定其所属类别.实验结果表明,该算法具有较快的训练、分类速度和较高的分类精度.     

Boosted multi-class semi-supervised learning for human action recognition

Human action recognition is a challenging task due to significant intra-class variations, occlusion, and background clutter. Most of the existing work use the action models based on statistic learning algorithms for classification. To achieve good performance on recognition, a large amount of the labeled samples are therefore required to train the sophisticated action models. However, collecting labeled samples is labor-intensive. To tackle this problem, we propose a boosted multi-class semi-supervised learning algorithm in which the co-EM algorithm is adopted to leverage the information from unlabeled data. Three key issues are addressed in this paper. Firstly, we formulate the action recognition in a multi-class semi-supervised learning problem to deal with the insufficient labeled data and high computational expense. Secondly, boosted co-EM is employed for the semi-supervised model construction. To overcome the high dimensional feature space, weighted multiple discriminant analysis (WMDA) is used to project the features into low dimensional subspaces in which the Gaussian mixture models (GMM) are trained and boosting scheme is used to integrate the subspace models. Thirdly, we present the upper bound of the training error in multi-class framework, which is able to guide the novel classifier construction. In theory, the proposed solution is proved to minimize this upper error bound. Experimental results have shown good performance on public datasets.     

Hybrid evolutionary learning for synthesizing multi-class pattern recognition systems

This paper describes one aspect of a machine-learning system called HELPR that blends the best aspects of different evolutionary techniques to bootstrap-up a complete recognition system from primitive input data. HELPR uses a multi-faceted representation consisting of a growing sequence of non-linear mathematical expressions. Individual features are represented as tree structures and manipulated using the techniques of genetic programming. Sets of features are represented as list structures that are manipulated using genetic algorithms and evolutionary programming. Complete recognition systems are formed in this version of HELPR by attaching the evolved features to multiple perceptron discriminators. Experiments on datasets from the University of California at Irvine (UCI) machine-learning repository show that HELPR’s performance meets or exceeds accuracies previously published.     

A new discriminant subspace analysis approach for multi-class problems

Fukunaga–Koontz Transform (FKT) is a famous feature extraction method in statistical pattern recognition, which aims to find a set of vectors that have the best representative power for one class while the poorest representative power for the other class. Li and Savvides [1] propose a one-against-all strategy to deal with multi-class problems, in which the two-class FKT method can be directly applied to find the presentative vectors of each class. Motivated by the FKT method, in this paper we propose a new discriminant subspace analysis (DSA) method for the multi-class feature extraction problems. To solve DSA, we propose an iterative algorithm for the joint diagonalization (JD) problem. Finally, we generalize the linear DSA method to handle nonlinear feature extraction problems via the kernel trick. To demonstrate the effectiveness of the proposed method for pattern recognition problems, we conduct extensive experiments on real data sets and show that the proposed method outperforms most commonly used feature extraction methods.     

面向多类别模式分类问题的新型阴性选择算法

针对阴性选择算法缺乏高效的分类器生成机制和“过拟合”抑制机制的缺陷,提出了一种面向多类别模式分类的阴性选择算法CS-NSA。通过引入克隆选择机制,根据分类器的分类效果和刺激度对其进行自适应学习;针对多类别模式分类的“过拟合”问题,引入了检测器集合的修剪机制,增强了检测器的分类推广能力。对比实验结果证明：与著名的人工免疫分类器AIRS相比,CS-NSA体现出更高的正确识别率。     

A fast kernel-based nonlinear discriminant analysis for multi-class problems

Nonlinear discriminant analysis may be transformed into the form of kernel-based discriminant analysis. Thus, the corresponding discriminant direction can be solved by linear equations. From the view of feature space, the nonlinear discriminant analysis is still a linear method, and it is provable that in feature space the method is equivalent to Fisher discriminant analysis. We consider that one linear combination of parts of training samples, called “significant nodes”, can replace the total training samples to express the corresponding discriminant vector in feature space to some extent. In this paper, an efficient algorithm is proposed to determine “significant nodes” one by one. The principle of determining “significant nodes” is simple and reasonable, and the consequent algorithm can be carried out with acceptable computation cost. Depending on the kernel functions between test samples and all “significant nodes”, classification can be implemented. The proposed method is called fast kernel-based nonlinear method (FKNM). It is noticeable that the number of “significant nodes” may be much smaller than that of the total training samples. As a result, for two-class classification problems, the FKNM will be much more efficient than the naive kernel-based nonlinear method (NKNM). The FKNM can be also applied to multi-class via two approaches: one-against-the-rest and one-against-one. Although there is a view that one-against-one is superior to one-against-the-rest in classification efficiency, it seems that for the FKNM one-against-the-rest is more efficient than one-against-one. Experiments on benchmark and real datasets illustrate that, for two-class and multi-class classifications, the FKNM is effective, feasible and much efficient.     

A dynamic over-sampling procedure based on sensitivity for multi-class problems

Classification with imbalanced datasets supposes a new challenge for researches in the framework of machine learning. This problem appears when the number of patterns that represents one of the classes of the dataset (usually the concept of interest) is much lower than in the remaining classes. Thus, the learning model must be adapted to this situation, which is very common in real applications. In this paper, a dynamic over-sampling procedure is proposed for improving the classification of imbalanced datasets with more than two classes. This procedure is incorporated into a memetic algorithm (MA) that optimizes radial basis functions neural networks (RBFNNs). To handle class imbalance, the training data are resampled in two stages. In the first stage, an over-sampling procedure is applied to the minority class to balance in part the size of the classes. Then, the MA is run and the data are over-sampled in different generations of the evolution, generating new patterns of the minimum sensitivity class (the class with the worst accuracy for the best RBFNN of the population). The methodology proposed is tested using 13 imbalanced benchmark classification datasets from well-known machine learning problems and one complex problem of microbial growth. It is compared to other neural network methods specifically designed for handling imbalanced data. These methods include different over-sampling procedures in the preprocessing stage, a threshold-moving method where the output threshold is moved toward inexpensive classes and ensembles approaches combining the models obtained with these techniques. The results show that our proposal is able to improve the sensitivity in the generalization set and obtains both a high accuracy level and a good classification level for each class.     

A sparse nearest mean classifier for high dimensional multi-class problems

The analysis of small datasets in high dimensional spaces is inherently difficult. For two-class classification problems there are a few methods that are able to face the so-called curse of dimensionality. However, for multi-class sparsely sampled datasets there are hardly any specific methods. In this paper, we propose four multi-class classifier alternatives that effectively deal with this type of data. Moreover, these methods implicitly select a feature subset optimized for class separation. Accordingly, they are especially interesting for domains where an explanation of the problem in terms of the original features is desired.In the experiments, we applied the proposed methods to an MDMA powders dataset, where the problem was to recognize the production process. It turns out that the proposed multi-class classifiers perform well, while the few utilized features correspond to known MDMA synthesis ingredients. In addition, to show the general applicability of the methods, we applied them to several other sparse datasets, ranging from bioinformatics to chemometrics datasets having as few as tens of samples in tens to even thousands of dimensions and three to four classes. The proposed methods had the best average performance, while very few dimensions were effectively utilized.     

基于分解策略处理多分类不均衡问题的方法

针对多分类不均衡问题,提出了一种新的基于一对一（one-versus-one,OVO）分解策略的方法。首先基于OVO分解策略将多分类不均衡问题分解成多个二值分类问题;再利用处理不均衡二值分类问题的算法建立二值分类器;接着利用SMOTE过抽样技术处理原始数据集;然后采用基于距离相对竞争力加权方法处理冗余分类器;最后通过加权投票法获得输出结果。在KEEL不均衡数据集上的大量实验结果表明,所提算法比其他经典方法具有显著的优势。     

以多类支持向量机为基础的小样本信息融合策略*

通过将多类支持向量机作为分类器,运用Dempster-Shafer理论等信息融合方法对分类结果进行融合,实现对小样本的分类。主要采用对多类支持向量机的分类结果进行求和后取最大值、Dempster-Shafer理论以及使用Dempster-Shafer理论后第二次使用支持向量机三种方式进行融合。由于支持向量机本身是适用于小样本的机器学习算法,Dempster-Shafer理论又可以较好地处理不确定性,两者的结合可以较好地处理小样本分类问题,并提高最终的分类精度。实验结果表明,提出的几种融合策略确实可以在小样     

An ensemble approach of dual base learners for multi-class classification problems

In this work, we formalise and evaluate an ensemble of classifiers that is designed for the resolution of multi-class problems. To achieve a good accuracy rate, the base learners are built with pairwise coupled binary and multi-class classifiers. Moreover, to reduce the computational cost of the ensemble and to improve its performance, these classifiers are trained using a specific attribute subset. This proposal offers the opportunity to capture the advantages provided by binary decomposition methods, by attribute partitioning methods, and by cooperative characteristics associated with a combination of redundant base learners. To analyse the quality of this architecture, its performance has been tested on different domains, and the results have been compared to other well-known classification methods. This experimental evaluation indicates that our model is, in most cases, as accurate as these methods, but it is much more efficient.     

A supervised learning neural network coprocessor for soft-decisionmaximum-likelihood decoding

A supervised learning neural network (SLNN) coprocessor which enhances the performance of a digital soft-decision Viterbi decoder used for forward error correction in a digital communication channel with either fading plus additive white Gaussian noise (AWGN) or pure AWGN has been investigated and designed. The SLNN is designed to cooperate with a phase shift keying (PSK) demodulator, an automatic gain control (AGC) circuit, and a 3-bit quantizer which is an analog to digital convertor. It is trained to learn the best uniform quantization step-size Delta (BEST) as a function of the mean and the standard deviation of various sets of Gaussian distributed random variables. The channel cutoff rate (R(0)) of the channel is employed to determine the best quantization threshold step-size (Delta(BEST)) that results in the minimization of the Viterbi decoder output bit error rate (BER). For a digital communication system with a SLNN coprocessor, consistent and substantial BER performance improvements are observed. The performance improvement ranges from a minimum of 9% to a maximum of 25% for a pure AWGN channel and from a minimum of 25% to a maximum of 70% for a fading channel. This neural network coprocessor approach can be generalized and applied to any digital signal processing system to decrease the performance losses associated with quantization and/or signal instability.     

Cost-sensitive probability for weighted voting in an ensemble model for multi-class classification problems

Applied Intelligence - Ensemble learning is an algorithm that utilizes various types of classification models. This algorithm can enhance the prediction efficiency of component models. However, the...     

非平稳数据流下的持续学习灾难性遗忘问题求解策略综述

持续学习作为一种在非平稳数据流中不断学习新任务并能保持旧任务性能的特殊机器学习范例,是视觉计算、自主机器人等领域的研究热点,但现阶段灾难性遗忘问题仍然是持续学习的一个巨大挑战。围绕持续学习灾难性遗忘问题展开综述研究,分析了灾难性遗忘问题缓解机理,并从模型参数、训练数据和网络架构三个层面探讨了灾难性遗忘问题求解策略,包括正则化策略、重放策略、动态架构策略和联合策略;根据现有文献凝练了灾难性遗忘方法的评估指标,并对比了不同灾难性遗忘问题的求解策略性能。最后对持续学习相关研究指出了未来的研究方向,以期为研究持续学习灾难性遗忘问题提供借鉴和参考。     

Asymptotic single-trial strategies for GMD decoding with arbitrary error-erasure tradeoff

Generalized minimum distance (GMD) decoders allow for combining some virtues of probabilistic and algebraic decoding approaches at a low complexity. We investigate single-trial strategies for GMD decoding with arbitrary error-erasure tradeoff, based on either erasing a fraction of the received symbols or erasing all symbols whose reliability values are below a certain threshold. The fraction/threshold may be either static or adaptive, where adaptive means that the erasing is a function of the channel output. Adaptive erasing based on a threshold is a new technique that has not been investigated before. An asymptotic approach is used to evaluate the error-correction radius for each strategy. Both known and new results appear as special cases of this general framework.     

A fast learning method for streaming and randomly ordered multi-class data chunks by using one-pass-throw-away class-wise learning concept

Presently, the amount of data occurring in several business and academic areas such as ATM transactions, web searches, and sensor data are tremendously and continuously increased. Classifying as well as recognizing patterns among these data in a limited memory space complexity are very challenging. Various incremental learning methods have proposed to achieve highly accurate results but both already learned data and new incoming data must be retained throughout the learning process, causing high space and time complexities. In this paper, a new neural learning method based on radial-shaped function and discard-after-learn concept in the data streaming environment was proposed to reduce the space and time complexities. The experimental results showed that the proposed method used 1 to 95 times fewer neurons and 1.2 to 2,700 times faster than the results produced by MLP, RBF, SVM, VEBF, ILVQ, ASC, and other incremental learning methods. It is also robust to the incoming order of data chunks.