On combining classifiers

We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically     

On combining classifiers for speaker authentication

Speaker verification and utterance verification are examples of techniques that can be used for speaker authentication purposes.Speaker verification consists of accepting or rejecting the claimed identity of a speaker by processing samples of his/her voice. Usually, these systems are based on HMM models that try to represent the characteristics of the speakers’ vocal tracts.Utterance verification systems make use of a set of speaker-independent speech models to recognize a certain utterance. If the utterances consist of passwords, this can be used for identity verification purposes.Up to now, both techniques have been used separately. This paper is focused on the problem of how to combine these two sources of information. New architectures are presented to join an utterance verification system and a speaker verification system in order to improve the performance in a speaker verification task.     

Pairwise fusion matrix for combining classifiers

Various fusion functions for classifier combination have been designed to optimize the results of ensembles of classifiers (EoC). We propose a pairwise fusion matrix (PFM) transformation, which produces reliable probabilities for the use of classifier combination and can be amalgamated with most existent fusion functions for combining classifiers. The PFM requires only crisp class label outputs from classifiers, and is suitable for high-class problems or problems with few training samples. Experimental results suggest that the performance of a PFM can be a notch above that of the simple majority voting rule (MAJ), and a PFM can work on problems where a behavior-knowledge space (BKS) might not be applicable.     

Extended decision template presentation for combining classifiers

In this paper, a new method in classifier fusion is introduced for decision making based on internal structure of base classifiers. Amongst methods used in combining classifiers, there are some methods which work on decision template as a tool for modeling behavior of base classifiers in order to label data. This tool models their behavior only based on their final outputs. Our new method, introduces a special structure for decision template such that internal behavior of a neural network base classifier can be modeled in a proper manner suitable for classifiers fusion. The new method builds decision template for each layer of the neural network including all hidden layers. Therefore, the process of making decision in each base classifier is also available for classifiers fusion. Efficiency of the new method is compared with some known benchmark datasets to show how it can improve efficiency of classifiers fusion.     

一个新的多分类器组合模型

分类在数据挖掘中扮演着很重要的角色,然而单个分类器有很多缺点,包括适用范围十分有限和分类准确度不高等。把多个单分类器的分类结果融合起来是克服这些缺点的有效途径,因此存在很高的研究价值。组合多分类器的一个核心内容是融合规则,现存的融合规则有积规则、和规则、中值规则与投票规则等,但这些规则性能还不够稳定。提出了一个新的基于神经网络的融合规则,并依此建立一个新的多分类器组合模型,实验表明它能提高分类准确度和稳定性。     

A genetic encoding approach for learning methods for combining classifiers

Several studies have reported that the ensemble of classifiers can improve the performance of a stand-alone classifier. In this paper, we propose a learning method for combining the predictions of a set of classifiers.The method described in this paper uses a genetic-based version of the correspondence analysis for combining classifiers. The correspondence analysis is based on the orthonormal representation of the labels assigned to the patterns by a pool of classifiers. In this paper instead of the orthonormal representation we use a pool of representations obtained by a genetic algorithm. Each single representation is used to train a different classifiers, these classifiers are combined by vote rule.The performance improvement with respect to other learning-based fusion methods is validated through experiments with several benchmark datasets.     

Statistical approaches to combining binary classifiers for multi-class classification

One of the popular methods for multi-class classification is to combine binary classifiers. In this paper, we propose a new approach for combining binary classifiers. Our method trains a combining method of binary classifiers using statistical techniques such as penalized logistic regression, stacking, and a sparsity promoting penalty. Our approach has several advantages. Firstly, our method outperforms existing methods even if the base classifiers are well-tuned. Secondly, an estimate of conditional probability for each class can be naturally obtained. Furthermore, we propose selecting relevant binary classifiers by adding the group lasso type penalty in training the combining method.     

Double-bagging: combining classifiers by bootstrap aggregation

The combination of classifiers leads to substantial reduction of misclassification error in a wide range of applications and benchmark problems. We suggest using an out-of-bag sample for combining different classifiers. In our setup, a linear discriminant analysis is performed using the observations in the out-of-bag sample, and the corresponding discriminant variables computed for the observations in the bootstrap sample are used as additional predictors for a classification tree. Two classifiers are combined and therefore method and variable selection bias is no problem for the corresponding estimate of misclassification error, the need of an additional test sample disappears. Moreover, the procedure performs comparable to the best classifiers used in a number of artificial examples and applications.     

基于置信度的多分类器互补集成手写数字识别

离线手写数字识别是光学字符识别的一个重要分支,在银行票据识别、邮政编码识别等领域有着广泛的应用。由于单一分类器在识别率上很难达到要求,人们提出了各种集成分类器识别方案。通过对离线手写数字的特征提取,从特征互补的角度出发,采用了最小距离分类器、树分类器和BP网络分类器进行多分类器互补集成,提出了基于置信度的多分类器互补集成方法。通过实验对比,基于置信度的多分类器互补集成手写数字识别在识别率和识别速度上达到了满意的结果。     

On the independence requirement in Dempster-Shafer theory for combining classifiers providing statistical evidence

In classifier combination, the relative values of beliefs assigned to different hypotheses are more important than accurate estimation of the combined belief function representing the joint observation space. Because of this, the independence requirement in Dempster’s rule should be examined from classifier combination point of view. In this study, it is investigated whether there is a set of dependent classifiers which provides a better combined accuracy than independent classifiers when Dempster’s rule of combination is used. The analysis carried out for three different representations of statistical evidence has shown that the combination of dependent classifiers using Dempster’s rule may provide much better combined accuracies compared to independent classifiers.     

Switching between selection and fusion in combining classifiers: anexperiment

This paper presents a combination of classifier selection and fusion by using statistical inference to switch between the two. Selection is applied in those regions of the feature space where one classifier strongly dominates the others from the pool [called clustering-and-selection or (CS)] and fusion is applied in the remaining regions. Decision templates (DT) method is adopted for the classifier fusion part. The proposed combination scheme (called CS+DT) is compared experimentally against its two components, and also against majority vote, naive Bayes, two joint-distribution methods (BKS and a variant due to Wernecke (1988)), the dynamic classifier selection (DCS) algorithm DCS_LA based on local accuracy (Woods et al. (1997)), and simple fusion methods such as maximum, minimum, average, and product. Based on the results with five data sets with homogeneous ensembles [multilayer perceptrons (NLPs)] and ensembles of different classifiers, we offer a discussion on when to combine classifiers and how classifier selection (static or dynamic) can be misled by the differences in the classifier team.     

Gender classification by combining clothing, hair and facial component classifiers

In this paper, we propose a novel gender classification framework, which utilizes not only facial features, but also external information, i.e. hair and clothing. Instead of using the whole face, we consider five facial components: forehead, eyes, nose, mouth and chin. We also design feature extraction methods for hair and clothing; these features have seldom been used in previous work because of their large variability. For each type of feature, we train a single support vector machine classifier with probabilistic output. The outputs of these classifiers are combined using various strategies, namely fuzzy integral, maximal, sum, voting, and product rule. The major contributions of this paper are (1) investigating the gender discriminative ability of clothing information; (2) using facial components instead of the whole face to obtain higher robustness for occlusions and noise; (3) exploiting hair and clothing information to facilitate gender classification. Experimental results show that our proposed framework improves classification accuracy, even when images contain occlusions, noise, and illumination changes.     

Relationships between combination methods and measures of diversity in combining classifiers

This study looks at the relationships between different methods of classifier combination and different measures of diversity. We considered 10 combination methods and 10 measures of diversity on two benchmark data sets. The relationship was sought on ensembles of three classifiers built on all possible partitions of the respective feature sets into subsets of pre-specified sizes. The only positive finding was that the Double-Fault measure of diversity and the measure of difficulty both showed reasonable correlation with Majority Vote and Naive Bayes combinations. Since both these measures have an indirect connection to the ensemble accuracy, this result was not unexpected. However, our experiments did not detect a consistent relationship between the other measures of diversity and the 10 combination methods.     

Normal maps vs. visible images: Comparing classifiers and combining modalities

This work investigates face recognition based on normal maps, and the performance improvement that can be obtained when exploiting it within a multimodal system, where a further independent module processes visible images. We first propose a technique to align two 3D models of a face by means of normal maps, which is very fast while providing an accuracy comparable to well-known and more general techniques such as Iterative Closest Point (ICP). Moreover, we propose a matching criterion based on a technique which exploits difference maps. It does not reduce the dimension of the feature space, but performs a weighted matching between two normal maps. In the second place, we explore the range of performances offered by different linear and nonlinear classifiers, when applied to the normal maps generated from the above aligned models. Such experiments highlight the added value of chromatic information contained in normal maps. We analyse a solid list of classifiers which were selected due to their historical reference value (e.g. Principal Component Analysis) or to their good performances in the bidimensional setting (Linear Discriminant Analysis, Partitioned Iterated Function Systems). Last but not least, we perform experiments to measure how different ways of combining normal maps and visible images can enhance the results obtained by the single recognition systems, given that specific characteristics of the images are taken into account. For these last experiments we only consider the classifier giving the best average results in the preceding ones, namely the PIFS-based one.     

多分类器组合及其应用于手写体数字识别

在许多模式识别的应用中经常遇到这样的问题:组合多个分类器.提出了一种新的组合多个分类器的方法,这个方法由反向传播神经网络来控制,一个无标号的模式输入到每一个单独的分类器,它也同时输入到神经网络中来决定哪两个分类器作为冠军和亚军.让这两个分类器通过一个随机数发生器来决定最终的胜者.并且将这个方法应用到识别手写体数字.实验显示单个分类器的性能能够得到可观的改变.     

Improving BCI-based emotion recognition by combining EEG feature selection and kernel classifiers

Current emotion recognition computational techniques have been successful on associating the emotional changes with the EEG signals, and so they can be identified and classified from EEG signals if appropriate stimuli are applied. However, automatic recognition is usually restricted to a small number of emotions classes mainly due to signal’s features and noise, EEG constraints and subject-dependent issues. In order to address these issues, in this paper a novel feature-based emotion recognition model is proposed for EEG-based Brain–Computer Interfaces. Unlike other approaches, our method explores a wider set of emotion types and incorporates additional features which are relevant for signal pre-processing and recognition classification tasks, based on a dimensional model of emotions: Valenceand Arousal. It aims to improve the accuracy of the emotion classification task by combining mutual information based feature selection methods and kernel classifiers. Experiments using our approach for emotion classification which combines efficient feature selection methods and efficient kernel-based classifiers on standard EEG datasets show the promise of the approach when compared with state-of-the-art computational methods.     

"Fuzzy" versus "nonfuzzy" in combining classifiers designed by Boosting

Boosting is recognized as one of the most successful techniques for generating classifier ensembles. Typically, the classifier outputs are combined by the weighted majority vote. The purpose of this study is to demonstrate the advantages of some fuzzy combination methods for ensembles of classifiers designed by Boosting. We ran two-fold cross-validation experiments on six benchmark data sets to compare the fuzzy and nonfuzzy combination methods. On the "fuzzy side" we used the fuzzy integral and the decision templates with different similarity measures. On the "nonfuzzy side" we tried the weighted majority vote as well as simple combiners such as the majority vote, minimum, maximum, average, product, and the Naive-Bayes combination. In our experiments, the fuzzy combination methods performed consistently better than the nonfuzzy methods. The weighted majority vote showed a stable performance, though slightly inferior to the performance of the fuzzy combiners.     

A review of selection combining receivers over correlated Rician fading

Selection combiner output (SCO) cumulative distribution functions (cdfs) of multivariate equally-correlated and exponentially-correlated (e.c.) Rician, and other correlated fading environments for positive channel correlation coefficients are rigorously surveyed and mathematically reviewed. Specifically, (i) new findings for the cdf of bivariate correlated and non-identically distributed Rician fading, and (ii) a simplified cdf of trivariate e.c. Rician fading, are reported. Applications of correlated fading into physical layer security are highlighted. Distributions of multivariate equally-correlated fading are discussed for even-degree-of-freedom (DoF), and even-and-odd-DoF non-central chi-square distributed envelopes. Schematic diagrams are employed to offer top-down view on progress for SCO distributions of multivariate correlated Rician fading, and their mathematical inter-relations. From that, knowledge gaps can systematically be identified. Detailed verification revealing mathematical links among published results is shown in detail, from which unidentified special cases can be unified to their corresponding generalised cases. Simulation results are obtained under two scenarios: (i) equal-channel gains, and (ii) unequal-channel gains. Apart from reviewing cdfs of multivariate correlated Rician fading, this paper serves as a tutorial, in which detailed insight into key mathematical developments are given in Appendices A–I, which (i) attract new researchers to the field, and (ii) progress this topic to fruitful ground.     

A method of combining multiple probabilistic classifiers through soft competition on different feature sets

A novel method is proposed for combining multiple probabilistic classifiers on different feature sets. In order to achieve the improved classification performance, a generalized finite mixture model is proposed as a linear combination scheme and implemented based on radial basis function networks. In the linear combination scheme, soft competition on different feature sets is adopted as an automatic feature rank mechanism so that different feature sets can be always simultaneously used in an optimal way to determine linear combination weights. For training the linear combination scheme, a learning algorithm is developed based on Expectation–Maximization (EM) algorithm. The proposed method has been applied to a typical real-world problem, viz., speaker identification, in which different feature sets often need consideration simultaneously for robustness. Simulation results show that the proposed method yields good performance in speaker identification.     

The effects of combining classifiers with the same training statistics using Bayesian decision rules

Multiple classifier systems (MCS) are attracting increasing interest in the field of pattern recognition and machine learning. Recently, MCS are also being introduced in the remote sensing field where the importance of classifier diversity for image classification problems has not been examined. In this article, Satellite Pour l'Observation de la Terre (SPOT) IV panchromatic and multispectral satellite images are classified into six land cover classes using five base classifiers: contextual classifier, k-nearest neighbour classifier, Mahalanobis classifier, maximum likelihood classifier and minimum distance classifier. The five base classifiers are trained with the same feature sets throughout the experiments and a posteriori probability, derived from the confusion matrix of these base classifiers, is applied to five Bayesian decision rules (product rule, sum rule, maximum rule, minimum rule and median rule) for constructing different combinations of classifier ensembles. The performance of these classifier ensembles is evaluated for overall accuracy and kappa statistics. Three statistical tests, the McNemar's test, the Cochran's Q test and the Looney's F-test, are used to examine the diversity of the classification results of the base classifiers compared to the results of the classifier ensembles. The experimental comparison reveals that (a) significant diversity amongst the base classifiers cannot enhance the performance of classifier ensembles; (b) accuracy improvement of classifier ensembles can only be found by using base classifiers with similar and low accuracy; (c) increasing the number of base classifiers cannot improve the overall accuracy of the MCS and (d) none of the Bayesian decision rules outperforms the others.