An experimental study of one- and two-level classifier fusion for different sample sizes

Due to the wide variety of fusion techniques available for combining multiple classifiers into a more accurate classifier, a number of good studies have been devoted to determining in what situations some fusion methods should be preferred over other ones. However, the sample size behavior of the various fusion methods has hitherto received little attention in the literature of multiple classifier systems. The main contribution of this paper is thus to investigate the effect of training sample size on their relative performance and to gain more insight into the conditions for the superiority of some combination rules.A large experiment is conducted to study the performance of some fixed and trainable combination rules for executing one- and two-level classifier fusion for different training sample sizes. The experimental results yield the following conclusions: when implementing one-level fusion to combine homogeneous or heterogeneous base classifiers, fixed rules outperform trainable ones in nearly all cases, with only one exception of merging heterogeneous classifiers for large sample size. Moreover, the best classification for any considered sample size is generally achieved by a second level of combination (namely, utilizing one fusion rule to further combine a set of ensemble classifiers with each of them constructed by fusing base classifiers). Under these circumstances, it seems that adopting different types of fusion rules (fixed or trainable) as the combiners for two levels of fusion is appropriate.     

A ‘No Panacea Theorem’ for classifier combination

We introduce the ‘No Panacea Theorem’ (NPT) for multiple classifier combination, previously proved only in the case of two classifiers and two classes. In this paper, we extend the NPT to cases of multiple classifiers and multiple classes. We prove that if the combination function is continuous and diverse, there exists a situation in which the combination algorithm will give very bad performance. The proof relies on constructing ‘pathological’ probability density distributions that have high densities in particular areas such that the combination functions give incorrect classification. Thus, there is no optimal combination algorithm that is suitable in all situations. It can be seen from this theorem that the probability density functions (pdfs) play an important role in the performance of combination algorithms, so studying the pdfs becomes the first step of finding a good combination algorithm. Although devised for classifier combination, the NPT is also relevant to all supervised classification problems.     

Combining classifiers: A theoretical framework

The problem of classifier combination is considered in the context of the two main fusion scenarios: fusion of opinions based on identical and on distinct representations. We develop a theoretical framework for classifier combination for these two scenarios. For multiple experts using distinct representations we argue that many existing schemes such as the product rule, sum rule, min rule, max rule, majority voting, and weighted combination, can be considered as special cases of compound classification. We then consider the effect of classifier combination in the case of multiple experts using a shared representation where the aim of fusion is to obtain a better estimate of the appropriatea posteriori class probabilities. We also show that the two theoretical frameworks can be used for devising fusion strategies when the individual experts use features some of which are shared and the remaining ones distinct. We show that in both cases (distinct and shared representations), the expert fusion involves the computation of a linear or nonlinear function of thea posteriori class probabilities estimated by the individual experts. Classifier combination can therefore be viewed as a multistage classification process whereby thea posteriori class probabilities generated by the individual classifiers are considered as features for a second stage classification scheme. Most importantly, when the linear or nonlinear combination functions are obtained by training, the distinctions between the two scenarios fade away, and one can view classifier fusion in a unified way.     

Switching between selection and fusion in combining classifiers: anexperiment

This paper presents a combination of classifier selection and fusion by using statistical inference to switch between the two. Selection is applied in those regions of the feature space where one classifier strongly dominates the others from the pool [called clustering-and-selection or (CS)] and fusion is applied in the remaining regions. Decision templates (DT) method is adopted for the classifier fusion part. The proposed combination scheme (called CS+DT) is compared experimentally against its two components, and also against majority vote, naive Bayes, two joint-distribution methods (BKS and a variant due to Wernecke (1988)), the dynamic classifier selection (DCS) algorithm DCS_LA based on local accuracy (Woods et al. (1997)), and simple fusion methods such as maximum, minimum, average, and product. Based on the results with five data sets with homogeneous ensembles [multilayer perceptrons (NLPs)] and ensembles of different classifiers, we offer a discussion on when to combine classifiers and how classifier selection (static or dynamic) can be misled by the differences in the classifier team.     

Combination of hyperbolic functions for multimodal biometrics data fusion.

In this paper, we treat the problem of combining fingerprint and speech biometric decisions as a classifier fusion problem. By exploiting the specialist capabilities of each classifier, a combined classifier may yield results which would not be possible in a single classifier. The Feedforward Neural Network provides a natural choice for such data fusion as it has been shown to be a universal approximator. However, the training process remains much to be a trial-and-error effort since no learning algorithm can guarantee convergence to optimal solution within finite iterations. In this work, we propose a network model to generate different combinations of the hyperbolic functions to achieve some approximation and classification properties. This is to circumvent the iterative training problem as seen in neural networks learning. In many decision data fusion applications, since individual classifiers or estimators to be combined would have attained a certain level of classification or approximation accuracy, this hyperbolic functions network can be used to combine these classifiers taking their decision outputs as the inputs to the network. The proposed hyperbolic functions network model is first applied to a function approximation problem to illustrate its approximation capability. This is followed by some case studies on pattern classification problems. The model is finally applied to combine the fingerprint and speaker verification decisions which show either better or comparable results with respect to several commonly used methods.     

分类器的动态选择与循环集成方法

针对多分类器系统设计中最优子集选择效率低下、集成方法缺乏灵活性等问题, 提出了分类器的动态选择与循环集成方法 (Dynamic selection and circulating combination, DSCC). 该方法利用不同分类器模型之间的互补性, 动态选择出对目标有较高识别率的分类器组合, 使参与集成的分类器数量能够随识别目标的复杂程度而自适应地变化, 并根据可信度实现系统的循环集成. 在手写体数字识别实验中, 与其他常用的分类器选择方法相比, 所提出的方法灵活高效, 识别率更高.     

Diversity measures for multiple classifier system analysis and design

In the context of Multiple Classifier Systems, diversity among base classifiers is known to be a necessary condition for improvement in ensemble performance. In this paper the ability of several pair-wise diversity measures to predict generalisation error is compared. A new pair-wise measure, which is computed between pairs of patterns rather than pairs of classifiers, is also proposed for two-class problems. It is shown experimentally that the proposed measure is well correlated with base classifier test error as base classifier complexity is systematically varied. However, correlation with unity-weighted sum and vote is shown to be weaker, demonstrating the difficulty in choosing base classifier complexity for optimal fusion. An alternative strategy based on weighted combination is also investigated and shown to be less sensitive to number of training epochs.     

Local Classifier Weighting by Quadratic Programming

It has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers. However, how to combine multiple classifiers with various (potentially conflicting) decisions is still an open problem. A rich collection of classifier combination procedures-many of which are heuristic in nature-have been developed for this goal. In this brief, we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space. To this end, we use local classifier accuracy estimates to weight classifier outputs. Specifically, we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors, and then use these estimates to find the best weights of classifiers to label the query. The problem is formulated as a convex quadratic optimization problem, which returns optimal nonnegative classifier weights with respect to the chosen objective function, and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample. Experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes, particularly on problems with complex decision boundaries. Hence, the results indicate that local classification-accuracy-based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space.     

NMC: nearest matrix classification – A new combination model for pruning One-vs-One ensembles by transforming the aggregation problem

The One-vs-One strategy is among the most used techniques to deal with multi-class problems in Machine Learning. This way, any binary classifier can be used to address the original problem, since one classifier is learned for each possible pair of classes. As in every ensemble method, classifier combination becomes a vital step in the classification process. Even though many combination models have been developed in the literature, none of them have dealt with the possibility of reducing the number of generated classifiers after the training phase, i.e., ensemble pruning, since every classifier is supposed to be necessary.On this account, our objective in this paper is two-fold: (1) We propose a transformation of the aggregation step, which lead us to a new combination strategy where instances are classified on the basis of the similarities among score-matrices. (2) This fact allows us to introduce the possibility of reducing the number of binary classifiers without affecting the final accuracy. We will show that around 50% of classifiers can be removed (depending on the base learner and the specific problem) and that the confidence degrees obtained by these base classifiers have a strong influence on the improvement in the final accuracy.A thorough experimental study is carried out in order to show the behavior of the proposed approach in comparison with the state-of-the-art combination models in the One-vs-One strategy. Different classifiers from various Machine Learning paradigms are considered as base classifiers and the results obtained are contrasted with the proper statistical analysis.     

二叉树型多分类器融合的轴承故障诊断方法

多分类器融合能有效集成多种分类算法的优势,实现优势互补,提高智能诊断模型的稳健性和诊断精度。但在利用多数投票法构建多分类器融合决策系统时,要求成员分类器数目多于要识别的设备状态数,否则会出现无法融合的情况。针对此问题,提出了一种基于二叉树的多分类器融合算法,利用二叉树将多类分类问题转化为多个二值分类问题,从而各个节点上的成员分类器个数只要大于2即可,有效避免了成员分类器数目不足的问题。实验结果表明,相比单一分类器的诊断方法,该方法能有效地实现滚动轴承故障智能诊断,并具有对各神经网络初始值不敏感、识别率高且稳定等优势。     

Extended decision template presentation for combining classifiers

In this paper, a new method in classifier fusion is introduced for decision making based on internal structure of base classifiers. Amongst methods used in combining classifiers, there are some methods which work on decision template as a tool for modeling behavior of base classifiers in order to label data. This tool models their behavior only based on their final outputs. Our new method, introduces a special structure for decision template such that internal behavior of a neural network base classifier can be modeled in a proper manner suitable for classifiers fusion. The new method builds decision template for each layer of the neural network including all hidden layers. Therefore, the process of making decision in each base classifier is also available for classifiers fusion. Efficiency of the new method is compared with some known benchmark datasets to show how it can improve efficiency of classifiers fusion.     

Building ensemble classifiers using belief functions and OWA operators

A pervasive task in many forms of human activity is classification. Recent interest in the classification process has focused on ensemble classifier systems. These types of systems are based on a paradigm of combining the outputs of a number of individual classifiers. In this paper we propose a new approach for obtaining the final output of ensemble classifiers. The method presented here uses the Dempster–Shafer concept of belief functions to represent the confidence in the outputs of the individual classifiers. The combing of the outputs of the individual classifiers is based on an aggregation process which can be seen as a fusion of the Dempster rule of combination with a generalized form of OWA operator. The use of the OWA operator provides an added degree of flexibility in expressing the way the aggregation of the individual classifiers is performed.     

基于Sugeno模糊积分神经网络分类器融合方法在手写数字识别中的应用

神经网络是模式识别中一种常见的分类器.针对同一个分类问题,构建多个分类器并把多个分类器进行融合可以提高分类系统的分类正确率、改善系统的稳健性.首先介绍了Sugeno模糊积分及Sugeno模糊积分神经网络分类器融合方法的一般原理,而后将其应用于手写数字识别,通过实际的案例验证了该融合方法的有效性和可行性.     

CSBF: A static ensemble fusion method based on the centrality score of complex networks

Ensemble of classifiers can improve classification accuracy by combining several models. The fusion method plays an important role in the ensemble performance. Usually, a criterion for weighting the decision of each ensemble member is adopted. Frequently, this can be done using some heuristic based on accuracy or confidence. Then, the used fusion rule must consider the established criterion for providing a most reliable ensemble output through a kind of competition among the ensemble members. This article presents a new ensemble fusion method, named centrality score-based fusion, which uses the centrality concept in the context of social network analysis (SNA) as a criterion for the ensemble decision. Centrality measures have been applied in the SNA to measure the importance of each person inside of a social network, taking into account the relationship of each person with all others. Thus, the idea is to derive the classifier weight considering the overall classifier prominence inside the ensemble network, which reflects the relationships among pairs of classifiers. We hypothesized that the prominent position of a classifier based on its pairwise relationship with the other ensemble members could be its weight in the fusion process. A robust experimental protocol has confirmed that centrality measures represent a promising strategy to weight the classifiers of an ensemble, showing that the proposed fusion method performed well against the literature.     

Diversity-based combination of non-parametric classifiers for EMG signal decomposition

Non-parametric classification procedures based on a certainty measure and nearest neighbour rule for motor unit potential classification (MUP) during electromyographic (EMG) signal decomposition were explored. A diversity-based classifier fusion approach is developed and evaluated to achieve improved classification performance. The developed system allows the construction of a set of non-parametric base classifiers and then automatically chooses, from the pool of base classifiers, subsets of classifiers to form candidate classifier ensembles. The system selects the classifier ensemble members by exploiting a diversity measure for selecting classifier teams. The kappa statistic is used as the diversity measure to estimate the level of agreement between base classifier outputs, i.e., to measure the degree of decision similarity between base classifiers. The pool of base classifiers consists of two kinds of classifiers: adaptive certainty-based classifiers (ACCs) and adaptive fuzzy k-NN classifiers (AFNNCs) and both utilize different types of features. Once the patterns are assigned to their classes, by the classifier fusion system, firing pattern consistency statistics for each class are calculated to detect classification errors in an adaptive fashion. Performance of the developed system was evaluated using real and simulated EMG signals and was compared with the performance of the constituent base classifiers and the performance of the fixed ensemble containing the full set of base classifiers. Across the EMG signal data sets used, the diversity-based classifier fusion approach had better average classification performance overall, especially in terms of reducing classification errors.     

基于标准数据集的分类器融合学习模型

判别式分类器通过生成不同复杂度的指示函数去调节算法与所解决问题的适应性,能有效地避免过拟合现象。分类器融合方法就是应用单个分类器对特定样本预报的特异性来提高模型的整体预测精度,应用支持向量机(SVM)对乳腺癌数据进行建模,通过选取不同的模型参数(径向基核函数参数gamma和正则化约束参数cost)构建9个单分类器,通过投票策略在单分类器上构建融合分类器,融合模型对乳腺癌数据的预测精度为98.59%,相比单分类模型对此数据集的预测精度97.72%有明显的竞争力,试验结果表明融合模型能有效提升分类器的泛化能力。     

Adaptive classifier integration for robust pattern recognition

The integration of multiple classifiers promises higher classification accuracy and robustness than can be obtained with a single classifier. This paper proposes a new adaptive technique for classifier integration based on a linear combination model. The proposed technique is shown to exhibit robustness to a mismatch between test and training conditions. It often outperforms the most accurate of the fused information sources. A comparison between adaptive linear combination and non-adaptive Bayesian fusion shows that, under mismatched test and training conditions, the former is superior to the latter in terms of identification accuracy and insensitivity to information source distortion.     

Measure oriented training: a targeted approach to imbalanced classification problems

Since the overall prediction error of a classifier on imbalanced problems can be potentially misleading and biased, alternative performance measures such as G-mean and F-measure have been widely adopted. Various techniques including sampling and cost sensitive learning are often employed to improve the performance of classifiers in such situations. However, the training process of classifiers is still largely driven by traditional error based objective functions. As a result, there is clearly a gap between themeasure according to which the classifier is evaluated and how the classifier is trained. This paper investigates the prospect of explicitly using the appropriate measure itself to search the hypothesis space to bridge this gap. In the case studies, a standard threelayer neural network is used as the classifier, which is evolved by genetic algorithms (GAs) with G-mean as the objective function. Experimental results on eight benchmark problems show that the proposed method can achieve consistently favorable outcomes in comparison with a commonly used sampling technique. The effectiveness of multi-objective optimization in handling imbalanced problems is also demonstrated.     

On the combination of locally optimal pairwise classifiers

Classification methods generally rely on some idea about the data structure. If the specific assumptions are not met, a classifier may fail. In this paper, the possibility of combining classifiers in multi-class problems is investigated. Multi-class classification problems are split into two class problems. For each of the latter problems an optimal classifier is determined. The results of applying the optimal classifiers on the two class problems can be combined using a pairwise coupling algorithm.In this paper, exemplary situations are investigated where the respective assumptions of Naive Bayes or the classical Linear Discriminant Analysis (LDA) fail. It is investigated at which degree of violations of the assumptions it may be advantageous to use single methods or a classifier combination by pairwise coupling.     

Central-tendency estimation and nearest-estimate classification of event related potentials

Event related potentials (ERPs) are modeled as random vectors in order to determine multivariate central-tendency (C-T) estimates of ERPs such as the arithmetic mean, geometric mean, harmonic mean, median, tri-mean, trimmed-mean, and the Winsorized mean. Additionally, it is shown that the C-T estimates can be used to implement various forms of minimum-distance classifiers for individual channels and for single-channel heterogeneous, multi-channel homogeneous, and multi-channel heterogeneous-homogenous ERP classification through decision fusion. The study also focuses on answering the following related questions: (a) How do the C-T ERP estimates compare with each other? (b) How do the performances of nearest-estimate classifiers compare with each other? (c) For a given ERP channel, do the heterogeneous nearest-estimate classifiers offer complementary information for improving performance through decision fusion? (d) Do the homogeneous nearest-estimate classifiers of different channels offer complementary information for improving performance through decision fusion? (e) Can the performance be improved by fusing the decisions of all or a selected subset of the entire classifier ensemble? These questions are answered by designing estimation and classification experiments using real 6-channel ERPs. It is shown that although the operations to compute the vector C-T estimates can be quite different, the ERP estimates are similar with respect to their overall waveform shapes and peak latencies. Furthermore, the results of the classification experiments show that by fusing homogeneous nearest-estimate classifier decisions across multiple channels, the classification accuracy can be improved significantly when compared with the accuracies of individual channel classifiers.