Classifier combination based on confidence transformation

This paper investigates the effects of confidence transformation in combining multiple classifiers using various combination rules. The combination methods were tested in handwritten digit recognition by combining varying classifier sets. The classifier outputs are transformed to confidence measures by combining three scaling functions (global normalization, Gaussian density modeling, and logistic regression) and three confidence types (linear, sigmoid, and evidence). The combination rules include fixed rules (sum-rule, product-rule, median-rule, etc.) and trained rules (linear discriminants and weighted combination with various parameter estimation techniques). The experimental results justify that confidence transformation benefits the combination performance of either fixed rules or trained rules. Trained rules mostly outperform fixed rules, especially when the classifier set contains weak classifiers. Among the trained rules, the support vector machine with linear kernel (linear SVM) performs best while the weighted combination with optimized weights performs comparably well. I have also attempted the joint optimization of confidence parameters and combination weights but its performance was inferior to that of cascaded confidence transformation-combination. This justifies that the cascaded strategy is a right way of multiple classifier combination.     

Probabilistic classification of acute myocardial infarction from multiple cardiac markers

Logistic regression and Gaussian mixture model (GMM) classifiers have been trained to estimate the probability of acute myocardial infarction (AMI) in patients based upon the concentrations of a panel of cardiac markers. The panel consists of two new markers, fatty acid binding protein (FABP) and glycogen phosphorylase BB (GPBB), in addition to the traditional cardiac troponin I (cTnI), creatine kinase MB (CKMB) and myoglobin. The effect of using principal component analysis (PCA) and Fisher discriminant analysis (FDA) to preprocess the marker concentrations was also investigated. The need for classifiers to give an accurate estimate of the probability of AMI is argued and three categories of performance measure are described, namely discriminatory ability, sharpness, and reliability. Numerical performance measures for each category are given and applied. The optimum classifier, based solely upon the samples take on admission, was the logistic regression classifier using FDA preprocessing. This gave an accuracy of 0.85 (95% confidence interval: 0.78–0.91) and a normalised Brier score of 0.89. When samples at both admission and a further time, 1–6 h later, were included, the performance increased significantly, showing that logistic regression classifiers can indeed use the information from the five cardiac markers to accurately and reliably estimate the probability AMI.     

Sparse multinomial logistic regression: fast algorithms and generalization bounds

Recently developed methods for learning sparse classifiers are among the state-of-the-art in supervised learning. These methods learn classifiers that incorporate weighted sums of basis functions with sparsity-promoting priors encouraging the weight estimates to be either significantly large or exactly zero. From a learning-theoretic perspective, these methods control the capacity of the learned classifier by minimizing the number of basis functions used, resulting in better generalization. This paper presents three contributions related to learning sparse classifiers. First, we introduce a true multiclass formulation based on multinomial logistic regression. Second, by combining a bound optimization approach with a component-wise update procedure, we derive fast exact algorithms for learning sparse multiclass classifiers that scale favorably in both the number of training samples and the feature dimensionality, making them applicable even to large data sets in high-dimensional feature spaces. To the best of our knowledge, these are the first algorithms to perform exact multinomial logistic regression with a sparsity-promoting prior. Third, we show how nontrivial generalization bounds can be derived for our classifier in the binary case. Experimental results on standard benchmark data sets attest to the accuracy, sparsity, and efficiency of the proposed methods.     

基于置信度的手写体数字识别多分类器动态组合

多分类器组合利用不同分类器、不同特征之间的互补性，提高了组合分类器的识别率。传统的组合方法里，各分类器在组合中所承担的角色是固定的，而实际应用中，对于不同的测试样本，每个分类器识别结果的可信度是不同的。该文根据分类器置信度理论，提出了各类别的置信度。用测试样本自身的置信度信息实现分类器的动态组合，并把这种动态组合方法具体应用到手写体数字的识别。这种方法还可以在不影响已有数据的情况下添加新的分类器进行组合。     

Generalised Gaussian radial basis function neural networks

The mixed use of different shapes of radial basis functions (RBFs) in radial basis functions neural networks (RBFNNs) is investigated in this paper. For this purpose, we propose the use of a generalised version of the standard RBFNN, based on the generalised Gaussian distribution. The generalised radial basis function (GRBF) proposed in this paper is able to reproduce other different radial basis functions (RBFs) by changing a real parameter τ. In the proposed methodology, a hybrid evolutionary algorithm (HEA) is employed to estimate the number of hidden neuron, the centres, type and width of each RBF associated with each radial unit. In order to test the performance of the proposed methodology, an experimental study is presented with 20 datasets from the UCI repository. The GRBF neural network (GRBFNN) was compared to RBFNNs with Gaussian, Cauchy and inverse multiquadratic RBFs in the hidden layer and to other classifiers, including different RBFNN design methods, support vector machines (SVMs), a sparse probabilistic classifier (sparse multinominal logistic regression, SMLR) and other non-sparse (but regularised) probabilistic classifiers (regularised multinominal logistic regression, RMLR). The GRBFNN models were found to be better than the alternative RBFNNs for almost all datasets, producing the highest mean accuracy rank.     

Pairwise fusion matrix for combining classifiers

Various fusion functions for classifier combination have been designed to optimize the results of ensembles of classifiers (EoC). We propose a pairwise fusion matrix (PFM) transformation, which produces reliable probabilities for the use of classifier combination and can be amalgamated with most existent fusion functions for combining classifiers. The PFM requires only crisp class label outputs from classifiers, and is suitable for high-class problems or problems with few training samples. Experimental results suggest that the performance of a PFM can be a notch above that of the simple majority voting rule (MAJ), and a PFM can work on problems where a behavior-knowledge space (BKS) might not be applicable.     

Financial distress prediction using support vector machines: Ensemble vs. individual

Financial distress prediction (FDP) is of great importance to both inner and outside parts of companies. Though lots of literatures have given comprehensive analysis on single classifier FDP method, ensemble method for FDP just emerged in recent years and needs to be further studied. Support vector machine (SVM) shows promising performance in FDP when compared with other single classifier methods. The contribution of this paper is to propose a new FDP method based on SVM ensemble, whose candidate single classifiers are trained by SVM algorithms with different kernel functions on different feature subsets of one initial dataset. SVM kernels such as linear, polynomial, RBF and sigmoid, and the filter feature selection/extraction methods of stepwise multi discriminant analysis (MDA), stepwise logistic regression (logit), and principal component analysis (PCA) are applied. The algorithm for selecting SVM ensemble's base classifiers from candidate ones is designed by considering both individual performance and diversity analysis. Weighted majority voting based on base classifiers’ cross validation accuracy on training dataset is used as the combination mechanism. Experimental results indicate that SVM ensemble is significantly superior to individual SVM classifier when the number of base classifiers in SVM ensemble is properly set. Besides, it also shows that RBF SVM based on features selected by stepwise MDA is a good choice for FDP when individual SVM classifier is applied.     

分类器的动态选择与循环集成方法

针对多分类器系统设计中最优子集选择效率低下、集成方法缺乏灵活性等问题, 提出了分类器的动态选择与循环集成方法 (Dynamic selection and circulating combination, DSCC). 该方法利用不同分类器模型之间的互补性, 动态选择出对目标有较高识别率的分类器组合, 使参与集成的分类器数量能够随识别目标的复杂程度而自适应地变化, 并根据可信度实现系统的循环集成. 在手写体数字识别实验中, 与其他常用的分类器选择方法相比, 所提出的方法灵活高效, 识别率更高.     

Bayesian classification of hyperspectral images using spatially-varying Gaussian mixture model

We propose a spatially-varying Gaussian mixture model for joint spectral and spatial classification of hyperspectral images. The model provides a robust estimation framework for small sample size training sets. Defining prior distributions for the mean vector and the covariance matrix enables us to regularize the parameter estimation problem. More specifically, we can obtain invertible positive definite covariance matrices by the help of this regularization. Moreover, the proposed model also takes into account the spatial alignments of the pixels by using spatially-varying mixture proportions. The spatially-varying mixture model is based on spatial multinomial logistic regression. The classification results obtained on Indian Pines, Pavia Centre, Pavia University, and Salinas data sets show that the proposed methods perform better especially for small-sized training sets compared to the state-of-the-art classifiers.     

基于模糊积分和遗传算法的分类器组合算法

将多个分类器进行组合能提高分类精度。基于模糊测度的Sugeno和Choquet积分具有理想的特性,因此该文利用其进行分类器组合。然而在实际中难以求得模糊测度。该文利用两种方法求取模糊测度,一是分类器对样本数据的分类能力,另一种是根据遗传算法。这两种方法均考虑了每个分类器对不同类的分类能力不同这一经验知识。实验中对UCI中的几个数据库进行了测试,同时将该组合方法应用于一多传感器融合工件识别系统。测试结果表明了该算法是一种计算简便、精度较高的分类器组合方法。     

On hybrid classification using model assisted posterior estimates

Traditional parametric and nonparametric classifiers used for statistical pattern recognition have their own strengths and limitations. While parametric methods assume some specific parametric models for density functions or posterior probabilities of competing classes, nonparametric methods are free from such assumptions. So, when these model assumptions are correct, parametric methods outperform nonparametric classifiers, especially when the training sample is small. But, violations of these assumptions often lead to poor performance by parametric classifiers, where nonparametric methods work well. In this article, we make an attempt to overcome these limitations of parametric and nonparametric approaches and combine their strengths. The resulting classifiers, denoted the hybrid classifiers, perform like parametric classifiers when the model assumptions are valid, but unlike parametric classifiers, they also provide safeguards against possible deviations from parametric model assumptions. In this article, we propose some multiscale methods for hybrid classification, and their performance is evaluated using several simulated and benchmark data sets.     

Mixed group ranks: preference and confidence in classifier combination

Classifier combination holds the potential of improving performance by combining the results of multiple classifiers. For domains with very large numbers of classes, such as biometrics, we present an axiomatic framework of desirable mathematical properties for combination functions of rank-based classifiers. This framework represents a continuum of combination rules, including the Borda Count, Logistic Regression, and Highest Rank combination methods as extreme cases. Intuitively, this framework captures how the two complementary concepts of general preference for specific classifiers and the confidence it has in any specific result (as indicated by ranks) can be balanced while maintaining consistent rank interpretation. Mixed Group Ranks (MGR) is a new combination function that balances preference and confidence by generalizing these other functions. We demonstrate that MGR is an effective combination approach by performing multiple experiments on data sets with large numbers of classes and classifiers from the FERET face recognition study.     

Efficient approximate leave-one-out cross-validation for kernel logistic regression

Kernel logistic regression (KLR) is the kernel learning method best suited to binary pattern recognition problems where estimates of a-posteriori probability of class membership are required. Such problems occur frequently in practical applications, for instance because the operational prior class probabilities or equivalently the relative misclassification costs are variable or unknown at the time of training the model. The model parameters are given by the solution of a convex optimization problem, which may be found via an efficient iteratively re-weighted least squares (IRWLS) procedure. The generalization properties of a kernel logistic regression machine are however governed by a small number of hyper-parameters, the values of which must be determined during the process of model selection. In this paper, we propose a novel model selection strategy for KLR, based on a computationally efficient closed-form approximation of the leave-one-out cross-validation procedure. Results obtained on a variety of synthetic and real-world benchmark datasets are given, demonstrating that the proposed model selection procedure is competitive with a more conventional k-fold cross-validation based approach and also with Gaussian process (GP) classifiers implemented using the Laplace approximation and via the Expectation Propagation (EP) algorithm.     

Weighted pseudometric discriminatory power improvement using a Bayesian logistic regression model based on a variational method

In this paper, we investigate the effectiveness of a Bayesian logistic regression model to compute the weights of a pseudo-metric, in order to improve its discriminatory capacity and thereby increase image retrieval accuracy. In the proposed Bayesian model, the prior knowledge of the observations is incorporated and the posterior distribution is approximated by a tractable Gaussian form using variational transformation and Jensen's inequality, which allow a fast and straightforward computation of the weights. The pseudo-metric makes use of the compressed and quantized versions of wavelet decomposed feature vectors, and in our previous work, the weights were adjusted by classical logistic regression model. A comparative evaluation of the Bayesian and classical logistic regression models is performed for content-based image retrieval as well as for other classification tasks, in a decontextualized evaluation framework. In this same framework, we compare the Bayesian logistic regression model to some relevant state-of-the-art classification algorithms. Experimental results show that the Bayesian logistic regression model outperforms these linear classification algorithms, and is a significantly better tool than the classical logistic regression model to compute the pseudo-metric weights and improve retrieval and classification performance. Finally, we perform a comparison with results obtained by other retrieval methods.     

COMBINING MULTIPLE CLASSIFIERS USING DEMPSTER'S RULE FOR TEXT CATEGORIZATION

In this paper we investigate the combination of four machine learning methods for text categorization using Dempster's rule of combination. These methods include Support Vector Machine (SVM), kNN (Nearest Neighbor), kNN model-based approach (kNNM), and Rocchio. We first present a general representation of the outputs of different classifiers, in particular, modeling it as a piece of evidence by using a novel evidence structure called focal element triplet. Furthermore, we investigate an effective method for combining pieces of evidence derived from classifiers generated by a 10-fold cross-validation. Finally, we evaluate our methods on the 20-newsgroup and Reuters-21578 benchmark data sets and perform the comparative analysis with majority voting in combining multiple classifiers along with the previous result. Our experimental results show that the best combined classifier can improve the performance of the individual classifiers and Dempster's rule of combination outperforms majority voting in combining multiple classifiers.     

基于最小代价的多分类器动态集成

本文提出一种基于最小代价准则的分类器动态集成方法．与一般方法不同,动态集成是根据“性能预测特征”,动态地为每一样本选择最适合的一组分类器进行集成．该选择基于使误识代价与时间代价最小化的准则,改变代价函数的定义可以方便地达到识别率与识别速度之间的不同折衷．本文中提出了两种分类器动态集成的方法,并介绍了在联机手写汉字识别中的具体应用．在实验中使了３个分类器进行动态集成,因此,得到７种分类组合．在预先定义的代价意义下,我们比较了动态集成方法和其它７种固定方法的性能．实验结果证明了动态集成方法的高灵活性、实用性和提高系统综合性能的能力．     

Spectral clustering with fuzzy similarity measure

Spectral clustering algorithms have been successfully used in the field of pattern recognition and computer vision. The widely used similarity measure for spectral clustering is Gaussian kernel function which measures the similarity between data points. However, it is difficult for spectral clustering to choose the suitable scaling parameter in Gaussian kernel similarity measure. In this paper, utilizing the prototypes and partition matrix obtained by fuzzy c-means clustering algorithm, we develop a fuzzy similarity measure for spectral clustering (FSSC). Furthermore, we introduce the K-nearest neighbor sparse strategy into FSSC and apply the sparse FSSC to texture image segmentation. In our experiments, we firstly perform some experiments on artificial data to verify the efficiency of the proposed fuzzy similarity measure. Then we analyze the parameters sensitivity of our method. Finally, we take self-tuning spectral clustering and Nyström methods for baseline comparisons, and apply these three methods to the synthetic texture and remote sensing image segmentation. The experimental results show that the proposed method is significantly effective and stable.     

A unified neurofuzzy model for classification

This work proposes a unified neurofuzzy modelling scheme. To begin with, the initial fuzzy base construction method is based on fuzzy clustering utilising a Gaussian mixture model (GMM) combined with the analysis of covariance (ANOVA) decomposition in order to obtain more compact univariate and bivariate membership functions over the subspaces of the input features. The mean and covariance of the Gaussian membership functions are found by the expectation maximisation (EM) algorithm with the merit of revealing the underlying density distribution of system inputs. The resultant set of membership functions forms the basis of the generalised fuzzy model (GFM) inference engine. The model structure and parameters of this neurofuzzy model are identified via the supervised subspace orthogonal least square (OLS) learning. Finally, instead of providing deterministic class label as model output by convention, a logistic regression model is applied to present the classifier’s output, in which the sigmoid type of logistic transfer function scales the outputs of the neurofuzzy model to the class probability. Experimental validation results are presented to demonstrate the effectiveness of the proposed neurofuzzy modelling scheme.     

On the impact of disproportional samples in credit scoring models: An application to a Brazilian bank data

Statistical methods have been widely employed to assess the capabilities of credit scoring classification models in order to reduce the risk of wrong decisions when granting credit facilities to clients. The predictive quality of a classification model can be evaluated based on measures such as sensitivity, specificity, predictive values, accuracy, correlation coefficients and information theoretical measures, such as relative entropy and mutual information. In this paper we analyze the performance of a naive logistic regression model (Hosmer & Lemeshow, 1989) and a logistic regression with state-dependent sample selection model (Cramer, 2004) applied to simulated data. Also, as a case study, the methodology is illustrated on a data set extracted from a Brazilian bank portfolio. Our simulation results so far revealed that there is no statistically significant difference in terms of predictive capacity between the naive logistic regression models and the logistic regression with state-dependent sample selection models. However, there is strong difference between the distributions of the estimated default probabilities from these two statistical modeling techniques, with the naive logistic regression models always underestimating such probabilities, particularly in the presence of balanced samples.     

Classification by ensembles from random partitions of high-dimensional data

A robust classification procedure is developed based on ensembles of classifiers, with each classifier constructed from a different set of predictors determined by a random partition of the entire set of predictors. The proposed methods combine the results of multiple classifiers to achieve a substantially improved prediction compared to the optimal single classifier. This approach is designed specifically for high-dimensional data sets for which a classifier is sought. By combining classifiers built from each subspace of the predictors, the proposed methods achieve a computational advantage in tackling the growing problem of dimensionality. For each subspace of the predictors, we build a classification tree or logistic regression tree. Our study shows, using four real data sets from different areas, that our methods perform consistently well compared to widely used classification methods. For unbalanced data, our approach maintains the balance between sensitivity and specificity more adequately than many other classification methods considered in this study.