Plurality voting-based multiple classifier systems: statistically independent with respect to dependent classifier sets

The simultaneous use of multiple classifiers has been shown to provide performance improvement in classification problems. The selection of an optimal set of classifiers is an important part of multiple classifier systems and the independence of classifier outputs is generally considered to be an advantage for obtaining better multiple classifier systems. In this paper, the need for the classifier independence is interrogated from classification performance point of view. The performance achieved with the use of classifiers having independent joint distributions is compared to some other classifiers which are defined to have best and worst joint distributions. These distributions are obtained by formulating the combination operation as an optimization problem. The analysis revealed several important observations about classifier selection which are then used to analyze the problem of selecting an additional classifier to be used with the available multiple classifier system.     

一种基于几何关系的多分类器差异性度量及其在多分类器系统构造中的应用

多分类器系统是应对复杂模式识别问题的有效手段之一. 当子分类器之间存在差异性或互补性时,多分类器系统往往能够获得比单分类器更高的分类正确率. 因而差异性度量在多分类器系统设计中至关重要. 目前已有的差异性度量方法虽能够在一定程度上刻画分类器之间的差异,但在应用中可能出现诸如差异性淹没等问题. 本文提出了一种基于几何关系的多分类器差异性度量,并在此基础上提出了一种多分类器系统构造方法,同时通过实验对比了使用新差异性度量方法和传统方法对多分类器系统融合分类正确率的影响. 结果表明,本文所提出的差异性度量能够很好地刻画分类器之间的差异,能从很大程度上抑制差异性淹没问题,并能有效应用于多分类器系统构造.     

An improved genetic-fuzzy system for classification and data analysis

Interpretability of classification systems, which refers to the ability of these systems to express their behavior in an understandable way, has recently gained more attention and it is considered as an important requirement especially for knowledge-based systems. The main objective of this study is to improve the ability of a well-known fuzzy classifier proposed in Ishibuchi and Nojima (2007) to maximize the accuracy while preserve its interpretability. To achieve the above-mentioned objective, we propose two variants of the original fuzzy classifier. In the first variant classifier, the same components of the original classifier were used except NSGA-II which was replaced by an enhanced version called Controlled Elitism NSGA-II. This replacement aims at improving the ability of the first variant classifier to find non-dominated solutions with better interpretability-accuracy trade-off. In the second variant classifier, we further improve the first variant classifier by enhancing the selection method of the antecedent conditions of the rules generated in the initial population of genetic algorithm. Unlike the method applied in the original classifier and the first variant classifier, which uses a random selection of the antecedent conditions, we proposed a feature-based selection method to favor the antecedent conditions associated with the most relevant features. The results show that the two variant classifiers find more non-dominated fuzzy rule-based systems with better generalization ability than the original method which suggests that Controlled Elitism NSGA-II algorithm is more efficient than NSGA-II. In addition, feature-based selection method applied in the second variant classifier allowed this method to successfully obtain high-quality solutions as it has consistently achieved the best error rates for all the data sets compared to the original method and the first variant classifier.     

A Tale of Two Classifier Systems

This paper describes two classifier systems that learn. These are rule-based systems that use genetic algorithms, which are based on an analogy with natural selection and genetics, as their principal learning mechanism, and an economic model as their principal mechanism for apportioning credit. CFS-C is a domain-independent learning system that has been widely tested on serial computers. CFS is a parallel implementation of CFS-C that makes full use of the inherent parallelism of classifier systems and genetic algorithms, and that allows the exploration of large-scale tasks that were formerly impractical. As with other approaches to learning, classifier systems in their current form work well for moderately-sized tasks but break down for larger tasks. In order to shed light on this issue, we present several empirical studies of known issues in classifier systems, including the effects of population size, the actual contribution of genetic algorithms, the use of rule chaining in solving higher-order tasks, and issues of task representation and dynamic population convergence. We conclude with a discussion of some major unresolved issues in learning classifier systems and some possible approaches to making them more effective on complex tasks.     

Vision-based Classification of Driving Postures by Efficient Feature Extraction and Bayesian Approach

Reports of traffic accidents show that a considerable percentage of the accidents are caused by human factors. Human-centric driver assistance systems, with integrated sensing, processing and networking, aim to find solutions to this problem and other relevant issues. The key technology in such systems is the capability to automatically understand and characterize driver behaviors. In this paper, we propose a novel, efficient feature extraction approach for driving postures from a video camera, which consists of Homomorphic filter, skin-like regions segmentation, canny edge detection, connected regions detection, small connected regions deletion and spatial scale ratio calculation. With features extracted from a driving posture dataset we created at Southeast University (SEU), holdout and cross-validation experiments on driving posture classification are then conducted using Bayes classifier. Compared with a number of commonly used classification methods including naive Bayes classifier, subspace classifier, linear perception classifier and Parzen classifier, the holdout and cross-validation experiments show that the Bayes classifier offers better classification performance than the other four classifiers. Among the four predefined classes, i.e., grasping the steering wheel, operating the shift gear, eating a cake and talking on a cellular phone, the class of talking on a cellular phone is the most difficult to classify. With Bayes classifier, the classification accuracies of talking on a cellular phone are over 90 % in holdout and cross-validation experiments, which shows the effectiveness of the proposed feature extraction method and the importance of Bayes classifier in automatically understanding and characterizing driver behaviors towards human-centric driver assistance systems.     

An approach to the automatic design of multiple classifier systems

Multiple classifier systems (MCSs) based on the combination of outputs of a set of different classifiers have been proposed in the field of pattern recognition as a method for the development of high performance classification systems. Previous work clearly showed that multiple classifier systems are effective only if the classifiers forming them are accurate and make different errors. Therefore, the fundamental need for methods aimed to design “accurate and diverse” classifiers is currently acknowledged. In this paper, an approach to the automatic design of multiple classifier systems is proposed. Given an initial large set of classifiers, our approach is aimed at selecting the subset made up of the most accurate and diverse classifiers. A proof of the optimality of the proposed design approach is given. Reported results on the classification of multisensor remote sensing images show that this approach allows the design of effective multiple classifier systems.     

Building compact MQDF classifier for large character set recognition by subspace distribution sharing

Quadratic classifier with modified quadratic discriminant function (MQDF) has been successfully applied to recognition of handwritten characters to achieve very good performance. However, for large category classification problem such as Chinese character recognition, the storage of the parameters for the MQDF classifier is usually too large to make it practical to be embedded in the memory limited hand-held devices. In this paper, we aim at building a compact and high accuracy MQDF classifier for these embedded systems. A method by combining linear discriminant analysis and subspace distribution sharing is proposed to greatly compress the storage of the MQDF classifier from 76.4 to 2.06 MB, while the recognition accuracy still remains above 97%, with only 0.88% accuracy loss. Furthermore, a two-level minimum distance classifier is employed to accelerate the recognition process. Fast recognition speed and compact dictionary size make the high accuracy quadratic classifier become practical for hand-held devices.     

Learning classifier systems from a reinforcement learning perspective

 We analyze learning classifier systems in the light of tabular reinforcement learning. We note that although genetic algorithms are the most distinctive feature of learning classifier systems, it is not clear whether genetic algorithms are important to learning classifiers systems. In fact, there are models which are strongly based on evolutionary computation (e.g., Wilson's XCS) and others which do not exploit evolutionary computation at all (e.g., Stolzmann's ACS). To find some clarifications, we try to develop learning classifier systems “from scratch”, i.e., starting from one of the most known reinforcement learning technique, Q-learning. We first consider thebasics of reinforcement learning: a problem modeled as a Markov decision process and tabular Q-learning. We introduce a formal framework to define a general purpose rule-based representation which we use to implement tabular Q-learning. We formally define generalization within rules and discuss the possible approaches to extend our rule-based Q-learning with generalization capabilities. We suggest that genetic algorithms are probably the most general approach for adding generalization although they might be not the only solution.     

A Universal Image Steganalysis System Based On Double Sparse Representation Classification (DSRC)

Achieving high rates of detection in low rates of embedding is still a challenging problem in many steganalysis systems. The newly proposed steganalysis system based on sparse representation classifier has shown remarkable detection rates in low embedding rate. In this paper, we propose a new steganalysis system based on double sparse representation classifier. We compare our proposed method with other steganalysis systems which use different classifier (including nearest neighbor, support vector machine, ensemble support vector machine and sparse representation). In all of our experiments, input features to the classifier are fixed and the ability of classifier is examined. Also we provide a complexity analysis in terms of execution time for different classifier. In most of experiments, our proposed method shows superior performance in terms of detection rate and complexity for low embedding rates.     

Rules discovery in fuzzy classifier systems with PSO for scheduling in grid computational infrastructures

Particle swarm optimization (PSO) is a bio-inspired optimization strategy founded on the movement of particles within swarms. PSO can be encoded in a few lines in most programming languages, it uses only elementary mathematical operations, and it is not costly as regards memory demand and running time. This paper discusses the application of PSO to rules discovery in fuzzy classifier systems (FCSs) instead of the classical genetic approach and it proposes a new strategy, Knowledge Acquisition with Rules as Particles (KARP). In KARP approach every rule is encoded as a particle that moves in the space in order to cooperate in obtaining high quality rule bases and in this way, improving the knowledge and performance of the FCS. The proposed swarm-based strategy is evaluated in a well-known problem of practical importance nowadays where the integration of fuzzy systems is increasingly emerging due to the inherent uncertainty and dynamism of the environment: scheduling in grid distributed computational infrastructures. Simulation results are compared to those of classical genetic learning for fuzzy classifier systems and the greater accuracy and convergence speed of classifier discovery systems using KARP is shown.     

A sentiment classification model based on multiple classifiers

With the widespread usage of social networks, forums and blogs, customer reviews emerged as a critical factor for the customers’ purchase decisions. Since the beginning of 2000s, researchers started to focus on these reviews to automatically categorize them into polarity levels such as positive, negative, and neutral. This research problem is known as sentiment classification. The objective of this study is to investigate the potential benefit of multiple classifier systems concept on Turkish sentiment classification problem and propose a novel classification technique. Vote algorithm has been used in conjunction with three classifiers, namely Naive Bayes, Support Vector Machine (SVM), and Bagging. Parameters of the SVM have been optimized when it was used as an individual classifier. Experimental results showed that multiple classifier systems increase the performance of individual classifiers on Turkish sentiment classification datasets and meta classifiers contribute to the power of these multiple classifier systems. The proposed approach achieved better performance than Naive Bayes, which was reported the best individual classifier for these datasets, and Support Vector Machines. Multiple classifier systems (MCS) is a good approach for sentiment classification, and parameter optimization of individual classifiers must be taken into account while developing MCS-based prediction systems.     

The effects of combining classifiers with the same training statistics using Bayesian decision rules

Multiple classifier systems (MCS) are attracting increasing interest in the field of pattern recognition and machine learning. Recently, MCS are also being introduced in the remote sensing field where the importance of classifier diversity for image classification problems has not been examined. In this article, Satellite Pour l'Observation de la Terre (SPOT) IV panchromatic and multispectral satellite images are classified into six land cover classes using five base classifiers: contextual classifier, k-nearest neighbour classifier, Mahalanobis classifier, maximum likelihood classifier and minimum distance classifier. The five base classifiers are trained with the same feature sets throughout the experiments and a posteriori probability, derived from the confusion matrix of these base classifiers, is applied to five Bayesian decision rules (product rule, sum rule, maximum rule, minimum rule and median rule) for constructing different combinations of classifier ensembles. The performance of these classifier ensembles is evaluated for overall accuracy and kappa statistics. Three statistical tests, the McNemar's test, the Cochran's Q test and the Looney's F-test, are used to examine the diversity of the classification results of the base classifiers compared to the results of the classifier ensembles. The experimental comparison reveals that (a) significant diversity amongst the base classifiers cannot enhance the performance of classifier ensembles; (b) accuracy improvement of classifier ensembles can only be found by using base classifiers with similar and low accuracy; (c) increasing the number of base classifiers cannot improve the overall accuracy of the MCS and (d) none of the Bayesian decision rules outperforms the others.     

The Influence of Multi-class Feature Selection on the Prediction of Diagnostic Phenotypes

In this work, we evaluate two schemes for incorporating feature selection processes in multi-class classifier systems on high-dimensional data of low cardinality. These schemes operate on the level of the systems’ individual base classifiers and therefore do not perfectly fit in the traditional categories of filter, wrapper and embedded feature selection strategies. They can be seen as two examples of feature selection networks that are only loosely related to the structure of the multi-class classifier system. The architectures are tested for their application in predicting diagnostic phenotypes from gene expression profiles. Their selection stability and the overall generalization ability are evaluated in \(10 \times 10\) cross-validation experiments with support vector machines, random forests and nearest neighbor classifiers on eight publicly available multi-class microarray datasets. Overall the feature selecting multi-class classifier systems were able to outperform their counterparts on at least five of eight datasets.     

Classifier Systems and the Animat Problem

This paper characterizes and investigates, from the perspective of machine learning and, particularly, classifier systems, the learning problem faced by animals and autonomous robots (here collectively termed animats). We suggest that, to survive in their environments, animats must in effect learn multiple disjunctive concepts incrementally under payoff (needs-satisfying) feedback. A review of machine learning techniques indicates that most relax at least one of these constraints. In theory, classifier systems satisfy the constraints, but tests have been limited. We show how the standard classifier system model applies to the animat learning problem. Then, in the experimental part of the paper, we specialize the model and test it in a problem environment satisfying the constraints and consisting of a difficult, disjunctive Boolean function drawn from the machine learning literature. Results include: learning the function in significantly fewer trials than a neural-network method; learning under payoff regimes that include both noisy payoff and partial reward for suboptimal performance; demonstration, in a classifier system, of a theoretically predicted property of genetic algorithms: the superiority of crossovers to point mutations; and automatic control of variation (search) rate based on system entropy. We conclude that the results support the classifier system approach to the animat problem, but suggest work aimed at the emergence of behavioral hierarchies of classifiers to offset slower learning rates in larger problems.     

A new classifier ensemble methodology based on subspace learning

Different classifiers with different characteristics and methodologies can complement each other and cover their internal weaknesses; so classifier ensemble is an important approach to handle the weakness of single classifier based systems. In this article we explore an automatic and fast function to approximate the accuracy of a given classifier on a typical dataset. Then employing the function, we can convert the ensemble learning to an optimisation problem. So, in this article, the target is to achieve a model to approximate the performance of a predetermined classifier over each arbitrary dataset. According to this model, an optimisation problem is designed and a genetic algorithm is employed as an optimiser to explore the best classifier set in each subspace. The proposed ensemble methodology is called classifier ensemble based on subspace learning (CEBSL). CEBSL is examined on some datasets and it shows considerable improvements.     

Design of a real time automatic speech recognition system using Modified One Against All SVM classifier

In this paper, Texas Instruments TMS320C6713 DSP based real-time speech recognition system using Modified One Against All Support Vector Machine (SVM) classifier is proposed. The major contributions of this paper are: the study and evaluation of the performance of the classifier using three feature extraction techniques and proposal for minimizing the computation time for the classifier. From this study, it is found that the recognition accuracies of 93.33%, 98.67% and 96.67% are achieved for the classifier using Mel Frequency Cepstral Coefficients (MFCC) features, zerocrossing (ZC) and zerocrossing with peak amplitude (ZCPA) features respectively. To reduce the computation time required for the systems, two techniques – one using optimum threshold technique for the SVM classifier and another using linear assembly are proposed. The ZC based system requires the least computation time and the above techniques reduce the execution time by a factor of 6.56 and 5.95 respectively. For the purpose of comparison, the speech recognition system is also implemented using Altera Cyclone II FPGA with Nios II soft processor and custom instructions. Of the two approaches, the DSP approach requires 87.40% less number of clock cycles. Custom design of the recognition system on the FPGA without using the soft-core processor would have resulted in less computational complexity. The proposed classifier is also found to reduce the number of support vectors by a factor of 1.12–3.73 when applied to speaker identification and isolated letter recognition problems. The techniques proposed here can be adapted for various other SVM based pattern recognition systems.     

Classifier systems and genetic algorithms

Classifier systems are massively parallel, message-passing, rule-based systems that learn through credit assignment (the bucket brigade algorithm) and rule discovery (the genetic algorithm). They typically operate in environments that exhibit one or more of the following characteristics: (1) perpetually novel events accompanied by large amounts of noisy or irrelevant data; (2) continual, often real-time, requirements for action; (3) implicitly or inexactly defined goals; and (4) sparse payoff or reinforcement obtainable only through long action sequences. Classifier systems are designed to absorb new information continuously from such environments, devising sets of competing hypotheses (expressed as rules) without disturbing significantly capabilities already acquired. This paper reviews the definition, theory, and extant applications of classifier systems, comparing them with other machine learning techniques, and closing with a discussion of advantages, problems, and possible extensions of classifier systems.     

A measure of competence based on random classification for dynamic ensemble selection

In this paper, a measure of competence based on random classification (MCR) for classifier ensembles is presented. The measure selects dynamically (i.e. for each test example) a subset of classifiers from the ensemble that perform better than a random classifier. Therefore, weak (incompetent) classifiers that would adversely affect the performance of a classification system are eliminated. When all classifiers in the ensemble are evaluated as incompetent, the classification accuracy of the system can be increased by using the random classifier instead. Theoretical justification for using the measure with the majority voting rule is given. Two MCR based systems were developed and their performance was compared against six multiple classifier systems using data sets taken from the UCI Machine Learning Repository and Ludmila Kuncheva Collection. The systems developed had typically the highest classification accuracies regardless of the ensemble type used (homogeneous or heterogeneous).     

Combined Classifiers for Invariant Face Recognition

This paper presents a system for invariant face recognition. A combined classifier uses the generalisation capabilities of both Learning Vector Quantisation (LVQ) and Radial Basis Function (RBF) neural networks to build a representative model of a face from a variety of training patterns with different poses, details and facial expressions. The combined generalisation error of the classifier is found to be lower than that of each individual classifier. A new face synthesis method is implemented for reducing the false acceptance rate and enhancing the rejection capability of the classifier. The system is capable of recognising a face in less than one second. The well-known ORL database is used for testing the combined classifier. Comparisons with several other systems show that our system compares favourably with the state-of-the-art systems. In the case of the ORL database, a correct recognition rate of 99.5% at 0.5% rejection rate is achieved.     

Tabu search-based classification for eye-movement behavioral decisions

Adaptive human–computer interfaces (HCIs) are fundamental to designing adaptive websites and adaptive decision support systems. Integrating these intelligent systems with modern eye trackers provides more effective ways to exploit eye fixation data and offers improved services to users. We develop an exemplar-based classifier using the tabu search algorithm to predict which decision strategy may underlie an empirical search behavior. Our algorithm reduces the size of decision concept representations to find the best exemplars for each concept. Experimental results show that our classifier is highly accurate in classifying the sequence of empirical eye fixations, demonstrating the promise of integrating adaptive HCIs with modern eye trackers.