Learning classifier systems: a survey

Learning classifier systems (LCSs) are rule- based systems that automatically build their ruleset. At the origin of Holland’s work, LCSs were seen as a model of the emergence of cognitive abilities thanks to adaptive mechanisms, particularly evolutionary processes. After a renewal of the field more focused on learning, LCSs are now considered as sequential decision problem-solving systems endowed with a generalization property. Indeed, from a Reinforcement Learning point of view, LCSs can be seen as learning systems building a compact representation of their problem thanks to generalization. More recently, LCSs have proved efficient at solving automatic classification tasks. The aim of the present contribution is to describe the state-of- the-art of LCSs, emphasizing recent developments, and focusing more on the sequential decision domain than on automatic classification.     

A generalized adaptive ensemble generation and aggregation approach for multiple classifier systems

In this paper, a generalized adaptive ensemble generation and aggregation (GAEGA) method for the design of multiple classifier systems (MCSs) is proposed. GAEGA adopts an “over-generation and selection” strategy to achieve a good bias-variance tradeoff. In the training phase, different ensembles of classifiers are adaptively generated by fitting the validation data globally with different degrees. The test data are then classified by each of the generated ensembles. The final decision is made by taking into consideration both the ability of each ensemble to fit the validation data locally and reducing the risk of overfitting. In this paper, the performance of GAEGA is assessed experimentally in comparison with other multiple classifier aggregation methods on 16 data sets. The experimental results demonstrate that GAEGA significantly outperforms the other methods in terms of average accuracy, ranging from 2.6% to 17.6%.     

Learning classifier systems resources

 This article lists currently available sources of information on classifier systems and classifier systems research, both on-line and in print. The need for new resources, and improvements to certain existing ones, are suggested.     

An artificial immune classifier for credit scoring analysis

The primary concern of the rating policies for a banking industry is to develop a more objective, accurate and competitive scoring model to avoid losses from potential bad debt. This study proposes an artificial immune classifier based on the artificial immune network (named AINE-based classifier) to evaluate the applicants’ credit scores. Two experimental credit datasets are used to show the accuracy rate of the artificial immune classifier. The ten-fold cross-validation method is applied to evaluate the performance of the classifier. The classifier is compared with other data mining techniques. Experimental results show that for the AINE-based classifier in credit scoring is more competitive than the SVM and hybrid SVM-based classifiers, except the BPN classifier. We further compare our classifier with other three AIS-based classifiers in the benchmark datasets, and show that the AINE-based classifier can rival the AIRS-based classifiers and outperforms the SAIS classifier when the number of attributes and classes increase. Our classifier can provide the credit card issuer with accurate and valuable information of credit scoring analyses to avoid making incorrect decisions that result in the loss of applicants’ bad debt.     

Learning control of autonomous robots using an instance-based classifier generator in continuous state space

A classifier system for the reinforcement learning control of autonomous mobile robots is proposed. The classifier system contains action selection, rules reproduction, and credit assignment mechanisms. An important feature of the classifier system is that it operates with continuous sensor and action spaces. The system is applied to the control of mobile robots. The local controllers use independent classifiers specified at the wheel-level. The controllers work autonomously, and with respect to each other represent dynamic systems connected through the external environment. The feasibility of the proposed system is tested in an experiment with a Khepera robot. It is shown that some patterns of global behavior can emerge from locally organized classifiers. This work was presented, in part, at the Third International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–21, 1998     

A comparative study of classifier ensembles for bankruptcy prediction

The aim of bankruptcy prediction in the areas of data mining and machine learning is to develop an effective model which can provide the higher prediction accuracy. In the prior literature, various classification techniques have been developed and studied, in/with which classifier ensembles by combining multiple classifiers approach have shown their outperformance over many single classifiers. However, in terms of constructing classifier ensembles, there are three critical issues which can affect their performance. The first one is the classification technique actually used/adopted, and the other two are the combination method to combine multiple classifiers and the number of classifiers to be combined, respectively. Since there are limited, relevant studies examining these aforementioned disuses, this paper conducts a comprehensive study of comparing classifier ensembles by three widely used classification techniques including multilayer perceptron (MLP) neural networks, support vector machines (SVM), and decision trees (DT) based on two well-known combination methods including bagging and boosting and different numbers of combined classifiers. Our experimental results by three public datasets show that DT ensembles composed of 80–100 classifiers using the boosting method perform best. The Wilcoxon signed ranked test also demonstrates that DT ensembles by boosting perform significantly different from the other classifier ensembles. Moreover, a further study over a real-world case by a Taiwan bankruptcy dataset was conducted, which also demonstrates the superiority of DT ensembles by boosting over the others.     

Accuracy-based learning classifier systems: models, analysis and applications to classification tasks

Recently, Learning Classifier Systems (LCS) and particularly XCS have arisen as promising methods for classification tasks and data mining. This paper investigates two models of accuracy-based learning classifier systems on different types of classification problems. Departing from XCS, we analyze the evolution of a complete action map as a knowledge representation. We propose an alternative, UCS, which evolves a best action map more efficiently. We also investigate how the fitness pressure guides the search towards accurate classifiers. While XCS bases fitness on a reinforcement learning scheme, UCS defines fitness from a supervised learning scheme. We find significant differences in how the fitness pressure leads towards accuracy, and suggest the use of a supervised approach specially for multi-class problems and problems with unbalanced classes. We also investigate the complexity factors which arise in each type of accuracy-based LCS. We provide a model on the learning complexity of LCS which is based on the representative examples given to the system. The results and observations are also extended to a set of real world classification problems, where accuracy-based LCS are shown to perform competitively with respect to other learning algorithms. The work presents an extended analysis of accuracy-based LCS, gives insight into the understanding of the LCS dynamics, and suggests open issues for further improvement of LCS on classification tasks.     

Data dependency in multiple classifier systems

In this paper, the data dependency of aggregation modules in multiple classifier system is being investigated. We first propose a new categorization scheme, in which combining methods are grouped into data-independent, implicitly data-dependent and explicitly data-dependent. It is argued that data-dependent approaches present the highest potential for improved performance. In this study, we intend to provide a comprehensive investigation of this argument and explore the impact of data dependency on the performance of multiple classifiers. We evaluate this impact based on two criteria, prediction accuracy and stability. In addition, we examine the effect of class imbalance and uneven data distribution on these two criteria. This paper presents the findings of an extensive set of comparative experiments. Based on the findings, it can be concluded that data-dependent aggregation methods are generally more stable and less sensitive to class imbalance. In addition, data-dependent methods exhibited superior or identical generalization ability for most of the data sets.     

Plurality voting-based multiple classifier systems: statistically independent with respect to dependent classifier sets

The simultaneous use of multiple classifiers has been shown to provide performance improvement in classification problems. The selection of an optimal set of classifiers is an important part of multiple classifier systems and the independence of classifier outputs is generally considered to be an advantage for obtaining better multiple classifier systems. In this paper, the need for the classifier independence is interrogated from classification performance point of view. The performance achieved with the use of classifiers having independent joint distributions is compared to some other classifiers which are defined to have best and worst joint distributions. These distributions are obtained by formulating the combination operation as an optimization problem. The analysis revealed several important observations about classifier selection which are then used to analyze the problem of selecting an additional classifier to be used with the available multiple classifier system.     

An efficient algorithm for the incremental construction of a piecewise linear classifier

In this paper the problem of finding piecewise linear boundaries between sets is considered and is applied for solving supervised data classification problems. An algorithm for the computation of piecewise linear boundaries, consisting of two main steps, is proposed. In the first step sets are approximated by hyperboxes to find so-called “indeterminate” regions between sets. In the second step sets are separated inside these “indeterminate” regions by piecewise linear functions. These functions are computed incrementally starting with a linear function. Results of numerical experiments are reported. These results demonstrate that the new algorithm requires a reasonable training time and it produces consistently good test set accuracy on most data sets comparing with mainstream classifiers.     

Pipelines on heterogeneous systems: models and tools

We study the performance of pipeline algorithms in heterogeneous networks. The concept of heterogeneity is not only restricted to the differences in computational power of the nodes, but also refers to the network capabilities. We develop a skeleton tool that allows us an efficient block‐cyclic mapping of pipelines on heterogeneous systems. The tool supports pipelines with a number of stages much larger than the number of physical processors available. We derive an analytical formula that allows us to predict the performance of pipelines in heterogeneous systems. According to the analytical complexity formula, numerical strategies to solve the optimal mapping problem are proposed. The computational results prove the accuracy of the predictions and effectiveness of the approach. Copyright © 2005 John Wiley & Sons, Ltd.     

Positive-realness analysis of sampled-data systems and its applications

In this paper, we first study strong positive-realness of sampled-data systems and introduce a measure called positive-realness gap index. We show that this index can be computed efficiently with a bisection method, and provide state space formulas for its computation. The importance of this index lies in that it is useful for robust stability analysis of sampled-data systems. An iterative procedure for computing an exact robust stability margin is given and illustrated through a numerical example.     

Effective classification of noisy data streams with attribute-oriented dynamic classifier selection

Recently, mining from data streams has become an important and challenging task for many real-world applications such as credit card fraud protection and sensor networking. One popular solution is to separate stream data into chunks, learn a base classifier from each chunk, and then integrate all base classifiers for effective classification. In this paper, we propose a new dynamic classifier selection (DCS) mechanism to integrate base classifiers for effective mining from data streams. The proposed algorithm dynamically selects a single “best” classifier to classify each test instance at run time. Our scheme uses statistical information from attribute values, and uses each attribute to partition the evaluation set into disjoint subsets, followed by a procedure that evaluates the classification accuracy of each base classifier on these subsets. Given a test instance, its attribute values determine the subsets that the similar instances in the evaluation set have constructed, and the classifier with the highest classification accuracy on those subsets is selected to classify the test instance. Experimental results and comparative studies demonstrate the efficiency and efficacy of our method. Such a DCS scheme appears to be promising in mining data streams with dramatic concept drifting or with a significant amount of noise, where the base classifiers are likely conflictive or have low confidence. A preliminary version of this paper was published in the Proceedings of the 4th IEEE International Conference on Data Mining, pp 305–312, Brighton, UK Xingquan Zhu received his Ph.D. degree in Computer Science from Fudan University, Shanghai, China, in 2001. He spent four months with Microsoft Research Asia, Beijing, China, where he was working on content-based image retrieval with relevance feedback. From 2001 to 2002, he was a Postdoctoral Associate in the Department of Computer Science, Purdue University, West Lafayette, IN. He is currently a Research Assistant Professor in the Department of Computer Science, University of Vermont, Burlington, VT. His research interests include Data mining, machine learning, data quality, multimedia computing, and information retrieval. Since 2000, Dr. Zhu has published extensively, including over 40 refereed papers in various journals and conference proceedings. Xindong Wu is a Professor and the Chair of the Department of Computer Science at the University of Vermont. He holds a Ph.D. in Artificial Intelligence from the University of Edinburgh, Britain. His research interests include data mining, knowledge-based systems, and Web information exploration. He has published extensively in these areas in various journals and conferences, including IEEE TKDE, TPAMI, ACM TOIS, IJCAI, ICML, KDD, ICDM, and WWW, as well as 11 books and conference proceedings. Dr. Wu is the Editor-in-Chief of the IEEE Transactions on Knowledge and Data Engineering (by the IEEE Computer Society), the founder and current Steering Committee Chair of the IEEE International Conference on Data Mining (ICDM), an Honorary Editor-in-Chief of Knowledge and Information Systems (by Springer), and a Series Editor of the Springer Book Series on Advanced Information and Knowledge Processing (AI&KP). He is the 2004 ACM SIGKDD Service Award winner. Ying Yang received her Ph.D. in Computer Science from Monash University, Australia in 2003. Following academic appointments at the University of Vermont, USA, she currently holds a Research Fellow at Monash University, Australia. Dr. Yang is recognized for contributions in the fields of machine learning and data mining. She has published many scientific papers and book chapters on adaptive learning, proactive mining, noise cleansing and discretization. Contact her at yyang@mail.csse.monash.edu.au.     

Soft systems and systems engineering: on the use of conceptual models in information system development

Abstract. Many recent studies have shown that computer-based systems continue to ‘fail’ at a number of different levels (Romtec, 1988; KPMG, 1990) and it is increasingly apparent (Maclaren et al., 1991) that the most serious failures of information technology (IT) lie in the continuing inability to address those concerns which are central to the successful achievement of individual, organizational and social goals. It is the contention of this paper that this failing is precisely because these are the areas which are ignored or inadequately treated by conventional system development methods. There is, of course, a vast body of literature concerned with the understanding of complex human activity systems. This literature often reflects a mass of contradictions at the epistemological and the ontological level about the behaviour of such systems and has also spawned numerous methods (and methodologies) which seek to guide the individual in making successful interventions into organizational situations (Rosenhead, 1989). Despite this multiplicity of viewpoints many writers have posited a dichotomy between so-called 'soft and ‘hard’ approaches to problem situations and use this dichotomy to inform the choice of an appropriate problem-solving methodology (Checkland, 1985). In this paper we characterize these two approaches as being concerned with either the purpose(s) of the human activity system (i.e. ‘doing the right thing’) or with the design of the efficient means of achieving such purpose(s) (i.e. ‘doing the thing right’). It is our belief that much of the literature and work in either area has not concerned itself with the issues of the other. Writers on ‘hard’ engineering methods often assume the question of purpose to be either straightforward (e.g. given in the project brief) or, paradoxically, too difficult (e.g. it is not our concern as mere systems analysts). Writers on ‘soft’ methods on the other hand rarely have anything to say about the design and implementation of well-engineered computer-based systems, giving the impression that this is a somewhat mundane activity better left to technical experts. This paper, therefore, attempts to set out a rationale for the bringing together of principles from both ‘hard’ engineering and ‘soft’ inquiry methods without doing epistemological damage to either. To illustrate our argument we concentrate on JSD (Jackson system development) as an example of system engineering (Cameron, 1983) and SSM (soft systems methodology) as an example of system inquiry (Checkland, 1981; Checkland & Scholes, 1990). Our general thesis, however, does not depend upon either of these two approaches per se but applies to the overall issue of bringing together insights from two apparently opposed epistemological positions in an effort better to harness the power of IT in pursuit of purposeful human activity.     

Corrective classification: Learning from data imperfections with aggressive and diverse classifier ensembling

Learning from imperfect (noisy) information sources is a challenging and reality issue for many data mining applications. Common practices include data quality enhancement by applying data preprocessing techniques or employing robust learning algorithms to avoid developing overly complicated structures that overfit the noise. The essential goal is to reduce noise impact and eventually enhance the learners built from noise-corrupted data. In this paper, we propose a novel corrective classification (C2) design, which incorporates data cleansing, error correction, Bootstrap sampling and classifier ensembling for effective learning from noisy data sources. C2 differs from existing classifier ensembling or robust learning algorithms in two aspects. On one hand, a set of diverse base learners of C2 constituting the ensemble are constructed via a Bootstrap sampling process; on the other hand, C2 further improves each base learner by unifying error detection, correction and data cleansing to reduce noise impact. Being corrective, the classifier ensemble is built from data preprocessed/corrected by the data cleansing and correcting modules. Experimental comparisons demonstrate that C2 is not only more accurate than the learner built from original noisy sources, but also more reliable than Bagging [4] or aggressive classifier ensemble (ACE) [56], which are two degenerated components/variants of C2. The comparisons also indicate that C2 is more stable than Boosting and DECORATE, which are two state-of-the-art ensembling methods. For real-world imperfect information sources (i.e. noisy training and/or test data), C2 is able to deliver more accurate and reliable prediction models than its other peers can offer.     

Multirate sampled-data systems: computing fast-rate models

This paper studies identification of a general multirate sampled-data system. Using the lifting technique, we associate the multirate system with an equivalent linear time-invariant system, from which a fast-rate discrete-time system is extracted. Uniqueness of the fast-rate system, controllability and observability of the lifted system, and other related issues are discussed. The effectiveness is demonstrated through simulation and real-time implementation.     

Construction of discrete-time models for port-controlled Hamiltonian systems with applications

The issues of constructing a discrete-time model for Hamiltonian systems are in general different from those for dissipative systems. We propose an algorithm for constructing an approximate discrete-time model, which guarantees Hamiltonian conservation. We show that the algorithm also preserves, in a weaker sense, the losslessness property of a class of port-controlled Hamiltonian systems. An application of the algorithm to port-controlled Hamiltonian systems with quadratic Hamiltonian is presented, and we use this to solve the stabilization problem for this class of systems based on the approximate discrete-time model constructed using the proposed algorithm. We illustrate the usefulness of the algorithm in designing a discrete-time controller to stabilize the angular velocity of the dynamics of a rigid body.     

Improving the management of microfinance institutions by using credit scoring models based on Statistical Learning techniques

A wide range of supervised classification algorithms have been successfully applied for credit scoring in non-microfinance environments according to recent literature. However, credit scoring in the microfinance industry is a relatively recent application, and current research is based, to the best of our knowledge, on classical statistical methods. This lack is surprising since the implementation of credit scoring based on supervised classification algorithms should contribute towards the efficiency of microfinance institutions, thereby improving their competitiveness in an increasingly constrained environment. This paper explores an extensive list of Statistical Learning techniques as microfinance credit scoring tools from an empirical viewpoint. A data set of microcredits belonging to a Peruvian Microfinance Institution is considered, and the following models are applied to decide between default and non-default credits: linear and quadratic discriminant analysis, logistic regression, multilayer perceptron, support vector machines, classification trees, and ensemble methods based on bagging and boosting algorithm. The obtained results suggest the use of a multilayer perceptron trained in the R statistical system with a second order algorithm. Moreover, our findings show that, with the implementation of this MLP-based model, the MFI? misclassification costs could be reduced to 13.7% with respect to the application of other classic models.     

Supporting teachers in adaptive educational systems through predictive models: A proof of concept

Adaptive educational systems (AESs) guide students through the course materials in order to improve the effectiveness of the learning process. However, AES cannot replace the teacher. Instead, teachers can also benefit from the use of adaptive educational systems enabling them to detect situations in which students experience problems (when working with the AES). To this end the teacher needs to monitor, understand and evaluate the students’ activity within the AES. In fact, these systems can be enhanced if tools for supporting teachers in this task are provided. In this paper, we present the experiences with predictive models that have been undertaken to assist the teacher in PDinamet, a web-based adaptive educational system for teaching Physics in secondary education. Although the obtained models are still very simple, our findings suggest the feasibility of predictive modeling in the area of supporting teachers in adaptive educational systems.     

Classification cost: An empirical comparison among traditional classifier, Cost-Sensitive Classifier, and MetaCost

Loan fraud is a critical factor in the insolvency of financial institutions, so companies make an effort to reduce the loss from fraud by building a model for proactive fraud prediction. However, there are still two critical problems to be resolved for the fraud detection: (1) the lack of cost sensitivity between type I error and type II error in most prediction models, and (2) highly skewed distribution of class in the dataset used for fraud detection because of sparse fraud-related data. The objective of this paper is to examine whether classification cost is affected both by the cost-sensitive approach and by skewed distribution of class. To that end, we compare the classification cost incurred by a traditional cost-insensitive classification approach and two cost-sensitive classification approaches, Cost-Sensitive Classifier (CSC) and MetaCost. Experiments were conducted with a credit loan dataset from a major financial institution in Korea, while varying the distribution of class in the dataset and the number of input variables. The experiments showed that the lowest classification cost was incurred when the MetaCost approach was used and when non-fraud data and fraud data were balanced. In addition, the dataset that includes all delinquency variables was shown to be most effective on reducing the classification cost.