Automated design of linear tree classifiers

An automated method is presented for the design of linear tree classifiers, i.e. tree classifiers in which a decision based on a linear sum of features is carried out at each node. The method exploits the discriminability of Tomek links joining opposed pairs of data points in multidimensional feature space to produce a hierarchically structured piecewise linear decision function. The corresponding decision surface is optimized by a gradient descent that maximizes the number of Tomek links cut by each linear segment of the decision surface, followed by training each node's linear decision segment on the data associated with that node. Experiments on real data obtained from ship images and character images suggest that the accuracy of the tree classifier designed by this scheme is comparable to that of the k-NN classifier while providing much greater decision speeds, and that the trade-off between the speed and the accuracy in pattern classification can be controlled by bounding the number of features to be used at each node of the tree. Further experiments comparing the classification errors of our tree classifier with the tree classifier produced by the Mui/Fu technique indicate that our tree classifier is no less accurate and often much faster than the Mui/Fu classifier.     

Symmetric Veronese classifiers with application to materials design

To solve the materials classification problem, we propose a fast, exhaustive approach. We propose to test every feature (chemical property), every pair of features, every three features, etc., against every classifier architecture from a certain group of classifiers known as Support Vector Machines. This approach generalizes Pierre Villars’ work to higher dimensions and more operations. We have at present duplicated his result in identifying the Mendeleev Number as the single best feature, and we have produced a new result for the case of two features: namely, we have identified the Mendeleev number with the valence electron number as the best combination of two features.     

A new boosting design of Support Vector Machine classifiers

Boosting algorithms pay attention to the particular structure of the training data when learning, by means of iteratively emphasizing the importance of the training samples according to their difficulty for being correctly classified. If common kernel Support Vector Machines (SVMs) are used as basic learners to construct a Real AdaBoost ensemble, the resulting ensemble can be easily compacted into a monolithic architecture by simply combining the weights that correspond to the same kernels when they appear in different learners, avoiding to increase the operation computational effort for the above potential advantage. This way, the performance advantage that boosting provides can be obtained for monolithic SVMs, i.e., without paying in classification computational effort because many learners are needed. However, SVMs are both stable and strong, and their use for boosting requires to unstabilize and to weaken them. Yet previous attempts in this direction show a moderate success.In this paper, we propose a combination of a new and appropriately designed subsampling process and an SVM algorithm which permits sparsity control to solve the difficulties in boosting SVMs for obtaining improved performance designs. Experimental results support the effectiveness of the approach, not only in performance, but also in compactness of the resulting classifiers, as well as that combining both design ideas is needed to arrive to these advantageous designs.     

A method for the design of binary tree classifiers

A method based on multivariate stepwise regression is proposed for the design of binary tree classifiers. Experimental results of cell classification are given to illustrate the efficiency of the proposed method.     

A novel approach to design classifiers using genetic programming

We propose a new approach for designing classifiers for a c-class (c/spl ges/2) problem using genetic programming (GP). The proposed approach takes an integrated view of all classes when the GP evolves. A multitree representation of chromosomes is used. In this context, we propose a modified crossover operation and a new mutation operation that reduces the destructive nature of conventional genetic operations. We use a new concept of unfitness of a tree to select trees for genetic operations. This gives more opportunity to unfit trees to become fit. A new concept of OR-ing chromosomes in the terminal population is introduced, which enables us to get a classifier with better performance. Finally, a weight-based scheme and some heuristic rules characterizing typical ambiguous situations are used for conflict resolution. The classifier is capable of saying "don't know" when faced with unfamiliar examples. The effectiveness of our scheme is demonstrated on several real data sets.     

基于差异性度量的多分类器集成系统设计

为了解决在分类器集成过程中分类性能要求高和集成过程复杂等问题,分析常规集成方法的优缺点,研究已有的分类器差异性度量方法,提出了筛选差异性尽可能大的分类器作为基分类器而构建的一个层级式分类器集成系统.构建不同的基分类器,选择准确率较高的备选,分析其差异性,选出差异大的分类器作为系统所需基分类器,构成集成系统.通过在UCI数据集上进行的试验,获得了很好的分类识别效果,验证了这种分类集成系统的优越性.     

An automated approach to the design of decision tree classifiers

The classification of large dimensional data sets arising from the merging of remote sensing data with more traditional forms of ancillary data causes a significant computational problem. Decision tree classification is a popular approach to the problem. This type of classifier is characterized by the property that samples are subjected to a sequence of decision rules before they are assigned to a unique class. If a decision tree classifier is well designed, the result in many cases is a classification scheme which is accurate, flexible, and computationally efficient. This correspondence provides an automated technique for effective decision tree design which relies only on a priori statistics. This procedure utilizes canonical transforms and Bayes table look-up decision rules. An optimal design at each node is derived based on the associated decision table. A procedure for computing the global probability of correct classification is also provided. An example is given in which class statistics obtained from an actual Landsat scene are used as input to the program. The resulting decision tree design has an associated probability of correct classification of 0.75 compared to the theoretically optimum 0.79 probability of correct classification associated with a full dimensional Bayes classifier. Recommendations for future research are included.     

An intuitive general rank-based correlation coefficient

Correlation analysis is an effective mechanism for studying patterns in data and making predictions. Many interesting discoveries have been made by formulating correlations in seemingly unrelated data. We propose an algorithm to quantify the theory of correlations and to give an intuitive, more accurate correlation coefficient. We propose a predictive metric to calculate correlations between paired values, known as the general rank-based correlation coefficient. It fulfills the five basic criteria of a predictive metric: independence from sample size, value between ?1 and 1, measuring the degree of monotonicity, insensitivity to outliers, and intuitive demonstration. Furthermore, the metric has been validated by performing experiments using a real-time dataset and random number simulations. Mathematical derivations of the proposed equations have also been provided. We have compared it to Spearman’s rank correlation coefficient. The comparison results show that the proposed metric fares better than the existing metric on all the predictive metric criteria.     

Exploiting application locality to design low-complexity, highly performing, and power-aware embedded classifiers

Temporal and spatial locality of the inputs, i.e., the property allowing a classifier to receive the same samples over time-or samples belonging to a neighborhood-with high probability, can be translated into the design of embedded classifiers. The outcome is a computational complexity and power aware design particularly suitable for implementation. A classifier based on the gated-parallel family has been found particularly suitable for exploiting locality properties: Subclassifiers are generally small, independent each other, and controlled by a master-enabling module granting that only a subclassifier is active at a time, the others being switched off. By exploiting locality properties we obtain classifiers with accuracy comparable with the ones designed without integrating locality but gaining a significant reduction in computational complexity and power consumption.     

Analysis of error-reject trade-off in linearly combined multiple classifiers

In this paper, a theoretical and experimental analysis of the error-reject trade-off achievable by linearly combining the outputs of an ensemble of classifiers is presented. To this aim, the theoretical framework previously developed by Tumer and Ghosh for the analysis of the simple average rule without the reject option has been extended. Analytical results that allow to evaluate the improvement of the error-reject trade-off achievable by simple averaging their outputs under different assumptions about the distributions of the estimation errors affecting a posteriori probabilities, are provided. The conditions under which the weighted average can provide a better error-reject trade-off than the simple average are then determined. From the theoretical results obtained under the assumption of unbiased and uncorrelated estimation errors, simple guidelines for the design of multiple classifier systems using linear combiners are given. Finally, an experimental evaluation and comparison of the error-reject trade-off of the simple and weighted averages is reported for five real data sets. The results show the practical relevance of the proposed guidelines in the design of linear combiners.     

Ensemble-based classifiers

The idea of ensemble methodology is to build a predictive model by integrating multiple models. It is well-known that ensemble methods can be used for improving prediction performance. Researchers from various disciplines such as statistics and AI considered the use of ensemble methodology. This paper, review existing ensemble techniques and can be served as a tutorial for practitioners who are interested in building ensemble based systems.     

Mapping classifiers and datasets

Given the posterior probability estimates of 14 classifiers on 38 datasets, we plot two-dimensional maps of classifiers and datasets using principal component analysis (PCA) and Isomap. The similarity between classifiers indicate correlation (or diversity) between them and can be used in deciding whether to include both in an ensemble. Similarly, datasets which are too similar need not both be used in a general comparison experiment. The results show that (i) most of the datasets (approximately two third) we used are similar to each other, (ii) multilayer perceptrons and k-nearest neighbor variants are more similar to each other than support vector machine and decision tree variants, (iii) the number of classes and the sample size has an effect on similarity.     

Multiscale rank-based ordered dither algorithm for digital halftoning

In this paper, we proposed a simple and efficient algorithm that is based on the ranking of brightness within block. It builds the cumulative intensity quadtree for each block and generates the digital halftone׳s image using its information. We did the quality assessment for the result image of the proposed algorithm. As a result of experiment, it generates the digital halftone׳s image that is a little worse than that generated by error diffusion method, but it has merits to be implemented easily in parallel hardware because of block processing and simple operations. So we can conclude that the proposed algorithm is a very practical method to apply to the hardware implementation.     

Extensions to rank-based prototype selection in k-Nearest Neighbour classification

The $k$-nearest neighbour rule is commonly considered for classification tasks given its straightforward implementation and good performance in many applications. However, its efficiency represents an obstacle in real-case scenarios because the classification requires computing a distance to every single prototype of the training set. Prototype Selection (PS) is a typical approach to alleviate this problem, which focuses on reducing the size of the training set by selecting the most interesting prototypes. In this context, rank methods have been postulated as a good solution: following some heuristics, these methods perform an ordering of the prototypes according to their relevance in the classification task, which is then used to select the most relevant ones. This work presents a significant improvement of existing rank methods by proposing two extensions: (i) a greater robustness against noise at label level by considering the parameter ‘k’ of the classification in the selection process; and (ii) a new parameter-free rule to select the prototypes once they have been ordered. The experiments performed in different scenarios and datasets demonstrate the goodness of these extensions. Also, it is empirically proved that the new full approach is competitive with respect to existing PS algorithms.     

Multi-category classifiers and sample width

In a recent paper, the authors introduced the notion of sample width for binary classifiers defined on the set of real numbers. It was shown that the performance of such classifiers could be quantified in terms of this sample width. This paper considers how to adapt the idea of sample width so that it can be applied in cases where the classifiers are multi-category and are defined on some arbitrary metric space.     

GP-based secondary classifiers

Genetic programming (GP) is used to evolve secondary classifiers for disambiguating between pairs of handwritten digit images. The inherent property of feature selection accorded by GP is exploited to make sharper decision between conflicting classes. Classification can be done in several steps with an available feature set and a mixture of strategies. A two-step classification strategy is presented in this paper. After the first step of the classification using the full feature set, the high confidence recognition result will lead to an end of the recognition process. Otherwise a secondary classifier designed using a sub-set of the original feature set and the information available from the earlier classification step will help classify the input further. The feature selection mechanism employed by GP selects important features that provide maximum separability between classes under consideration. In this way, a sharper decision on fewer classes is obtained at the secondary classification stage. The full feature set is still available in both stages of classification to retain complete information. An intuitive motivation and detailed analysis using confusion matrices between digit classes is presented to describe how this strategy leads to improved recognition performance. In comparison with the existing methods, our method is aimed for increasing recognition accuracy and reliability. Results are reported for the BHA test-set and the NIST test-set of handwritten digits.     

Optimal robust classifiers

Qualitatively, a filter is said to be “robust” if its performance degradation is acceptable for distributions close to the one for which it is optimal, that is, the one for which it has been designed. This paper adapts the signal-processing theory of optimal robust filters to classifiers. The distribution (class conditional distributions) to which the classifier is to be applied is parameterized by a state vector and the principle issue is to choose a design state that is optimal in comparison to all other states relative to some measure of robustness. A minimax robust classifier is one whose worst performance over all states is better than the worst performances of the other classifiers (defined at the other states). A Bayesian robust classifier is one whose expected performance is better than the expected performances of the other classifiers. The state corresponding to the Bayesian robust classifier is called the maximally robust state. Minimax robust classifiers tend to give too much weight to states for which classification is very difficult and therefore our effort is focused on Bayesian robust classifiers. Whereas the signal-processing theory of robust filtering concentrates on design with full distributional knowledge and a fixed number of observation variables (features), design via training from sample data and feature selection are so important for classification that robustness optimality must be considered from these perspectives—in particular, for small samples. In this context, for a given sample size, we will be concerned with the maximally robust state-feature pair. All definitions are independent of the classification rule; however, applications are only considered for linear and quadratic discriminant analysis, for which there are parametric forms for the optimal discriminants.     

Combinations of weak classifiers

To obtain classification systems with both good generalization performance and efficiency in space and time, we propose a learning method based on combinations of weak classifiers, where weak classifiers are linear classifiers (perceptrons) which can do a little better than making random guesses. A randomized algorithm is proposed to find the weak classifiers. They are then combined through a majority vote. As demonstrated through systematic experiments, the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications. Theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works. In particular, when the strength of weak classifiers is properly chosen, combinations of weak classifiers can achieve a good generalization performance with polynomial space- and time-complexity.     

On combining classifiers

We develop a common theoretical framework for combining classifiers which use distinct pattern representations and show that many existing schemes can be considered as special cases of compound classification where all the pattern representations are used jointly to make a decision. An experimental comparison of various classifier combination schemes demonstrates that the combination rule developed under the most restrictive assumptions-the sum rule-outperforms other classifier combinations schemes. A sensitivity analysis of the various schemes to estimation errors is carried out to show that this finding can be justified theoretically     

On output independence and complementariness in rank-based multiple classifier decision systems

This study presents a theoretical analysis of output independence and complementariness between classifiers in a rank-based multiple classifier decision system in the context of the partitioned observation space theory. To enable such an analysis, an information theoretic interpretation of a rank-based multiple classifier system is developed and basic concepts from information theory are applied to develop measures for output independence and complementariness. It is shown that output independence of classifiers is not a requirement for achieving complementariness between these classifiers. Namely, output independence does not imply a performance improvement by combining multiple classifiers. A condition called dominance is shown to be important instead. The information theoretic measures proposed for output independence and complementariness are justified by simulated examples.