Improving experimental studies about ensembles of classifiers for bankruptcy prediction and credit scoring

Previous studies about ensembles of classifiers for bankruptcy prediction and credit scoring have been presented. In these studies, different ensemble schemes for complex classifiers were applied, and the best results were obtained using the Random Subspace method. The Bagging scheme was one of the ensemble methods used in the comparison. However, it was not correctly used. It is very important to use this ensemble scheme on weak and unstable classifiers for producing diversity in the combination. In order to improve the comparison, Bagging scheme on several decision trees models is applied to bankruptcy prediction and credit scoring. Decision trees encourage diversity for the combination of classifiers. Finally, an experimental study shows that Bagging scheme on decision trees present the best results for bankruptcy prediction and credit scoring.     

基于混合特征参数和BP_Adaboost的方言辨识

着眼于非特定人孤立词湖南地区的方言辨识,提出一种将BP神经网络和Adaboost算法相结合的辨识模型。为反映方言的动态特性及其声道特性,采用LPCC、MFCC和各自一阶差分系数相组合作为方言特征系数。利用多个BP神经网络作为弱分类器对方言进行初步辨识,借助Adaboost迭代算法将这些弱分类器组合起来构成强分类器,得出最终辨识结果。实验证明,该混合模型较单纯的BP神经网络具有更强的噪声鲁棒性和较高的识别率。     

Boosting recombined weak classifiers

Boosting is a set of methods for the construction of classifier ensembles. The differential feature of these methods is that they allow to obtain a strong classifier from the combination of weak classifiers. Therefore, it is possible to use boosting methods with very simple base classifiers. One of the most simple classifiers are decision stumps, decision trees with only one decision node.

This work proposes a variant of the most well-known boosting method, AdaBoost. It is based on considering, as the base classifiers for boosting, not only the last weak classifier, but a classifier formed by the last r selected weak classifiers (r is a parameter of the method). If the weak classifiers are decision stumps, the combination of r weak classifiers is a decision tree.

The ensembles obtained with the variant are formed by the same number of decision stumps than the original AdaBoost. Hence, the original version and the variant produce classifiers with very similar sizes and computational complexities (for training and classification). The experimental study shows that the variant is clearly beneficial.     

用Boosting方法组合增强Stumps进行文本分类

为提高文本分类的精度,Schapire和Singer尝试了一个用Boosting来组合仅有一个划分的简单决策树(Stumps)的方法.其基学习器的划分是由某个特定词项是否在待分类文档中出现决定的.这样的基学习器明显太弱,造成最后组合成的Boosting分类器精度不够理想,而且需要的迭代次数很大,因而效率很低.针对这个问题,提出由文档中所有词项来决定基学习器划分以增强基学习器分类能力的方法.它把以VSM表示的文档与类代表向量之间的相似度和某特定阈值的大小关系作为基学习器划分的标准.同时,为提高算法的收敛速度,在类代表向量的计算过程中动态引入Boosting分配给各学习样本的权重.实验结果表明,这种方法提高了用Boosting组合Stump分类器进行文本分类的性能(精度和效率),而且问题规模越大,效果越明显.     

Fast and Robust Face Detection Using Evolutionary Pruning

Face detection task can be considered as a classifier training problem. Finding the parameters of the classifier model by using training data is a complex process. To solve such a complex problem, evolutionary algorithms can be employed in cascade structure of classifiers. This paper proposes evolutionary pruning to reduce the number of weak classifiers in AdaBoost-based cascade detector, while maintaining the detection accuracy. The computation time is proportional to the number of weak classifiers and, therefore, a reduction in the number of weak classifiers results in an increased detection speed. Three kinds of cascade structures are compared by the number of weak classifiers. The efficiency in computation time of the proposed cascade structure is shown experimentally. It is also compared with the state-of-the-art face detectors, and the results show that the proposed method outperforms the previous studies. A multiview face detector is constructed by incorporating the three face detectors: frontal, left profile, and right profile.     

AdaBoost-based algorithm for network intrusion detection.

Network intrusion detection aims at distinguishing the attacks on the Internet from normal use of the Internet. It is an indispensable part of the information security system. Due to the variety of network behaviors and the rapid development of attack fashions, it is necessary to develop fast machine-learning-based intrusion detection algorithms with high detection rates and low false-alarm rates. In this correspondence, we propose an intrusion detection algorithm based on the AdaBoost algorithm. In the algorithm, decision stumps are used as weak classifiers. The decision rules are provided for both categorical and continuous features. By combining the weak classifiers for continuous features and the weak classifiers for categorical features into a strong classifier, the relations between these two different types of features are handled naturally, without any forced conversions between continuous and categorical features. Adaptable initial weights and a simple strategy for avoiding overfitting are adopted to improve the performance of the algorithm. Experimental results show that our algorithm has low computational complexity and error rates, as compared with algorithms of higher computational complexity, as tested on the benchmark sample data.     

The use of vicinal-risk minimization for training decision trees

We propose the use of Vapnik's vicinal risk minimization (VRM) for training decision trees to approximately maximize decision margins. We implement VRM by propagating uncertainties in the input attributes into the labeling decisions. In this way, we perform a global regularization over the decision tree structure. During a training phase, a decision tree is constructed to minimize the total probability of misclassifying the labeled training examples, a process which approximately maximizes the margins of the resulting classifier. We perform the necessary minimization using an appropriate meta-heuristic (genetic programming) and present results over a range of synthetic and benchmark real datasets. We demonstrate the statistical superiority of VRM training over conventional empirical risk minimization (ERM) and the well-known C4.5 algorithm, for a range of synthetic and real datasets. We also conclude that there is no statistical difference between trees trained by ERM and using C4.5. Training with VRM is shown to be more stable and repeatable than by ERM.     

Approaches to construction of linear classifiers in the case of many classes

Some approaches to the problem of constructing linear classifiers, including embedded ones, are studied for the case of many classes. Sufficient conditions for linear separability of classes are formulated, and specifics of the problem statement when sets are not linearly separable are considered. Different approaches to construction of optimal linear classifiers are studied, and the results of numerical experiments are presented. The properties of embedded (convex piecewise linear) classifiers are studied. It is shown that, for an arbitrary family of finite nonintersecting sets, there is an embedded linear classifier that correctly separates the points of these sets.     

Bagging, Boosting and the Random Subspace Method for Linear Classifiers

Recently bagging, boosting and the random subspace method have become popular combining techniques for improving weak classifiers. These techniques are designed for, and usually applied to, decision trees. In this paper, in contrast to a common opinion, we demonstrate that they may also be useful in linear discriminant analysis. Simulation studies, carried out for several artificial and real data sets, show that the performance of the combining techniques is strongly affected by the small sample size properties of the base classifier: boosting is useful for large training sample sizes, while bagging and the random subspace method are useful for critical training sample sizes. Finally, a table describing the possible usefulness of the combining techniques for linear classifiers is presented. Received: 03 November 2000, Received in revised form: 02 November 2001, Accepted: 13 December 2001     

The random subspace method for constructing decision forests

Much of previous attention on decision trees focuses on the splitting criteria and optimization of tree sizes. The dilemma between overfitting and achieving maximum accuracy is seldom resolved. A method to construct a decision tree based classifier is proposed that maintains highest accuracy on training data and improves on generalization accuracy as it grows in complexity. The classifier consists of multiple trees constructed systematically by pseudorandomly selecting subsets of components of the feature vector, that is, trees constructed in randomly chosen subspaces. The subspace method is compared to single-tree classifiers and other forest construction methods by experiments on publicly available datasets, where the method's superiority is demonstrated. We also discuss independence between trees in a forest and relate that to the combined classification accuracy     

Combinations of weak classifiers

To obtain classification systems with both good generalization performance and efficiency in space and time, we propose a learning method based on combinations of weak classifiers, where weak classifiers are linear classifiers (perceptrons) which can do a little better than making random guesses. A randomized algorithm is proposed to find the weak classifiers. They are then combined through a majority vote. As demonstrated through systematic experiments, the method developed is able to obtain combinations of weak classifiers with good generalization performance and a fast training time on a variety of test problems and real applications. Theoretical analysis on one of the test problems investigated in our experiments provides insights on when and why the proposed method works. In particular, when the strength of weak classifiers is properly chosen, combinations of weak classifiers can achieve a good generalization performance with polynomial space- and time-complexity.     

Hyperspectral discrimination of tropical dry forest lianas and trees: Comparative data reduction approaches at the leaf and canopy levels

A dataset of spectral signatures (leaf level) of tropical dry forest trees and lianas and an airborne hyperspectral image (crown level) are used to test three hyperspectral data reduction techniques (principal component analysis, forward feature selection and wavelet energy feature vectors) along with pattern recognition classifiers to discriminate between the spectral signatures of lianas and trees. It was found at the leaf level the forward waveband selection method had the best results followed by the wavelet energy feature vector and a form of principal component analysis. For the same dataset our results indicate that none of the pattern recognition classifiers performed the best across all reduction techniques, and also that none of the parametric classifiers had the overall lowest training and testing errors. At the crown level, in addition to higher testing error rates (7%), it was found that there was no optimal data reduction technique. The significant wavebands were also found to be different between the leaf and crown levels. At the leaf level, the visible region of the spectrum was the most important for discriminating between lianas and trees whereas at the crown level the shortwave infrared was also important in addition to the visible and near infrared.     

Structural diversity for decision tree ensemble learning

Decision trees are a kind of off-the-shelf predictive models, and they have been successfully used as the base learners in ensemble learning. To construct a strong classifier ensemble, the individual classifiers should be accurate and diverse. However, diversity measure remains a mystery although there were many attempts. We conjecture that a deficiency of previous diversity measures lies in the fact that they consider only behavioral diversity, i.e., how the classifiers behave when making predictions, neglecting the fact that classifiers may be potentially different even when they make the same predictions. Based on this recognition, in this paper, we advocate to consider structural diversity in addition to behavioral diversity, and propose the TMD (tree matching diversity) measure for decision trees. To investigate the usefulness of TMD, we empirically evaluate performances of selective ensemble approaches with decision forests by incorporating different diversity measures. Our results validate that by considering structural and behavioral diversities together, stronger ensembles can be constructed. This may raise a new direction to design better diversity measures and ensemble methods.     

A method of exploiting spatial information for improving classification rules: Application to the construction of polygon-based land cover maps

This article proposes a method of exploiting spatial information to improve classification rules constructed by automated methods such as k-nearest neighbour or linear discriminant analysis. The method is intended for polygonbased, land cover type mapping using remote sensing information in a GIS. Our approach differs from contextual allocation methods used in lattice- or pixel-based mapping because it does not rely on spatial dependence models. Instead, the method uses a Bayes-type formula to modify the estimated posterior probabilities of group membership produced by automated classifiers. The method is found to substantially improve classification accuracy estimates in areas where there is a moderate or greater degree of physiographic variation across the map extent.     

基于改进在线多示例学习算法的机器人目标跟踪

提出基于改进的在线多示例学习算法(Improved multiple instance learning, IMIL)的移动机器人目标跟踪方法. 该方法利用射频识别系统(Radio frequency identification, RFID)粗定位IMIL算法的搜索区域, 然后应用IMIL算法实现目标跟踪. 该方法保证了机器人跟踪系统的连续性, 解决了目标突然转弯时的跟踪问题. IMIL算法采用从低维空间提取的压缩特征描述包中示例, 以降低算法耗时. 通过最大化弱分类器与极大似然概率的内积, 选择判别能力强的弱分类器, 避免了弱分类器选择过程中多次计算包概率和示例概率, 进一步提高算法的实时处理能力. 计算包概率时该算法平等对待各示例, 保证概率高的示例对包概率的贡献度, 克服跟踪漂移问题. 跟踪过程中, 结合当前跟踪结果与目标模板间的相似性分数在线实时调整分类器, 提高了算法的自适应能力. 最后将本文方法在视频和移动机器人上进行实验. 实验结果表明, 该方法在目标运动突变及外观改变时具有较强的鲁棒性和准确性, 并满足系统的实时性要求.     

新的软间隔AdaBoost弱分类器权重调整算法

为避免硬间隔算法过分强调较难分类样本而导致泛化性能下降的问题,提出一种新的基于软间隔的AdaBoost-QP算法。在样本硬间隔中加入松弛项,得到软间隔的概念,以优化样本间隔分布、调整弱分类器的权重。实验结果表明,该算法能降低泛化误差,提高 AdaBoost算法的泛化性能。     

Multilevel classifiers based on a tree-structured set of Gaussian densities

This paper considers an approach to solving the problem of binary classification of objects. This approach is based on representing one of the classes by a sequence of Gaussian mixtures with further introduction of threshold decision rules. A method of constructing hierarchical sequences of Gaussian mixtures using the partial EM algorithm is proposed. We compare classifiers that use single Gaussian mixtures, cascades based on sequences of independent mixtures, cascades based on hierarchical sequences of mixtures, and classifiers that use trees of Gaussian densities for decision making. The theoretical estimates of computational costs for these classifiers are provided. The classifiers are tested on simulated data. The results are presented as the relations between the computational cost of classification and the obtained values of error criteria.     

人脸检测的继承式集成学习方法

基于"遗传+变异"模式,提出继承式集成学习方法框架,它可以训练出四种不同形式的层叠分类器。除了基于"无遗传"模式的基本层叠分类器与基于"全部遗传"模式的嵌入式层叠分类器两种传统方法之外,还有基于"部分遗传+部分变异"模式的特征继承层叠分类器与弱分类器继承层叠分类器。虽然后两种层叠分类器都有一定的继承代价,但是其拟合性更好,可以更好地均衡收敛速度和扩展性能,其综合性能优于传统方法。基于RAB、GAB算法与LUT弱分类器的正面直立人脸检测实验结果表明了新的继承式集成学习方法的有效性。     

On the generalization error of fixed combinations of classifiers

We consider the generalization error of concept learning when using a fixed Boolean function of the outputs of a number of different classifiers. Here, we take into account the ‘margins’ of each of the constituent classifiers. A special case is that in which the constituent classifiers are linear threshold functions (or perceptrons) and the fixed Boolean function is the majority function. This corresponds to a ‘committee of perceptrons,’ an artificial neural network (or circuit) consisting of a single layer of perceptrons (or linear threshold units) in which the output of the network is defined to be the majority output of the perceptrons. Recent work of Auer et al. studied the computational properties of such networks (where they were called ‘parallel perceptrons’), proposed an incremental learning algorithm for them, and demonstrated empirically that the learning rule is effective. As a corollary of the results presented here, generalization error bounds are derived for this special case that provide further motivation for the use of this learning rule.     

An intrusion detection and alert correlation approach based on revising probabilistic classifiers using expert knowledge

Bayesian networks are important knowledge representation tools for handling uncertain pieces of information. The success of these models is strongly related to their capacity to represent and handle dependence relations. Some forms of Bayesian networks have been successfully applied in many classification tasks. In particular, naive Bayes classifiers have been used for intrusion detection and alerts correlation. This paper analyses the advantage of adding expert knowledge to probabilistic classifiers in the context of intrusion detection and alerts correlation. As examples of probabilistic classifiers, we will consider the well-known Naive Bayes, Tree Augmented Naïve Bayes (TAN), Hidden Naive Bayes (HNB) and decision tree classifiers. Our approach can be applied for any classifier where the outcome is a probability distribution over a set of classes (or decisions). In particular, we study how additional expert knowledge such as “it is expected that 80 % of traffic will be normal” can be integrated in classification tasks. Our aim is to revise probabilistic classifiers’ outputs in order to fit expert knowledge. Experimental results show that our approach improves existing results on different benchmarks from intrusion detection and alert correlation areas.