基于免疫克隆特征选择和欠采样集成的垃圾网页检测

为解决垃圾网页检测过程中的“维数灾难”和不平衡分类问题,提出一种基于免疫克隆特征选择和欠采样（US）集成的二元分类器算法。首先,使用欠采样技术将训练样本集大类抽样成多个与小类样本数相近的样本集,再将其分别与小类样本合并构成多个平衡的子训练样本集;然后,设计一种免疫克隆算法遴选出多个最优的特征子集;基于最优特征子集对平衡的子样本集进行投影操作,生成平衡数据集的多个视图;最后,用随机森林（RF）分类器对测试样本进行分类,采用简单投票法确定测试样本的最终类别。在WEBSPAM UK-2006数据集上的实验结果表明,该集成分类器算法应用于垃圾网页检测：与随机森林算法及其Bagging和AdaBoost集成分类器算法相比,准确率、F1测度、AUC等指标均提高11%以上;与其他最优的研究结果相比,该集成分类器算法在F1测度上提高2%,在AUC上达到最优。     

基于随机森林和欠采样集成的垃圾网页检测

为解决垃圾网页检测过程中的不平衡分类和"维数灾难"问题,提出一种基于随机森林(RF)和欠采样集成的二元分类器算法。首先使用欠采样技术将训练样本集大类抽样成多个子样本集,再将其分别与小类样本集合并构成多个平衡的子训练样本集;然后基于各个子训练样本集训练出多个随机森林分类器;最后用多个随机森林分类器对测试样本集进行分类,采用投票法确定测试样本的最终所属类别。在WEBSPAM UK-2006数据集上的实验表明,该集成分类器算法应用于垃圾网页检测比随机森林算法及其Bagging和Adaboost集成分类器算法效果更好,准确率、F1测度、ROC曲线下面积(AUC)等指标提高至少14%,13%和11%。与Web spam challenge 2007 优胜团队的竞赛结果相比,该集成分类器算法在F1测度上提高至少1%,在AUC上达到最优结果。     

三种用于垃圾网页检测的随机欠采样集成分类器

针对垃圾网页检测过程中轻微的不平衡分类问题,提出三种随机欠采样集成分类器算法,分别为一次不放回随机欠采样（RUS-once）、多次不放回随机欠采样（RUS-multiple）和有放回随机欠采样（RUS-replacement）算法。首先使用其中一种随机欠采样技术将训练样本集转换成平衡样本集,然后对每个平衡样本集使用分类回归树（CART）分类器算法进行分类,最后采用简单投票法构建集成分类器对测试样本进行分类。实验表明,三种随机欠采样集成分类器均取得了良好的分类效果,其中RUS-multiple和RUS-replacement比RUS-once的分类效果更好。与CART及其Bagging和Adaboost集成分类器相比,在WEBSPAM UK-2006数据集上,RUS-multiple和RUS-replacement方法的AUC指标值提高了10%左右,在WEBSPAM UK-2007数据集上,提高了25%左右;与其他最优研究结果相比,RUS-multiple和RUS-replacement方法在AUC指标上能达到最优分类结果。     

结合欠抽样与集成的软件缺陷预测

软件缺陷预测是提高测试效率、保证软件可靠性的重要途径。为了提高软件缺陷预测的准确率,提出一种结合欠抽样与决策树分类器集成的软件缺陷预测模型。考虑到软件缺陷数据的类不平衡特性,首先,通过数据的不平衡率确定抽样度,执行欠抽样实现数据的重新平衡;然后,采用Bagging随机抽样原理训练若干个决策树子分类器;最后,按照少数服从多数的原则生成预测模型。使用公开的NASA软件缺陷预测数据集进行了仿真实验。实验结果表明,与3种基准方法对比,所提模型在保证预报率的前提下,误报率(PF)降低了10%以上,综合评价指标均有显著提升。该模型的缺陷预测误报率较低,而且具有较高的预测准确率与稳定性。     

Evolutionary under-sampling based bagging ensemble method for imbalanced data classification

In the class imbalanced learning scenario, traditional machine learning algorithms focusing on optimizing the overall accuracy tend to achieve poor classification performance especially for the minority class in which we are most interested. To solve this problem, many effective approaches have been proposed. Among them, the bagging ensemble methods with integration of the under-sampling techniques have demonstrated better performance than some other ones including the bagging ensemble methods integrated with the over-sampling techniques, the cost-sensitive methods, etc. Although these under-sampling techniques promote the diversity among the generated base classifiers with the help of random partition or sampling for the majority class, they do not take any measure to ensure the individual classification performance, consequently affecting the achievability of better ensemble performance. On the other hand, evolutionary under-sampling EUS as a novel undersampling technique has been successfully applied in searching for the best majority class subset for training a good-performance nearest neighbor classifier. Inspired by EUS, in this paper, we try to introduce it into the under-sampling bagging framework and propose an EUS based bagging ensemble method EUS-Bag by designing a new fitness function considering three factors to make EUS better suited to the framework. With our fitness function, EUS-Bag could generate a set of accurate and diverse base classifiers. To verify the effectiveness of EUS-Bag, we conduct a series of comparison experiments on 22 two-class imbalanced classification problems. Experimental results measured using recall, geometric mean and AUC all demonstrate its superior performance.     

A novel hybrid credit scoring model based on ensemble feature selection and multilayer ensemble classification

Credit scoring focuses on the development of empirical models to support the financial decision‐making processes of financial institutions and credit industries. It makes use of applicants' historical data and statistical or machine learning techniques to assess the risk associated with an applicant. However, the historical data may consist of redundant and noisy features that affect the performance of credit scoring models. The main focus of this paper is to develop a hybrid model, combining feature selection and a multilayer ensemble classifier framework, to improve the predictive performance of credit scoring. The proposed hybrid credit scoring model is modeled in three phases. The initial phase constitutes preprocessing and assigns ranks and weights to classifiers. In the next phase, the ensemble feature selection approach is applied to the preprocessed dataset. Finally, in the last phase, the dataset with the selected features is used in a multilayer ensemble classifier framework. In addition, a classifier placement algorithm based on the Choquet integral value is designed, as the classifier placement affects the predictive performance of the ensemble framework. The proposed hybrid credit scoring model is validated on real‐world credit scoring datasets, namely, Australian, Japanese, German‐categorical, and German‐numerical datasets.     

Structural diversity for decision tree ensemble learning

Decision trees are a kind of off-the-shelf predictive models, and they have been successfully used as the base learners in ensemble learning. To construct a strong classifier ensemble, the individual classifiers should be accurate and diverse. However, diversity measure remains a mystery although there were many attempts. We conjecture that a deficiency of previous diversity measures lies in the fact that they consider only behavioral diversity, i.e., how the classifiers behave when making predictions, neglecting the fact that classifiers may be potentially different even when they make the same predictions. Based on this recognition, in this paper, we advocate to consider structural diversity in addition to behavioral diversity, and propose the TMD (tree matching diversity) measure for decision trees. To investigate the usefulness of TMD, we empirically evaluate performances of selective ensemble approaches with decision forests by incorporating different diversity measures. Our results validate that by considering structural and behavioral diversities together, stronger ensembles can be constructed. This may raise a new direction to design better diversity measures and ensemble methods.     

基于动态集成决策树的多类别时间序列分类模型

针对以往时间序列分类技术忽略了数据间自相关性对算法影响的不足,通过对传统决策树算法进行扩展,提出了序列熵和序列对信息增益的概念,并以此构建针对时间序列的决策树（Time Series Decision Tree,简称TSDT）。在此基础上,以TSDT为基分类器,通过动态分类器集成技术,提出了时间序列动态集成分类算法（En-TSDT）。在UCR数据集上的实验表明,与目前应用最广泛的1NN-DTW分类器相比,En-TSDT克服了时间序列数据的自相关性对分类算法的影响,具有更好的分类性能和鲁棒性。     

Improved response modeling based on clustering, under-sampling, and ensemble

The purpose of response modeling for direct marketing is to identify those customers who are likely to purchase a campaigned product, based upon customers’ behavioral history and other information available. Contrary to mass marketing strategy, well-developed response models used for targeting specific customers can contribute profits to firms by not only increasing revenues, but also lowering marketing costs. Endemic in customer data used for response modeling is a class imbalance problem: the proportion of respondents is small relative to non-respondents. In this paper, we propose a novel data balancing method based on clustering, under-sampling, and ensemble to deal with the class imbalance problem, and thus improve response models. Using publicly available response modeling data sets, we compared the proposed method with other data balancing methods in terms of prediction accuracy and profitability. To investigate the usability of the proposed algorithm, we also employed various prediction algorithms when building the response models. Based on the response rate and profit analysis, we found that our proposed method (1) improved the response model by increasing response rate as well as reducing performance variation, and (2) increased total profit by significantly boosting revenue.     

A comparison of decision tree ensemble creation techniques

We experimentally evaluate bagging and seven other randomization-based approaches to creating an ensemble of decision tree classifiers. Statistical tests were performed on experimental results from 57 publicly available data sets. When cross-validation comparisons were tested for statistical significance, the best method was statistically more accurate than bagging on only eight of the 57 data sets. Alternatively, examining the average ranks of the algorithms across the group of data sets, we find that boosting, random forests, and randomized trees are statistically significantly better than bagging. Because our results suggest that using an appropriate ensemble size is important, we introduce an algorithm that decides when a sufficient number of classifiers has been created for an ensemble. Our algorithm uses the out-of-bag error estimate, and is shown to result in an accurate ensemble for those methods that incorporate bagging into the construction of the ensemble     

Research on decision tree induction from self-map space based on web

This paper proposes an improved decision tree method for web information retrieval with self-map attributes. Our self-map tree has a value of self-map attribute in its internal node, and information based on dissimilarity between a pair of map sequences. Our method selects self-map which exists between data by exhaustive search based on relation and attribute information. Experimental results confirm that our improved method constructs comprehensive and accurate decision tree. Moreover, an example shows that our self-map decision tree is promising for data mining and knowledge discovery.     

欺诈网页检测中基于遗传算法的特征优选

针对网页欺诈检测中特征的高维、冗余问题,提出一个基于信息增益和遗传算法的改进特征选择算法（IFS-BIGGA）。首先,通过信息增益（IG）给出特征重要性排序,设定动态阈值减少冗余特征;其次,改进遗传算法（GA）中染色体编码函数和选择算子,并结合随机森林（RF）的受试者工作特征曲线面积（AUC）作为适应度函数,选择高辨识度特征;最后,增加实验迭代次数避免算法随机性,产生最佳最小的特征集合（OMFS）。实验验证表明,应用IFS-BIGGA生成的OMFS与高维特征集合相比,尽管RF下的AUC减小了2%,但是真阳性率（TPR）提高了21%,并且特征维度减少了92%;同时多个常用分类器的平均检测时间减少了83%;另外,IFS-BIGGA的F1值相比传统的遗传算法（TGA）和帝国主义竞争算法（ICA）分别提高了4.2%和3.5%。实验结果表明,IFS-BIGGA可以进行高效特征降维,在实际的网页检测工程中,有效减少计算代价,提高检测效率。     

Random feature weights for decision tree ensemble construction

This paper proposes a method for constructing ensembles of decision trees, random feature weights (RFW). The method is similar to Random Forest, they are methods that introduce randomness in the construction method of the decision trees. In Random Forest only a random subset of attributes are considered for each node, but RFW considers all of them. The source of randomness is a weight associated with each attribute. All the nodes in a tree use the same set of random weights but different from the set of weights in other trees. So, the importance given to the attributes will be different in each tree and that will differentiate their construction. The method is compared to Bagging, Random Forest, Random-Subspaces, AdaBoost and MultiBoost, obtaining favourable results for the proposed method, especially when using noisy data sets. RFW can be combined with these methods. Generally, the combination of RFW with other method produces better results than the combined methods. Kappa-error diagrams and Kappa-error movement diagrams are used to analyse the relationship between the accuracies of the base classifiers and their diversity.     

基于贝叶斯和决策树的入侵检测方法

针对目前基于贝叶斯或决策树的入侵检测方法存在检测率低、误检率高的问题,提出了一种基于贝叶斯和决策树的入侵检测方法。该检测方法首先采用基于特征相似度的朴素贝叶斯方法对训练集中的样本进行分类,更新每个样本的类值;然后对训练集中的样本再次使用朴素贝叶斯方法进行分类,对存在误分类样本的类采用决策树的信息增益来确定属性划分子类,再对子类进行分类和划分操作;最后建立贝叶斯和决策树的混合模型进行入侵检测。实验结果表明,与单独使用贝叶斯或者决策树的检测方法相比,该检测方法具有较高的检测率。     

一种新的决策树选择性集成学习方法

个体学习器的差异度是集成学习中的关键因素。流行的集成学习算法如Bagging通过重取样技术产生个体学习器的差异度。选择性集成从集成学习算法产生的个体学习器中选择一部分来集成,结果表明比原集成更好。但如何选择学习器是个难题。使用Q统计量度量两个学习器的差异度,提出一种新的决策树选择性集成学习方法。与C4.5,Bagging方法相比,表现出很好的效果。     

垃圾邮件处理中LDA特征选择方法

垃圾邮件处理是一项长期研究课题,越来越多的文本分类技术被移植到垃圾邮件处理应用当中。LDA（Latent Dirichlet Allocation）等topic模型在自动摘要、信息获取和其他离散数据应用中受到越来越多的关注。将LDA模型作为一种特征选择方法,引入垃圾邮件处理应用中。将LDA特征选择方法与质心＋KNN分类器结合,得到简单的测试用垃圾邮件过滤器。初步实验结果表明,基于LDA的特征选择方法优于通常的IG、MI特征选择方法;测试过滤器的过滤性能与其他过滤器相当。     

Visual concept detection of web images based on group sparse ensemble learning

Due to the huge intra-class variations for visual concept detection, it is necessary for concept learning to collect large scale training data to cover a wide variety of samples as much as possible. But it presents great challenges on both how to collect and how to train the large scale data. In this paper, we propose a novel web image sampling approach and a novel group sparse ensemble learning approach to tackle these two challenging problems respectively. For data collection, in order to alleviate manual labeling efforts, we propose a web image sampling approach based on dictionary coherence to select coherent positive samples from web images. We propose to measure the coherence in terms of how dictionary atoms are shared because shared atoms represent common features with regard to a given concept and are robust to occlusion and corruption. For efficient training of large scale data, in order to exploit the hidden group structures of data, we propose a novel group sparse ensemble learning approach based on Automatic Group Sparse Coding (AutoGSC). After AutoGSC, we present an algorithm to use the reconstruction errors of data instances to calculate the ensemble gating function for ensemble construction and fusion. Experiments show that our proposed methods can achieve promising results and outperforms existing approaches.     

伪标签置信选择的半监督集成学习视频语义检测

在视频语义检测中，有标记样本不足会严重影响检测的性能，而且伪标签样本中的噪声也会导致集成学习基分类器性能提升不足。为此，提出一种伪标签置信选择的半监督集成学习算法。首先，在三个不同的特征空间上训练出三个基分类器，得到基分类器的标签矢量；然后，引入加权融合样本所属某个类别的最大概率与次大概率的误差和样本所属某个类别的最大概率与样本所属其他各类别的平均概率的误差，作为基分类器的标签置信度，并融合标签矢量和标签置信度得到样本的伪标签和集成置信度；接着，选择集成置信度高的样本加入到有标签的样本集，迭代训练基分类器；最后，采用训练好的基分类器集成协作检测视频语义概念。该算法在实验数据集UCF11上的平均准确率到达了83.48%，与Co-KNN-SVM算法相比，平均准确率提高了3.48个百分点。该算法选择的伪标签能体现样本所属类别与其他类别的总体差异性，又能体现所属类别的唯一性，可减少利用伪标签样本的风险，有效提高视频语义概念检测的准确率。     

基于堆稀疏自编码的二叉树集成入侵检测方法

为了解决大规模入侵数据的分类问题,提出了堆稀疏自编码的lightGBM（light gridient boosting model）二叉树算法。首先将类别标签分为五类,构造成二叉树结构;然后通过上采样方法解决数据分布的不平衡问题,以上处理可以将大规模的数据分解开来以便之后分开训练;再采用稀疏自编码器网络进行特征降维,采用该种降维方法可以保证在原始数据中抽取出更深层特征的基础上节省降维时间;最后通过lightGBM集成算法进行分类,而采用lightGBM模型相比其他模型可以在保证分类性能的情况下节省训练时间。实验利用NSL-KDD数据集测量了所提方法的准确率、精确率、召回率,并且综合评价指标◢F▼◣▽1在五类分类上平均分别达到了87.42%、98.20%、91.31%,优于对比算法,且明显节省了运算时间。     

基于决策树与朴素贝叶斯分类的入侵检测模型

入侵检测要求系统能够快速准确地找出网络中的入侵行为,因此对检测算法的效率有较高的要求。针对入侵检测系统效率和准确率偏低,系统的误报率和漏报率偏高的问题,在充分分析C4.5算法和朴素贝叶斯(NB)算法后,提出一种二者相结合的H-C4.5-NB入侵检测模型。该模型以概率的形式来描述决策类别的分布,并由C4.5和NB概率加权和的形式给出最终的决策结果,最后使用KDD 99数据集测试模型性能。实验结果表明,与传统的C4.5、NB和NBTree方法相比,在H-C4.5-NB中对拒绝服务(DoS)攻击的分类准确率提高了约9%,对U2R和R2L攻击的准确率提高约20%~30%。