使用PCA建立基于规则的组合分类器

提出了一种使用基于规则的基分类器建立组合分类器的新方法PCARules。尽管新方法也采用基分类器预测的加权投票来决定待分类样本的类,但是为基分类器创建训练数据集的方法与bagging和boosting完全不同。该方法不是通过抽样为基分类器创建数据集,而是随机地将特征划分成K个子集,使用PCA得到每个子集的主成分,形成新的特征空间,并将所有训练数据映射到新的特征空间作为基分类器的训练集。在UCI机器学习库的30个随机选取的数据集上的实验表明:算法不仅能够显著提高基于规则的分类方法的分类性能,而且与bagging和boosting等传统组合方法相比,在大部分数据集上都具有更高的分类准确率。     

基于k-means聚类的神经网络分类器集成方法研究

针对差异性是集成学习的必要条件,研究了基于k-means聚类技术提高神经网络分类器集成差异性的方法。通过训练集并使用神经网络分类器学习算法训练许多分类器模型,在验证集中利用每个分类器的分类结果作为聚类的数据对象;然后应用k-means聚类方法对这些数据聚类,在聚类结果的每个簇中选择一个分类器代表模型,以此构成集成学习的成员;最后应用投票方法实验研究了这种提高集成学习差异性方法的性能,并与常用的集成学习方法bagging、adaboost进行了比较。     

结合主动学习与置信度投票的集成自训练方法

基于集成学习的自训练算法是一种半监督算法,不少学者通过集成分类器类别投票或平均置信度的方法选择可靠样本。基于置信度的投票策略倾向选择置信度高的样本或置信度低但投票却一致的样本进行标记,后者这种情形可能会误标记靠近决策边界的样本,而采用异构集成分类器也可能会导致各基分类器对高置信度样本的类别标记不同,从而无法将其有效加入到有标记样本集。提出了结合主动学习与置信度投票策略的集成自训练算法用来解决上述问题。该算法合理调整了投票策略,选择置信度高且投票一致的无标记样本加以标注,同时利用主动学习对投票不一致而置信度较低的样本进行人工标注,以弥补集成自训练学习只关注置信度高的样本,而忽略了置信度低的样本的有用信息的缺陷。在UCI数据集上的对比实验验证了该算法的有效性。     

基于近似约简与最优采样的集成剪枝

集成学习被广泛用于提高分类精度, 近年来的研究表明, 通过多模态扰乱策略来构建集成分类器可以进一步提高分类性能. 本文提出了一种基于近似约简与最优采样的集成剪枝算法(EPA_AO). 在EPA_AO中, 我们设计了一种多模态扰乱策略来构建不同的个体分类器. 该扰乱策略可以同时扰乱属性空间和训练集, 从而增加了个体分类器的多样性. 我们利用证据KNN (K-近邻)算法来训练个体分类器, 并在多个UCI数据集上比较了EPA_AO与现有同类型算法的性能. 实验结果表明, EPA_AO是一种有效的集成学习方法.     

基于仿生模式识别的用户概貌攻击集成检测方法

针对有监督方法在检测用户概貌攻击时准确率不高的问题,通过引入仿生模式识别理论和集成学习技术提出一种集成检测方法.首先,通过计算被覆盖直线段与最近邻真实概貌的距离,提出一种自适应神经元超球半径计算算法,为每个神经元确定合适的超球半径;然后利用该超球半径对现有的一个3层神经网络进行重新设计,使其能够对攻击概貌样本进行更合理覆盖,以提高分类性能;最后,提出一种用户概貌攻击集成检测框架,通过组合多种攻击类型,利用提出的基训练集生成算法建立不同的基训练集,以训练新设计的神经网络生成基分类器,基于信息论得分(information theoretic score,ITS)算法提出一种选择性集成检测算法对基分类器进行筛选,并采用多数投票策略融合基分类器的输出结果.在MovieLens和Netflix两个不同规模的真实数据集上的实验结果表明,所提出的集成检测方法能够在保持较高召回率的条件下有效提高用户概貌攻击检测的准确率.     

基于类别信息的分类器集成方法Cagging

提出一种基于类别信息的分类器集成方法Cagging.基于类别信息重复选择样本生成基本分类器的训练集,增强了基本分类器之间的差异性;利用基本分类器对不同模式类的分类能力为每个基本分类器设置一组权重.使用权重对各分类器输出结果进行加权决策,较好地利用了各个基本分类器之间的差异性.在人脸图像库ORL上的实验验证了Cagging的有效性.此外,Cagging方法的基本分类器生成方式适合于通过增量学习生成集成分类器,扩展Cagging设计了基于增量学习的分类器集成方法Cagging-Ⅰ,实验验证了它的有效性.     

基于拆分集成的不均衡数据分类算法

为改进SVM对不均衡数据的分类性能,提出一种基于拆分集成的不均衡数据分类算法,该算法对多数类样本依据类别之间的比例通过聚类划分为多个子集,各子集分别与少数类合并成多个训练子集,通过对各训练子集进行学习获得多个分类器,利用WE集成分类器方法对多个分类器进行集成,获得最终分类器,以此改进在不均衡数据下的分类性能.在UCI数据集上的实验结果表明,该算法的有效性,特别是对少数类样本的分类性能.     

多分类器选择集成方法

针对目前人们对分类性能的高要求和多分类器集成实现的复杂性,从基分类器准确率和基分类器间差异性两方面出发,提出了一种新的多分类器选择集成算法。该算法首先从生成的基分类器中选择出分类准确率较高的,然后利用分类器差异性度量来选择差异性大的高性能基分类器,在分类器集成之前先对分类器集进行选择获得新的分类器集。在UCI数据库上的实验结果证明,该方法优于bagging方法,取得了很好的分类识别效果。     

基于选择性集成的最大化软间隔算法?

当前,boosting 集成学习算法研究主要集中于最大化弱学习器凸组合的间隔或软间隔,该凸组合几乎使用了生成的所有弱学习器,然而这些弱学习器间存在大量的相关性和冗余,增加了训练和分类过程的时空复杂度.针对这一问题,在LPBoost基础上提出了一种选择性boosting集成学习算法,称为SelectedBoost.在每次迭代生成新的弱学习器以后,通过计算新生成的弱学习器与已有弱学习器的相关度和差异度,并结合当前集成的强学习器的准确率来判断是否选择该弱学习器.另外,当前的一系列boosting算法(如AdaBoost,LPBoost,ERLPBoost等),本质上是基于已生成的1个或者多个弱学习器来更新样本权重,但与弱学习器相比,强学习器更能代表当前的决策面.因此, SelectedBoost 通过在带约束的间隔最大化问题中引入更加严格的强学习器边界约束条件,使得该算法不仅参考弱学习器边界,同时还参考已生成的强学习器来更新样本权重,进而提高算法的收敛速度.最后,与其他有代表性的集成学习算法进行实验比较,结果表明,该方法在收敛率、分类准确性以及泛化能力等方面均具有比较明显的优势.     

基于选择性集成的最大化软间隔算法

当前,boosting集成学习算法研究主要集中于最大化弱学习器凸组合的间隔或软间隔,该凸组合几乎使用了生成的所有弱学习器,然而这些弱学习器间存在大量的相关性和冗余,增加了训练和分类过程的时空复杂度.针对这一问题,在LPBoost基础上提出了一种选择性boosting集成学习算法,称为SelectedBoost.在每次迭代生成新的弱学习器以后,通过计算新生成的弱学习器与已有弱学习器的相关度和差异度,并结合当前集成的强学习器的准确率来判断是否选择该弱学习器.另外,当前的一系列boosting算法(如AdaBoost,LPBoost,ERLPBoost等),本质上是基于已生成的1个或者多个弱学习器来更新样本权重,但与弱学习器相比,强学习器更能代表当前的决策面.因此,SelectedBoost通过在带约束的间隔最大化问题中引入更加严格的强学习器边界约束条件,使得该算法不仅参考弱学习器边界,同时还参考已生成的强学习器来更新样本权重,进而提高算法的收敛速度.最后,与其他有代表性的集成学习算法进行实验比较,结果表明,该方法在收敛率、分类准确性以及泛化能力等方面均具有比较明显的优势.     

Ensembling local learners through multimodal perturbation.

Ensemble learning algorithms train multiple component learners and then combine their predictions. In order to generate a strong ensemble, the component learners should be with high accuracy as well as high diversity. A popularly used scheme in generating accurate but diverse component learners is to perturb the training data with resampling methods, such as the bootstrap sampling used in bagging. However, such a scheme is not very effective on local learners such as nearest-neighbor classifiers because a slight change in training data can hardly result in local learners with big differences. In this paper, a new ensemble algorithm named Filtered Attribute Subspace based Bagging with Injected Randomness (FASBIR) is proposed for building ensembles of local learners, which utilizes multimodal perturbation to help generate accurate but diverse component learners. In detail, FASBIR employs the perturbation on the training data with bootstrap sampling, the perturbation on the input attributes with attribute filtering and attribute subspace selection, and the perturbation on the learning parameters with randomly configured distance metrics. A large empirical study shows that FASBIR is effective in building ensembles of nearest-neighbor classifiers, whose performance is better than that of many other ensemble algorithms.     

Bagging with Adaptive Costs

Ensemble methods have proven to be highly effective in improving the performance of base learners under most circumstances. In this paper, we propose a new algorithm that combines the merits of some existing techniques, namely, bagging, arcing, and stacking. The basic structure of the algorithm resembles bagging. However, the misclassification cost of each training point is repeatedly adjusted according to its observed out-of-bag vote margin. In this way, the method gains the advantage of arcing-building the classifier the ensemble needs - without fixating on potentially noisy points. Computational experiments show that this algorithm performs consistently better than bagging and arcing with linear and nonlinear base classifiers. In view of the characteristics of bacing, a hybrid ensemble learning strategy, which combines bagging and different versions of bacing, is proposed and studied empirically.     

基于动态权重的Adaboost算法研究 *

针对Adaboost算法只能静态分配基分类器权重,不能自适应地对每个测试样本动态调整权重的问题,提出了一种基于动态权重的Adaboost算法。算法通过对训练样本集合进行聚类,并分析每个基分类器和每个类簇的适应性,进而为每个基分类器在不同类簇上设置不同权重,最终根据测试样本与类簇之间的相似性来计算基分类器在测试样本上的权重。在UCI数据集上的实验结果表明本文提出算法有效利用了测试样本之间的差异性,得到了比Adaboost算法更好的效果。     

集成学习算法的研究与应用

集成学习算法的思想就是集成多个学习器,并组合它们的预测结果,以形成最终的结论。典型的学习模型组合方法有投票法,专家混合方法,堆叠泛化法与级联法,但这些方法的性能都有待进一步提高。提出了一种新颖的集成学习算法--增强的集成学习算法（ReinforcedEnsemble）。ReinforcedEnsemble集成算法由两大部分组成：ReinforcedEnsemble特征提取算法与ReinforcedEnsemble基分类器。通过实验,将ReinforcedEnsemble算法与其他集成学习算法进行了性能比较。实验结果表明,所提出的算法在多项指标上均达到最优。     

Building bagging on critical instances

The ensemble method is a powerful data mining paradigm, which builds a classification model by integrating multiple diversified component learners. Bagging is one of the most successful ensemble methods. It is made of bootstrap-inspired classifiers and uses these classifiers to get an aggregated classifier. However, in bagging, bootstrapped training sets become more and more similar as redundancy is increasing. Besides redundancy, any training set is usually subject to noise. Moreover, the training set might be imbalanced. Thus, each training instance has a different impact on the learning process. This paper explores some properties of the ensemble margin and its use in improving the performance of bagging. We introduce a new approach to measure the importance of training data in learning, based on the margin theory. Then, a new bagging method concentrating on critical instances is proposed. This method is more accurate than bagging and more robust than boosting. Compared to bagging, it reduces the bias while generally keeping the same variance. Our findings suggest that (a) examples with low margins tend to be more critical for the classifier performance; (b) examples with higher margins tend to be more redundant; (c) misclassified examples with high margins tend to be noisy examples. Our experimental results on 15 various data sets show that the generalization error of bagging can be reduced up to 2.5% and its resilience to noise strengthened by iteratively removing both typical and noisy training instances, reducing the training set size by up to 75%.     

Boosting neural networks

Boosting is a general method for improving the performance of learning algorithms. A recently proposed boosting algorithm, AdaBoost, has been applied with great success to several benchmark machine learning problems using mainly decision trees as base classifiers. In this article we investigate whether AdaBoost also works as well with neural networks, and we discuss the advantages and drawbacks of different versions of the AdaBoost algorithm. In particular, we compare training methods based on sampling the training set and weighting the cost function. The results suggest that random resampling of the training data is not the main explanation of the success of the improvements brought by AdaBoost. This is in contrast to bagging, which directly aims at reducing variance and for which random resampling is essential to obtain the reduction in generalization error. Our system achieves about 1.4% error on a data set of on-line handwritten digits from more than 200 writers. A boosted multilayer network achieved 1.5% error on the UCI letters and 8.1% error on the UCI satellite data set, which is significantly better than boosted decision trees.     

Cluster‐based ensemble of classifiers

This paper presents cluster‐based ensemble classifier – an approach toward generating ensemble of classifiers using multiple clusters within classified data. Clustering is incorporated to partition data set into multiple clusters of highly correlated data that are difficult to separate otherwise and different base classifiers are used to learn class boundaries within the clusters. As the different base classifiers engage on different difficult‐to‐classify subsets of the data, the learning of the base classifiers is more focussed and accurate. A selection rather than fusion approach achieves the final verdict on patterns of unknown classes. The impact of clustering on the learning parameters and accuracy of a number of learning algorithms including neural network, support vector machine, decision tree and k‐NN classifier is investigated. A number of benchmark data sets from the UCI machine learning repository were used to evaluate the cluster‐based ensemble classifier and the experimental results demonstrate its superiority over bagging and boosting.     

SUCE:基于聚类集成的半监督二分类方法

半监督学习和集成学习是目前机器学习领域中的重要方法。半监督学习利用未标记样本,而集成学习综合多个弱学习器,以提高分类精度。针对名词型数据,本文提出一种融合聚类和集成学习的半监督分类方法SUCE。在不同的参数设置下,采用多个聚类算法生成大量的弱学习器;利用已有的类标签信息,对弱学习器进行评价和选择;通过集成弱学习器对测试集进行预分类,并将置信度高的样本放入训练集;利用扩展的训练集,使用ID3、Nave Bayes、 kNN、C4.5、OneR、Logistic等基础算法对其他样本进行分类。在UCI数据集上的实验结果表明,当训练样本较少时,本方法能稳定提高多数基础算法的准确性。     

基于成对差异性度量的选择性集成方法

有效地产生泛化能力强、差异大的个体学习器,是集成学习算法的关键。为了提高学习器的差异性和精度,文中提出一种基于成对差异性度量的选择性集成方法。同时研究一种改进方法,进一步提高方法的运算速度,且支持并行计算。最后通过使用BP神经网络作为基学习器,在UCI数据集上进行实验,并与Bagging、基于遗传算法的选择性集成(GASEN)算法进行比较。实验结果表明,该改进算法在性能上与GASEN算法相近的前提下,训练速度得到大幅提高。