选择策略的集成学习方法研究

差异性是提高分类器集成泛化性能的重要因素。采用熵差异性度量及数据子集法训练基分类器,研究了爬山选择、集成前序选择、集成后序选择以及聚类选择策略选取个体模型的集成学习。实验结果表明,由选择策略选取差异性较大的个体模型,其集成性能表现出较好的优势;从总体角度考虑,爬山选择策略的集成性能优于集成前序选择和集成后序选择的集成性能;另外,由聚类技术选取的集成模型,当集成正确率较稳定时,则模型间的差异性变化较小;簇数也对集成性能与集成模型间的差异性产生一定的影响。     

A novel ensemble learning approach to unsupervised record linkage

Record linkage is a process of identifying records that refer to the same real-world entity. Many existing approaches to record linkage apply supervised machine learning techniques to generate a classification model that classifies a pair of records as either match or non-match. The main requirement of such an approach is a labelled training dataset. In many real-world applications no labelled dataset is available hence manual labelling is required to create a sufficiently sized training dataset for a supervised machine learning algorithm. Semi-supervised machine learning techniques, such as self-learning or active learning, which require only a small manually labelled training dataset have been applied to record linkage. These techniques reduce the requirement on the manual labelling of the training dataset. However, they have yet to achieve a level of accuracy similar to that of supervised learning techniques. In this paper we propose a new approach to unsupervised record linkage based on a combination of ensemble learning and enhanced automatic self-learning. In the proposed approach an ensemble of automatic self-learning models is generated with different similarity measure schemes. In order to further improve the automatic self-learning process we incorporate field weighting into the automatic seed selection for each of the self-learning models. We propose an unsupervised diversity measure to ensure that there is high diversity among the selected self-learning models. Finally, we propose to use the contribution ratios of self-learning models to remove those with poor accuracy from the ensemble. We have evaluated our approach on 4 publicly available datasets which are commonly used in the record linkage community. Our experimental results show that our proposed approach has advantages over the state-of-the-art semi-supervised and unsupervised record linkage techniques. In 3 out of 4 datasets it also achieves comparable results to those of the supervised approaches.     

Multi-armed bandit heterogeneous ensemble learning for imbalanced data

One of the most widely used approaches to the class-imbalanced issue is ensemble learning. The base classifier is trained using an unbalanced training set in the conventional ensemble learning approach. We are unable to select the best suitable resampling method or base classifier for the training set, despite the fact that researchers have examined employing resampling strategies to balance the training set. A multi-armed bandit heterogeneous ensemble framework was developed as a solution to these issues. This framework employs the multi-armed bandit technique to pick the best base classifier and resampling techniques to build a heterogeneous ensemble model. To obtain training sets, we first employ the bagging technique. Then, we use the instances from the out-of-bag set as the validation set. In general, we consider the basic classifier combination with the highest validation set score to be the best model on the bagging subset and add it to the pool of model. The classification performance of the multi-armed bandit heterogeneous ensemble model is then assessed using 30 real-world imbalanced data sets that were gathered from UCI, KEEL, and HDDT. The experimental results demonstrate that, under the two assessment metrics of AUC and Kappa, the proposed heterogeneous ensemble model performs competitively with other nine state-of-the-art ensemble learning methods. At the same time, the findings of the experiment are confirmed by the statistical findings of the Friedman test and Holm's post-hoc test.     

Hybrid ensemble approach for classification

This paper presents a novel hybrid ensemble approach for classification in medical databases. The proposed approach is formulated to cluster extracted features from medical databases into soft clusters using unsupervised learning strategies and fuse the decisions using parallel data fusion techniques. The idea is to observe associations in the features and fuse the decisions made by learning algorithms to find the strong clusters which can make impact on overall classification accuracy. The novel techniques such as parallel neural-based strong clusters fusion and parallel neural network based data fusion are proposed that allow integration of various clustering algorithms for hybrid ensemble approach. The proposed approach has been implemented and evaluated on the benchmark databases such as Digital Database for Screening Mammograms, Wisconsin Breast Cancer, and Pima Indian Diabetics. A comparative performance analysis of the proposed approach with other existing approaches for knowledge extraction and classification is presented. The experimental results demonstrate the effectiveness of the proposed approach in terms of improved classification accuracy on benchmark medical databases.     

Convex ensemble learning with sparsity and diversity

Classifier ensemble has been broadly studied in two prevalent directions, i.e., to diversely generate classifier components, and to sparsely combine multiple classifiers. While most current approaches are emphasized on either sparsity or diversity only, we investigate classifier ensemble focused on both in this paper. We formulate the classifier ensemble problem with the sparsity and diversity learning in a general mathematical framework, which proves beneficial for grouping classifiers. In particular, derived from the error-ambiguity decomposition, we design a convex ensemble diversity measure. Consequently, accuracy loss, sparseness regularization, and diversity measure can be balanced and combined in a convex quadratic programming problem. We prove that the final convex optimization leads to a closed-form solution, making it very appealing for real ensemble learning problems. We compare our proposed novel method with other conventional ensemble methods such as Bagging, least squares combination, sparsity learning, and AdaBoost, extensively on a variety of UCI benchmark data sets and the Pascal Large Scale Learning Challenge 2008 webspam data. Experimental results confirm that our approach has very promising performance.     

A survey on ensemble learning

Despite significant successes achieved in knowledge discovery,traditional machine learning methods may fail to obtain satisfactory performances when dealing with complex data,such as imbalanced,high-dimensional,noisy data,etc.The reason behind is that it is difficult for these methods to capture multiple characteristics and underlying structure of data.In this context,it becomes an important topic in the data mining field that how to effectively construct an efficient knowledge discovery and mining model.Ensemble learning,as one research hot spot,aims to integrate data fusion,data modeling,and data mining into a unified framework.Specifically,ensemble learning firstly extracts a set of features with a variety of transformations.Based on these learned features,multiple learning algorithms are utilized to produce weak predictive results.Finally,ensemble learning fuses the informative knowledge from the above results obtained to achieve knowledge discovery and better predictive performance via voting schemes in an adaptive way.In this paper,we review the research progress of the mainstream approaches of ensemble learning and classify them based on different characteristics.In addition,we present challenges and possible research directions for each mainstream approach of ensemble learning,and we also give an extra introduction for the combination of ensemble learning with other machine learning hot spots such as deep learning,reinforcement learning,etc.     

A comparative assessment of ensemble learning for credit scoring

Both statistical techniques and Artificial Intelligence (AI) techniques have been explored for credit scoring, an important finance activity. Although there are no consistent conclusions on which ones are better, recent studies suggest combining multiple classifiers, i.e., ensemble learning, may have a better performance. In this study, we conduct a comparative assessment of the performance of three popular ensemble methods, i.e., Bagging, Boosting, and Stacking, based on four base learners, i.e., Logistic Regression Analysis (LRA), Decision Tree (DT), Artificial Neural Network (ANN) and Support Vector Machine (SVM). Experimental results reveal that the three ensemble methods can substantially improve individual base learners. In particular, Bagging performs better than Boosting across all credit datasets. Stacking and Bagging DT in our experiments, get the best performance in terms of average accuracy, type I error and type II error.     

A compressed sensing approach for efficient ensemble learning

This paper presents a method for improved ensemble learning, by treating the optimization of an ensemble of classifiers as a compressed sensing problem. Ensemble learning methods improve the performance of a learned predictor by integrating a weighted combination of multiple predictive models. Ideally, the number of models needed in the ensemble should be minimized, while optimizing the weights associated with each included model. We solve this problem by treating it as an example of the compressed sensing problem, in which a sparse solution must be reconstructed from an under-determined linear system. Compressed sensing techniques are then employed to find an ensemble which is both small and effective. An additional contribution of this paper, is to present a new performance evaluation method (a new pairwise diversity measurement) called the roulette-wheel kappa-error. This method takes into account the different weightings of the classifiers, and also reduces the total number of pairs of classifiers needed in the kappa-error diagram, by selecting pairs through a roulette-wheel selection method according to the weightings of the classifiers. This approach can greatly improve the clarity and informativeness of the kappa-error diagram, especially when the number of classifiers in the ensemble is large. We use 25 different public data sets to evaluate and compare the performance of compressed sensing ensembles using four different sparse reconstruction algorithms, combined with two different classifier learning algorithms and two different training data manipulation techniques. We also give the comparison experiments of our method against another five state-of-the-art pruning methods. These experiments show that our method produces comparable or better accuracy, while being significantly faster than the compared methods.     

Random forests-based extreme learning machine ensemble for multi-regime time series prediction

Accurate and timely predicting values of performance parameters are currently strongly needed for important complex equipment in engineering. In time series prediction, two problems are urgent to be solved. One problem is how to achieve the accuracy, stability and efficiency together, and the other is how to handle time series with multiple regimes. To solve these two problems, random forests-based extreme learning machine ensemble model and a novel multi-regime approach are proposed respectively, and these two approaches can be integrated to achieve better performance. First, the extreme learning machine (ELM) is used in the proposed model because of its efficiency. Then the regularized ELM and ensemble learning strategy are used to improve generalization performance and prediction accuracy. The bootstrap sampling technique is used to generate training sample sets for multiple base-level ELM models, and then the random forests (RF) model is used as the combiner to aggregate these ELM models to achieve more accurate and stable performance. Next, based on the specific properties of turbofan engine time series, a multi-regime approach is proposed to handle it. Regimes are first separated, then the proposed RF-based ELM ensemble model is used to learn models of all regimes, individually, and last, all the learned regime models are aggregated to predict performance parameter at the future timestamp. The proposed RF-based ELM ensemble model and multi-regime approaches are evaluated by using NN3 time series and NASA turbofan engine time series, and then the proposed model is applied to the exhaust gas temperature prediction of CFM engine. The results demonstrate that the proposed RF-based ELM ensemble model and multi-regime approach can be accurate, stable and efficient in predicting multi-regime time series, and it can be robust against overfitting.     

Evolutionary under-sampling based bagging ensemble method for imbalanced data classification

In the class imbalanced learning scenario, traditional machine learning algorithms focusing on optimizing the overall accuracy tend to achieve poor classification performance especially for the minority class in which we are most interested. To solve this problem, many effective approaches have been proposed. Among them, the bagging ensemble methods with integration of the under-sampling techniques have demonstrated better performance than some other ones including the bagging ensemble methods integrated with the over-sampling techniques, the cost-sensitive methods, etc. Although these under-sampling techniques promote the diversity among the generated base classifiers with the help of random partition or sampling for the majority class, they do not take any measure to ensure the individual classification performance, consequently affecting the achievability of better ensemble performance. On the other hand, evolutionary under-sampling EUS as a novel undersampling technique has been successfully applied in searching for the best majority class subset for training a good-performance nearest neighbor classifier. Inspired by EUS, in this paper, we try to introduce it into the under-sampling bagging framework and propose an EUS based bagging ensemble method EUS-Bag by designing a new fitness function considering three factors to make EUS better suited to the framework. With our fitness function, EUS-Bag could generate a set of accurate and diverse base classifiers. To verify the effectiveness of EUS-Bag, we conduct a series of comparison experiments on 22 two-class imbalanced classification problems. Experimental results measured using recall, geometric mean and AUC all demonstrate its superior performance.     

Feature-based ensemble learning for hyperspectral chemical plume detection

Feature-based ensemble learning, where weak hypotheses are learned within the associated feature subspaces constructed by repeated random feature selection, is described. The proposed ensemble approach is less affected by noisy features or outliers unique to the training set than the bagging and boosting algorithms due to the randomized selection of feature subsets from the entire training set. The individual weak hypotheses perform their own generalization processes, within the associated feature subspaces, independently of each other. This allows the proposed ensemble to provide improved performance on unseen data over other ensemble learning methods that randomly choose subsets of training samples in an input space. The weak hypotheses are combined through three different aggregating strategies: majority voting, weighted average and neural network-based aggregation. The proposed ensemble technique has been applied to hyperspectral chemical plume data and a performance comparison of the proposed and other existing ensemble methods is presented.     

极限学习机的分类器集成模型研究

将极限学习机算法与旋转森林算法相结合,提出了以ELM算法为基分类器并以旋转森林算法为框架的RF-ELM集成学习模型。在8个数据集上进行了3组预测实验,根据实验结果讨论了ELM算法中隐含层神经元个数对预测结果的影响以及单个ELM模型预测结果不稳定的缺陷;将RF-ELM模型与单ELM模型和基于Bagging算法集成的ELM模型相比较,由稳定性和预测精度的两组对比实验的实验结果表明,对ELM的集成学习可以有效地提高ELM模型的性能,且RF-ELM模型较其他两个模型具有更好的稳定性和更高的准确率,验证了RF-ELM是一种有效的ELM集成学习模型。     

Exploration of classification confidence in ensemble learning

Ensemble learning has attracted considerable attention owing to its good generalization performance. The main issues in constructing a powerful ensemble include training a set of diverse and accurate base classifiers, and effectively combining them. Ensemble margin, computed as the difference of the vote numbers received by the correct class and the another class received with the most votes, is widely used to explain the success of ensemble learning. This definition of the ensemble margin does not consider the classification confidence of base classifiers. In this work, we explore the influence of the classification confidence of the base classifiers in ensemble learning and obtain some interesting conclusions. First, we extend the definition of ensemble margin based on the classification confidence of the base classifiers. Then, an optimization objective is designed to compute the weights of the base classifiers by minimizing the margin induced classification loss. Several strategies are tried to utilize the classification confidences and the weights. It is observed that weighted voting based on classification confidence is better than simple voting if all the base classifiers are used. In addition, ensemble pruning can further improve the performance of a weighted voting ensemble. We also compare the proposed fusion technique with some classical algorithms. The experimental results also show the effectiveness of weighted voting with classification confidence.     

Coevolutionary learning of neural network ensemble for complex classification tasks

Ensemble approaches to classification have attracted a great deal of interest recently. This paper presents a novel method for designing the neural network ensemble using coevolutionary algorithm. The bootstrap resampling procedure is employed to obtain different training subsets that are used to estimate different component networks of the ensemble. Then the cooperative coevolutionary algorithm is developed to optimize the ensemble model via the divide-and-cooperative mechanism. All component networks are coevolved in parallel in the scheme of interacting co-adapted subpopulations. The fitness of an individual from a particular subpopulation is assessed by associating it with the representatives from other subpopulations. In order to promote the cooperation of all component networks, the proposed method considers both the accuracy and the diversity among the component networks that are evaluated using the multi-objective Pareto optimality measure. A hybrid output-combination method is designed to determine the final ensemble output. Experimental results illustrate that the proposed method is able to obtain neural network ensemble models with better classification accuracy in comparison with currently popular ensemble algorithms.     

集成学习中的多样性度量

在集成学习中, 基分类器之间的多样性对于解释多分类器系统的工作机理和构造有效的集成系统具有重要的作用, 但至今仍没有统一的度量多样性的方法. 首先总结介绍常用的多样性度量方法, 阐述每种方法评估多样性的角度和方式; 然后从对多样性新的解释和度量、多样性度量在选择性集成中的应用、多样性度量和集成学习精度的关系3 个方面探讨多样性度量的研究进展; 最后给出关于多样性度量进一步的研究方向.

     

Enhancing deep learning sentiment analysis with ensemble techniques in social applications

Deep learning techniques for Sentiment Analysis have become very popular. They provide automatic feature extraction and both richer representation capabilities and better performance than traditional feature based techniques (i.e., surface methods). Traditional surface approaches are based on complex manually extracted features, and this extraction process is a fundamental question in feature driven methods. These long-established approaches can yield strong baselines, and their predictive capabilities can be used in conjunction with the arising deep learning methods. In this paper we seek to improve the performance of deep learning techniques integrating them with traditional surface approaches based on manually extracted features. The contributions of this paper are sixfold. First, we develop a deep learning based sentiment classifier using a word embeddings model and a linear machine learning algorithm. This classifier serves as a baseline to compare to subsequent results. Second, we propose two ensemble techniques which aggregate our baseline classifier with other surface classifiers widely used in Sentiment Analysis. Third, we also propose two models for combining both surface and deep features to merge information from several sources. Fourth, we introduce a taxonomy for classifying the different models found in the literature, as well as the ones we propose. Fifth, we conduct several experiments to compare the performance of these models with the deep learning baseline. For this, we use seven public datasets that were extracted from the microblogging and movie reviews domain. Finally, as a result, a statistical study confirms that the performance of these proposed models surpasses that of our original baseline on F1-Score.     

Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms

Improving accuracies of machine learning algorithms is vital in designing high performance computer-aided diagnosis (CADx) systems. Researches have shown that a base classifier performance might be enhanced by ensemble classification strategies. In this study, we construct rotation forest (RF) ensemble classifiers of 30 machine learning algorithms to evaluate their classification performances using Parkinson's, diabetes and heart diseases from literature.While making experiments, first the feature dimension of three datasets is reduced using correlation based feature selection (CFS) algorithm. Second, classification performances of 30 machine learning algorithms are calculated for three datasets. Third, 30 classifier ensembles are constructed based on RF algorithm to assess performances of respective classifiers with the same disease data. All the experiments are carried out with leave-one-out validation strategy and the performances of the 60 algorithms are evaluated using three metrics; classification accuracy (ACC), kappa error (KE) and area under the receiver operating characteristic (ROC) curve (AUC).Base classifiers succeeded 72.15%, 77.52% and 84.43% average accuracies for diabetes, heart and Parkinson's datasets, respectively. As for RF classifier ensembles, they produced average accuracies of 74.47%, 80.49% and 87.13% for respective diseases.RF, a newly proposed classifier ensemble algorithm, might be used to improve accuracy of miscellaneous machine learning algorithms to design advanced CADx systems.     

基于集成学习的口令强度评估模型

针对现有的口令评估模型通用性差,没有一个可以对从简单口令到非常复杂口令都适用的评估模型的问题,设计了一种基于多模型的集成学习的口令评估模型。首先,使用真实的口令训练集训练多个现有的口令评估模型作为子模型;其次,将多个经过训练的子模型作为基学习器进行集成学习,采用偏弱项投票法的结合策略实现各个子模型的优势集成;最后,实现一个以高准确性为前提的通用口令评估模型。实验中使用网络泄露的真实用户口令数据集作为实验数据,实验结果表明,基于多模型集成学习模型针对不同复杂程度的口令进行口令强度评估,其评估结果准确率高、通用性强,所提模型在口令评估方面具有较好的适用性。     

Network traffic classification based on ensemble learning and co-training

Classification of network traffic is the essential step for many network researches. However,with the rapid evolution of Internet applications the effectiveness of the port-based or payload-based identifi-cation approaches has been greatly diminished in recent years. And many researchers begin to turn their attentions to an alternative machine learning based method. This paper presents a novel machine learning-based classification model,which combines ensemble learning paradigm with co-training tech-niques. Compared to previous approaches,most of which only employed single classifier,multiple clas-sifiers and semi-supervised learning are applied in our method and it mainly helps to overcome three shortcomings:limited flow accuracy rate,weak adaptability and huge demand of labeled training set. In this paper,statistical characteristics of IP flows are extracted from the packet level traces to establish the feature set,then the classification model is created and tested and the empirical results prove its feasibility and effectiveness.     

The active grading ensemble framework for learning visual quality inspection from multiple humans

When applying machine learning technology to real-world applications, such as visual quality inspection, several practical issues need to be taken care of. One problem is posed by the reality that usually there are multiple human operators doing the inspection, who will inevitably contradict each other for some of the products to be inspected. In this paper an architecture for learning visual quality inspection is proposed which can be trained by multiple human operators, based on trained ensembles of classifiers. Most of the applicable ensemble techniques have however difficulties learning in these circumstances. In order to effectively train the system a novel ensemble framework is proposed as an enhancement of the grading ensemble technique—called active grading. The active grading algorithms are evaluated on data obtained from a real-world industrial system for visual quality inspection of the printing of labels on CDs, which was labelled independently by four different human operators and their supervisor, and compared to the standard grading algorithm and a range of other ensemble (classifier fusion) techniques.