Online error correcting output codes

This article proposes a general extension of the error correcting output codes framework to the online learning scenario. As a result, the final classifier handles the addition of new classes independently of the base classifier used. In particular, this extension supports the use of both online example incremental and batch classifiers as base learners. The extension of the traditional problem independent codings one-versus-all and one-versus-one is introduced. Furthermore, two new codings are proposed, unbalanced online ECOC and a problem dependent online ECOC. This last online coding technique takes advantage of the problem data for minimizing the number of dichotomizers used in the ECOC framework while preserving a high accuracy. These techniques are validated on an online setting of 11 data sets from UCI database and applied to two real machine vision applications: traffic sign recognition and face recognition. As a result, the online ECOC techniques proposed provide a feasible and robust way for handling new classes using any base classifier.     

Face verification via error correcting output codes

We propose a novel approach to face verification based on the Error Correcting Output Coding (ECOC) classifier design concept. In the training phase, the client set is repeatedly divided into two ECOC specified sub-sets (super-classes) to train a set of binary classifiers. The output of the classifiers defines the ECOC feature space, in which it is easier to separate transformed patterns representing clients and impostors. As a matching score in this space, we propose the average first order Minkowski distance between the probe and gallery images. The proposed method exhibits superior verification performance on the well known XM2VTS data set as compared with previously reported results.     

Exploration of classification confidence in ensemble learning

Ensemble learning has attracted considerable attention owing to its good generalization performance. The main issues in constructing a powerful ensemble include training a set of diverse and accurate base classifiers, and effectively combining them. Ensemble margin, computed as the difference of the vote numbers received by the correct class and the another class received with the most votes, is widely used to explain the success of ensemble learning. This definition of the ensemble margin does not consider the classification confidence of base classifiers. In this work, we explore the influence of the classification confidence of the base classifiers in ensemble learning and obtain some interesting conclusions. First, we extend the definition of ensemble margin based on the classification confidence of the base classifiers. Then, an optimization objective is designed to compute the weights of the base classifiers by minimizing the margin induced classification loss. Several strategies are tried to utilize the classification confidences and the weights. It is observed that weighted voting based on classification confidence is better than simple voting if all the base classifiers are used. In addition, ensemble pruning can further improve the performance of a weighted voting ensemble. We also compare the proposed fusion technique with some classical algorithms. The experimental results also show the effectiveness of weighted voting with classification confidence.     

An ensemble of support vector machines for predicting virulent proteins

It is important to develop a reliable system for predicting bacterial virulent proteins for finding novel drug/vaccine and for understanding virulence mechanisms in pathogens.In this work we have proposed a bacterial virulent protein prediction method based on an ensemble of classifiers where the features are extracted directly from the amino acid sequence of a given protein. It is well known in the literature that the features extracted from the evolutionary information of a given protein are better than the features extracted from the amino acid sequence. Our method tries to fill the gap between the amino acid sequence based approaches and the evolutionary information based approaches.An extensive evaluation according to a blind testing protocol, where the parameters of the system are calculated using the training set and the system is validated in three different independent datasets, has demonstrated the validity of the proposed method.     

A compressed sensing approach for efficient ensemble learning

This paper presents a method for improved ensemble learning, by treating the optimization of an ensemble of classifiers as a compressed sensing problem. Ensemble learning methods improve the performance of a learned predictor by integrating a weighted combination of multiple predictive models. Ideally, the number of models needed in the ensemble should be minimized, while optimizing the weights associated with each included model. We solve this problem by treating it as an example of the compressed sensing problem, in which a sparse solution must be reconstructed from an under-determined linear system. Compressed sensing techniques are then employed to find an ensemble which is both small and effective. An additional contribution of this paper, is to present a new performance evaluation method (a new pairwise diversity measurement) called the roulette-wheel kappa-error. This method takes into account the different weightings of the classifiers, and also reduces the total number of pairs of classifiers needed in the kappa-error diagram, by selecting pairs through a roulette-wheel selection method according to the weightings of the classifiers. This approach can greatly improve the clarity and informativeness of the kappa-error diagram, especially when the number of classifiers in the ensemble is large. We use 25 different public data sets to evaluate and compare the performance of compressed sensing ensembles using four different sparse reconstruction algorithms, combined with two different classifier learning algorithms and two different training data manipulation techniques. We also give the comparison experiments of our method against another five state-of-the-art pruning methods. These experiments show that our method produces comparable or better accuracy, while being significantly faster than the compared methods.     

Data representations and generalization error in kernel based learning machines

This paper focuses on the problem of how data representation influences the generalization error of kernel based learning machines like support vector machines (SVM) for classification. Frame theory provides a well founded mathematical framework for representing data in many different ways. We analyze the effects of sparse and dense data representations on the generalization error of such learning machines measured by using leave-one-out error given a finite amount of training data. We show that, in the case of sparse data representations, the generalization error of an SVM trained by using polynomial or Gaussian kernel functions is equal to the one of a linear SVM. This is equivalent to saying that the capacity of separating points of functions belonging to hypothesis spaces induced by polynomial or Gaussian kernel functions reduces to the capacity of a separating hyperplane in the input space. Moreover, we show that, in general, sparse data representations increase or leave unchanged the generalization error of kernel based methods. Dense data representations, on the contrary, reduce the generalization error in the case of very large frames. We use two different schemes for representing data in overcomplete systems of Haar and Gabor functions, and measure SVM generalization error on benchmarked data sets.     

Mixed continuous/binary quantum-inspired learning system with non-negative least square optimisation for automated design of regularised ensemble extreme learning machines

In this paper, a hybrid quantum-inspired evolutionary algorithm (QIEA) is proposed to automatically design regularised ensemble extreme learning machines (EELMs). Quantum evolutionary computing is a relatively recent spot-lighted concept which takes advantage from both the evolutionary and quantum computing laws. In general, QIEAs have been proven to be really powerful for optimising complex engineering tasks. The fascinating trait of observation operator in QIEA enables us to transform the quantum bits to both the binary and continuous spaces. Here, the authors present a mix continuous/binary version of QIEA, to find out whether it is suited for designing regularised EELMs. Indeed, the design process of EELM is conducted at two different levels, i.e. hyper and low levels. At the low level, some novel criteria are presented in the form of penalty functions to enable the optimiser searching for parsimonious, compact and accurate regularised extreme learning machines, as individual components of the ensemble. At the hyper-level, the non-negative least square error optimisation technique is utilised to deterministically find the most eligible components for designing the ensemble. Through extensive numerical experiments, the authors demonstrate that the proposed method is really efficient for the automated design of EELM identifiers.     

集成学习中基于离散化方法的基分类器构造研究

为构造集成学习中具有差异性的基分类器,提出基于数据离散化的基分类器构造方法,并用于支持向量机集成。该方法采用粗糙集和布尔推理离散化算法处理训练样本集,能有效删除不相关和冗余的属性,提高基分类器的准确性和差异性。实验结果表明,所提方法能取得比传统集成学习算法Bagging和Adaboost更好的性能。     

Combining features of negative correlation learning with mixture of experts in proposed ensemble methods

Both theoretical and experimental studies have shown that combining accurate neural networks (NNs) in the ensemble with negative error correlation greatly improves their generalization abilities. Negative correlation learning (NCL) and mixture of experts (ME), two popular combining methods, each employ different special error functions for the simultaneous training of NNs to produce negatively correlated NNs. In this paper, we review the properties of the NCL and ME methods, discussing their advantages and disadvantages. Characterization of both methods showed that they have different but complementary features, so if a hybrid system can be designed to include features of both NCL and ME, it may be better than each of its basis approaches. In this study, two approaches are proposed to combine the features of both methods in order to solve the weaknesses of one method with the strength of the other method, i.e., gated-NCL (G-NCL) and mixture of negatively correlated experts (MNCE). In the first approach, G-NCL, a dynamic combiner of ME is used to combine the outputs of base experts in the NCL method. The suggested combiner method provides an efficient tool to evaluate and combine the NCL experts by the weights estimated dynamically from the inputs based on the different competences of each expert regarding different parts of the problem. In the second approach, MNCE, the capability of a control parameter for NCL is incorporated in the error function of ME, which enables the training algorithm of ME to efficiently adjust the measure of negative correlation between the experts. This control parameter can be regarded as a regularization term added to the error function of ME to establish better balance in bias–variance–covariance trade-offs and thus improves the generalization ability. The two proposed hybrid ensemble methods, G-NCL and MNCE, are compared with their constituent methods, ME and NCL, in solving several benchmark problems. The experimental results show that our proposed methods preserve the advantages and alleviate the disadvantages of their basis approaches, offering significantly improved performance over the original methods.     

RS码技术在PDF417码纠错码编译码中的应用研究

在引入RS码编译码原理的基础上,通过对二维条码PDF417码和四一七国家条码规范的分析和研究,将RS码技术应用到PDF417码纠错码的编码、译码算法过程中,完成了PDF417纠错的实现,同时给出了在GF（929）域中计算3的幂值的方法,解决了中间结果过大导致数据溢出的问题。为PDF417码的纠错码提供了一个完整的解决方案。     

基于集成学习的Self-training在入侵检测中的应用

针对入侵检测的标记数据难以获得的问题,提出一种基于集成学习的Self-training方法——正则化Self-training。该方法结合主动学习和正则化理论,利用无标记数据对已有的分类器（该分类器对分类模式已学习得很好）作进一步的改进。对三种主要的集成学习方法在不同标记数据比例下进行对比实验,实验结果表明：借助大量无标记数据可以改善组合分类器的分类边界,算法能显著地降低结果分类器的错误率。     

监督与半监督学习下的数据流集成分类综述

在监督或半监督学习的条件下对数据流集成分类进行研究是一个很有意义的方向.从基分类器、关键技术、集成策略等三个方面进行介绍,其中,基分类器主要介绍了决策树、神经网络、支持向量机等;关键技术从增量、在线等方面介绍;集成策略主要介绍了boosting、stacking等.对不同集成方法的优缺点、对比算法和实验数据集进行了总结与分析.最后给出了进一步研究方向,包括监督和半监督学习下对于概念漂移的处理、对于同质集成和异质集成的研究,无监督学习下的数据流集成分类等.     

An emboli detection system based on Dual Tree Complex Wavelet Transform and ensemble learning

The traditional visual and acoustic embolic signal detection methods based on the expert analysis of individual spectral recordings and Doppler shift sounds are the gold standards. However, these types of detection methods are high-cost, subjective, and can only be applied by experts. In order to overcome these drawbacks, computer based automated embolic detection systems which employ spectral properties of emboli, speckle, and artifact using Fourier and Wavelet Transforms have been proposed. In this study, we propose a fast, accurate, and robust automated emboli detection system based on the Dual Tree Complex Wavelet Transform (DTCWT). Employing the DTCWT, which does not suffer from the lack of shift invariance property of ordinary Discrete Wavelet Transform (DWT), increases the robustness of the coefficients extracted from the Doppler ultrasound signals. In this study, a Doppler ultrasound dataset including 100 samples from each embolic, Doppler speckle, and artifact signal is used. Each sample obtained from forward and reverse blood flow directions is represented by 1024 points. In our method, we first extract the forward and reverse blood flow coefficients separately using DTCWT from the samples. Then dimensionality reduction is applied to each set of coefficients and both of the reduced set of coefficients are fed to classifiers individually. Subsequently, in the view that the forward and reverse blood flow coefficients carry different characteristics, the individual predictors of these classifiers are combined using ensemble stacking method. We compare the obtained results with Fast Fourier Transform and DWT based emboli detection systems, and show that the features extracted using DTCWT give the highest accuracy and emboli detection rate. It is also observed that combining forward and reverse coefficients using stacking ensemble method improves the emboli and artifact detection rates, and overall accuracy.     

Failure mode classification and bearing capacity prediction for reinforced concrete columns based on ensemble machine learning algorithm

Failure mode (FM) and bearing capacity of reinforced concrete (RC) columns are key concerns in structural design and/or performance assessment procedures. The failure types, i.e., flexure, shear, or mix of the above two, will greatly affect the capacity and ductility of the structure. Meanwhile, the design methodologies for structures of different failure types will be totally different. Therefore, developing efficient and reliable methods to identify the FM and predict the corresponding capacity is of special importance for structural design/assessment management. In this paper, an intelligent approach is presented for FM classification and bearing capacity prediction of RC columns based on the ensemble machine learning techniques. The most typical ensemble learning method, adaptive boosting (AdaBoost) algorithm, is adopted for both classification and regression (prediction) problems. Totally 254 cyclic loading tests of RC columns are collected. The geometric dimensions, reinforcing details, material properties are set as the input variables, while the failure types (for classification problem) and peak capacity forces (for regression problem) are set as the output variables. The results indicate that the model generated by the AdaBoost learning algorithm has a very high accuracy for both FM classification (accuracy = 0.96) and capacity prediction (R² = 0.98). Different learning algorithms are also compared and the results show that ensemble learning (especially AdaBoost) has better performance than single learning. In addition, the bearing capacity predicted by the AdaBoost is also compared to that by the empirical formulas provided by the design codes, which shows an obvious superior of the proposed method. In summary, the machine learning technique, especially the ensemble learning, can provide an alternate to the conventional mechanics-driven models in structural design in this big data time.     

基于纠错编码的CSNN及其在遥感图像分类中的应用

单输出组合神经网络(CSNN)克服了BP神经网络固有的缺陷，具有网络结构确定、分类行为易于解释、并行性好等优点，但分类精度比经过结构选择的BPNN略差．采用纠错编码可以提高CSNN的分类精度，首先根据类别数与纠错能力确定类别码组，每个码字对应一种类别，每个SNN子网对这些码字中的同一位进行训练，从而确定网络结构与每个子网所学习的二值函数；对未知类别的样本进行分类时，各SNN的结果组成一个输出码，计算该输出码与各类别码的汉明距离，选择与其距离最近的类别码所对应的类别为该样本的类别；基于纠错编码的CSNN的分类行为易于转化为规则集形式，可理解性强．将该网络结构用于遥感图像分类，并与其他分类算法进行比较，结果表明采用纠错编码技术，CSNN不仅具备原有的各项优点，而且分类精度得到显著提高．     

LDPC码的不等差错保护策略在SVC中的应用

在分析信源信道联合编码的基础上,提出了一种使用LDPC码对SVC比特流进行不等差错保护的策略,能够使端到端的视频序列失真最小化。该不等差错保护策略根据各帧对重建图像的贡献量大小为其进行合理的比特分配,并对每一帧的各质量层实施最佳的非均衡差错保护。实验结果表明,与基于拉格朗日的优化方法相比,该方法更为简单,重建视频的峰值信噪比（PSNR）性能也有明显改进。     

集成优化核极限学习机的冠心病无创性诊断

冠心病的早期无创性诊断一直是医疗诊断领域的研究热点,为了提高冠心病诊断的准确率和诊断效率,提出了一种新颖的局部Fisher判别分析(LFDA)特征提取方法和集成核极限学习机(KELM)相结合的冠心病诊断模型(LFDA-EKELM)。首先使用LFDA方法剔除不相关特征和冗余特征,找出对分类结果贡献度较高的特征子集,产生不同的训练集以训练粒子群优化的KELM分类器PSO-KELM,并基于旋转森林(RF)构建集成分类器,实现冠心病的智能诊断。实验结果表明,与基于ELM、SVM和BPNN方法相比,提出方法有效提高了冠心病诊断准确率,提升了诊断效率,且分类结果高于已有方法和相似方法,是一种有效冠心病诊断模型。     

On-line transfer learning for multi-fidelity data fusion with ensemble of deep neural networks

Deep Neural Network (DNN) is widely used in engineering applications for its ability to handle problems with almost any nonlinearities. However, it is generally difficult to obtain sufficient high-fidelity (HF) sample points for expensive optimization tasks, which may affect the generalization performance of DNN and result in inaccurate predictions. To solve this problem and improve the prediction accuracy of DNN, this paper proposes an on-line transfer learning based multi-fidelity data fusion (OTL-MFDF) method including two parts. In the first part, the ensemble of DNNs is established. Firstly, a large number of low-fidelity sample points and a few HF sample points are generated, which are used as the source dataset and target dataset, respectively. Then, the Bayesian Optimization (BO) is utilized to obtain several groups of hyperparameters, based on which DNNs are pre-trained using the source dataset. Next, these pre-trained DNNs are re-trained by fine-tuning on the target dataset, and the ensemble of DNNs is established by assigning different weights to each pre-trained DNN. In the second part, the on-line learning system is developed for adaptive updating of the ensemble of DNNs. To evaluate the uncertainty error of the predicted values of DNN and determine the location of the updated HF sample point, the query-by-committee strategy based on the ensemble of DNNs is developed. The Covariance Matrix Adaptation Evolutionary Strategies is employed as the optimizer to find out the location where the maximal disagreement is achieved by the ensemble of DNNs. The design space is partitioned by the Voronoi diagram method, and then the selected point is moved to its nearest Voronoi cell boundary to avoid clustering between the updated point and the existing sample points. Three different types of test problems and an engineering example are adopted to illustrate the effectiveness of the OTL-MFDF method. Results verify the outstanding efficiency, global prediction accuracy and applicability of the OTL-MFDF method.     

基于极端学习机的MANET移动性预测模型

针对移动自组织网络移动性在管理无线网络带宽资源可用性方面的重要性,为了更好地规划连续服务可用性和有效能源管理以提升网络的整体服务质量,提出了一种基于极端学习机的MANET移动性预测模型。利用ELM对MANET中的任意节点进行建模;假设已知每个移动节点当前的移动性信息（位置、速度和运动方向角度）,以这种方式预测节点未来的位置和相邻节点之间未来的距离;基于几个标准移动性模型,产生更加真实、精确的移动性预测,从而更好地捕捉任意节点直角坐标系之间现有交互/相关性。使用标准移动性模型的仿真结果验证了所提模型的有效性,实验结果表明,提出的预测模型明显改进了传统基于多层感知器的模型,此外,当预测相邻节点之间未来距离时,避免了当前算法对预测精度的限制。     

集成学习算法在中医证型分类预测中的应用

为提高中医诊断的智能化以及辩证的准确度,提出一种基于多模态扰动策略的集成学习算法(MPEL算法)。首先,在样本域多次抽样产生不同的样本子空间;其次,在属性域采用改进的层次聚类特征选择算法,划分不同的属性子空间,进而训练出具有较大差异性的基分类器;然后,采用贪心策略选取最优的基分类器组合,提高算法整体性能。选择中医哮喘病症状-证型病案进行验证,并与其它集成学习算法对比,实验结果表明,改进的集成学习算法在哮喘病症状-证型分类预测中训练速度较快、识别准确率更高,最高识别率高达98.16%。