Combining features of negative correlation learning with mixture of experts in proposed ensemble methods

Both theoretical and experimental studies have shown that combining accurate neural networks (NNs) in the ensemble with negative error correlation greatly improves their generalization abilities. Negative correlation learning (NCL) and mixture of experts (ME), two popular combining methods, each employ different special error functions for the simultaneous training of NNs to produce negatively correlated NNs. In this paper, we review the properties of the NCL and ME methods, discussing their advantages and disadvantages. Characterization of both methods showed that they have different but complementary features, so if a hybrid system can be designed to include features of both NCL and ME, it may be better than each of its basis approaches. In this study, two approaches are proposed to combine the features of both methods in order to solve the weaknesses of one method with the strength of the other method, i.e., gated-NCL (G-NCL) and mixture of negatively correlated experts (MNCE). In the first approach, G-NCL, a dynamic combiner of ME is used to combine the outputs of base experts in the NCL method. The suggested combiner method provides an efficient tool to evaluate and combine the NCL experts by the weights estimated dynamically from the inputs based on the different competences of each expert regarding different parts of the problem. In the second approach, MNCE, the capability of a control parameter for NCL is incorporated in the error function of ME, which enables the training algorithm of ME to efficiently adjust the measure of negative correlation between the experts. This control parameter can be regarded as a regularization term added to the error function of ME to establish better balance in bias–variance–covariance trade-offs and thus improves the generalization ability. The two proposed hybrid ensemble methods, G-NCL and MNCE, are compared with their constituent methods, ME and NCL, in solving several benchmark problems. The experimental results show that our proposed methods preserve the advantages and alleviate the disadvantages of their basis approaches, offering significantly improved performance over the original methods.     

A novel evolutionary-negative correlated mixture of experts model in tourism demand estimation

Mixtures of experts (ME) model are widely used in many different areas as a recognized ensemble learning approach to account for nonlinearities and other complexities in the data, such as time series estimation. With the aim of developing an accurate tourism demand time series estimation model, a mixture of experts model called LSPME (Lag Space Projected ME) is presented by combining ideas from subspace projection methods and negative correlation learning (NCL). The LSPME uses a new cluster-based lag space projection (CLSP) method to automatically obtain input space to train each expert focused on the difficult instances at each step of the boosting approach. For training experts of the LSPME, a new NCL algorithm called Sequential Evolutionary NCL algorithm (SENCL) is proposed that uses a moving average for the correlation penalty term in the error function of each expert to measure the error correlation between it and its previous experts. The LSPME model was compared with other ensemble models using monthly tourist arrivals to Japan from four markets: The United States, United Kingdom, Hong Kong and Taiwan. The experimental results show that the estimation accuracy of the proposed LSPME model is significantly better than the other ensemble models and can be considered to be a promising alternative for time series estimation problems.     

Improving combination method of NCL experts using gating network

Negative Correlation Learning (NCL) is a popular combining method that employs special error function for the simultaneous training of base neural network (NN) experts. In this article, we propose an improved version of NCL method in which the capability of gating network, as the combining part of Mixture of Experts method, is used to combine the base NNs in the NCL ensemble method. The special error function of the NCL method encourages each NN expert to learn different parts or aspects of the training data. Thus, the local competence of the experts should be considered in the combining approach. The gating network provides a way to support this needed functionality for combining the NCL experts. So the proposed method is called Gated NCL. The improved ensemble method is compared with the previous approaches were used for combining NCL experts, including winner-take-all (WTA) and average (AVG) combining techniques, in solving several classification problems from UCI machine learning repository. The experimental results show that our proposed ensemble method significantly improved performance over the previous combining approaches.     

基于人工神经网络的负相关学习研究

人工神经网络集成技术是神经计算技术的一个研究热点,在许多领域中已经有了成熟的应用.神经网络集成是用有限个神经网络对同一个问题进行学习,集成在某输入示例下的输出由构成集成的各神经网络在该示例下的输出共同决定.负相关学习法是一种神经网络集成的训练方法,它鼓励集成中的不同个体网络学习训练集的不同部分,以使整个集成能更好地学习整个训练数据.改进的负相关学习法是在误差函数中使用一个带冲量的BP算法,给合了原始负相关学习法和带冲量的BP算法的优点,使改进的算法成为泛化能力强、学习速度快的批量学习算法.     

Root-quatric mixture of experts for complex classification problems

Mixture of experts (ME) as an ensemble method consists of several experts and a gating network to decompose the input space into some subspaces regarding to the experts specialties. To increase the diversity between experts in ME, this paper incorporates a correlation penalty function into the error function of ME. The significant of this modification is providing an occasion to encourage experts to specialize on different parts of the input space and to create decorrelated experts. The experimental results of this approach reveals that the impacts of this penalty function is extremely improved the diversity of experts and the tradeoff between the accuracy and the diversity in ME. Moreover in the implementation of this method, the experts are trained simultaneously and they can communicate by the aid of the correlation penalty function. The performance of the proposed method on ten classification benchmark datasets shows that the average of accuracy of this method improves 1.94%, 3.7%, and 3.74% compared with the mixture of negatively correlated experts, ME and the negative correlation learning, respectively. Thus the proposed method can be considered as a better classifier for healthy and medical problems and also when the great non-stationary data should be classified.     

Evolutionary ensembles with negative correlation learning

Based on negative correlation learning and evolutionary learning, this paper presents evolutionary ensembles with negative correlation learning (EENCL) to address the issues of automatic determination of the number of individual neural networks (NNs) in an ensemble and the exploitation of the interaction between individual NN design and combination. The idea of EENCL is to encourage different individual NNs in the ensemble to learn different parts or aspects of the training data so that the ensemble can learn better the entire training data. The cooperation and specialization among different individual NNs are considered during the individual NN design. This provides an opportunity for different NNs to interact with each other and to specialize. Experiments on two real-world problems demonstrate that EENCL can produce NN ensembles with good generalization ability.     

Bagging and boosting negatively correlated neural networks.

In this paper, we propose two cooperative ensemble learning algorithms, i.e., NegBagg and NegBoost, for designing neural network (NN) ensembles. The proposed algorithms incrementally train different individual NNs in an ensemble using the negative correlation learning algorithm. Bagging and boosting algorithms are used in NegBagg and NegBoost, respectively, to create different training sets for different NNs in the ensemble. The idea behind using negative correlation learning in conjunction with the bagging/boosting algorithm is to facilitate interaction and cooperation among NNs during their training. Both NegBagg and NegBoost use a constructive approach to automatically determine the number of hidden neurons for NNs. NegBoost also uses the constructive approach to automatically determine the number of NNs for the ensemble. The two algorithms have been tested on a number of benchmark problems in machine learning and NNs, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, satellite, soybean, and waveform problems. The experimental results show that NegBagg and NegBoost require a small number of training epochs to produce compact NN ensembles with good generalization.     

Neural-Based Learning Classifier Systems

UCS is a supervised learning classifier system that was introduced in 2003 for classification in data mining tasks. The representation of a rule in UCS as a univariate classification rule is straightforward for a human to understand. However, the system may require a large number of rules to cover the input space. Artificial neural networks (NNs), on the other hand, normally provide a more compact representation. However, it is not a straightforward task to understand the network. In this paper, we propose a novel way to incorporate NNs into UCS. The approach offers a good compromise between compactness, expressiveness, and accuracy. By using a simple artificial NN as the classifier's action, we obtain a more compact population size, better generalization, and the same or better accuracy while maintaining a reasonable level of expressiveness. We also apply negative correlation learning (NCL) during the training of the resultant NN ensemble. NCL is shown to improve the generalization of the ensemble.     

Regularized Negative Correlation Learning for Neural Network Ensembles

Negative correlation learning (NCL) is a neural network ensemble learning algorithm that introduces a correlation penalty term to the cost function of each individual network so that each neural network minimizes its mean square error (MSE) together with the correlation of the ensemble. This paper analyzes NCL and reveals that the training of NCL (when $lambda=1$) corresponds to training the entire ensemble as a single learning machine that only minimizes the MSE without regularization. This analysis explains the reason why NCL is prone to overfitting the noise in the training set. This paper also demonstrates that tuning the correlation parameter $lambda$ in NCL by cross validation cannot overcome the overfitting problem. The paper analyzes this problem and proposes the regularized negative correlation learning (RNCL) algorithm which incorporates an additional regularization term for the whole ensemble. RNCL decomposes the ensemble's training objectives, including MSE and regularization, into a set of sub-objectives, and each sub-objective is implemented by an individual neural network. In this paper, we also provide a Bayesian interpretation for RNCL and provide an automatic algorithm to optimize regularization parameters based on Bayesian inference. The RNCL formulation is applicable to any nonlinear estimator minimizing the MSE. The experiments on synthetic as well as real-world data sets demonstrate that RNCL achieves better performance than NCL, especially when the noise level is nontrivial in the data set.      

A constructive algorithm for training cooperative neural network ensembles

Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability.     

基于双重扰动的选择性支持向量机集成

为了进一步提升支持向量机泛化性能,提出一种基于双重扰动的选择性支持向量机集成算法。利用Boosting方法对训练集进行扰动基础上,采用基于相对核的粗糙集相对约简与重采样技术相结合的动态约简算法进行特征扰动以生成个体成员,然后基于负相关学习理论构造遗传个体适应度函数,利用加速遗传算法选择权重大于阈值的最优个体进行加权集成。实验结果表明,该算法具有较高的泛化性能和较低的时、空复杂性,是一种高效的集成方法。     

基于加速遗传算法的选择性支持向量机集成*

为有效提升支持向量机的泛化性能,提出基于加速遗传算法的选择性支持向量机集成。通过Bootstrap技术产生并训练得到多个独立子SVM,基于负相关学习理论构造适应度函数,提高子SVM的泛化性能,并增大其之间差异度。利用加速遗传算法计算各子SVM在加权平均中的最优权重,然后选择权值大于一定阈值的部分SVM进行加权集成。实验结果表明,该算法是一种有效的集成方法,能进一步提高SVM的集成效率和泛化性能。     

多模态粒子群集成神经网络

提出一种基于多模态粒子群算法的神经网络集成方法,在网络训练每个迭代周期内利用改进的快速聚类算法在权值搜索空间上动态地把搜索粒子分为若干类,求得每一类的最优粒子,然后计算最优个体两两之间的输出空间相异度,合并相异度过低的两类粒子,最终形成不但权值空间相异、而且输出空间也相异的若干类粒子,每类粒子负责一个成员网络权值的搜索,其中最优粒子对应于一个成员网络,所有类的最优粒子组成神经网络集成,成员网络的个数是由算法自动确定的.算法控制网络多样性的方法更直接、更有效.与负相关神经网络集成、bagging和boosting方法比较,实验结果表明,此算法较好地提高了神经网络集成的泛化能力.     

Negatively correlated neural networks for classification

This paper presents a new algorithm for designing neural network ensembles for classification problems with noise. The idea behind this new algorithm is to encourage different individual networks in an ensemble to learn different parts or aspects of the training data so that the whole ensemble can learn the whole training data better. Negatively correlated neural networks are trained with a novel correlation penalty term in the error function to encourage such specialization. In our algorithm, individual networks are trained simultaneously rather than independently or sequentially. This provides an opportunity for different networks to interact with each other and to specialize. Experiments on two real-world problems demonstrate that the new algorithm can produce neural network ensembles with good generalization ability. This work was presented, in part, at the Third International Symposium on Artificial Life and Robotics, Oita, Japan January 19–21, 1998     

Simultaneous training of negatively correlated neural networks inan ensemble

This paper presents a new cooperative ensemble learning system (CELS) for designing neural network ensembles. The idea behind CELS is to encourage different individual networks in an ensemble to learn different parts or aspects of a training data so that the ensemble can learn the whole training data better. In CELS, the individual networks are trained simultaneously rather than independently or sequentially. This provides an opportunity for the individual networks to interact with each other and to specialize. CELS can create negatively correlated neural networks using a correlation penalty term in the error function to encourage such specialization. This paper analyzes CELS in terms of bias-variance-covariance tradeoff. CELS has also been tested on the Mackey-Glass time series prediction problem and the Australian credit card assessment problem. The experimental results show that CELS can produce neural network ensembles with good generalization ability.     

基于动态粗糙集约简的选择性支持向量机集成

支持向量机集成是提高支持向量机泛化性能的有效手段,个体支持向量机的泛化能力及其之间的差异性是影响集成性能的关键因素。为了进一步提升支持向量机整体泛化性能,提出利用动态粗糙集的选择性支持向量机集成算法。首先在利用Boosting算法对样本进行扰动基础上,采用遗传算法改进的粗糙集与重采样技术相结合的动态约简算法进行特征扰动,获得稳定、泛化能力较强的属性约简集,继而生成差异性较大的个体学习器;然后利用模糊核聚类根据个体学习器在验证集上的泛化误差来选择最优个体;并用支持向量机算法对最优个体进行非线性集成。通过在UCI数据集进行仿真,结果表明算法能明显提高支持向量机的泛化性能,具有较低的时、空复杂性,是一种高效、稳定的集成方法。     

基于分层采样的集成k近邻说话人识别算法

k近邻学习器将复杂的全局非线性关系映射为大量局部线性关系的组合,具有易解释、易扩展、抗噪能力强等优点,被广泛应用于说话人识别领域并取得了良好的效果。而集成学习算法因其强泛化能力和易于应用的特性得到了许多领域研究者的关注,但是研究表明通过重采样产生训练集差异的集成算法并不能有效地提高k近邻学习器系统的泛化能力。提出了一种新的BagWithProb采样算法产生训练集。实验表明,该算法可以有效地扩展训练集差异,提高集成系统性能。此外,还提出了基于环域分层采样的算法以加快k近邻识别算法在识别阶段的运算速度。     

一种基于神经网络集成的决策树构造方法

神经网络集成方法具有比单个神经网络更强的泛化能力，却因为其黑箱性而难以理解；决策树算法因为分类结果显示为树型结构而具有良好的可理解性，泛化能力却比不上神经网络集成。该文将这两种算法相结合，提出一种决策树的构造算法：使用神经网络集成来预处理训练样本，使用C4．5算法处理预处理后的样本并生成决策树。该文在UCI数据上比较了神经网络集成方法、决策树C4．5算法和该文算法，实验表明：该算法具有神经网络集成方法的强泛化能力的优点，其泛化能力明显优于C4．5算法；该算法的最终结果昆示为决策树，显然具有良好的可理解性。     

An exploration of the uncertainty relation satisfied by BP network learning ability and generalization ability

An overfit phenomenon exists in the BP network. The so-called overfit means that as long as the network is allowed to be sufficiently complicated, the BP network can minimize the error of the training sample set; however, in the case of a limited number of samples, the generalization ability of the network will decrease. This indicates that there is a relation between the learning ability and the generalization ability. Therefore, studying the relationship between the learning ability is the…     

用于处理不平衡样本的改进近似支持向量机新算法

近似支持向量机(PSVM)在处理不平衡样本时,会过拟合样本点数较多的一类,低估样本点数较少的类的错分误差,从而导致整体样本的分类准确率下降。针对该问题,提出一种用于处理不平衡样本的改进的PSVM新算法。新算法不仅给正、负类样本赋予不同的惩罚因子,而且在约束条件中新增参数,使得分类面更具灵活性。该算法先对训练集训练获得最优参数,然后再对测试集进行训练获得分类超平面,最后输出分类结果。UCI数据库中9组数据集的实验结果表明:新算法提高了样本的分类准确率,在线性的情况下平均提高了2.19个百分点,在非线性的情况下平均提高了3.14个百分点,有效地提高了模型的泛化能力。