基于数据自律分发的神经网络集成模型

针对神经网络在求解大规模数据时表现出的计算能力不足的瓶颈问题,本文在对神经网络集成理论及其算法进行分析研究的基础上,结合自律分散系统提出一种新的基于数据自律分发实现的神经网络集成模型,设计了该模型下由Pull-MA和Push-MA实现的确保时序一致性的通信机制,并给出了用于实现网络中数据自律分发和结果自律收集评价的训练算法.实验结果表明,所构建模型和集成算法对大规模数据的处理能达到理想的训练效果,网络具有良好的泛化和分类能力.     

神经网络集成在图书剔旧分类中的应用

在分析图书剔旧工作的基础上，指出用智能的方法解决图书剔旧问题的必要性。提出了可以用神经网络集成技术来解决该问题，并给出一种动态构建神经网络集成的方法，该方法在训练神经网络集成成员网络时不仅调整网络的连接权值，而且动态地构建神经网络集成中各成员神经网络的结构，从而在提高单个网络精度的同时，增加了各网络成员之间的差异度，减小了集成的泛化误差。实验证明该方法可以有效地用于图书剔旧分类。     

构造性层状神经网络集成方法

在分析构造性神经网络集成和层状神经网络集成方法的基础上,提出了一种构造性层状神经网络集成方法。该方法自动确定神经网络集成中成员神经网络的数目,以及成员神经网络的结构等。集成在保证成员神经网络精度的同时,又保证了成员网络之间的差异度。用户只需要简单定义一些参数,就可以构造出性能较好的神经网络集成。     

全卷积多并联残差神经网络

随着人工智能的火热发展,深度学习已经在很多领域占有了一席之地.作为深度学习中一个典型网络--残差神经网络模型自提出之日起就成为了众多研究者的关注点.然而,残差神经网络还有很大的改进空间.为了更好地解决反向传播中梯度减小的问题,本文提出了一种改进的残差神经网络,称为全卷积多并联残差神经网络.在该网络中,每一层的特征信息不仅传输到下一层还输出到最后的平均池化层.为了测试该网络的性能,分别在三个数据集(MNIST,CIFAR-10和CIFAR-100)上对比图像分类的结果.实验结果表明,改进后的全卷积多并联残差神经网络与残差网络相比具有更高的分类准确率和更好的泛化能力.     

基于神经网络集成的专家系统模型

提出一种基于神经网络集成的专家系统模型,并给出神经网络集成的构造算法.在该模型中神经网络集成作为专家系统的一个内嵌模块,用于专家系统的知识获取,克服了传统专家系统在知识获取中的＂瓶颈＂问题.并将该模型用于图书剔旧系统中,初步建成基于神经网络集成的图书剔旧专家系统原型.     

CNN-ELM混合短文本分类模型

针对目前自然语言处理研究中,使用卷积神经网络（CNN）进行短文本分类任务时可以结合不同神经网络结构与分类算法以提高分类性能的问题,提出了一种结合卷积神经网络与极速学习机的CNN-ELM混合短文本分类模型。使用词向量训练构成文本矩阵作为输入数据,然后使用卷积神经网络提取特征并使用Highway网络进行特征优化,最后使用误差最小化极速学习机(EM-ELM)作为分类器完成短文本分类任务。与其他模型相比,该混合模型能够提取更具代表性的特征并能快速准确地输出分类结果。在多种英文数据集上的实验结果表明提出的CNN-ELM混合短文本分类模型比传统机器学习模型与深度学习模型更适合完成短文本分类任务。     

基于排错等待延迟的广义动态集成神经网络模型

软件可靠性增长模型在可靠性评估与保障中具有重要作用,针对软件测试过程中的故障检测和排错等待延迟问题,提出了一种考虑故障排错等待延迟的广义动态集成神经网络模型(RWD-SRGM)。该模型考虑软件工程的多样性,利用神经网络方法构建广义动态集成模型,并考虑排错等待延迟现象完成故障检测和预测。通过2组真实失效数据集(DS1和DS2)的实验,将所提模型与现有的软件可靠性增长模型进行了比较,结果显示考虑故障排错等待延迟的神经网络模型拟合效果最优,表现出了更好的软件可靠性评估性能和模型通用性。     

基于选择性神经网络集成的地基云状识别算法

针对传统地基云图云状识别模型精度较低的问题,提出一种基于K均值算法的选择性神经网络集成的方法。该方法以BP神经网络集成模型为基础,采用K均值聚类算法选择部分有差异性的个体神经网络进行集成,建立了云状分类模型。通过对云图样本进行仿真实验,结果表明所提出的算法相对于单个BP神经网络及传统的BP_AdaBoost集成算法用于云图的分类,能有效地提高云图识别分类的精度。     

随机初始化神经网络剪枝的稀疏二值规划方法

传统深度神经网络剪枝方法往往以预训练模型为初始网络并需要在剪枝后进行微调。受到近年来edgepopup等基于随机初始化网络的剪枝算法优异性能的启发,提出了一种基于稀疏二值规划的随机初始化网络剪枝算法。该算法将剪枝训练过程建模为一个稀疏二值约束优化问题。其核心思想是利用稀疏二值规划来学习一个二值掩膜,利用该掩膜可以从随机初始化的神经网络上裁剪出一个未经训练却性能良好的稀疏网络。与之前基于随机初始化网络的剪枝算法相比,该算法找到的稀疏网络在多个稀疏度下都具有更好的分类泛化性能。与edge-popup算法相比,在ImageNet数据集分类任务中,模型在稀疏度为70%时精度提升7.98个百分点。在CIFAR-10数据集分类任务中,模型在稀疏度为50%时精度提升2.48个百分点。     

神经网络集成的设计与应用

传统的神经网络一般采用个体网络,其应用效果很大程度上取决于使用者的经验,且网络的泛化能力不强.一种改进的神经网络集成方法,为传统神经网络存在的问题提供了一个简易的解决方案.由理论分析和实验结果可以得出结论,神经网络集成方法比传统的个体网络方法的效果更好.     

Cooperative coevolution of artificial neural network ensembles for pattern classification

This paper presents a cooperative coevolutive approach for designing neural network ensembles. Cooperative coevolution is a recent paradigm in evolutionary computation that allows the effective modeling of cooperative environments. Although theoretically, a single neural network with a sufficient number of neurons in the hidden layer would suffice to solve any problem, in practice many real-world problems are too hard to construct the appropriate network that solve them. In such problems, neural network ensembles are a successful alternative. Nevertheless, the design of neural network ensembles is a complex task. In this paper, we propose a general framework for designing neural network ensembles by means of cooperative coevolution. The proposed model has two main objectives: first, the improvement of the combination of the trained individual networks; second, the cooperative evolution of such networks, encouraging collaboration among them, instead of a separate training of each network. In order to favor the cooperation of the networks, each network is evaluated throughout the evolutionary process using a multiobjective method. For each network, different objectives are defined, considering not only its performance in the given problem, but also its cooperation with the rest of the networks. In addition, a population of ensembles is evolved, improving the combination of networks and obtaining subsets of networks to form ensembles that perform better than the combination of all the evolved networks. The proposed model is applied to ten real-world classification problems of a very different nature from the UCI machine learning repository and proben1 benchmark set. In all of them the performance of the model is better than the performance of standard ensembles in terms of generalization error. Moreover, the size of the obtained ensembles is also smaller.     

Design of effective neural network ensembles for image classification purposes

In the field of pattern recognition, the combination of an ensemble of neural networks has been proposed as an approach to the development of high performance image classification systems. However, previous work clearly showed that such image classification systems are effective only if the neural networks forming them make different errors. Therefore, the fundamental need for methods aimed to design ensembles of ‘error-independent’ networks is currently acknowledged. In this paper, an approach to the automatic design of effective neural network ensembles is proposed. Given an initial large set of neural networks, our approach is aimed to select the subset formed by the most error-independent nets. Reported results on the classification of multisensor remote-sensing images show that this approach allows one to design effective neural network ensembles.     

Bio-inspired and gradient-based algorithms to train MLPs: The influence of diversity

This paper has three main goals: (i) to employ two classes of algorithms: bio-inspired and gradient-based to train multi-layer perceptron (MLP) neural networks for pattern classification; (ii) to combine the trained neural networks into ensembles of classifiers; and (iii) to investigate the influence of diversity in the classification performance of individual and ensembles of classifiers. The optimization version of an artificial immune network, named opt-aiNet, particle swarm optimization (PSO) and an evolutionary algorithm (EA) are used as bio-inspired methods to train MLP networks. Besides, the standard backpropagation with momentum (BPM), a quasi-Newton method called DFP and a modified scaled-conjugate gradient (SCGM) are the gradient-based algorithms used to train MLP networks in this work. Comparisons among all the training methods are presented in terms of classification accuracy and diversity of the solutions found. The results obtained suggest that most bio-inspired algorithms deteriorate the diversity of solutions during the search, while immune-based methods, like opt-aiNet, and multiple initializations of standard gradient-based algorithms provide diverse solutions that result in good classification accuracy for the ensembles.     

Boosted Bayesian network classifiers

The use of Bayesian networks for classification problems has received a significant amount of recent attention. Although computationally efficient, the standard maximum likelihood learning method tends to be suboptimal due to the mismatch between its optimization criteria (data likelihood) and the actual goal of classification (label prediction accuracy). Recent approaches to optimizing classification performance during parameter or structure learning show promise, but lack the favorable computational properties of maximum likelihood learning. In this paper we present boosted Bayesian network classifiers, a framework to combine discriminative data-weighting with generative training of intermediate models. We show that boosted Bayesian network classifiers encompass the basic generative models in isolation, but improve their classification performance when the model structure is suboptimal. We also demonstrate that structure learning is beneficial in the construction of boosted Bayesian network classifiers. On a large suite of benchmark data-sets, this approach outperforms generative graphical models such as naive Bayes and TAN in classification accuracy. Boosted Bayesian network classifiers have comparable or better performance in comparison to other discriminatively trained graphical models including ELR and BNC. Furthermore, boosted Bayesian networks require significantly less training time than the ELR and BNC algorithms.     

The build of a dynamic classifier selection ICBP system and its application to pattern recognition

Abstract  

In neural network ensemble, the diversity of its constitutive component networks is a crucial factor to boost its generalization performance. In terms of how each ensemble system solves the problem, we can roughly categorize the existing ensemble mechanism into two groups: data-driven and model-driven ensembles. The former engenders diversity to ensemble members by manipulating the data, while the latter realizes ensemble diversity by manipulating the component models themselves. Within a neural network ensemble, standard back-propagation (BP) networks are usually used as a base component. However, in this article, we will use our previously designed improved circular back-propagation (ICBP) neural network to establish such an ensemble. ICBP differentiates from BP network not only because an extra anisotropic input node is added, but also more importantly, because of the introduction of the extra node, it possesses an interesting property apart from the BP network, i.e., just through directly assigning different sets of values 1 and −1 to the weights connecting the extra node to all the hidden nodes, we can construct a set of heterogeneous ICBP networks with different hidden layer activation functions, among which we select four typical heterogeneous ICBPs to build a dynamic classifier selection ICBP system (DCS-ICBP). The system falls into the category of model-driven ensemble. The aim of this article is to explore the relationship between the explicitly constructed ensemble and the diversity scale, and further to verify feasibility and effectiveness of the system on classification problems through empirical study. Experimental results on seven benchmark classification tasks show that our DCS-ICBP outperforms each individual ICBP classifier and surpasses the performance of combination of ICBP using the majority voting technique, i.e. majority voting ICBP system (MVICBP). The successful simulation results validate that in DCS-ICBP we provide a new constructive method for diversity enforcement for ICBP ensemble systems.     

Negatively correlated neural networks for classification

This paper presents a new algorithm for designing neural network ensembles for classification problems with noise. The idea behind this new algorithm is to encourage different individual networks in an ensemble to learn different parts or aspects of the training data so that the whole ensemble can learn the whole training data better. Negatively correlated neural networks are trained with a novel correlation penalty term in the error function to encourage such specialization. In our algorithm, individual networks are trained simultaneously rather than independently or sequentially. This provides an opportunity for different networks to interact with each other and to specialize. Experiments on two real-world problems demonstrate that the new algorithm can produce neural network ensembles with good generalization ability. This work was presented, in part, at the Third International Symposium on Artificial Life and Robotics, Oita, Japan January 19–21, 1998     

Using GMDH-based networks for improved spam detection and email feature analysis

Unsolicited or spam email has recently become a major threat that can negatively impact the usability of electronic mail. Spam substantially wastes time and money for business users and network administrators, consumes network bandwidth and storage space, and slows down email servers. In addition, it provides a medium for distributing harmful code and/or offensive content. In this paper, we explore the application of the GMDH (Group Method of Data Handling) based inductive learning approach in detecting spam messages by automatically identifying content features that effectively distinguish spam from legitimate emails. We study the performance for various network model complexities using spambase, a publicly available benchmark dataset. Results reveal that classification accuracies of 91.7% can be achieved using only 10 out of the available 57 attributes, selected through abductive learning as the most effective feature subset (i.e. 82.5% data reduction). We also show how to improve classification performance using abductive network ensembles (committees) trained on different subsets of the training data. Comparison with other techniques such as neural networks and naïve Bayesian classifiers shows that the GMDH-based learning approach can provide better spam detection accuracy with false-positive rates as low as 4.3% and yet requires shorter training time.     

A cooperative constructive method for neural networks for pattern recognition

In this paper, we propose a new constructive method, based on cooperative coevolution, for designing automatically the structure of a neural network for classification. Our approach is based on a modular construction of the neural network by means of a cooperative evolutionary process. This process benefits from the advantages of coevolutionary computation as well as the advantages of constructive methods. The proposed methodology can be easily extended to work with almost any kind of classifier.The evaluation of each module that constitutes the network is made using a multiobjective method. So, each new module can be evaluated in a comprehensive way, considering different aspects, such as performance, complexity, or degree of cooperation with the previous modules of the network. In this way, the method has the advantage of considering not only the performance of the networks, but also other features.The method is tested on 40 classification problems from the UCI machine learning repository with very good performance. The method is thoroughly compared with two other constructive methods, cascade correlation and GMDH networks, and other classification methods, namely, SVM, C4.5, and k nearest-neighbours, and an ensemble of neural networks constructed using four different methods.