Cooperative coevolution of artificial neural network ensembles for pattern classification

This paper presents a cooperative coevolutive approach for designing neural network ensembles. Cooperative coevolution is a recent paradigm in evolutionary computation that allows the effective modeling of cooperative environments. Although theoretically, a single neural network with a sufficient number of neurons in the hidden layer would suffice to solve any problem, in practice many real-world problems are too hard to construct the appropriate network that solve them. In such problems, neural network ensembles are a successful alternative. Nevertheless, the design of neural network ensembles is a complex task. In this paper, we propose a general framework for designing neural network ensembles by means of cooperative coevolution. The proposed model has two main objectives: first, the improvement of the combination of the trained individual networks; second, the cooperative evolution of such networks, encouraging collaboration among them, instead of a separate training of each network. In order to favor the cooperation of the networks, each network is evaluated throughout the evolutionary process using a multiobjective method. For each network, different objectives are defined, considering not only its performance in the given problem, but also its cooperation with the rest of the networks. In addition, a population of ensembles is evolved, improving the combination of networks and obtaining subsets of networks to form ensembles that perform better than the combination of all the evolved networks. The proposed model is applied to ten real-world classification problems of a very different nature from the UCI machine learning repository and proben1 benchmark set. In all of them the performance of the model is better than the performance of standard ensembles in terms of generalization error. Moreover, the size of the obtained ensembles is also smaller.     

Reduced Pattern Training Based on Task Decomposition Using Pattern Distributor

Task decomposition with pattern distributor (PD) is a new task decomposition method for multilayered feedforward neural networks (NNs). Pattern distributor network is proposed that implements this new task decomposition method. We propose a theoretical model to analyze the performance of pattern distributor network. A method named reduced pattern training (RPT) is also introduced, aiming to improve the performance of pattern distribution. Our analysis and the experimental results show that RPT improves the performance of pattern distributor network significantly. The distributor module's classification accuracy dominates the whole network's performance. Two combination methods, namely, crosstalk-based combination and genetic-algorithm (GA)-based combination, are presented to find suitable grouping for the distributor module. Experimental results show that this new method can reduce training time and improve network generalization accuracy when compared to a conventional method such as constructive backpropagation or a task decomposition method such as output parallelism (OP).     

构造性层状神经网络集成方法

在分析构造性神经网络集成和层状神经网络集成方法的基础上,提出了一种构造性层状神经网络集成方法。该方法自动确定神经网络集成中成员神经网络的数目,以及成员神经网络的结构等。集成在保证成员神经网络精度的同时,又保证了成员网络之间的差异度。用户只需要简单定义一些参数,就可以构造出性能较好的神经网络集成。     

Entropy-based generation of supervised neural networks for classification of structured patterns

Sperduti and Starita proposed a new type of neural network which consists of generalized recursive neurons for classification of structures. In this paper, we propose an entropy-based approach for constructing such neural networks for classification of acyclic structured patterns. Given a classification problem, the architecture, i.e., the number of hidden layers and the number of neurons in each hidden layer, and all the values of the link weights associated with the corresponding neural network are automatically determined. Experimental results have shown that the networks constructed by our method can have a better performance, with respect to network size, learning speed, or recognition accuracy, than the networks obtained by other methods.     

Counter-Propagation Neural Networks for Molecular Sequence Classification: Supervised LVQ and Dynamic Node Allocation

A modified counter-propagation (CP) algorithm with supervised learning vector quantizer (LVQ) and dynamic node allocation has been developed for rapid classification of molecular sequences. The molecular sequences were encoded into neural input vectors using an n–gram hashing method for word extraction and a singular value decomposition (SVD) method for vector compression. The neural networks used were three-layered, forward-only CP networks that performed nearest neighbor classification. Several factors affecting the CP performance were evaluated, including weight initialization, Kohonen layer dimensioning, winner selection and weight update mechanisms. The performance of the modified CP network was compared with the back-propagation (BP) neural network and the k–nearest neighbor method. The major advantages of the CP network are its training and classification speed and its capability to extract statistical properties of the input data. The combined BP and CP networks can classify nucleic acid or protein sequences with a close to 100% accuracy at a rate of about one order of magnitude faster than other currently available methods.     

A PDP constructive algorithm for system modelling

A new constructive algorithm is presented for building neural networks that learn to reproduce output temporal sequences based on one or several input sequences. This algorithm builds a network for the task of system modelling, dealing with continuous variables in the discrete time domain. The constructive scheme makes it user independent. The network's structure consists of an ordinary set and a classification set, so it is a hybrid network like that of Stokbro et al. [6], but with a binary classification. The networks can easily be interpreted, so the learned representation can be transferred to a human engineer, unlike many other network models. This allows for a better understanding of the system structure than just its simulation. This constructive algorithm limits the network complexity automatically, hence preserving extrapolation capabilities. Examples with real data from three totally different sources show good performance and allow for a promising line of research.     

Combining linear discriminant functions with neural networks for supervised learning

A novel supervised learning method is proposed by combining linear discriminant functions with neural networks. The proposed method results in a tree-structured hybrid architecture. Due to constructive learning, the binary tree hierarchical architecture is automatically generated by a controlled growing process for a specific supervised learning task. Unlike the classic decision tree, the linear discriminant functions are merely employed in the intermediate level of the tree for heuristically partitioning a large and complicated task into several smaller and simpler subtasks in the proposed method. These subtasks are dealt with by component neural networks at the leaves of the tree accordingly. For constructive learning, growing and credit-assignment algorithms are developed to serve for the hybrid architecture. The proposed architecture provides an efficient way to apply existing neural networks (e.g. multi-layered perceptron) for solving a large scale problem. We have already applied the proposed method to a universal approximation problem and several benchmark classification problems in order to evaluate its performance. Simulation results have shown that the proposed method yields better results and faster training in comparison with the multilayered perceptron.     

无人机目标分类的深度卷积网络设计与优化

针对传统无人机目标分类方法效率低、特征提取能力不足和适应性差等问题,通过对无人机自身特点和现有分类方法的分析,提出了引入注意力机制优化深度卷积神经网络的无人机分类方法.设计多组对比实验,根据实验效果设计出模型结构为3层卷积层、3层池化层、2层全连接层的卷积神经网络进行训练,得到最优的无人机目标分类模型,再引入卷积注意力模块对特征图元素进行加强和抑制,引入批归一化层加速模型收敛,提升泛化能力.实验结果表明:引入卷积注意力模块和批归一化层优化后的无人机目标分类模型的识别率达到92.44％,较优化前提升1.5％,相比于其它神经网络模型具有识别率高、收敛速度快的优点,可以基本满足实际场景中无人机目标分类的要求.     

Rules extraction from constructively trained neural networks based on genetic algorithms

The application of neural networks in the data mining has become wider. Although neural networks may have complex structure, long training time, and the representation of results is not comprehensible, neural networks have high acceptance ability for noisy data, high accuracy and are preferable in data mining. On the other hand, It is an open question as to what is the best way to train and extract symbolic rules from trained neural networks in domains like classification. In this paper, we train the neural networks by constructive learning and present the analysis of the convergence rate of the error in a neural network with and without threshold which have been learnt by a constructive method to obtain the simple structure of the network.The response of ANN is acquired but its result is not in understandable form or in a black box form. It is frequently desirable to use the model backwards and identify sets of input variable which results in a desired output value. The large numbers of variables and nonlinear nature of many materials models that can help finding an optimal set of difficult input variables. We will use a genetic algorithm to solve this problem. The method is evaluated on different public-domain data sets with the aim of testing the predictive ability of the method and compared with standard classifiers, results showed comparatively high accuracy.     

A constructive algorithm to synthesize arbitrarily connected feedforward neural networks

In this work we present a constructive algorithm capable of producing arbitrarily connected feedforward neural network architectures for classification problems. Architecture and synaptic weights of the neural network should be defined by the learning procedure. The main purpose is to obtain a parsimonious neural network, in the form of a hybrid and dedicate linear/nonlinear classification model, which can guide to high levels of performance in terms of generalization. Though not being a global optimization algorithm, nor a population-based metaheuristics, the constructive approach has mechanisms to avoid premature convergence, by mixing growing and pruning processes, and also by implementing a relaxation strategy for the learning error. The synaptic weights of the neural networks produced by the constructive mechanism are adjusted by a quasi-Newton method, and the decision to grow or prune the current network is based on a mutual information criterion. A set of benchmark experiments, including artificial and real datasets, indicates that the new proposal presents a favorable performance when compared with alternative approaches in the literature, such as traditional MLP, mixture of heterogeneous experts, cascade correlation networks and an evolutionary programming system, in terms of both classification accuracy and parsimony of the obtained classifier.     

构造性核覆盖算法在图像识别中的应用

构造性神经网络的主要特点是：在对给定的具体数据的处理过程中,能同时给出网络的结构和参数;支持向量机就是先通过引入核函数的非线性变换,然后在这个核空间中求取最优线性分类面,其所求得的分类函数,形式上类似于一个神经网络,而构造性核覆盖算法(简称为CKCA)则是一种将神经网络中的构造性学习方法(如覆盖算法)与支持向量机(SVM)中的核函数法相结合的方法。CKCA方法具有运算量小、构造性强、直观等特点,适于处理大规模分类问题和图像识别问题。为验证CKCA算法的应用效果,利用图像质量不高的车牌字符进行了识别实验,并取得了较好的结果。     

一种基于多进化神经网络的分类方法

分类问题是目前数据挖掘和机器学习领域的重要内容.提出了一种基于多进化神经网络的分类方法CABEN(classification approach based on evolutionary neural networks).利用改进的进化策略和Levenberg-Marquardt方法对多个三层前馈神经网络同时进行训练.训练好各个分类模型以后,将待识别数据分别输入,最后根据绝对多数投票法决定最终分类结果.实验结果表明,该方法可以较好地进行数据分类,而且与传统的神经网络方法以及贝叶斯方法和决策树方法相比,在     

基于双池化特征加权结构CNN的图像分类

传统的池化方式会造成特征信息丢失,导致卷积神经网络中提取的特征信息不足。为了提高卷积神经网络在图像分类过程中的准确率,优化其学习性能,本文在传统池化方式的基础上提出一种双池化特征加权结构的池化算法,利用最大池化和平均池化2种方式保留更多的有价值的特征信息,并通过遗传算法对模型进行优化。通过训练不同池化方式的卷积神经网络,研究卷积神经网络在不同数据集上的分类准确率和收敛速度。实验在遥感图像数据集NWPU-RESISC45和彩色图像数据集Cifar-10上对采用几种池化方式的卷积神经网络分类结果进行对比验证,结果分析表明：双池化特征加权结构使得卷积神经网络的分类准确率有很大程度的提高,同时模型的收敛速度得到进一步提高。     

高光谱图像小样本分类的卷积神经网络方法

目的 与传统分类方法相比,基于深度学习的高光谱图像分类方法能够提取出高光谱图像更深层次的特征。针对现有深度学习的分类方法网络结构简单、特征提取不够充分的问题,提出一种堆叠像元空间变换信息的数据扩充方法,用于解决训练样本不足的问题,并提出一种基于不同尺度的双通道3维卷积神经网络的高光谱图像分类模型,来提取高光谱图像的本质空谱特征。方法 通过对高光谱图像的每一像元及其邻域像元进行旋转、行列变换等操作,丰富中心像元的潜在空间信息,达到数据集扩充的作用。将扩充之后的像素块输入到不同尺度的双通道3维卷积神经网络学习训练集的深层特征,实现更高精度的分类。结果 5次重复实验后取平均的结果表明,在随机选取了10%训练样本并通过8倍数据扩充的情况下,Indian Pines数据集实现了98.34%的总体分类精度,Pavia University数据集总体分类精度达到99.63%,同时对比了不同算法的运行时间,在保证分类精度的前提下,本文算法的运行时间短于对比算法,保证了分类模型的稳定性、高效性。结论 本文提出的基于双通道卷积神经网络的高光谱图像分类模型,既解决了训练样本不足的问题,又综合了高光谱图像的光谱特征和空间特征,提高了高光谱图像的分类精度。     

一种新型的神经网络集成模型

神经网络集成作为神经网络技术的延伸,被广泛的用于解决分类问题。很多实际应用表明:神经网络集成表现出比单个神经网络更好的性能。而传统的神经网络集成模型中网络的构建和集成是分两个阶段完成的。论文提出一种新的神经网络集成结构模型“层状集成”。该模型中网络的构建和集成同时完成,且每个成员网络的输出流入到下一个神经网络,作为下一个神经网络的输入,以这种方式生成一种层状神经网络集成。该模型用于解决分类问题,表现出比传统神经网络集成更好的性能。     

Constructive Backpropagation for Recurrent Networks

Choosing a network size is a difficult problem in neural network modelling. In many recent studies, constructive or destructive methods that add or delete connections, neurons or layers have been studied in order to solve this problem. In this work we consider the constructive approach, which is in many cases a very computationally efficient approach. In particular, we address the construction of recurrent networks by the use of constructive backpropagation. The benefits of the proposed scheme are firstly that fully recurrent networks with an arbitrary number of layers can be constructed efficiently. Secondly, after the network has been constructed we can continue the adaptation of the network weights as well as we can of its structure. This includes both addition and deletion of neurons/layers in a computationally efficient manner. Thus, the investigated method is very flexible compared to many previous methods. In addition, according to our time series prediction experiments, the proposed method is competitive in terms of modelling performance and training time compared to the well-known recurrent cascade-correlation method.     

A new genetic feature selection with neural network ensemble

A neural network ensemble is a learning paradigm in which a finite collection of neural networks is trained for the same task. Ensembles generally show better classification and generalization performance than a single neural network does. In this paper, a new feature selection method for a neural network ensemble is proposed for pattern classification. The proposed method selects an adequate feature subset for each constituent neural network of the ensemble using a genetic algorithm. Unlike the conventional feature selection method, each neural network is only allowed to have some (not all) of the considered features. The proposed method can therefore be applied to huge-scale feature classification problems. Experiments are performed with four databases to illustrate the performance of the proposed method.     

图像分类的深度卷积神经网络模型综述

图像分类是计算机视觉中的一项重要任务,传统的图像分类方法具有一定的局限性。随着人工智能技术的发展,深度学习技术越来越成熟,利用深度卷积神经网络对图像进行分类成为研究热点,图像分类的深度卷积神经网络结构越来越多样,其性能远远好于传统的图像分类方法。本文立足于图像分类的深度卷积神经网络模型结构,根据模型发展和模型优化的历程,将深度卷积神经网络分为经典深度卷积神经网络模型、注意力机制深度卷积神经网络模型、轻量级深度卷积神经网络模型和神经网络架构搜索模型等4类,并对各类深度卷积神经网络模型结构的构造方法和特点进行了全面综述,对各类分类模型的性能进行了对比与分析。虽然深度卷积神经网络模型的结构设计越来越精妙,模型优化的方法越来越强大,图像分类准确率在不断刷新的同时,模型的参数量也在逐渐降低,训练和推理速度不断加快。然而深度卷积神经网络模型仍有一定的局限性,本文给出了存在的问题和未来可能的研究方向,即深度卷积神经网络模型主要以有监督学习方式进行图像分类,受到数据集质量和规模的限制,无监督式学习和半监督学习方式的深度卷积神经网络模型将是未来的重点研究方向之一;深度卷积神经网络模型的速度和资源消耗仍不尽人意,应用于移动式设备具有一定的挑战性;模型的优化方法以及衡量模型优劣的度量方法有待深入研究;人工设计深度卷积神经网络结构耗时耗力,神经架构搜索方法将是未来深度卷积神经网络模型设计的发展方向。     

Coevolutionary learning of neural network ensemble for complex classification tasks

Ensemble approaches to classification have attracted a great deal of interest recently. This paper presents a novel method for designing the neural network ensemble using coevolutionary algorithm. The bootstrap resampling procedure is employed to obtain different training subsets that are used to estimate different component networks of the ensemble. Then the cooperative coevolutionary algorithm is developed to optimize the ensemble model via the divide-and-cooperative mechanism. All component networks are coevolved in parallel in the scheme of interacting co-adapted subpopulations. The fitness of an individual from a particular subpopulation is assessed by associating it with the representatives from other subpopulations. In order to promote the cooperation of all component networks, the proposed method considers both the accuracy and the diversity among the component networks that are evaluated using the multi-objective Pareto optimality measure. A hybrid output-combination method is designed to determine the final ensemble output. Experimental results illustrate that the proposed method is able to obtain neural network ensemble models with better classification accuracy in comparison with currently popular ensemble algorithms.     

基于深度学习的网络流时空特征自动提取方法

流量异常检测是网络入侵检测的主要途径之一,也是网络安全领域的一个热门研究方向。通过对网络流量进行实时监控,可及时有效地对网络异常进行预警。目前,网络流量异常检测方法主要分为基于规则和基于特征工程的方法,但现有方法需针对网络流量特征的变化需重新人工收集规则或 构造特征,工作量大且繁杂。为解决上述问题,该文提出一种基于卷积神经网络和循环神经网络的深度学习方法来自动提取网络流量的时空特征,可同时提取不同数据包之间的时序特征和同一数据包内字节流的空间特征,并减少了大量的人工工作。在 MAWILab 网络轨迹数据集上进行的验证分析结果表明,该文所提出的网络流时空特征提取方法优于已有的深度表示学习方法。