Constructive learning of recurrent neural networks: limitations ofrecurrent cascade correlation and a simple solution

It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a "preliminary" approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.     

Entropy-based generation of supervised neural networks for classification of structured patterns

Sperduti and Starita proposed a new type of neural network which consists of generalized recursive neurons for classification of structures. In this paper, we propose an entropy-based approach for constructing such neural networks for classification of acyclic structured patterns. Given a classification problem, the architecture, i.e., the number of hidden layers and the number of neurons in each hidden layer, and all the values of the link weights associated with the corresponding neural network are automatically determined. Experimental results have shown that the networks constructed by our method can have a better performance, with respect to network size, learning speed, or recognition accuracy, than the networks obtained by other methods.     

A cooperative constructive method for neural networks for pattern recognition

In this paper, we propose a new constructive method, based on cooperative coevolution, for designing automatically the structure of a neural network for classification. Our approach is based on a modular construction of the neural network by means of a cooperative evolutionary process. This process benefits from the advantages of coevolutionary computation as well as the advantages of constructive methods. The proposed methodology can be easily extended to work with almost any kind of classifier.The evaluation of each module that constitutes the network is made using a multiobjective method. So, each new module can be evaluated in a comprehensive way, considering different aspects, such as performance, complexity, or degree of cooperation with the previous modules of the network. In this way, the method has the advantage of considering not only the performance of the networks, but also other features.The method is tested on 40 classification problems from the UCI machine learning repository with very good performance. The method is thoroughly compared with two other constructive methods, cascade correlation and GMDH networks, and other classification methods, namely, SVM, C4.5, and k nearest-neighbours, and an ensemble of neural networks constructed using four different methods.     

SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons

Many biological neural network models face the problem of scalability because of the limited computational power of today's computers. Thus, it is difficult to assess the efficiency of these models to solve complex problems such as image processing. Here, we describe how this problem can be tackled using event-driven computation. Only the neurons that emit a discharge are processed and, as long as the average spike discharge rate is low, millions of neurons and billions of connections can be modelled. We describe the underlying computation and implementation of such a mechanism in SpikeNET, our neural network simulation package. The type of model one can build is not only biologically compliant, it is also computationally efficient as 400 000 synaptic weights can be propagated per second on a standard desktop computer. In addition, for large networks, we can set very small time steps (< 0.01 ms) without significantly increasing the computation time. As an example, this method is applied to solve complex cognitive tasks such as face recognition in natural images.     

A constructive algorithm for binary neural networks: the oil-spot algorithm.

This paper presents a constructive training algorithm for supervised neural networks. The algorithm relies on a topological approach, based on the representation of the mapping of interest onto the binary hypercube of the input space. It dynamically constructs a two-layer neural network by involving successively binary examples. A convenient treatment of real-valued data is possible by means of a suitable real-to-binary codification. In the case of target functions that have efficient halfspace union representations, simulations show the constructed networks result optimized in terms of number of neurons.     

一种多权值神经元覆盖网络的构造方法

仿生模式识别利用多权值神经元覆盖网络构造模式类的覆盖来进行相应事物的识别.但在构造多权值神经元覆盖网络的过程中,关于构造神经元个数的确定方法没有相关讨论,即需要使用多少个神经元才能完成对模式类的覆盖.较多的神经元在精确的对模式类进行覆盖同时,也增大了网络的复杂度.文中提出了一种多权值神经元覆盖网络的构造方法.在保持神经网络对模式类的覆盖能力的基础上采用尽量少的神经元,从而能有效的降低神经网络构造代价.最后,通过实验作者验证了算法的有效性.     

Controlling activity fluctuations in large, sparsely connected random networks

Controlling activity in recurrent neural network models of brain regions is essential both to enable effective learning and to reproduce the low activities that exist in some cortical regions such as hippocampal region CA3. Previous studies of sparse, random, recurrent networks constructed with McCulloch-Pitts neurons used probabilistic arguments to set the parameters that control activity. Here, we extend this work by adding an additional, biologically appropriate, parameter to control the magnitude and stability of activity oscillations. The new constant can be considered to be the rest conductance in a shunting model or the threshold when subtractive inhibition is used. This new parameter is critical for large networks run at low activity levels. Importantly, extreme activity fluctuations that act to turn large networks totally on or totally off can now be avoided. We also show how the size of external input activity interacts with this parameter to affect network activity. Then the model based on fixed weights is extended to estimate activities in networks with distributed weights. Because the theory provides accurate control of activity fluctuations, the approach can be used to design a predictable amount of pseudorandomness into deterministic networks. Such nonminimal fluctuations improve learning in simulations trained on the transitive inference problem.     

基于LMBP改进算法的神经网络结构优化

提出了一种基于增长法的神经网络结构优化算法。在函数逼近的BP神经网络中引入一种改进的BP算法(LMBP算法),通过二次误差下降与梯度下降,利用误差变化规律分析网络结构的优化程度,自适应地增加隐层神经元或网络层次,从而得到一个合适的网络结构。进行了仿真实验及该算法与RAN算法用于逼近函数的对比实验,实验结果表明了该算法的有效性。     

Recurrent neural network approach for cyclic job shop scheduling problem

While cyclic scheduling is involved in numerous real-world applications, solving the derived problem is still of exponential complexity. This paper focuses specifically on modelling the manufacturing application as a cyclic job shop problem and we have developed an efficient neural network approach to minimise the cycle time of a schedule. Our approach introduces an interesting model for a manufacturing production, and it is also very efficient, adaptive and flexible enough to work with other techniques. Experimental results validated the approach and confirmed our hypotheses about the system model and the efficiency of neural networks for such a class of problems.     

Solving the problem of negative synaptic weights in cortical models

In cortical neural networks, connections from a given neuron are either inhibitory or excitatory but not both. This constraint is often ignored by theoreticians who build models of these systems. There is currently no general solution to the problem of converting such unrealistic network models into biologically plausible models that respect this constraint. We demonstrate a constructive transformation of models that solves this problem for both feedforward and dynamic recurrent networks. The resulting models give a close approximation to the original network functions and temporal dynamics of the system, and they are biologically plausible. More precisely, we identify a general form for the solution to this problem. As a result, we also describe how the precise solution for a given cortical network can be determined empirically.     

A genetic approach to automatic neural network architecture optimization

The use of artificial neural networks for various problems has provided many benefits in various fields of research and engineering. Yet, depending on the problem, different architectures need to be developed and most of the time the design decision relies on a trial and error basis as well as on the experience of the developer. Many approaches have been investigated concerning the topology modelling, training algorithms, data processing. This paper proposes a novel automatic method for the search of a neural network architecture given a specific task. When selecting the best topology, our method allows the exploration of a multidimensional space of possible structures, including the choice of the number of neurons, the number of hidden layers, the types of synaptic connections, and the use of transfer functions. Whereas the backpropagation algorithm is being conventionally used in the field of neural networks, one of the known disadvantages of the technique represents the possibility of the method to reach saddle points or local minima, hence overfitting the output data. In this work, we introduce a novel strategy which is capable to generate a network topology with overfitting being avoided in the majority of the cases at affordable computational cost. In order to validate our method, we provide several numerical experiments and discuss the outcomes.

     

Multifeedback-Layer Neural Network

The architecture and training procedure of a novel recurrent neural network (RNN), referred to as the multifeedback-layer neural network (MFLNN), is described in this paper. The main difference of the proposed network compared to the available RNNs is that the temporal relations are provided by means of neurons arranged in three feedback layers, not by simple feedback elements, in order to enrich the representation capabilities of the recurrent networks. The feedback layers provide local and global recurrences via nonlinear processing elements. In these feedback layers, weighted sums of the delayed outputs of the hidden and of the output layers are passed through certain activation functions and applied to the feedforward neurons via adjustable weights. Both online and offline training procedures based on the backpropagation through time (BPTT) algorithm are developed. The adjoint model of the MFLNN is built to compute the derivatives with respect to the MFLNN weights which are then used in the training procedures. The Levenberg-Marquardt (LM) method with a trust region approach is used to update the MFLNN weights. The performance of the MFLNN is demonstrated by applying to several illustrative temporal problems including chaotic time series prediction and nonlinear dynamic system identification, and it performed better than several networks available in the literature     

Pruning recurrent neural networks for improved generalizationperformance

Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.     

多元多项式函数的三层前向神经网络逼近方法

该文首先用构造性方法证明:对任意r阶多元多项式,存在确定权值和确定隐元个数的三层前向神经网络.它能以任意精度逼近该多项式.其中权值由所给多元多项式的系数和激活函数确定,而隐元个数由r与输入变量维数确定.作者给出算法和算例,说明基于文中所构造的神经网络可非常高效地逼近多元多项式函数.具体化到一元多项式的情形,文中结果比曹飞龙等所提出的网络和算法更为简单、高效;所获结果对前向神经网络逼近多元多项式函数类的网络构造以及逼近等具有重要的理论与应用意义,为神经网络逼近任意函数的网络构造的理论与方法提供了一条途径.     

基于特征可视化分析深度神经网络的内部表征

基于可视化的方式理解深度神经网络能直观地揭示其工作机理,即提供了黑盒模型做出决策的解释,在医疗诊断、自动驾驶等领域尤其重要。大部分现有工作均基于激活值最大化框架,即选定待观测神经元,通过优化输入值(如隐藏层特征图谱、原始图片),定性地将待观测神经元产生最大激活值时输入值的改变作为一种解释。然而,这种方法缺乏对深度神经网络深入的定量分析。文中提出了结构可视化和基于规则可视化两种可视化的元方法。结构可视化从浅至深依层可视化,发现浅层神经元具有一般性的全局特征,而深层神经元更针对细节特征。基于规则可视化包括交集与差集规则,可以帮助发现共享神经元与抑制神经元的存在,它们分别学习了不同类别的共有特征与抑制不相关的特征。实验针对代表性卷积网络VGG和残差网络ResNet在ImageNet和微软COCO数据集上进行了分析。通过量化分析发现,ResNet和VGG均有很高的稀疏性,通过屏蔽一些低激活值的“噪音”神经元,发现其对深度神经网络分类准确率均没有影响,甚至有一定程度的提高作用。文中通过可视化和量化分析深度神经网络的隐藏层特征,揭示其内部特征表达,从而为高性能深度神经网络的设计提供指导和借鉴。     

Simulations of construction learning for neuron-computer resources

It is well known that information processing in the brain depends on neuron systems. Simple neuron systems are neural networks, and their learning methods have been studied. However, we believe that research on large-scale neural network systems is still incomplete. Here, we propose a learning method for millions of neurons as resources for a neuron computer. The method is a type of recurrent path-selection, so the neural network objective must have nesting structures. This method is executed at high speed. When information processing is executed by analogue signals, the accumulation of errors is a grave problem. We equipped a neural network with a digitizer and AD/DA (Analogue Digital) converters constructed of neurons. They retain all information signals and guarantee precision in complex operations. By using these techniques, we generated an image shifter constructed of 8.6 million neurons. We believe that there is the potential to design a neuron computer using this scheme. This work was presented in part at the Fifth International Symposium on Artificial Life and Robotics, Oita, Japan, January 26–28, 2000     

Estimation of all-terminal network reliability using an artificial neural network

The exact calculation of all-terminal network reliability is an NP-hard problem, with computational effort growing exponentially with the number of nodes and links in the network. During optimal network design, a huge number of candidate topologies are typically examined with each requiring a network reliability calculation. Because of the impracticality of calculating all-terminal network reliability for networks of moderate to large size, Monte Carlo simulation methods to estimate network reliability and upper and lower bounds to bound reliability have been used as alternatives. This paper puts forth another alternative to the estimation of all-terminal network reliability — that of artificial neural network (ANN) predictive models. Neural networks are constructed, trained and validated using the network topologies, the link reliabilities, and a network reliability upperbound as inputs and the exact network reliability as the target. A hierarchical approach is used: a general neural network screens all network topologies for reliability followed by a specialized neural network for highly reliable network designs. Both networks with identical link reliability and networks with varying link reliability are studied. Results, using a grouped cross-validation approach, show that the ANN approach yields more precise estimates than the upperbound, especially in the worst cases. Using the reliability estimation methods of the ANN, the upperbound and backtracking, optimal network design by simulated annealing is considered. Results show that the ANN regularly produces superior network designs at a reasonable computational cost.Scope and purposeAn important application area of operations research is the design of structures, products or systems where both technical and business aspects must be considered. One expanding design domain is the design of computer or communications networks. While cost is a prime consideration, reliability is equally important. A common reliability measure is all-terminal reliability, the probability that all nodes (computers or terminals) on the network can communicate with all others. Exact calculation of all-terminal reliability is an NP-hard problem, precluding its use during optimal network topology design, where this calculation must be made thousands or millions of times. This paper presents a novel computationally practical method for estimating all-terminal network reliability. Is shown how a neural network can be used to estimate all-terminal network reliability by using the network topology, the link reliabilities and an upperbound on all-terminal network reliability as inputs. The neural network is trained and validated on a very minute fraction of possible network topologies, and once trained, it can be used without restriction during network design for a topology of a fixed number of nodes. The trained neural network is extremely fast computationally and can accommodate a variety of network design problems. The neural network approach, an upper bound approach and an exact backtracking calculation are compared for network design using simulated annealing for optimization and show that the neural network approach yields superior designs at manageable computational cost.     

Rate and synchrony in feedforward networks of coincidence detectors: analytical solution

We provide an analytical recurrent solution for the firing rates and cross-correlations of feedforward networks with arbitrary connectivity, excitatory or inhibitory, in response to steady-state spiking input to all neurons in the first network layer. Connections can go between any two layers as long as no loops are produced. Mean firing rates and pairwise cross-correlations of all input neurons can be chosen individually. We apply this method to study the propagation of rate and synchrony information through sample networks to address the current debate regarding the efficacy of rate codes versus temporal codes. Our results from applying the network solution to several examples support the following conclusions: (1) differential propagation efficacy of rate and synchrony to higher layers of a feedforward network is dependent on both network and input parameters, and (2) previous modeling and simulation studies exclusively supporting either rate or temporal coding must be reconsidered within the limited range of network and input parameters used. Our exact, analytical solution for feedforward networks of coincidence detectors should prove useful for further elucidating the efficacy and differential roles of rate and temporal codes in terms of different network and input parameter ranges.     

A PDP constructive algorithm for system modelling

A new constructive algorithm is presented for building neural networks that learn to reproduce output temporal sequences based on one or several input sequences. This algorithm builds a network for the task of system modelling, dealing with continuous variables in the discrete time domain. The constructive scheme makes it user independent. The network's structure consists of an ordinary set and a classification set, so it is a hybrid network like that of Stokbro et al. [6], but with a binary classification. The networks can easily be interpreted, so the learned representation can be transferred to a human engineer, unlike many other network models. This allows for a better understanding of the system structure than just its simulation. This constructive algorithm limits the network complexity automatically, hence preserving extrapolation capabilities. Examples with real data from three totally different sources show good performance and allow for a promising line of research.