Structural Synthesis of Modular Weakly Connected Neural Networks. II. Nuclear Neural Networks

The simplest modular neural network is investigated, in which each module is a single-layer perceptron (a neural kernel). Numerical indices of quality by the criteria of operating speed and plasticity are proposed to characterize structural properties of nuclear neural networks. Based on the indices introduced, optimum procedures of structural synthesis are constructed.     

一种分式过程神经元网络及其应用研究

针对带有奇异值复杂时变信号的模式分类和系统建模问题，提出了一种分式过程神经元网络．该模型是基于有理式函数具有的对复杂过程信号的逼近性质和过程神经元网络对时变信息的非线性变换机制构建的。其基本信息处理单元由两个过程神经元成对偶组成。逻辑上构成一个分式过程神经元，是人工神经网络在结构和信息处理机制上的一种扩展．分析了分式过程神经元网络的连续性和泛函数逼近能力，给出了基于函数正交基展开的学习算法．实验结果表明，分式过程神经元网络对于带有奇异值时变函数样本的学习性质和泛化性质要优于BP网络和一般过程神经元网络。网络隐层数和节点数可较大减少，且算法的学习性质与传统BP算法相同．     

记忆神经网络的研究与发展

首先,根据记忆神经网络训练形式的不同,介绍了强监督模型和弱监督模型的结构特征和各自应用场景以及处理方式,总结了两类主要模型的优缺点;随后,对两类模型的发展和应用（包括模型创新和应用创新）进行了简要综述,总结了各类新模型在处理自然语言过程中所起的关键作用;最后梳理了记忆神经网络处理自然语言所面临的复杂性挑战,并预测了记忆神经网络未来的发展方向.     

基于小波网络和多模块网络的数字识别

本文研究一种新的数字识别方法,这种方法用小波神经网络抽取特征、用多模块结构神经网络作模式分类器。小波分解的函数近似能力和人工神经网络的学习能力结合起来形成的小波神经网络,有着良好的特征描述性能,可用作特征抽取工具。多模块结构的神经网络将一个k类的模式分类问题转换为k个互相独立的2类分类问题。这种结构将一个复杂的分类问题化解为多个简单的分类问题,各个模块互相并联,各自负责一种模式的识别。用这种修改过的多模块结构网络的BP训练方法,可加速训练和提高训练精度,并且各模块可互相独立地进行训练。用美国NIST数字样本进行训练及测试,结果良好。这种方法可用于更广泛的平面图形识别。     

Structural Properties of Recurrent Neural Networks

In this article we research the impact of the adaptive learning process of recurrent neural networks (RNN) on the structural properties of the derived graphs. A trained fully connected RNN can be converted to a graph by defining edges between pairs od nodes having significant weights. We measured structural properties of the derived graphs, such as characteristic path lengths, clustering coefficients and degree distributions. The results imply that a trained RNN has significantly larger clustering coefficient than a random network with a comparable connectivity. Besides, the degree distributions show existence of nodes with a large degree or hubs, typical for scale-free networks. We also show analytically and experimentally that this type of degree distribution has increased entropy.     

全连接回归神经网络的稳定性分析

讨论全连接回归神经网络的稳定性,具体来说,就是分析网络输入和网络权值的扰动对网络输出的影响,得到了上述扰动下,网络输出误差不累积的条件,即整个网络稳定的分析。     

小波神经网络学习的结构风险最小化方法

针对大噪声、小样本情形下神经网络学习的外推能力弱这一突出的问题，根据统计学习理论中结构风险最小化准则的基本原理，提出了一种基于小波神经基元频率谱分布的小波神经网络阵列结构和基于小波多分辨逼近、综合风险分析的小波网络学习算法．该方法充分发挥了小波神经网络的优点，理论基础可靠，实际意义明确，算法实现简便，自适应性强．仿真实验结果和应用实例说明了该方法对于非线性系统在线辨识的有效性，同时也为统计学习理论的工程应用提供了新的途径．     

Efficient Training of Multi-Layer Neural Networks to Achieve Faster Validation

Artificial neural networks (ANNs) are one of the hottest topics in computer science and artificial intelligence due to their potential and advantages in analyzing real-world problems in various disciplines, including but not limited to physics, biology, chemistry, and engineering. However, ANNs lack several key characteristics of biological neural networks, such as sparsity, scale-freeness, and small-worldness. The concept of sparse and scale-free neural networks has been introduced to fill this gap. Network sparsity is implemented by removing weak weights between neurons during the learning process and replacing them with random weights. When the network is initialized, the neural network is fully connected, which means the number of weights is four times the number of neurons. In this study, considering that a biological neural network has some degree of initial sparsity, we design an ANN with a prescribed level of initial sparsity. The neural network is tested on handwritten digits, Arabic characters, CIFAR-10, and Reuters newswire topics. Simulations show that it is possible to reduce the number of weights by up to 50% without losing prediction accuracy. Moreover, in both cases, the testing time is dramatically reduced compared with fully connected ANNs.     

一种基于混合建模的模块化神经网络控制方法

针对传统神经网络用于复杂过程系统的控制时难于收敛的问题,文章提出了基于混合建模的模块化的神经网络模型。采取运行机理建模和神经网络建模相结合的方式,把输入样本空间进行划分,实现基于混合专家网络的建模。试验结果表明,对大型燃煤锅炉供热系统,文章提出的方法可以较好地提高供热系统的稳定性和供热质量。     

基于多重结构神经网络的歼击机结构故障检测

提出了一种基于多重结构神经网络的故障检测方法。针对以歼击机为代表的非线性系统中存在的突发结构故障，构造了一个多重结构神经网络，在输入层对残差信号进行二进离散小波变换，提取其在多尺度下的细节系数作为故障特征向量，并将其输入到神经网络分类器进行相应的模式分类，之后再利用下一级的故障度辨识神经网络对故障的大小进行辨识。仿真结果表明，本文方法为歼击机组合结构故障的检测提供了有效的方法和途径。     

MIGA, A Software Tool for Nonlinear System Modelling with Modular Neural Networks

This paper presents a software tool suitable for dynamic system modelling. The models generated by this tool are modular neural networks, see [1]. Each module behaves like a functional block and is connected to the other modules like in classical block diagrams. This tool allows the inclusion of a priori knowledge and, furthermore, to extract physical information from the models, once the system has learned. The modelling tool is capable of automatic model generation, parameter estimation and model validation.     

Evolutionary Learning of Modular Neural Networks with Genetic Programming

Evolutionary design of neural networks has shown a great potential as a powerful optimization tool. However, most evolutionary neural networks have not taken advantage of the fact that they can evolve from modules. This paper presents a hybrid method of modular neural networks and genetic programming as a promising model for evolutionary learning. This paper describes the concepts and methodologies for the evolvable model of modular neural networks, which might not only develop new functionality spontaneously, but also grow and evolve its own structure autonomously. We show the potential of the method by applying an evolved modular network to a visual categorization task with handwritten digits. Sophisticated network architectures as well as functional subsystems emerge from an initial set of randomly-connected networks. Moreover, the evolved neural network has reproduced some of the characteristics of natural visual system, such as the organization of coarse and fine processing of stimuli in separate pathways.     

前向神经网络的容错性问题

在讨论神经网络的容错性的文献中,主要涉及的一直是关于输入噪音的容错问题.在这些文献中通常把该问题转换为某种优化问题,并用现成的优化方法进行求解,但很少涉及由网络故障所引起的容错问题,即结构容错问题.利用覆盖算法分析结构容错问题,给出一个神经网络容纳所有单节点故障的充要条件和构造这种网络的算法.这些结果揭示了神经网络结构容错能力的本质,并提供了一种分析神经网络容错的新方法.     

Application of Genetic Algorithms and Systems of Generating Graphs for Creation of Modular Neural Networks

There are many methods based on joint use of genetic algorithms and neural networks. In the majority of these methods, the architecture of the network and/or its weights are coded into chromosomes directly, which results in a huge number of chromosomes and increases their dimensions. From this point of view, methods based on coding rules that generate networks are highly promising. In this paper, a method of creation of architectures of modular neural networks based on the use of grammatical systems for fractal generation is considered. Genetic algorithms are used as an optimizer of the neural architectures obtained.     

Pinning Stabilization of Linearly Coupled Stochastic Neural Networks via Minimum Number of Controllers

Pinning stabilization problem of linearly coupled stochastic neural networks (LCSNNs) is studied in this paper. A minimum number of controllers are used to force the LCSNNs to the desired equilibrium point by fully utilizing the structure of the network. In order to pinning control the LCSNNs to a certain desired state, only one controller is required for strongly connected network topology, and m controllers, which will be shown to be the minimum number, are needed for LCSNNs with m -reducible coupling matrix. The isolate node of the LCSNNs can be stable, periodic, or even chaotic. The coupling Laplacian matrix of the LCSNNs can be symmetric irreducible, asymmetric irreducible, or m-reducible, which means that the network topology can be strongly connected, weakly connected, or even unconnected. There is no constraint on the network topology. Some criteria are derived to judge whether the LCSNNs can be controlled in mean square by using designed controllers. The given criteria are expressed in terms of strict linear matrix inequalities, which can be easily checked by resorting to recently developed algorithm. Moreover, numerical examples including small-world and scale-free networks are also given to demonstrate that our theoretical results are valid and efficient for large systems.     

离散时间反馈神经网络的渐近行为研究

Hopfield神经网络是一类应用非常成功的人工神经网络模型，它是研究这个反馈神经网络的基础．该文主要研究离散时间、连续状态的反馈神经网络，它是Hopfield神经网络的推广．众所周知，研究反馈神经网络的稳定性不仅被认为是神经网络最基本、最主要的问题之一，同时也是神经网络各种应用的基础．文中主要研究离散时间反馈神经网络的稳定性，给出了连接权矩阵非对称的并且输入-输出函数是一般的S-函数的新的渐近收敛性条件及相应的收敛性结论．所获结果不仅推广了一些已有的结论，而且为反馈神经网络的应用提供了一定的理论基础．     

Single-Iteration Training Algorithm for Multi-Layer Feed-Forward Neural Networks

A new methodology for neural learning is presented. Only a single iteration is needed to train a feed-forward network with near-optimal results. This is achieved by introducing a key modification to the conventional multi-layer architecture. A virtual input layer is implemented, which is connected to the nominal input layer by a special nonlinear transfer function, and to the first hidden layer by regular (linear) synapses. A sequence of alternating direction singular value decompositions is then used to determine precisely the inter-layer synaptic weights. This computational paradigm exploits the known separability of the linear (inter-layer propagation) and nonlinear (neuron activation) aspects of information transfer within a neural network. Examples show that the trained neural networks generalize well.     

具有FIR突触的积单元神经网络预测时间序列

提出一种具有有限脉冲响应(FIR)突触的积单元神经网络(PUNN)结构，并用于预测混沌时间序列。这种神经网络结构既继承了标准PUNN的结构简单、信息存储能力强的优点，又更适合预测混沌时间序列，特别是在小的学习样本情况。分别用具有FIR突触的PUNN、标准PUNN以及模糊神经网络(FNN)等3种神经网络对小的样本混沌时间序列做了1步多步预测对比实验。结果显示具有FIR突触的PUNN比其他2种神经网络预测精度都高。这说明具有FIR突触的PUNN是预测小学习样本时间序列的一种有效方法。     

Evolutionary Multi-task Learning for Modular Knowledge Representation in Neural Networks

The brain can be viewed as a complex modular structure with features of information processing through knowledge storage and retrieval. Modularity ensures that the knowledge is stored in a manner where any complications in certain modules do not affect the overall functionality of the brain. Although artificial neural networks have been very promising in prediction and recognition tasks, they are limited in terms of learning algorithms that can provide modularity in knowledge representation that could be helpful in using knowledge modules when needed. Multi-task learning enables learning algorithms to feature knowledge in general representation from several related tasks. There has not been much work done that incorporates multi-task learning for modular knowledge representation in neural networks. In this paper, we present multi-task learning for modular knowledge representation in neural networks via modular network topologies. In the proposed method, each task is defined by the selected regions in a network topology (module). Modular knowledge representation would be effective even if some of the neurons and connections are disrupted or removed from selected modules in the network. We demonstrate the effectiveness of the method using single hidden layer feedforward networks to learn selected n-bit parity problems of varying levels of difficulty. Furthermore, we apply the method to benchmark pattern classification problems. The simulation and experimental results, in general, show that the proposed method retains performance quality although the knowledge is represented as modules.     

基于新型动态神经元网络的逆系统方法

针对提高逆系统建模中神经网络的逼近效果和动态性能问题,根据PID神经元网络工作原理,提出一种具有动态激励函数的新型PID神经元模型—输出反馈型PID神经元(OFPID),输出激励采用连续的Sigmoidal函数,使神经元具有等效的IIR突触,采用梯度下降法实现OFPID神经元网络的权值调整,将其应用于非线性系统的神经网络逆控制系统,从而提高非线性系统的解耦效果和控制性能。仿真实验证明,提出的新型神经元网络是一种良好的非线性系统建模和控制工具。