Cognitive engineering based knowledge representation in neural networks

Abstract

A model of a human neural knowledge processing system is presented that suggests the following. First, an entity in the outside world lends to be locally encoded in neural networks so that the conceptual information structure is mirrored in its physical implementation. Second, the knowledge of problem solving is implemented in a quite implicit way in the internal structure of the neural network (a functional group of associated hidden neurons and their connections to entity neurons) not in individual neurons or connections. Third, the knowledge system is organized and implemented in a modular fashion in neural networks according to the local specialization of problem solving where a module of neural network implements an inter-related group of knowledge such as a schema, and different modules have similar processing mechanisms, but differ in their input and output patterns. A neural network module can be tuned just as a schema structure can be adapted for changing environments. Three experiments were conducted to try to validate the suggested cognitive engineering based knowledge structure in neural networks through computer simulation. The experiments, which were based on a task of modulo arithmetic, provided some insights into the plausibility of the suggested model of a neural knowledge processing system.     

Self-organizing learning array and its application to economic and financial problems

A new self-organizing learning array (SOLAR) system has been implemented in software. It is an information theory based learning machine capable of handling a wide variety of classification problems. It has self-reconfigurable processing cells (neurons) and an evolvable system structure. Entropy based learning is performed locally at each neuron, where neural functions and connections that correspond to the minimum entropy are adaptively learned. By choosing connections for each neuron, the system sets up the wiring and completes its self-organization. SOLAR classifies input data based on weighted statistical information from all neurons. Unlike artificial neural networks, its multi-layer structure scales well to large systems capable of solving complex pattern recognition and classification tasks. This paper shows its application in economic and financial fields. A reference to influence diagrams is also discussed. Several prediction and classification cases are studied. The results have been compared with the existing methods.     

Evolutionary Multi-task Learning for Modular Knowledge Representation in Neural Networks

The brain can be viewed as a complex modular structure with features of information processing through knowledge storage and retrieval. Modularity ensures that the knowledge is stored in a manner where any complications in certain modules do not affect the overall functionality of the brain. Although artificial neural networks have been very promising in prediction and recognition tasks, they are limited in terms of learning algorithms that can provide modularity in knowledge representation that could be helpful in using knowledge modules when needed. Multi-task learning enables learning algorithms to feature knowledge in general representation from several related tasks. There has not been much work done that incorporates multi-task learning for modular knowledge representation in neural networks. In this paper, we present multi-task learning for modular knowledge representation in neural networks via modular network topologies. In the proposed method, each task is defined by the selected regions in a network topology (module). Modular knowledge representation would be effective even if some of the neurons and connections are disrupted or removed from selected modules in the network. We demonstrate the effectiveness of the method using single hidden layer feedforward networks to learn selected n-bit parity problems of varying levels of difficulty. Furthermore, we apply the method to benchmark pattern classification problems. The simulation and experimental results, in general, show that the proposed method retains performance quality although the knowledge is represented as modules.     

基于深度学习OFDM信道补偿技术硬件实现

为了解决部分高性能深度学习神经网络因存在复杂度高及计算量大等缺陷在嵌入式设备中应用效果不理想的问题;以小型化集成智能无线电设备AIR-T为平台实现了基于深度学习的OFDM信道补偿技术;在FPGA芯片上不仅实现了OFDM信号传输系统模块,也实现了传统信道估计与均衡模块,模块对数据进行预处理减轻神经网络工作量以完成神经网络信道补偿技术模块在Jetson TX2平台GPU上的高效实现;由实验记录神经网络训练过程中的计算复杂度和参数拟合速度得知,传统信道估计与均衡模块有效降低了网络训练时的运算次数;由测试性能方面可知,经过神经网络信道补偿后的数据误码率比之前传统信道估计与均衡后的误码率有明显降低;     

Quantum holography and neurocomputer architectures

It is generally accepted among neuroscientists that the sensory cortex of the brain is arranged in a layered structure. Based on a unified quantum holographic approach to artificial neural network models implemented with coherent, hybrid optoelectronic, or analog electronic neurocomputer architectures, the present paper establishes a novel identity for the matching polynomials of complete bichromatic graphs which implement the intrinsic connections between neurons of local networks located in neural layers.     

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Optimizing the structure of neural networks is an essential step for the discovery of knowledge from data. This paper deals with a new approach which determines the insignificant input and hidden neurons to detect the optimum structure of a feedforward neural network. The proposed pruning algorithm, called as neural network pruning by significance (N2PS), is based on a new significant measure which is calculated by the Sigmoidal activation value of the node and all the weights of its outgoing connections. It considers all the nodes with significance value below the threshold as insignificant and eliminates them. The advantages of this approach are illustrated by implementing it on six different real datasets namely iris, breast-cancer, hepatitis, diabetes, ionosphere and wave. The results show that the proposed algorithm is quite efficient in pruning the significant number of neurons on the neural network models without sacrificing the networks performance.     

TWO-LAYER STABILIZATION OF CONTINUOUS NEURAL NETWORKS WITH FEEDBACKS

Methods of stabilization as applied to Hopfield-type continuous neural networks with a unique equilibrium point are considered. These methods permit the design of stable networks where the elements of the interconnection matrix and nonlinear activation functions of separate neurons vary with time. For stabilization with a variable interconnection matrix it is suggested that a new second layer of neurons be introduced to the initial single-layer network and some additional connections be added between the new and old layers. This approach gives us a system with a unique equilibrium point that is globally asymptotically stable, i.e. the entire space serves as the domain of attraction of this point, and the stability does not depend on the interconnection matrix of the system. In the case of the variable activation functions, some results from a recent investigation of the absolute stability problem for neural networks are presented, along with some recommendations.     

Rough sets for adapting wavelet neural networks as a new classifier system

Classification is an important theme in data mining. Rough sets and neural networks are two techniques applied to data mining problems. Wavelet neural networks have recently attracted great interest because of their advantages over conventional neural networks as they are universal approximations and achieve faster convergence. This paper presents a hybrid system to extract efficiently classification rules from decision table. The neurons of such hybrid network instantiate approximate reasoning knowledge gleaned from input data. The new model uses rough set theory to help in decreasing the computational effort needed for building the network structure by using what is called reduct algorithm and a rules set (knowledge) is generated from the decision table. By applying the wavelets, frequencies analysis, rough sets and dynamic scaling in connection with neural network, novel and reliable classifier architecture is obtained and its effectiveness is verified by the experiments comparing with traditional rough set and neural networks approaches.     

自反馈神经网络的椭球学习算法

本文讨论自反馈神经网络的学习问题，指出联想记忆的神经网络的学习可以化为某种规划（优化）的问题来解，于是可借用规划数学中发展得成熟的优化技术来解自反馈神经网络的学习问题，文中给出一种称为椭球算法的学习方法，其计算复杂性是多项式型。     

Behavior control of coherent-type neural networks bycarrier-frequency modulation

Coherent-type artificial neural networks whose behavior is controlled by carrier-frequency modulation are proposed. The network learns teacher signals associated with an information-carrier frequency as a network parameter. The total network system forms a self-homodyne circuit. The learning process is realized by adjusting delay time and conductance of neural connections. Experiments demonstrate that the network behavior is successfully controlled by the carrier-frequency modulation. This result will be applicable not only to signal processing but also to frequency-multiplexed optical neural computing and quantum neural devices such as carrier-energy-controlled neurons in the future.     

Clearly defined architectures of neural networks and multilayer perceptron

Neural networks with clearly defined architecture differ in the fact that they make it possible to determine the structure of neural network (number of neurons, layers, connections) on the basis of initial parameters of recognition problem. For these networks, the value of weights determined also analytically. In this paper, we consider the problem of networks with clearly defined architecture transformation into the classical schemes of multilayer perceptron architectures. Such possibility may allow us to combine the advantages of neural networks with clearly defined architecture with the capabilities of multilayer perceptron, that eventually may enable us to speed up and simplify the process of creating and training a neural network.     

The design of a neural network with a biologically motivatedarchitecture

An associative neural network whose architecture is greatly influenced by biological data is described. The proposed neural network is significantly different in architecture and connectivity from previous models. Its emphasis is on high parallelism and modularity. The network connectivity is enriched by recurrent connections within the modules. Each module is, effectively, a Hopfield net. Connections within a module are plastic and are modified by associative learning. Connections between modules are fixed and thus not subject to learning. Although the network is tested with character recognition, it cannot be directly used as such for real-world applications. It must be incorporated as a module in a more complex structure. The architectural principles of the proposed network model can be used in the design of other modules of a whole system. Its architecture is such that it constitutes a good mathematical prototype to analyze the properties of modularity, recurrent connections, and feedback. The model does not make any contribution to the subject of learning in neural networks.     

Object detection using pulse coupled neural networks

Describes an object detection system based on pulse coupled neural networks. The system is designed and implemented to illustrate the power, flexibility and potential the pulse coupled neural networks have in real-time image processing. In the preprocessing stage, a pulse coupled neural network suppresses noise by smoothing the input image. In the segmentation stage, a second pulse coupled neural-network iteratively segments the input image. During each iteration, with the help of a control module, the segmentation network deletes regions that do not satisfy the retention criteria from further processing and produces an improved segmentation of the retained image. In the final stage each group of connected regions that satisfies the detection criteria is identified as an instance of the object of interest.     

Constructive Backpropagation for Recurrent Networks

Choosing a network size is a difficult problem in neural network modelling. In many recent studies, constructive or destructive methods that add or delete connections, neurons or layers have been studied in order to solve this problem. In this work we consider the constructive approach, which is in many cases a very computationally efficient approach. In particular, we address the construction of recurrent networks by the use of constructive backpropagation. The benefits of the proposed scheme are firstly that fully recurrent networks with an arbitrary number of layers can be constructed efficiently. Secondly, after the network has been constructed we can continue the adaptation of the network weights as well as we can of its structure. This includes both addition and deletion of neurons/layers in a computationally efficient manner. Thus, the investigated method is very flexible compared to many previous methods. In addition, according to our time series prediction experiments, the proposed method is competitive in terms of modelling performance and training time compared to the well-known recurrent cascade-correlation method.     

An efficient model of neural networks for dynamic programming

Systems based on artificial neural networks have high computational rates owing to the use of a massive number of simple processing elements and the high degree of connectivity between these elements. Neural networks with feedback connections provide a computing model capable of solving a large class of optimization problems. This paper presents a novel approach for solving dynamic programming problems using artificial neural networks. More specifically, a modified Hopfield network is developed and its internal parameters are computed using the valid-subspace technique. These parameters guarantee the convergence of the network to the equilibrium points. Simulated examples are presented and compared with other neural networks. The results demonstrate that the proposed method gives a significant improvement.     

基于健壮半径求解的循环神经网络形式化验证方法

随着软硬件技术的发展,神经网络在各行各业取得了广泛的应用,然而在应用过程中也暴露出健壮性的不足。因此,采用形式化的手段来检验和保障神经网络的安全性是至关重要的。然而,由于循环神经网络结构复杂、激活函数非线性等因素,目前关于这类神经网络的形式化验证工作非常有限。针对循环神经网络难于验证的问题,本文提出了VR-RRS,一种基于健壮半径求解的循环神经网络形式化验证方法。首先,基于神经网络验证的经典近似求解框架,通过逐层回溯迭代的方式得到循环神经网络各层神经元近似区间上下界关于输入的线性表达式,在此基础上利用赫尔德不等式推导出各层神经元的近似上下界关于扰动半径的解析解。随后,针对经典近似求解方法精度不高的问题,通过对激活函数的近似方式进行分析和优化,提出一种基于多路径回溯的求解策略。该策略以细粒度近似方法构造不同的回溯路径,在此基础上将这些回溯路径求解的近似区间取交集,能够得到更为精确的近似区间。最后,采用改进的二分法对健壮半径进行求解,主要针对经典二分法中精度不足的问题,优化了判断神经网络健壮性的标准。通过在不同结构的循环神经网络上与已有方法开展对比实验,结果表明了该方法在求解出的健壮半径和验证成功率上具有明显优势。     

知识堆叠降噪自编码器EI北大核心CSCD

深度神经网络是具有复杂结构和多个非线性处理单元的模型,广泛应用于计算机视觉、自然语言处理等领域.但是,深度神经网络存在不可解释这一致命缺陷,即“黑箱问题”,这使得深度学习在各个领域的应用仍然存在巨大的障碍.本文提出了一种新的深度神经网络模型——知识堆叠降噪自编码器(Knowledge-based stacked denoising autoencoder,KBSDAE).尝试以一种逻辑语言的方式有效解释网络结构及内在运作机理,同时确保逻辑规则可以进行深度推导.进一步通过插入提取的规则到深度网络,使KBSDAE不仅能自适应地构建深度网络模型并具有可解释和可视化特性,而且有效地提高了模式识别性能.大量的实验结果表明,提取的规则不仅能够有效地表示深度网络,还能够初始化网络结构以提高KBSDAE的特征学习性能、模型可解释性与可视化,可应用性更强.     

Evolving modular networks with genetic algorithms: application to nonlinear time series

Abstract: A key problem of modular neural networks is finding the optimal aggregation of the different subtasks (or modules) of the problem at hand. Functional networks provide a partial solution to this problem, since the inter‐module topology is obtained from domain knowledge (functional relationships and symmetries). However, the learning process may be too restrictive in some situations, since the resulting modules (functional units) are assumed to be linear combinations of selected families of functions. In this paper, we present a non‐parametric learning approach for functional networks using feedforward neural networks for approximating the functional modules of the resulting architecture; we also introduce a genetic algorithm for finding the optimal intra‐module topology (the appropriate balance of neurons for the different modules according to the complexity of their respective tasks). Some benchmark examples from nonlinear time‐series prediction are used to illustrate the performance of the algorithm for finding optimal modular network architectures for specific problems.     

基于LMBP改进算法的神经网络结构优化

提出了一种基于增长法的神经网络结构优化算法。在函数逼近的BP神经网络中引入一种改进的BP算法(LMBP算法),通过二次误差下降与梯度下降,利用误差变化规律分析网络结构的优化程度,自适应地增加隐层神经元或网络层次,从而得到一个合适的网络结构。进行了仿真实验及该算法与RAN算法用于逼近函数的对比实验,实验结果表明了该算法的有效性。     

Investigation of possible neural architectures underlying information-geometric measures

A novel analytical method based on information geometry was recently proposed, and this method may provide useful insights into the statistical interactions within neural groups. The link between informationgeometric measures and the structure of neural interactions has not yet been elucidated, however, because of the ill-posed nature of the problem. Here, possible neural architectures underlying information-geometric measures are investigated using an isolated pair and an isolated triplet of model neurons. By assuming the existence of equilibrium states, we derive analytically the relationship between the information-geometric parameters and these simple neural architectures. For symmetric networks, the first- and second-order information-geometric parameters represent, respectively, the external input and the underlying connections between the neurons provided that the number of neurons used in the parameter estimation in the log-linear model and the number of neurons in the network are the same. For asymmetric networks, however, these parameters are dependent on both the intrinsic connections and the external inputs to each neuron. In addition, we derive the relation between the information-geometric parameter corresponding to the two-neuron interaction and a conventional cross-correlation measure. We also show that the information-geometric parameters vary depending on the number of neurons assumed for parameter estimation in the log-linear model. This finding suggests a need to examine the information-geometric method carefully. A possible criterion for choosing an appropriate orthogonal coordinate is also discussed. This article points out the importance of a model-based approach and sheds light on the possible neural structure underlying the application of information geometry to neural network analysis.