Information theoretic competitive learning in self-adaptive multi-layered networks

In this paper, we propose self-adaptive multi-layered networks in which information in each processing layer is always maximized. Using these multi-layered networks, we can solve complex problems and discover salient features that single-layered networks fail to extract. In addition, this successive information maximization enables networks gradually to extract important features. We applied the new method to the Iris data problem, the vertical-horizontal lines detection problem, a phonological data analysis problem and a medical data problem. Experimental results confirmed that information can repeatedly be maximized in multi-layered networks and that the networks can extract features that cannot be detected by single-layered networks. In addition, features extracted in successive layers are cumulatively combined to detect more macroscopic features.     

Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

In this paper, we proopose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition , because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This proceess continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results confirmed that our new method can detect salient features in input patterns more clearly than the other methods.     

Improving feature extraction performance of greedy network-growing algorithm by inverse euclidean distance

We propose here a new computational method for the information-theoretic method, called the greedy network-growing algorithm, to facilitate a process of information acquisition. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function are not very sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily faithfully describe input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance becomes smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than in the previous method with the sigmoidal activation function. We applied the new method to artificial data analysis and animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.     

Flexible feature discovery and structural information control

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.     

Information Content in Neural Net Optimization

Reduction in the size and complexity of neural networks is essential to improve generalization, reduce training error and improve network speed. Most of the known optimization methods heavily rely on weight-sharing concepts for pattern separation and recognition. In weight-sharing methods the redundant weights from specific areas of input layer are pruned and the value of weights and their information content play a very minimal role in the pruning process. The method presented here focuses on network topology and information content for optimization. We have studied the change in the network topology and its effects on information content dynamically during the optimization of the network. The primary optimization uses scaled conjugate gradient and the secondary method of optimization is a Boltzmann method. The conjugate gradient optimization serves as a connection creation operator and the Boltzmann method serves as a competitive connection annihilation operator. By combining these two methods, it is possible to generate small networks which have similar testing and training accuracy, i.e. good generalization, from small training sets. In this paper, we have also focused on network topology. Topological separation is achieved by changing the number of connections in the network. This method should be used when the size of the network is large enough to tackle real-life problems such as fingerprint classification. Our findings indicate that for large networks, topological separation yields a smaller network size, which is more suitable for VLSI implementation. Topological separation is based on the error surface and information content of the network. As such it is an economical way of reducing size, leading to overall optimization. The differential pruning of the connections is based on the weight content rather than the number of connections. The training error may vary with the topological dynamics but the correlation between the error surface and recognition rate decreases to a minimum. Topological separation reduces the size of the network by changing its architecture without degrading its performance,     

Learn more by training less: systematicity in sentence processing by recurrent networks

Connectionist models of sentence processing must learn to behave systematically by generalizing from a small training set. To what extent recurrent neural networks manage this generalization task is investigated. In contrast to Van der Velde et al. (Connection Sci., 16, pp. 21–46, 2004), it is found that simple recurrent networks do show so-called weak combinatorial systematicity, although their performance remains limited. It is argued that these limitations arise from overfitting in large networks. Generalization can be improved by increasing the size of the recurrent layer without training its connections, thereby combining a large short-term memory with a small long-term memory capacity. Performance can be improved further by increasing the number of word types in the training set.     

Novelty-dependent learning and topological mapping

Unsupervised topological ordering, similar to Kohonen's (1982, Biological Cybernetics, 43: 59-69) self-organizing feature map, was achieved in a connectionist module for competitive learning (a CALM Map) by internally regulating the learning rate and the size of the active neighbourhood on the basis of input novelty. In this module, winner-take-all competition and the 'activity bubble' are due tograded lateral inhibition between units. It tends to separate representations as far apart as possible, which leads to interpolation abilities and an absence of catastrophic interference when the interfering set of patterns forms an interpolated set of the initial data set. More than the Kohonen maps, these maps provide an opportunity for building psychologically and neurophysiologically motivated multimodular connectionist models. As an example, the dual pathway connectionist model for fear conditioning by Armony et al. (1997, Trends in Cognitive Science, 1: 28-34) was rebuilt and extended with CALM Maps. If the detection of novelty enhances memory encoding in a canonical circuit, such as the CALM Map, this could explain the finding of large distributed networks for novelty detection (e.g. Knight and Scabini, 1998, Journal of Clinical Neurophysiology, 15: 3-13) in the brain.     

Minimally Connective,Auto-associative,Neural Networks

Classic barriers to using auto-associative neural networks to model mammalian memory include the unrealistically high synaptic connectivity of fully connected networks, and the relative paucity of information that has been stored in networks with realistic numbers of synapses per neuron and learning rules amenable to physiological implementation. We describe extremely large, auto-associative networks with low synaptic density. The networks have no direct connections between neurons of the same layer. Rather, the neurons of one layer are 'linked' by connections to neurons of some other layer. Patterns of projections of one layer on to another which form projective planes, or other cognate geometries, confer considerable computational power an the network.     

Reinforcement learning agents providing advice in complex video games

This article introduces a teacher–student framework for reinforcement learning, synthesising and extending material that appeared in conference proceedings [Torrey, L., & Taylor, M. E. (2013)]. Teaching on a budget: Agents advising agents in reinforcement learning. {Proceedings of the international conference on autonomous agents and multiagent systems}] and in a non-archival workshop paper [Carboni, N., &Taylor, M. E. (2013, May)]. Preliminary results for 1 vs. 1 tactics in StarCraft. {Proceedings of the adaptive and learning agents workshop (at AAMAS-13)}]. In this framework, a teacher agent instructs a student agent by suggesting actions the student should take as it learns. However, the teacher may only give such advice a limited number of times. We present several novel algorithms that teachers can use to budget their advice effectively, and we evaluate them in two complex video games: StarCraft and Pac-Man. Our results show that the same amount of advice, given at different moments, can have different effects on student learning, and that teachers can significantly affect student learning even when students use different learning methods and state representations.     

Extraction,Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks

Recurrent neural networks readily process, learn and generate temporal sequences. In addition, they have been shown to have impressive computational power. Recurrent neural networks can be trained with symbolic string examples encoded as temporal sequences to behave like sequential finite slate recognizers. We discuss methods for extracting, inserting and refining symbolic grammatical rules for recurrent networks. This paper discusses various issues: how rules are inserted into recurrent networks, how they affect training and generalization, and how those rules can be checked and corrected. The capability of exchanging information between a symbolic representation (grammatical rules)and a connectionist representation (trained weights) has interesting implications. After partially known rules are inserted, recurrent networks can be trained to preserve inserted rules that were correct and to correct through training inserted rules that were ‘incorrec’—rules inconsistent with the training data.     

PARALLEL SELF-ORGANIZING MAP

１ＩＮＴＲＯＤＵＣＴＩＯＮ“Ｏｎｃｅｓａｗ，ｎｅｖｅｒｆｏｒｇｏｔｅｎ”ｉｓａｓｅｎｔｅｎｃｅｗｈｉｃｈｕｓｅｄｔｏｄｅｓｃｒｉｂｅａｈｕｍａｎｓｅｎｓｅａｎｄｌｅａｒｎｉｎｇｓｅｑｕｅｎｃｅ．Ｆｏｒｅｘａｍｐｌｅ，ａｂｏｙｇｌａｎｃｅｄａｔａｌｏ...     

Robust manifold learning with CycleCut

Many manifold learning algorithms utilise graphs of local neighbourhoods to estimate manifold topology. When neighbourhood connections short-circuit between geodesically distant regions of the manifold, poor results are obtained due to the compromises that the manifold learner must make to satisfy the erroneous criteria. Also, existing manifold learning algorithms have difficulty in unfolding manifolds with toroidal intrinsic variables without introducing significant distortions to local neighbourhoods. An algorithm called CycleCut is presented, which prepares data for manifold learning by removing short-circuit connections and by severing toroidal connections in a manifold.     

Redefining the rules: Providing race models with a connectionist learning rule

In this paper, we describe the Parallel Race Network (PRN), a race model with the ability to learn stimulus-response associations using a formal framework that is very similar to the one used by the traditional connectionist networks. The PRN assumes that the connections represent abstract units of time rather than strengths of association. Consequently, the connections in the network indicate how rapidly the information should be sent to an output unit. The decision is based on a race between the outputs. To make learning functional and autonomous, the Delta rule was modified to fit the time-based assumption of the PRN. Finally, the PRN is used to simulate an identification task and the implications of its mode of representation are discussed.     

一种基于CNN深度学习的焊接机器人视觉模型

为了准确地识别复杂环境下的焊缝目标,建立了一种基于深度学习的焊接机器人视觉模型,该模型采用局部联接和全联接相结合的CNN卷积神经网络结构,局部联接由3个卷积层和子采样层交替组成,用于焊接目标的特征提取,全连接层由输入层、隐层和输出层组成,作为分类器用于焊缝目标识别. 采样了一千多幅焊接目标图像样本用于CNN的网络训练,分析了不同CNN网络结构参数对模型的影响. 结果表明,该视觉模型对焊接目标的平移、旋转和比例缩放表现出良好的鲁棒性,可以应用到焊接机器人的视觉导航.     

Application of neural networks to an expert system for cold forging

The technique of neural networks is applied to an expert system for cold forging in order to increase the consultation speed and to provide more reliable results. A three-layer neural network is used and the back-propagation algorithm is employed to train the network.

By utilizing the ability of pattern recognition of neural networks, a system is constructed to relate the shapes of rotationally symmetric products to their forming methods. The cross-sectional shapes of the products which can be formed by one blow are transformed into 16 × 16 black and white points and are given to the input layer. After learning about 23 products, the system is able to determine the forming methods for the products which are exactly the same or slightly different from the products used in the network training. To exploit the self-learning ability, the neural networks are applied to the prediction of the most probable number of forming steps, from information about the complexity of the product shape and the materials of the die and billet, and also to the generation of rules from the knowledge acquired from an FEM simulation. It is found that the prediction of the most probable number of forming steps can be made successfully and that the FEM results are represented better by the neural networks than by the statistical methods.     

A constrained neural learning rule for eliminating the border effect in online self-organising maps

The self-organising map (SOM) is a concise and powerful algorithm for clustering and visualisation of high-dimensional data. However, this robust algorithm still suffers from the border effect. Most of the approaches proposed to eliminate this effect use a borderless topological structure. We prefer to keep the original topological structure of the SOM for visualisation. A novel approach is proposed for the elimination of the border effect from the perspective of self-organising learning. Based on an assumption that the best matching unit (BMU) should be the most active unit, the approach proposes that the BMU should move more towards its associated input sample than its neighbours in the fine-tuned learning stage. Our constrained approach emphasises the effect of the lateral connections and neutralises the effect on the distance between the input sample and units. This approach is able to make units of the map stretch wider than the traditional SOM and thus the border effect is alleviated. Our proposed approach is proved to satisfy the requirements of the topologically ordered neural networks and is evaluated by both qualitative and quantitative criteria. All experiments conclude that performance is improved if the proposed constrained learning rule is used.     

Reuse of information in multi-layer sigma-pi neural networks

This new work is an extension of existing research into artificial neural networks (Neville and Stonham, Connection Sci.: J. Neural Comput. Artif. Intell. Cognitive Res., 7, pp. 29–60, 1995; Neville, Neural Net., 45, pp. 375–393, 2002b). These previous studies of the reuse of information (Neville, IEEE World Congress on Computational Intelligence, 1998b, pp. 1377–1382; Neville and Eldridge, Neural Net., pp. 375–393, 2002; Neville, IEEE World Congress on Computational Intelligence, 1998c, pp. 1095–1100; Neville, IEEE 2003 International Joint Conference on Neural Networks, 2003; Neville, IEEE IJCNN'04, 2004 International Joint Conference on Neural Networks, 2004) are associated with a methodology that prescribes the weights, as opposed to training them. In addition, they work with smaller networks. Here, this work is extended to include larger nets. This methodology is considered in the context of artificial neural networks: geometric reuse of information is described mathematically and then validated experimentally. The theory shows that the trained weights of a neural network can be used to prescribe the weights of other nets of the same architecture. Hence, the other nets have prescribed weights that enable them to map related geometric functions. This means the nets are a method of ‘reuse of information’. This work is significant in that it validates the statement that, ‘knowledge encapsulated in a trained multi-layer sigma-pi neural network (MLSNN) can be reused to prescribe the weights of other MLSNNs which perform similar tasks or functions’. The important point to note here is that the other MLSNNs weights are prescribed in order to represent related functions. This implies that the knowledge encapsulated in the initially trained MLSNN is of more use than may initially appear.     

Learning Simple Arithmetic Procedures

Rumelhan et al. (1986b) proposed a model of how symbolic processing may be achieved by parallel distributed processing (PDP) networks. Their idea is tested by training two types of recurrent networks to learn to add two numbers of arbitrary lengths. This turned out to be a fruitful exercise. We demonstrate: (1) that networks can learn simple programming constructs such as sequences, conditional branches and while loops; (2) that by lsquo;going sequential’ in this manner, we are able to process artibrarily long problems; (3) a manipulation of the training environment, called combined subset training (CST), that was found to be necessary to acquire a large training set; (4) a power difference between simple recurrent networks and Jordan networks by providing a simple procedure that one can learn and the other cannot.     

Feedforward Neural Network;Construction Algorithm;Maximum Likelihood Function;Quasi-newton Methods.

A new method for constructing a feedforward neural network is proposed. The method starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that completely recognizes all its input patterns is constructed. The novel idea about this method is that the network is trained to maximize a certain likelihood function and not to minimize the more widely used mean squared error function. We show that when a new hidden unit is added to the network, this likelihood function is guaranteed to increase and this increase ensures the finite termination of the method. We also provide a wide range of numerical results. The method was tested on the n -bit parity problems and the spiral problem. It was able to construct networks having less than n hidden units that solve the n -bit parity problems for n = 4, 5, 6, 7 and 8. The method was also tested on some real-world data and the networks it constructed were shown to be able to predict patterns not in the training set with more than 95% accuracy.     

基于卷积神经网络的网络节点异常数据检测方法

异常数据检测是保障无线传感器网络节点数据准确性和可靠性的重要步骤。针对无线传感器网络节点异常数据检测问题,提出一种基于卷积神经网络的异常数据检测方法。该方法是对正常数据和注入故障后生成的异常数据进行归一化处理后映射成的灰度图片作为卷积神经网络的输入数据,并且基于LeNet-5卷积神经网络设计了合适的卷积层特征面及全连接层的参数,构造了3种新的卷积神经网络模型。该模型通过卷积层自主学习数据特征,解决了传统检测算法的性能容易受到相关阈值影响的问题。通过网络公开数据集进行模型测试,结果表明该方法具有很好的检测性能和较高的可靠性