Teacher-directed learning: information-theoretic competitive learning in supervised multi-layered networks

In this paper, we propose a new type of efficient learning method called teacher-directed learning. The method can accept training patterns and correlated teachers, and we need not back-propagate errors between targets and outputs into networks. Information flows always from an input layer to an output layer. In addition, connections to be updated are those from an input layer to the first competitive layer. All other connections can take fixed values. Learning is realized as a competitive process by maximizing information on training patterns and correlated teachers. Because information is maximized, information is compressed into networks in simple ways, which enables us to discover salient features in input patterns. We applied this method to the vertical and horizontal lines detection problem, the analysis of US–Japan trade relations and a fairly complex syntactic analysis system. Experimental results confirmed that teacher information in an input layer forces networks to produce correct answers. In addition, because of maximized information in competitive units, easily interpretable internal representations can be obtained.     

B-RAAM: A Connectionist Model which Develops Holistic Internal Representations of Symbolic Structures

Connectionist models have been criticized as seemingly unable to represent data structures thought necessary to support symbolic processing. However, a class of model-recursive auto-associative memory (RAAM)-has been demonstrated to be capable of encoding/ decoding compositionally such symbolic structures as trees, lists and stacks. Despite RAAM's appeal, a number of shortcomings are apparent. These include: the large number of epochs often required to train RAAM models; the size of encoded representation (and, therefore, of hidden layer) needed; a bias in the (fed-back) representation for more recently-presented information; and a cumulative error effect that results from recursively processing the encoded pattern during decoding. In this paper, the RAAM model is modified to form a new encoder/decoder, called bi-coded RAAM (B-RAAM). In bicoding, there are two mechanisms for holding contextual information: the first is hiddento-input layer feedback as in RAAM but extended with a delay line; the second is an output layer which expands dynamically to hold the concatenation of past input symbols. A comprehensive series of experiments is described which demonstrates the superiority of B-RAAM over RAAM in terms of fewer training epochs, smaller hidden layer, improved ability to represent long-term time dependencies and reduction of the cumulative error effect during decoding.     

Connectionist learning: A comparison of neural networks and an optical thin-film multilayer model

Current work on connectionist models has been focused largely on artificial neural networks that are inspired by the networks of biological neurons in the human brain. However, there are also other connectionistarchitectures that differ significantly from this biological exemplar. We proposed a novel connectionist learning architecture inspired by the physics associated with optical coatings of multiple layers of thin-films in a previous paper (Li and Purvis 1999, Annals of Mathematics and Artificial Intelligence, 26: 1-4). The proposed model differs significantly from the widely used neuron-inspired models. With thin-film layer thicknesses serving as adjustable parameters (as compared with connection weights in a neural network) for the learning system, the optical thin-film multilayer model (OTFM) is capable of approximating virtually any kind of highly nonlinear mappings. The OTFM is not a physical implementation using optical devices. Instead, it is proposed as a new connectionist learning architecture with its distinct optical properties as compared with neural networks. In this paper we focus on a detailed comparison of neural networks and the OTFM (Li 2001, Proceedings ofINNS-IEEE International Joint Conference on Neural Networks, Washington, DC, pp. 1727-1732). We describe the architecture of the OTFM and show how it can be viewed as a connectionist learning model. We then present experimental results on solving a classification problem and a time series prediction problem that are typical of conventional connectionist architectures to demonstrate the OTFM's learning capability.     

Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.     

Holistic Computation: Reconstructing a Muddled Concept

It has been claimed that connectionist methods of encoding compositional structures, as Pollack's recursive auto-associative memory (RAAM), support a non-classical form structure-sensitive operation known as 'holistic computation', where symbol structures be acted upon holistically without the need to decompose them, or to perform a search locate or access their constituents. In this paper, it is argued that the concept as described in the literature is vague and confused, and a revised definition of holistic computation proposed which aims to clarify the issues involved. It is also argued that holistic computation neither requires a highly distributed or holistic representation, nor is it unique to connectionist methods of representing compositional structure.     

Learn more by training less: systematicity in sentence processing by recurrent networks

Connectionist models of sentence processing must learn to behave systematically by generalizing from a small training set. To what extent recurrent neural networks manage this generalization task is investigated. In contrast to Van der Velde et al. (Connection Sci., 16, pp. 21–46, 2004), it is found that simple recurrent networks do show so-called weak combinatorial systematicity, although their performance remains limited. It is argued that these limitations arise from overfitting in large networks. Generalization can be improved by increasing the size of the recurrent layer without training its connections, thereby combining a large short-term memory with a small long-term memory capacity. Performance can be improved further by increasing the number of word types in the training set.     

Temporal sequence detection with spiking neurons: towards recognizing robot language instructions

We present an approach for recognition and clustering of spatio temporal patterns based on networks of spiking neurons with active dendrites and dynamic synapses. We introduce a new model of an integrate-and-fire neuron with active dendrites and dynamic synapses (ADDS) and its synaptic plasticity rule. The neuron employs the dynamics of the synapses and the active properties of the dendrites as an adaptive mechanism for maximizing its response to a specific spatio-temporal distribution of incoming action potentials. The learning algorithm follows recent biological evidence on synaptic plasticity. It goes beyond the current computational approaches which are based only on the relative timing between single pre- and post-synaptic spikes and implements a functional dependence based on the state of the dendritic and somatic membrane potentials around the pre- and post-synaptic action potentials. The learning algorithm is demonstrated to effectively train the neuron towards a selective response determined by the spatio-temporal pattern of the onsets of input spike trains. The model is used in the implementation of a part of a robotic system for natural language instructions. We test the model with a robot whose goal is to recognize and execute language instructions. The research in this article demonstrates the potential of spiking neurons for processing spatio-temporal patterns and the experiments present spiking neural networks as a paradigm which can be applied for modelling sequence detectors at word level for robot instructions.     

Evolution of Neural Controllers for Locomotion and Obstacle Avoidance in a Six-legged Robot

This article describes how the SGOCE paradigm has been used within the context of a 'minimal simulation' strategy to evolve neural networks controlling locomotion and obstacle avoidance in a six-legged robot. A standard genetic algorithm has been used to evolve developmental programs according to which recurrent networks of leaky-integrator neurons were grown in a user-provided developmental substrate and were connected to the robot's sensors and actuators. Specific grammars have been used to limit the complexity of the developmental programs and of the corresponding neural controllers. Such controllers were first evolved through simulation and then successfully downloaded on the real robot.     

Tail-recursive Distributed Representations and Simple Recurrent Networks

Representation poses important challenges to connectionism. The ability to compose representations structurally is critical in achieving the capability considered necessary for cognition. We are investigating distributed patterns that represent structure as part of a larger effort to develop a natural language processor. Recursive auto-associative memory (RAAM) representations show unusual promise as a general vehicle for representing classical symbolic structures in a way that supports compositionality. However, RAAMs are limited to representations for fixed-valence structures and can often be difficult to train. We provide a technique for mapping any ordered collection (forest) of hierarchical structures (trees) into a set of training patterns which can be used effectively in training a simple recurrent network (SRN) to develop RAAM-style distributed representations. The advantages in our technique are three-fold: first, the fixed-valence restriction on structures represented by patterns trained with RAAMs is removed; second, the representations resulting from training correspond to ordered forests of labeled trees, thereby extending what can be represented in this fashion; third, training can be accomplished with an auto-associative SRN, making training a much more straightforward process and one which optimally utilizes the n-dimensional space of patterns.     

Analysis–adjustment–synthesis networks

This paper presents a novel neural network architecture, analysis–adjustment–synthesis network (AASN), and tests its efficiency and accuracy in modelling non-linear function and classification. The AASN is a composite of three sub-networks: analysis sub-network; adjustment sub-network; and synthesis sub-network. The analysis sub-network is a one-layered network that spreads the input values into a layer of ‘spread input neurons’. This synthesis sub-network is a one-layered network that spreads the output values back into a layer of ‘spread output neurons’. The adjustment sub-network, between the analysis sub-network and the synthesis sub-network, is a standard multi-layered network that operates as the learning mechanism. After training the adjustment sub-network in recalling phase, the synthesis sub-network receives the output values of spread output neurons and synthesizes them into output values with a weighted-average computation. The weights in the weighted-average computation are deduced from the method of Lagrange multipliers. The approach is tested using four function mapping problems and one classification problem. The results show that combining the analysis sub-network and the synthesis sub-network with a multi-layered network can significantly improve a network's efficiency and accuracy.     

Sequence Recognition with Recurrent Neural Networks

The simple recurrent network (SRN) introduced by Elman (1990) can be trained to predict each successive symbol of any sequence in a particular language, and thus act as a recognizer of the language. Here, we show several conditions occurring within the class of regular languages that result in recognition failure by any SRN with a limited number of nodes in the hidden layer. Simulation experiments show how modified versions of the SRN can overcome these failure conditions. In one case, it is found to be necessary to train the SRN to show at its output units both the current input symbol as well as the predicted symbol. In another case, the SRN must show the current contents of the context units. It is shown that the SRN with both modifications, called the auto-associative recurrent network (AARN), overcomes the identified conditions for SRN failure, even when they occur simultaneously. However, it cannot be trained to recognize all of the regular languages.     

Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs terms of predictive accuracy and simplicity.     

Evolving unipolar memristor spiking neural networks

Neuromorphic computing – brain-like computing in hardware – typically requires myriad complimentary metal oxide semiconductor spiking neurons interconnected by a dense mesh of nanoscale plastic synapses. Memristors are frequently cited as strong synapse candidates due to their statefulness and potential for low-power implementations. To date, plentiful research has focused on the bipolar memristor synapse, which is capable of incremental weight alterations and can provide adaptive self-organisation under a Hebbian learning scheme. In this paper, we consider the unipolar memristor synapse – a device capable of non-Hebbian switching between only two states (conductive and resistive) through application of a suitable input voltage – and discuss its suitability for neuromorphic systems. A self-adaptive evolutionary process is used to autonomously find highly fit network configurations. Experimentation on two robotics tasks shows that unipolar memristor networks evolve task-solving controllers faster than both bipolar memristor networks and networks containing constant non-plastic connections whilst performing at least comparably.     

Variable binding by synaptic strength change

Variable binding is a difficult problem for neural networks. Two new mechanisms for binding by synaptic change are presented, and in both, bindings are erased and can be reused. The first is based on the commonly used learning mechanism of permanent change of synaptic weight, and the second on synaptic change which decays. Both are biologically motivated models. Simulations of binding on a paired association task are shown with the first mechanism succeeding with a 97.5% F-Score, and the second performing perfectly. Further simulations show that binding by decaying synaptic change copes with cross talk, and can be used for compositional semantics. It can be inferred that binding by permanent change accounts for these, but it faces the stability plasticity dilemma. Two other existing binding mechanisms, synchrony and active links, are compatible with these new mechanisms. All four mechanisms are compared and integrated in a Cell Assembly theory.     

盐酸埃他卡林对大鼠海马神经元谷氨酸受体功能及突触活动的影响

目的: 研究盐酸埃他卡林(Ipt) 对脑神经元谷氨酸受体功能及突触活动的影响。方法: 采用原代培养的大鼠海马神经元, 应用膜片钳全细胞记录技术, 记录Ipt 对培养的海马神经元谷氨酸或天冬氨酸(NMDA) 诱发电流及神经元突触后电流的影响。结果: Ipt (1 ~ 100 μmol·L^-1) 可浓度依赖性地对抗培养的海马神经元谷氨酸或NMDA 诱发电流, 并为ATP敏感性钾通道拮抗剂格列本脲30 μmol·L^-1所对抗。Ipt 抑制培养的海马神经元之间突触联系形成的自发兴奋性突触后电流, 降低其发放频率, 抑制其电流幅度;但对微小兴奋性突触后电流无显著性影响。结论: Ipt 可阻断脑神经元谷氨酸受体功能, 抑制脑神经元谷氨酸的兴奋性突触传递, 其作用与ATP 敏感性钾通道相关。     

Dynamic Behavior c with Chemical Markers

In this paper we studied the dynamic behavior of neural networks consisting of discrete populations of formal neurons. Neurons are assumed to have the same probability of connection with other neurons carrying the same type of chemical marker and divided in such a way to neural subpopulations due to the chemical affinity of markers carried by the individual cells. The dynamics of isolated networks as well as the ones that receive sustained external inputs show multiple stability and hysteresis phenomena which lead to multiple memory domains.     

Ti-17合金本构关系的人工神经网络模型

开发了一个基于神经网络的Ｔｉ１７ 合金的本构关系模型。首先利用ＴｈｅｒｍｅｃｍａｓｔｏｒＺ 型热模拟机等温压缩Ｔｉ１７ 合金, 研究在不同变形温度、变形程度和应变速率等工艺参数条件下流动应力的变化情况。然后用实验所得的热变形工艺参数与性能间的数据训练人工神经网络。训练结束后的神经网络变成为一个知识基的本构关系模型。利用该模型预测的流动应力的值与实验结果间的误差较小。     

Self-organization of Multiple Winner-take-all Neural Networks

In this paper, analysis of the information content of discretely firing neurons in unsupervised neural networks is presented, where information is measured according to the network's ability to reconstruct its input from its output with minimum mean square Euclidean error. It is shown how this type of network can self-organize into multiple winner-take-all subnetworks, each of which tackles only a low-dimensional subspace of the input vector. This is a rudimentary example of a neural network that effectively subdivides a task into manageable subtasks.