Extraction,Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks

Recurrent neural networks readily process, learn and generate temporal sequences. In addition, they have been shown to have impressive computational power. Recurrent neural networks can be trained with symbolic string examples encoded as temporal sequences to behave like sequential finite slate recognizers. We discuss methods for extracting, inserting and refining symbolic grammatical rules for recurrent networks. This paper discusses various issues: how rules are inserted into recurrent networks, how they affect training and generalization, and how those rules can be checked and corrected. The capability of exchanging information between a symbolic representation (grammatical rules)and a connectionist representation (trained weights) has interesting implications. After partially known rules are inserted, recurrent networks can be trained to preserve inserted rules that were correct and to correct through training inserted rules that were ‘incorrec’—rules inconsistent with the training data.     

A critique of connectionist semantics

Amongst modellers ofnatural language comprehension, the suspicion that explicit semantic representations are inherently biased has led many to rely more heavily on the ability of networks to form their own internal semantic representations over the course of training. The concern over explicit semantics, however, betrays a lack of appreciation for the manner in which insidious biases can and cannot creep into models of comprehension. In fact, the trend of relying on networks to form their own internal semantic representations has done little to curtail one common form of insidious bias. Where models of natural language comprehension are concerned, the cause of inappropriate biases has everything to do with the manner in which regularities find their way into sentence/meaning pairs and little or nothing to do with the degree to which semantic information is made explicit. This is fortunate, as there may be drawbacks to relying too heavily on the ability of networks to form their own internal semantic representations.     

Constituent similarity and systematicity: The limits of first-order connectionism

Standard feedforward and recurrent networks cannot support strong systematicity when constituents are presented as local input/output vectors. To explain systematicity connectionists must either: (1) develop alternative models, or (2) justify the assumption of similar (non-local) constituent representations prior to the learning task. I show that the second commonly presumed option cannot account for systematicity, in general. This option, termed first-order connectionism, relies upon established spatial relationships between common-class constituents to account for systematic generalization: inferences (functions) learnt over, for example, cats, extend systematically to dogs by virtue of both being nouns with similar internal representations so that the function learnt to make inferences employing one simultaneously has the capacity to make inferences employing the other. But, humans generalize beyond common-class constituents. Cross-category generalization (e.g. inferences that require treating mango as a colour, rather than a fruit) makes having had the necessary common context to learn similar constituent representations highly unlikely. At best, the constituent similarity proposal encodes for one binary relationship between any two constituents, at any one time. It cannot account for inferences, such as transverse patterning, that require identifying and applying one of many possible binary constituent relationships that is contingent on a third constituent (i.e. ternary relationship). Connectionists are, therefore, left with the first option which amounts to developing models with the symbol-like capacity to represent explicitly constituent relations independent of constituent contents, such as in tensor-related models. However, rather just simply implementing symbol systems, I suggest reconciling connectionist and classical frameworks to overcome their individual limitations.     

Learning Simple Arithmetic Procedures

Rumelhan et al. (1986b) proposed a model of how symbolic processing may be achieved by parallel distributed processing (PDP) networks. Their idea is tested by training two types of recurrent networks to learn to add two numbers of arbitrary lengths. This turned out to be a fruitful exercise. We demonstrate: (1) that networks can learn simple programming constructs such as sequences, conditional branches and while loops; (2) that by lsquo;going sequential’ in this manner, we are able to process artibrarily long problems; (3) a manipulation of the training environment, called combined subset training (CST), that was found to be necessary to acquire a large training set; (4) a power difference between simple recurrent networks and Jordan networks by providing a simple procedure that one can learn and the other cannot.     

Quality driven distribution of intelligent containers in cold chain logistics networks

The ‘intelligent container’ represents a novel transport system with the ability to make autonomous decisions regarding the condition of its transported goods. For example, fruit in cold chain logistics networks is very sensitive to mould and tends to perish. This can cause huge losses during transport, because the state-of-the-art reefer containers are able to control the temperature but not in relation to the fruit condition. The ‘intelligent container’ is able to precisely monitor the condition of fruit, as well as track its geographical position. Thus, the transport losses can be reduced due to better climate control and enhanced distribution strategies. This paper focuses on the development of a new scheduling method for distribution by applying principles of quality-driven customer order decoupling corridors (qCODC). Such corridors allow the dynamic change of allocations of container to customer order assignments. These corridors increase the flexibility of the decision-making process. Therefore, a simulation model will be developed and used in order to evaluate the potential of the new scheduling method based on the concept of the ‘intelligent container’ and qCODC.     

Recall and Recognition in an Attractor Neural Network Model of Memory Retrieval

This paper presents an attractor neural network (ANN) model of recall and recognition. It is shown that an ANN Hopfield-based network can qualitatively account for a wide range of experimental psychological data pertaining to these two main aspects of memory retrieval. After providing simple, straightforward definitions of recall and recognition in the model, a wide variety of ‘high-level’ psychological phenomena are shown to emerge from the ‘low-level’ neural-like properties of the network. It is shown that modeling the effect of memory load on the network's retrieval properties requires the incorporation of noise into the network's dynamics. External projections may account for phenomena related with the stored items’ associative links, but are not sufficient for representing context. With low memory load, the network generates retrieval response times which have the same distribution form as that observed experimentally. Finally, estimations of the probabilities of successful recall and recognition are obtained, possibly enabling further quantitative examination of the model     

Kinematic motor learning

This paper focuses on adaptive motor control in the kinematic domain. Several motor-learning strategies from the literature are adopted to kinematic problems: ‘feedback-error learning’, ‘distal supervised learning’, and ‘direct inverse modelling’ (DIM). One of these learning strategies, DIM, is significantly enhanced by combining it with abstract recurrent neural networks. Moreover, a newly developed learning strategy (‘learning by averaging’) is presented in detail. The performance of these learning strategies is compared with different learning tasks on two simulated robot setups (a robot-camera-head and a planar arm). The results indicate a general superiority of DIM if combined with abstract recurrent neural networks. Learning by averaging shows consistent success if the motor task is constrained by special requirements.     

Connectionism and Cognitive Linguistics

Cognitive linguists hypothesize that language is the product of general cognitive abilities. Semantic and functional motivations are sought for grammatical patterns, sentence meaning is viewed as the result of constraint satisfaction, and highly regular linguistic patterns are thought to be mediated by the same processes as irregular patterns. In this paper, recent cognitive linguistics arguments emphasizing the schematicity continuum, the non-autonomy of syntax, and the non-compositionality of semantics are presented and their amenability to connectionist modeling described. Some of the conceptual matches between cognitive linguistics and connectionism are then illustrated by a back-propagation model of the diverse meanings of the preposition over. The pattern set consisted of a distribution of form-meaning pairs that was meant to be evocative of English usage in that the regularities implicit in the distribution spanned the spectrum from rules to partial regularities to exceptions. Under pressure to encode these regularities with limited resources, the network used one hidden layer to recode the inputs into a set of abstract properties. The properties discovered by the network correspond closely to semantic features that linguists have proposed when giving an account of the meaning of over.     

Analysis–adjustment–synthesis networks

This paper presents a novel neural network architecture, analysis–adjustment–synthesis network (AASN), and tests its efficiency and accuracy in modelling non-linear function and classification. The AASN is a composite of three sub-networks: analysis sub-network; adjustment sub-network; and synthesis sub-network. The analysis sub-network is a one-layered network that spreads the input values into a layer of ‘spread input neurons’. This synthesis sub-network is a one-layered network that spreads the output values back into a layer of ‘spread output neurons’. The adjustment sub-network, between the analysis sub-network and the synthesis sub-network, is a standard multi-layered network that operates as the learning mechanism. After training the adjustment sub-network in recalling phase, the synthesis sub-network receives the output values of spread output neurons and synthesizes them into output values with a weighted-average computation. The weights in the weighted-average computation are deduced from the method of Lagrange multipliers. The approach is tested using four function mapping problems and one classification problem. The results show that combining the analysis sub-network and the synthesis sub-network with a multi-layered network can significantly improve a network's efficiency and accuracy.     

Tensor Product Production System: a Modular Architecture and Representation

Can connectionist networks effectively represent and process structure? A technique called ‘tensor product representations’, which formalizes and generalizes the approaches of several previous connectionist models, was developed by Smolensky and shown to possess a number of desirable general properties. This paper shows how the technique can be effectively used to design a specific symbol-processing task: the serial execution of simple production rules requiring pattern matching, variable binding and structure manipulation. This ‘Tensor Product Production System’ is applied to one of the classes of production rules in Touretzky and Hinton's Distributed Connectionist Production System, and a number of comparisons are made between the two approaches. The mathematical simplicity and analyzability of the tensor product scheme allows the straightforward design of a simpler, more principled, and in some ways more efficient system.     

Connectionism;Backpropagation;Structure;Interpretation;Symbolic Logic

A particular backpropagation network, called a network of value units, was trained to detect problem type and validity of a set of logic problems. This network differs from standard networks in using a Gaussian activation function. After training was successfully completed, jittered density plots were computed for each hidden unit, and used to represent the distribution of activations produced in each hidden unit by the entire training set. The density plots revealed a marked banding. Further analysis revealed that almost all of these bands could be assigned featural interpretations, and played an important role in explaining how the network classified input patterns. These results are discussed in the context of other techniques for analyzing network structure, and in the context of other parallel distributed processing architectures.     

Reuse of information in multi-layer sigma-pi neural networks

This new work is an extension of existing research into artificial neural networks (Neville and Stonham, Connection Sci.: J. Neural Comput. Artif. Intell. Cognitive Res., 7, pp. 29–60, 1995; Neville, Neural Net., 45, pp. 375–393, 2002b). These previous studies of the reuse of information (Neville, IEEE World Congress on Computational Intelligence, 1998b, pp. 1377–1382; Neville and Eldridge, Neural Net., pp. 375–393, 2002; Neville, IEEE World Congress on Computational Intelligence, 1998c, pp. 1095–1100; Neville, IEEE 2003 International Joint Conference on Neural Networks, 2003; Neville, IEEE IJCNN'04, 2004 International Joint Conference on Neural Networks, 2004) are associated with a methodology that prescribes the weights, as opposed to training them. In addition, they work with smaller networks. Here, this work is extended to include larger nets. This methodology is considered in the context of artificial neural networks: geometric reuse of information is described mathematically and then validated experimentally. The theory shows that the trained weights of a neural network can be used to prescribe the weights of other nets of the same architecture. Hence, the other nets have prescribed weights that enable them to map related geometric functions. This means the nets are a method of ‘reuse of information’. This work is significant in that it validates the statement that, ‘knowledge encapsulated in a trained multi-layer sigma-pi neural network (MLSNN) can be reused to prescribe the weights of other MLSNNs which perform similar tasks or functions’. The important point to note here is that the other MLSNNs weights are prescribed in order to represent related functions. This implies that the knowledge encapsulated in the initially trained MLSNN is of more use than may initially appear.     

Differential retroactive interference in humans following exposure to structured or unstructured learning material: a single distributed neural network account

While retroactive interference (RI) is a well-known phenomenon in humans, the differential effect of the structure of the learning material was only seldom addressed. Mirman and Spivey (2001 Mirman, D and Spivey, M. 2001. Retroactive interference in neural networks and in humans: the effect of pattern-based learning. Connection Science, 13: 257–275.  [Google Scholar], Connection Science, 13: 257–275) reported on behavioural results that show more RI for the subjects exposed to ‘Structured’ items than for those exposed to ‘Unstructured’ items. These authors claimed that two complementary memory systems functioning on radically different neural mechanisms are required to account for the behavioural results they reported. Using the same paradigm but controlling for proactive interference, we found the opposite pattern of results, that is, more RI for subjects exposed to ‘Unstructured’ items than for those exposed to ‘Structured’ items (experiment 1). Two additional experiments showed that this structure effect on RI is a genuine one. Experiment 2 confirmed that the design of experiment 1 forced the subjects from the ‘Structured’ condition to learn the items at the exemplar level, thus allowing for a close match between the two to-be-compared conditions (as ‘Unstructured’ condition items can be learned only at the exemplar level). Experiment 3 verified that the subjects from the ‘Structured’ condition could generalize to novel items. Simulations conducted with a three-layer neural network, that is, a single-memory system, produced a pattern of results that mirrors the structure effect reported here. By construction, Mirman and Spivey's architecture cannot simulate this behavioural structure effect. The results are discussed within the framework of catastrophic interference in distributed neural networks, with an emphasis on the relevance of these networks to the modelling of human memory.     

利用人工神经网络预测板带力学性能的软件开发

本文开发了利用人工神经网络预测力学性能的软件。本软件的特点是直观、方便、稳定。软件共分三大部分:数据处理部分、人工神经网络训练部分、运用成熟网络预报部分。数据应从稳定生产的现场取得,人工神经网络训练部分是用BP网络建立原始化学成分和热轧生产的主要工艺参数与产品力学性能之间的关系。经过反复训练满意后即可运用成熟网络进行性能预报。作者用此软件对SS400钢的性能进行预报,经过10万次训练后,产品力学性能的预报值与实际值拟合良好,预报结果的相对误差很小。     

A Distributed Memory Model of the Associative Boost in Semantic Priming

Evidence from priming studies indicates that both semantic and associative relations between pairs of words facilitate word recognition, and that pairs of words related in both ways (e.g. hammer-nail, cat-dog) produce an additional ‘associative boost’. We argue that while semantic priming may result from overlapping patterns of micro-features in a distributed memory model (e.g. Masson, 1991), associative priming is a result of frequent co-occurrence of words in the language. We describe a simple recurrent network, with distributed phonological and semantic representations, which is sensitive to the sequential occurrence of phonological patterns during training, and which produces associative facilitation of word recognition in a simulation of the priming task.     

A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks

Most known learning algorithms for dynamic neural networks in non-stationary environments need global computations to perform credit assignment. These algorithms either are not local in time or not local in space. Those algorithms which are local in both time and space usually cannot deal sensibly with ‘hidden units’. In contrast, as far as we can judge, learning rules in biological systems with many ‘hidden units’ are local in both space and time. In this paper we propose a parallel on-line learning algorithms which performs local computations only, yet still is designed to deal with hidden units and with units whose past activations are ‘hidden in time’. The approach is inspired by Holland's idea of the bucket brigade for classifier systems, which is transformed to run on a neural network with fixed topology. The result is a feedforward or recurrent ‘neural’ dissipative system which is consuming ‘weight-substance’ and permanently trying to distribute this substance onto its connections in an appropriate way. Simple experiments demonstrating the feasibility of the algorithm are reported.     

Incorporating Supervised Learning in the Domains Account of Categorization

This note presents an extension to the ‘domains’ account of cognitive categorization described by Robins. Current connectionist models of various aspects of categorization are based on either ‘supervised learning’ (the use of explicit categorizing information) or ‘unsupervised learning’ (using just the observable structure of a population). The domains account is an example of an unsupervised account. It seems clear, however, that humans employ both supervised and unsupervised methods of category learning. This note describes an extension to the domains approach which incorporates supervised learning, thereby providing an account of both mechanisms in the same unifying framework.     

Artificial neural networks whispering to the brain: nonlinear system attractors induce familiarity with never seen items

Attractors of nonlinear neural systems are at the core of the memory self-refreshing mechanism of human memory models that suppose memories are dynamically maintained in a distributed network [Ans, B., and Rousset, S. (1997), ‘Avoiding Catastrophic Forgetting by Coupling Two Reverberating Neural Networks’ Comptes Rendus de l'Académie des Sciences Paris, Life Sciences, 320, 989–997; Ans, B., and Rousset, S. (2000), ‘Neural Networks with a Self-Refreshing Memory: Knowledge Transfer in Sequential Learning Tasks Without Catastrophic Forgetting’, Connection Science, 12, 1–19; Ans, B., Rousset, S., French, R.M., and Musca, S.C. (2002), ‘Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks’, in Proceedings of the 24th Annual Meeting of the Cognitive Science Society, eds. W.D. Gray and C.D. Schunn, Mahwah, NJ: Lawrence Erlbaum Associates, pp. 71–76; Ans, B., Rousset, S., French, R.M., and Musca, S.C. (2004), ‘Self-Refreshing Memory in Artificial Neural Networks: Learning Temporal Sequences Without Catastrophic Forgetting’, Connection Science, 16, 71–99; Ans, B. (2004), ‘Sequential Learning in Distributed Neural Networks Without Catastrophic Forgetting: A Single and Realistic Self-Refreshing Memory can do it’, Neural Information Processing-Letters and Reviews, 4, 27–32]. Are humans able to learn never seen items from attractor patterns generated by a highly distributed artificial neural network? First, an opposition method was implemented to ensure that the attractors are not the items used to train the network, the source items: attractors were selected to be more similar (both at the exemplar and the centroïd level) to some control items than to the source items. In spite of this very severe selection, blank networks trained only on selected attractors performed better at test on the never seen source items than on the never seen control items. The results of two behavioural experiments using the opposition method show that humans exhibit more familiarity with the never seen source items than with the never seen control items, just as networks do. Thus, humans are sensitive to the particular type of information that allows distributed artificial neural networks to dynamically maintain their memory, and this information does not amount to the exemplars used to train the network that produced the attractors.     

Generalization in Sigma-Pi Networks

In this paper, we investigate generalization in supervised feedforward Sigma-pi nets with particular reference to means of augmentation of generalization of the network for specific tasks. The work was initiated because logical (digital) neural networks of this type do not function in the same manner as the more normal semi-linear unit, hence the general principle behind Sigma-pi networks generalization required examination, to enable one to put forward means of augmenting their generalization abilities. The paper studies four methods, two of which are novel methodologies for enhancing Sigma-pi networks generalization abilities. The networks are hardware realizable and the Sigma-pi units are logical (digital) nodes that respond to their input patterns in addressable locations, the locations (site-values) then define the probability of the output being a logical ‘1’. In this paper, we evaluate the performance of Sigma-pi nets with perceptual problems (in pattern recognition). This was carried out by comparative studies, to evaluate how each of the methodologies improved the performance of these networks on previously unseen stimuli.     

Place cognition and active perception: a study with evolved robots

A study of place cognition and ‘place units’ in robots produced via artificial evolution is described. Previous studies have investigated the possible role of place cells as building blocks for ‘cognitive maps’ representing place, distance and direction. Studies also show, however, that when animals are restrained, the spatial selectivity of place cells is partially or completely lost. This suggests that the role of place cells in spatial cognition depends not only on the place cells themselves, but also on representations of the animal's physical interactions with its environment. This hypothesis is tested in a population of evolved robots. The results suggest that successful place cognition requires not only the ability to process spatial information, but also the ability to select the environmental stimuli to which the agent is exposed. If this is so, theories of active perception can make a useful contribution to explaining the role of place cells in spatial cognition.