(S)RAAM: An Analytical Technique for Fast and Reliable Derivation of Connectionist Symbol Structure Representations

Recursive auto-associative memory (RAAM) has become established in the connectionist literature as a key contribution in the strive to develop connectionist representations of symbol structures. However, RAAMs use the backpropagation algorithm and therefore can be difficult to train and slow to learn. In addition, it is often hard to analyze exactly what a network has learnt and, therefore, it is difficult to state what composition mechanism is used by a RAAM for constructing representations. In this paper, we present an analytical version of RAAM, denoted as simplified RAAM or (S)RAAM. (S)RAAM models a RAAM very closely in that a single constructor matrix is derived which can be applied recursively to construct connectionist representations of symbol structures. The derivation, like RAAM, exhibits a moving target effect because training patterns adjust during learning but, unlike RAAM, the training is very fast. The analytical model allows a clear statement to be made about generalization characteristics and it can be shown that, in practice, the model will converge.     

Distributed representations for extended syntactic transformation

This paper shows how the choice of representation substantially affects the generalization performance of connectionistnetworks. The starting point is Chalmers' simulations involving structure-sensitive processing. Chalmers argued that a connectionist network could handle structure sensitive processing without the use of syntactically structured representations. He trained a connectionist architecture to encode/decode distributed representations for simple sentences. These distributed representations were then holistically transformed such that active sentences were transformed into their passive counterpart. However, he noted that the recursive auto-associative memory (RAAM), which was used to encode and decode distributed representations for the structures, exhibited only a limited ability to generalize when trained to encode/decode a randomly selected sample of the total corpus. When the RAAM was trained to encode/decode all sentences, and a separate transformation network was trained to make some active-passive transformations of the RAAMencoded sentences, the transformation network demonstrated perfect generalization on the remaining test sentences. It is argued here that the main reason for the limited generalization is not the ability of the RAAM architecture per se, but the choice of representation for the tokens used. This paper shows that 100% generalization can be achieved for Chalmers' original set up (i.e. using only 30% of the total corpus for training). The key to this success is to use distributed representations for the tokens (capturing different characteristics for differentclasses of tokens, e.g. verbs or nouns).     

Sequence Recognition with Recurrent Neural Networks

The simple recurrent network (SRN) introduced by Elman (1990) can be trained to predict each successive symbol of any sequence in a particular language, and thus act as a recognizer of the language. Here, we show several conditions occurring within the class of regular languages that result in recognition failure by any SRN with a limited number of nodes in the hidden layer. Simulation experiments show how modified versions of the SRN can overcome these failure conditions. In one case, it is found to be necessary to train the SRN to show at its output units both the current input symbol as well as the predicted symbol. In another case, the SRN must show the current contents of the context units. It is shown that the SRN with both modifications, called the auto-associative recurrent network (AARN), overcomes the identified conditions for SRN failure, even when they occur simultaneously. However, it cannot be trained to recognize all of the regular languages.     

Syntactic Transformations on Distributed Representations

There has been much interest in the possibility of connectionist models whose representations can be endowed with compositional structure, and a variety of such models have been proposed. These models typically use distributed representations that arise from the functional composition of constituent parts. Functional composition and decomposition alone, however, yield only an implementation of classical symbolic theories. This paper explores the possibility of moving beyond implementation by exploiting holistic structure-sensitive operations on distributed representations. An experiment is performed using Pollack's Recursive Auto-Associative Memory (RAAM). RAAM is used to construct distributed representations of syntactically structured sentences. A feed-forward network is then trained to operate directly on these representations, modeling syntactic transformations of the represented sentences. Successful training and generalization is obtained, demonstrating that the implicit structure present in these representations can be used for a kind of structure-sensitive processing unique to the connectionist domain.     

Labelling Recursive Auto-associative Memory

In this paper, we propose an extension to the recursive auto-associative memory (RAAM) by Pollack. This extension, the labelling RAAM (LRAAM), can encode labelled graphs with cycles by representing pointers explicitly. Some technical problems encountered in the RAAM, such as the termination problem in the learning and decoding processes, are solved more naturally in the LRAAM framework. The representations developed for the pointers seem to be robust to recurrent decoding along a cycle. Theoretical and experimental results show that the performances of the proposed learning scheme depend on the way the graphs are represented in the training set. Critical features for the representation are cycles and confluent pointers. Data encoded in a LRAAM can be accessed by a pointer as well as by content. Direct access by content can be achieved by transforming the encoder network of the LRAAM into a particular bidirectional associative memory (BAM). Statistics performed on different instances of LRAAM show a strict connection between the associated BAM and a standard BAM. Different access procedures can be defined depending on the access key. The access procedures are not wholly reliable; however, they seem to have a good success rate. The generalization test for the RAAM is no longer complete for the LRAAM. Some suggestions on how to solve this problem are given. Some results on modular LRAAM, stability and application to neural dynamics control are summarized.     

B-RAAM: A Connectionist Model which Develops Holistic Internal Representations of Symbolic Structures

Connectionist models have been criticized as seemingly unable to represent data structures thought necessary to support symbolic processing. However, a class of model-recursive auto-associative memory (RAAM)-has been demonstrated to be capable of encoding/ decoding compositionally such symbolic structures as trees, lists and stacks. Despite RAAM's appeal, a number of shortcomings are apparent. These include: the large number of epochs often required to train RAAM models; the size of encoded representation (and, therefore, of hidden layer) needed; a bias in the (fed-back) representation for more recently-presented information; and a cumulative error effect that results from recursively processing the encoded pattern during decoding. In this paper, the RAAM model is modified to form a new encoder/decoder, called bi-coded RAAM (B-RAAM). In bicoding, there are two mechanisms for holding contextual information: the first is hiddento-input layer feedback as in RAAM but extended with a delay line; the second is an output layer which expands dynamically to hold the concatenation of past input symbols. A comprehensive series of experiments is described which demonstrates the superiority of B-RAAM over RAAM in terms of fewer training epochs, smaller hidden layer, improved ability to represent long-term time dependencies and reduction of the cumulative error effect during decoding.     

Holistic Computation: Reconstructing a Muddled Concept

It has been claimed that connectionist methods of encoding compositional structures, as Pollack's recursive auto-associative memory (RAAM), support a non-classical form structure-sensitive operation known as 'holistic computation', where symbol structures be acted upon holistically without the need to decompose them, or to perform a search locate or access their constituents. In this paper, it is argued that the concept as described in the literature is vague and confused, and a revised definition of holistic computation proposed which aims to clarify the issues involved. It is also argued that holistic computation neither requires a highly distributed or holistic representation, nor is it unique to connectionist methods of representing compositional structure.     

A Distributed Memory Model of the Associative Boost in Semantic Priming

Evidence from priming studies indicates that both semantic and associative relations between pairs of words facilitate word recognition, and that pairs of words related in both ways (e.g. hammer-nail, cat-dog) produce an additional ‘associative boost’. We argue that while semantic priming may result from overlapping patterns of micro-features in a distributed memory model (e.g. Masson, 1991), associative priming is a result of frequent co-occurrence of words in the language. We describe a simple recurrent network, with distributed phonological and semantic representations, which is sensitive to the sequential occurrence of phonological patterns during training, and which produces associative facilitation of word recognition in a simulation of the priming task.     

Ordered structures in Al-3Cu-(1.78Mg) alloys aged at 190 °C for 2 minutes

Precipitate structures in Al-3Cu (wt%) and Al-3Cu-1.78Mg (wt%) alloys aged at 190 °C for 2 minutes were studied by transmission electron microscopy and high resolution transmission electron microscopy. New models named ordered structures were proposed for the fine precipitates formed in the two aged alloys. Ordered structures in Al-Cu alloys were composed by Al and Cu atoms, and by Al, Cu and Mg atoms together in Al-Cu-Mg alloys. By simulating selected area electron diffraction (SAED) patterns of Al-Cu-(Mg) alloys containing plate-shaped ordered structures and taking into account the influence of the plate shape of ordered structures on their electron diffraction characteristics, [001] zone axis SAED patterns of the two alloys were interpreted. It was concluded that volume expansions of the two aged alloys occurring at the early aging stage are caused by the lattice expansions as ordered structures are formed, and ordered structures are the nuclei for the GP zones in Al-Cu alloys.     

The Distributed Representation of Type and Category

This paper addresses the connectionist accounts of two related topics, the representation of ‘type hierarchies’ and ‘cognitive categories’ (as the terms are used in knowledge representation research and cognitive psychology respectively). There are important problems with the current accounts of these topics, particularly the circularity of the definition of subvector type representations, and the inaccessibility to the network of prototype category representations. We introduce our own account of type and category, based on the identification of the ‘domain’ of units on which inputs of the same category are typically represented. Domains are formed by learning information about the co-activation of units in a population, and identified (for a given input by using this information to generate a ‘centrality’ value for each unit (collectively called a ‘centrality distribution’). The main claims of this paper are that centrality distributions can be used as representations of categories, and can be hierarchically organized to capture type information. We illustrate these claims with implementations, and discuss the representation of category gradedness, and mechanisms of property inheritance. The use of centrality distributions (as well (as patterns of activation)as representations is an extension to basic connectionist methods.     

A modular approach to self-organization of robot control based on language instruction

In this paper we focus on how instructions for actions can be modelled in a self-organizing memory. Our approach draws from the concepts of regional distributed modularity and self-organization. We describe a self-organizing model that clusters action representations into different locations dependent on the body part they are related to. In the first case study we consider semantic representations of action verb meaning and then extend this concept significantly in a second case study by using actual sensor readings from our MIRA robot. Furthermore, we outline a modular model for a self-organizing robot action control system using language for instruction. Our approach for robot control using language incorporates some evidence related to the architectural and processing characteristics of the brain (Wermter et al. 2001b). This paper focuses on the neurocognitive clustering of actions and regional modularity for language areas in the brain. In particular, we describe a self-organizing network that realizes action clustering (Pulvermüller 2003).     

Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks

A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.     

Visual illusions,solid/outline invariance and non-stationary activity patterns

Coupled map lattices (CMLs) offer a new framework for modelling visual information processes. The framework involves computing with non-stationary patterns of synchronized activity. In this framework structural features of the visual field emerge through the lateral interaction of locally coupled non-linear maps. Invariant representations develop independent of top-down, or re-entrant, feedback. These representations distort certain features of the pattern, giving rise to visual field illusions. Boundary contours, among others, are emphasized, which suggests that special cases of boundary contour problem could be solved by the system. Simulation studies were performed to test the hypothesis that the system represents visual patterns in a solid/outline invariant manner. A standard back-propagation neural network trained with a CML-filtered set of solid images and tested with CML-filtered outline versions of the same set of images (or vice versa) showed perfect generalization. Generalization failed to occur for unfiltered or contour-filtered images. The CML representations, therefore, were concluded to be solid/outline invariant.     

Improving feature extraction performance of greedy network-growing algorithm by inverse euclidean distance

We propose here a new computational method for the information-theoretic method, called the greedy network-growing algorithm, to facilitate a process of information acquisition. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function are not very sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily faithfully describe input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance becomes smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than in the previous method with the sigmoidal activation function. We applied the new method to artificial data analysis and animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.     

Connectionist Finite State Natural Language Processing

Unless one is prepared to argue that existing, ‘classical’formal language and automata theory, together with the natural language linguistics built on them, are fundamentally mistaken about the nature of language, any viable connectionist natural language processing (NLP) model will have to be characterizable, at least approximately, by some generative grammar or by an automaton of the corresponding class. An obvious way of ensuring that a connectionist NLP device is so characterizable is to specify it in classical terms and then to implement it in an artificial neural network, and that is what this paper does. It adopts the deterministic pushdown transducer (DPDT) as an adequate formal model for general NLP and shows how a simple recurrent network (SRN) can be trained to implement a finite state transducer (FST) which simulates the DPDT. A computer simulation of a parser for a small fragment of English is used to study the properties of the model. The conclusion is that such SRN implementation results in a device which is broadly consistent with its classical specification, but also has emergent properties relative to that specification which are desirable in an NLP device.     

Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma

In order to solve the 'sensitivity-stability' problem-and its immediate correlate, the problem of sequential learning-it is crucial to develop connectionist architectures that are simultaneously sensitive to, but not excessively disrupted by, new input. French (1992) suggested that to alleviate a particularly severe form of this disruption, catastrophic forgetting, it was necessary for networks to separate dynamically their internal representations during learning. McClelland et al. (1995) went even further. They suggested that nature's way of implementing this obligatory separation was the evolution of two separate areas of the brain, the hippocampus and the neocortex. In keeping with this idea of radical separation, a 'pseudo-recurrent' memory model is presented here that partitions a connectionist network into two functionally distinct, but continually interacting areas. One area serves as a final-storage area for representations; the other is an early-processing area where new representations are first learned by the system. The final-storage area continually supplies internally generated patterns (pseudopatterns; Robins, 1995), which are approximations of its content, to the early-processing area, where they are interleaved with the new patterns to be learned. Transfer of the new learning is done either by weight-copying from the early-processing area to the final-storage area or by pseudopattern transfer. A number of experiments are presented that demonstrate the effectiveness of this approach, allowing, in particular, effective sequential learning with gradual forgetting in the presence of new input. Finally, it is shown that the two interacting areas automatically produce representational compaction and it is suggested that similar representational streamlining may exist in the brain.     

Materials Design and Discovery with High-Throughput Density Functional Theory: The Open Quantum Materials Database (OQMD)

High-throughput density functional theory (HT DFT) is fast becoming a powerful tool for accelerating materials design and discovery by the amassing tens and even hundreds of thousands of DFT calculations in large databases. Complex materials problems can be approached much more efficiently and broadly through the sheer quantity of structures and chemistries available in such databases. Our HT DFT database, the Open Quantum Materials Database (OQMD), contains over 200,000 DFT calculated crystal structures and will be freely available for public use at http://oqmd.org. In this review, we describe the OQMD and its use in five materials problems, spanning a wide range of applications and materials types: (I) Li-air battery combination catalyst/electrodes, (II) Li-ion battery anodes, (III) Li-ion battery cathode coatings reactive with HF, (IV) Mg-alloy long-period stacking ordered (LPSO) strengthening precipitates, and (V) training a machine learning model to predict new stable ternary compounds.     

Controlling stress and diffusion for the low-temperature-ordering of L1<Subscript>0</Subscript> ordered FePt films

Low-temperature-ordering of L1₀ ordered FePt films have been extensively studied in recent years due to its potential for future application on magnetic perpendicular media. The predominant issue of the ordering process is the diffusion of iron and platinum atoms from a disordered to an ordered phase. The diffusion can be enhanced by adjusting diffusivity, providing extra energy, or reducing energy barriers. In addition to reducing the ordering temperature of FePt, (001)-oriented granular films require perpendicular media because the magnetic easy axis of the ordered phase is [001]. Atomic-scale multilayer deposition is proposed to achieve designed film structures.     

Ordering tendencies in Pd-Pt, Rh-Pt, and Ag-Au alloys

First-principles quantum-mechanical calculations indicate that the mixing enthalpies for Pd-Pt and Rh-Pt solid solutions are negative, in agreement with experiment. Calculations of the diffuse-scattering intensity due to short-range order also exhibits ordering tendencies. Further, the directly calculated enthalpies of formation of ordered intermetallic compounds are negative. These ordering tendencies are in direct conflict with a 1959 prediction of Raub that Pd-Pt and Rh-Pt will phase-separate below ~760 °C (hence their mixing energy will be positive), a position that has been adopted by all binary alloy phase diagram compilations. The present authors predict that Pd_1-xPt_x will order in the L1₂, L1₀, and L1₂ structures ([001] superstructures) at compositionsx = 1/4, 1/2, and 3/4, respectively, while the ordered structures of Rh_1-xPt_x are predicted to be superlattices stacked along the [012] directions. While the calculated ordering temperatures for these intermetallic compounds are too low to enable direct growth into the ordered phase, diffuse-scattering experiments at higher temperatures should reveal ordering rather than phase-separation characteristics (i.e., off-F peaks). The situation is very similar to the case of Ag-Au, where an ordering tendency is manifested both by a diffuse scattering intensity and by a negative enthalpy of mixing. An experimental reexamination of PdPt and Rh-Pt is needed.