High-level Inferencing in a Connectionist Network

Connectionist models have had problems representing and applying general knowledge rules that specifically require variables. This variable binding problem has barred them from performing the high-level inferencing necessary for planning, reasoning, and natural language understanding. This paper describes ROBIN, a structured neural network model capable of high-level inferencing requiring variable bindings and rule application. Variable bindings are handled by signatures—activation patterns which uniquely identify the concept bound to a role. Signatures allow multiple role-bindings to be propagated across the network in parallel for rule application and dynamic inference path instantiation. Signatures are integrated within a connectionist semantic network structure whose constraint-relaxation process selects between those newly-instantiated inferences. This allows ROBIN to handle an area of high-level inferencing difficult even for symbolic models, that of resolving multiple constraints from context to select the best interpretation from among several alternative and possibly ambiguous inference paths.     

On Variable Binding in Connectionist Networks

This paper deals with the problem of variable binding in connectionist networks. Specifically, a more thorough solution to the variable binding problem based on the Discrete Neuron formalism is proposed and a number of issues arising in the solution are examined in relation to logic: consistency checking, binding generation, unification and functions. We analyze what is needed in order to resolve these issues and, based on this analysis, a procedure is developed for systematically setting up connectionist networks for variable binding based on logic rules. This solution compares favorably to similar solutions in simplicity and completeness.     

Brain-structured Connectionist Networks that Perceive and Learn

This paper specifies the main features of connectionist and brain-like connectionist models; argues for the need for, and usefulness of, appropriate successively larger brainlike structures; and examines parallel-hierarchical Recognition Cone models of perception from this perspective, as examples of networks exploiting such structures (e.g. local receptive fields, global convergence-divergence). The anatomy, physiology, behavior, and development of the visual system are briefly summarized to motivate the architecture of brain-structured networks for perceptual recognition. Results are presented from simulations of carefully pre-designed Recognition Cone structures that perceive objects (e.g. houses) in digitized photographs. A framework for perceptual learning is introduced, including mechanisms for generation learning, i.e. the growth of new links and possibly, nodes, subject to brain-like topological constraints. The information processing transforms discovered through feedback-guided generation are fine-tuned by feedback-guided reweighting of links. Some preliminary results are presented of brain-structured networks that learn to recognize simple objects (e.g. letters of the alphabet, cups, apples, bananas) through generation and reweighting of transforms. These show large improvements over networks that either lack brain-like structure or/and learn by reweighting of links alone. It is concluded that brain-like structures and generation learning can significantly increase the power of connectionist models.     

Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks

A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.     

A Survey of Transfer Between Connectionist Networks

Connectionist networks that have learned one task can be reused on related tasks in a process that is called 'transfer'. This paper surveys recent work on transfer, and includes an overview of the articles in this volume. A number of distinctions between kinds of transfer are identified, and future directions for research are explored. The study of transfer has a long history in cognitive science. Discoveries about transfer in human cognition can inform applied efforts. Advances in applications can also inform cognitive studies.     

Improving Connectionist Learning with Symbolic Feature Construction

The ease of learning concepts from examples in connectionist learning is highly dependent on the attributes used for describing the training data. Decision-tree based feature construction may be used to improve the performance of back-propagation, an artificial neural network algorithm. We use disjunctive concepts and DNA-sequencing data to illustrate feature construction, which builds better representations and reduces concept difficulty. The performance of the combined feature construction and back-propagation strategy improves performance compared to either method alone, in terms of the accuracy of the concepts learned and the time taken to learn.     

Increasing the Power of Connectionist Networks (CN) by Improving Structures,Processes, Learning

A crucial dilemma is how to increase the power of connectionist networks (CN), since simply increasing the size of today's relatively small CNs often slows down and worsens learning and performance. There are three possible ways: (1) use more powerful structures; (2) increase the amount of stored information, and the power and the variety of the basic processes; (3) have the network modify itself (learn, evolve) in more powerful ways. Today's connectionist networks use only a few of the many possible topological structures, handle only numerical values using only very simple basic processes, and learn only by modifying weights associated with links. This paper examines the great variety of potentially muck more powerful possibilities, focusing on what appear to be the most promising: appropriate brain-like structures (e.g. local connectivity, global convergence and divergence); matching, symbol-handling, and list-manipulating capabilities; and learning by extraction-generation-discovery.     

Function Optimization using Connectionist Reinforcement Learning Algorithms

Any non-associative reinforcement learning algorithm can be viewed as a method for performing function optimization through (possibly noise-corrupted) sampling of function values. We describe the results of simulations in which the optima of several deterministic functions studied by Ackley were sought using variants of REINFORCE algorithms. Some of the algorithms used here incorporated additional heuristic features resembling certain aspects of some of the algorithms used in Ackley's studies. Differing levels of performance were achieved by the various algorithms investigated, but a number of them performed at a level comparable to the best found in Ackley's studies on a number of the tasks, in spite of their simplicity. One of these variants, called REINFORCE/MENT, represents a novel but principled approach to reinforcement learning in nontrivial networks which incorporates an entropy maximization strategy. This was found to perform especially well on more hierarchically organized tasks.     

Pseudo-recurrent Connectionist Networks: An Approach to the 'Sensitivity-Stability' Dilemma

In order to solve the 'sensitivity-stability' problem-and its immediate correlate, the problem of sequential learning-it is crucial to develop connectionist architectures that are simultaneously sensitive to, but not excessively disrupted by, new input. French (1992) suggested that to alleviate a particularly severe form of this disruption, catastrophic forgetting, it was necessary for networks to separate dynamically their internal representations during learning. McClelland et al. (1995) went even further. They suggested that nature's way of implementing this obligatory separation was the evolution of two separate areas of the brain, the hippocampus and the neocortex. In keeping with this idea of radical separation, a 'pseudo-recurrent' memory model is presented here that partitions a connectionist network into two functionally distinct, but continually interacting areas. One area serves as a final-storage area for representations; the other is an early-processing area where new representations are first learned by the system. The final-storage area continually supplies internally generated patterns (pseudopatterns; Robins, 1995), which are approximations of its content, to the early-processing area, where they are interleaved with the new patterns to be learned. Transfer of the new learning is done either by weight-copying from the early-processing area to the final-storage area or by pseudopattern transfer. A number of experiments are presented that demonstrate the effectiveness of this approach, allowing, in particular, effective sequential learning with gradual forgetting in the presence of new input. Finally, it is shown that the two interacting areas automatically produce representational compaction and it is suggested that similar representational streamlining may exist in the brain.     

Combining Connectionist and Symbolic Learning to Refine Certainty Factor Rule Bases

This paper describes RAPTURE—a system for revising probabilistic knowledge bases that combines connectionist and symbolic learning methods. RAPTURE uses a modified version of backpropagation to refine the certainty factors of a probabilistic rule base and it uses ID3's information-gain heuristic to add new rules. Results on refining three actual expert rule bases demonstrate that this combined approach generally performs better than previous methods.     

Contribution Analysis: A Technique for Assigning Responsibilities to Hidden Units in Connectionist Networks

Contributions, the products of hidden unit activations and weights, are presented as a valuable tool for investigating the inner workings of neural nets. Using a scaled-down version of NETtalk, a fully automated method for summarizing in a compact form both local and distributed hidden-unit responsibilities is demonstrated. Contributions are shown to be more useful for ascertaining hidden-unit responsibilities than either weights or hidden-unit activations. Among the results yielded by contribution analysis: for the example net, redundant output units are handled by identical patterns of hidden units, and the amount of responsibility a hidden unit takes on is inversely proportional to the number of hidden units.     

Integrating Neural and Symbolic Approaches: A Symbolic Learning Scheme for a Connectionist Associative Memory

This paper deals with the integration of neural and symbolic approaches. It focuses on associative memories where a connectionist architecture tries to provide a storage and retrieval component for the symbolic level. In this light, the classic model for associative memory, the Hopfield network is briefly reviewed. Then, a new model for associative memory, the hybrid Hopfield-clique network is presented in detail. Its application to a typically symbolic task, the post -processing of the output of an optical character recognizer, is also described. In the author's view, the hybrid Hopfield -clique network constitutes an example of a successful integration of the two approaches. It uses a symbolic learning scheme to train a connectionist network, and through this integration, it can provide perfect storage and recall. As a conclusion, an analysis of what can be learned from this specific architecture is attempted. In the case of this model, a guarantee for perfect storage and recall can only be given because it was possible to analyze the problem using the well-defined symbolic formalism of graph theory. In general, we think that finding an adequate formalism for a given problem is an important step towards solving it.     

Cryptotype,Overgeneralization and Competition: A Connectionist Model of the Learning of English Reversive Prefixes

This study examined the role of covert semantic classes or 'cryptotypes' in determining children's overgeneralizations of reversive prefixes such as un - in *unsqueeze or *unpress . A training corpus of 160 English verbs was presented incrementally to a backpropagation network. In three simulations, we showed that the network developed structured representations for the semantic cryptotype associated with the use of the reversive prefix un- . Overgeneralizations produced by the network, such as *unbury or *unpress , match up well with actual overgeneralizations observed in human children, showing that structured cryptotypic semantic representations underlie this overgeneralization behaviour. Simulation 2 points towards a role of lexical competition in morphological acquisition and overgeneralizations. Simulation 3 provides insight into the relationship between plasticity in network learning and the ability to recover from overgeneralizations. Together, these analyses paint a dynamic picture in which competing morphological devices work together to provide the best possible match to underlying covert semantic structures.     

Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.     

Recovery of Unrehearsed Items in Connectionist Models

When gradient-descent models with hidden units are retrained on a portion of a previously learned set of items, performance on both the relearned and unrelearned items improves. Previous explanations of this phenomenon have not adequately distinguished recovery, which is dependent on original learning, from generalization, which is independent of original learning. Using a measure of vector similarity to track global changes in the weight state of three-layer networks, we show that (a) unlike in networks without hidden units, recovery occurs in the absence of generalization in networks with hidden units, and (b) when the conditions of learning are varied, changes in the extent of recovery are reflected in changes in the extent to which the weights move back towards their values held after original learning. The implications of this work for rehabilitation studies, human relearning and models of human long-term memory are also considered.     

A Connectionist Account of Base-rate Biases in Categorization

In two experiments subjects were required to make medical diagnoses for simulated patients on the basis of the symptoms the patients had. Each patient had one of two diseases, the common disease, which occurred on 75% of trials, or the rare disease, which occurred on 25%, and these diseases occurred with 16 different combinations of four symptoms. In the first experiment, subjects learned across a large number of trials to associate the symptoms with the diseases, and the probability of diagnosing each disease for each symptom pattern was recorded. These probabilities were well modelled by a connectionist network. In the second experiment, a stronger test of the connectionist model was attempted. The crucial feature was that the probability of the rare disease given one particular symptom was equal to the probability of the common disease given the same symptom, but the contingency between the symptom and the rare disease was greater than that between the symptom and the common disease, which meant that the symptom was a better predictor of the rare disease. Subjects in one group were more likely to diagnose the rare disease than the common disease on these trials than were subjects in a control group, thus showing a bias associated with the differing base-rates of the diseases. The bias is consistent with the predictions of the connectionist account of categorization. In fact, while the data could be comfortably accommodated by a connectionist theory, they are difficult to reconcile with a variety of alternative theories of categorization. Finally, some possible limitations of connectionist accounts of categorization are considered.     

A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks

Most known learning algorithms for dynamic neural networks in non-stationary environments need global computations to perform credit assignment. These algorithms either are not local in time or not local in space. Those algorithms which are local in both time and space usually cannot deal sensibly with ‘hidden units’. In contrast, as far as we can judge, learning rules in biological systems with many ‘hidden units’ are local in both space and time. In this paper we propose a parallel on-line learning algorithms which performs local computations only, yet still is designed to deal with hidden units and with units whose past activations are ‘hidden in time’. The approach is inspired by Holland's idea of the bucket brigade for classifier systems, which is transformed to run on a neural network with fixed topology. The result is a feedforward or recurrent ‘neural’ dissipative system which is consuming ‘weight-substance’ and permanently trying to distribute this substance onto its connections in an appropriate way. Simple experiments demonstrating the feasibility of the algorithm are reported.     

A Connectionist Model of Choice and Reaction Time in Absolute Identification

A connectionist architecture is developed that can be used for modeling choice probabilities and reaction times in identification tasks. The architecture consists of a feedforward network and a decoding module, and learning is by mean-variance back-propagation, an extension of the standard back-propagation learning algorithm. We suggest that the new learning procedure leads to a better model of human learning in simple identification tasks than does standard back-propagation. Choice probabilities are modeled by the input-output relations of the network and reaction times are modeled by the time taken for the network, particularly the decoding module, to achieve a stable state. In this paper, the model is applied to the identification of unidimensional stimuli; applications to the identification of multidimensional stimuli—visual displays and words—is mentioned and presented in more detail in other papers. The strengths and weaknesses of this connectionist approach vis-à-vis other approaches are discussed