Computational models of location-invariant orthographic processing

We trained three topologies of backpropagation neural networks to discriminate 2000 words (lexical representations) presented at different positions of a horizontal letter array. The first topology (zero-deck) contains no hidden layer, the second (one-deck) has a single hidden layer, and for the last topology (two-deck), the task is divided in two subtasks implemented as two stacked neural networks, with explicit word-centred letters as intermediate representations. All topologies successfully simulated two key benchmark phenomena observed in skilled human reading: transposed-letter priming and relative-position priming. However, the two-deck topology most accurately simulated the ability to discriminate words from nonwords, while containing the fewest connection weights. We analysed the internal representations after training. Zero-deck networks implement a letter-based scheme with a position bias to differentiate anagrams. One-deck networks implement a holographic overlap coding in which representations are essentially letter-based and words are linear combinations of letters. Two-deck networks also implement holographic-coding.     

A Distributed Memory Model of the Associative Boost in Semantic Priming

Evidence from priming studies indicates that both semantic and associative relations between pairs of words facilitate word recognition, and that pairs of words related in both ways (e.g. hammer-nail, cat-dog) produce an additional ‘associative boost’. We argue that while semantic priming may result from overlapping patterns of micro-features in a distributed memory model (e.g. Masson, 1991), associative priming is a result of frequent co-occurrence of words in the language. We describe a simple recurrent network, with distributed phonological and semantic representations, which is sensitive to the sequential occurrence of phonological patterns during training, and which produces associative facilitation of word recognition in a simulation of the priming task.     

Emergent latent symbol systems in recurrent neural networks

Fodor and Pylyshyn [(1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1–2), 3–71] famously argued that neural networks cannot behave systematically short of implementing a combinatorial symbol system. A recent response from Frank et al. [(2009). Connectionist semantic systematicity. Cognition, 110(3), 358–379] claimed to have trained a neural network to behave systematically without implementing a symbol system and without any in-built predisposition towards combinatorial representations. We believe systems like theirs may in fact implement a symbol system on a deeper and more interesting level: one where the symbols are latent – not visible at the level of network structure. In order to illustrate this possibility, we demonstrate our own recurrent neural network that learns to understand sentence-level language in terms of a scene. We demonstrate our model's learned understanding by testing it on novel sentences and scenes. By paring down our model into an architecturally minimal version, we demonstrate how it supports combinatorial computation over distributed representations by using the associative memory operations of Vector Symbolic Architectures. Knowledge of the model's memory scheme gives us tools to explain its errors and construct superior future models. We show how the model designs and manipulates a latent symbol system in which the combinatorial symbols are patterns of activation distributed across the layers of a neural network, instantiating a hybrid of classical symbolic and connectionist representations that combines advantages of both.     

Position-specific effects within the SERIOL framework of letter-position coding

In the SERIOL (sequential encoding regulated by inputs to oscillations within letter units) framework, letter position within a word is encoded by the temporal firing pattern ofletter units. As opposed to channel-specific schemes, a letter unit can potentially represent any position. This lack of positional specificity is consistent with studies showing that priming can occur across different positions within a letter string (Grainger and Jacobs 1991, European Journal of Cognitive Psychology , 3: 413-434, Peressotti and Grainger 1995, Perception of Psychophysics , 57: 875-890). However, these studies also showed that same-position priming is more robust than this cross-position priming. This result seems inconsistent with non-position-specific letter units. Here we give an explanation of these results under the SERIOL framework. Using mathematical models, we show that broadly tuned feature detectors and an activational gradient can account for the complex experimental data on position-specific and cross-position letter priming.     

Lack of combinatorial productivity in language processing with simple recurrent networks

An astronomical set of sentences can be produced in natural language by combining relatively simple sentence structures with a human-size lexicon. These sentences are within the range of human language performance. Here, we investigate the ability of simple recurrent networks (SRNs) to handle such combinatorial productivity. We successfully trained SRNs to process sentences formed by combining sentence structures with different groups of words. Then, we tested the networks with test sentences in which words from different training sentences were combined. The networks failed to process these sentences, even though the sentence structures remained the same and all words appeared on the same syntactic positions as in the training sentences. In these combination cases, the networks produced work–word associations, similar to the condition in which words are presented in the context of a random word sequence. The results show that SRNs have serious difficulties in handling the combinatorial productivity that underlies human language performance. We discuss implications of this result for a potential neural architecture of human language processing.     

基于人工神经网络的气门电镦工艺参数的选择

由于气门电热镦粗的工艺参数大多数是借助于经验来选择的,某些参数的选择不合理或者参数之间配合不好,都会造成工艺不稳定,从而使生产的成品率下降.并且在成形过程中,电加热和镦粗同时进行,很难建立合理且实用的数学模型.本文利用神经网络具有黑箱特性和非线形映射能力强的特点,提出了一种组合神经网络结构（ANN）来逐步确定气门电热镦粗的工艺参数.以实际生产中的数据作为ANN的学习训练样本,经过训练的网络不仅可以达到描述确定气门电镦成形控制参数的目的,而且还具有一定的预测功能,从而为气门电镦工艺提供了较合理的控制参数.     

Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs terms of predictive accuracy and simplicity.     

Transfer in Cognition

The purpose of this paper is to review the cognitive literature regarding transfer in order to provide a context for the consideration of transfer in neural networks. We consider transfer under the three general headings of analogy, skill transfer and metaphor. The emphasis of the research in each of these areas is quite different and the literatures are largely distinct. Important common themes emerge, however, relating to the role of similarity, the importance of 'surface content' and the nature of the representations that are used. We will draw out these common themes and note ways of facilitating transfer. We also briefly note possible implications for the study of transfer in neural networks.     

Artificial neural networks whispering to the brain: nonlinear system attractors induce familiarity with never seen items

Attractors of nonlinear neural systems are at the core of the memory self-refreshing mechanism of human memory models that suppose memories are dynamically maintained in a distributed network [Ans, B., and Rousset, S. (1997), ‘Avoiding Catastrophic Forgetting by Coupling Two Reverberating Neural Networks’ Comptes Rendus de l'Académie des Sciences Paris, Life Sciences, 320, 989–997; Ans, B., and Rousset, S. (2000), ‘Neural Networks with a Self-Refreshing Memory: Knowledge Transfer in Sequential Learning Tasks Without Catastrophic Forgetting’, Connection Science, 12, 1–19; Ans, B., Rousset, S., French, R.M., and Musca, S.C. (2002), ‘Preventing Catastrophic Interference in Multiple-Sequence Learning Using Coupled Reverberating Elman Networks’, in Proceedings of the 24th Annual Meeting of the Cognitive Science Society, eds. W.D. Gray and C.D. Schunn, Mahwah, NJ: Lawrence Erlbaum Associates, pp. 71–76; Ans, B., Rousset, S., French, R.M., and Musca, S.C. (2004), ‘Self-Refreshing Memory in Artificial Neural Networks: Learning Temporal Sequences Without Catastrophic Forgetting’, Connection Science, 16, 71–99; Ans, B. (2004), ‘Sequential Learning in Distributed Neural Networks Without Catastrophic Forgetting: A Single and Realistic Self-Refreshing Memory can do it’, Neural Information Processing-Letters and Reviews, 4, 27–32]. Are humans able to learn never seen items from attractor patterns generated by a highly distributed artificial neural network? First, an opposition method was implemented to ensure that the attractors are not the items used to train the network, the source items: attractors were selected to be more similar (both at the exemplar and the centroïd level) to some control items than to the source items. In spite of this very severe selection, blank networks trained only on selected attractors performed better at test on the never seen source items than on the never seen control items. The results of two behavioural experiments using the opposition method show that humans exhibit more familiarity with the never seen source items than with the never seen control items, just as networks do. Thus, humans are sensitive to the particular type of information that allows distributed artificial neural networks to dynamically maintain their memory, and this information does not amount to the exemplars used to train the network that produced the attractors.     

Distributed representations for extended syntactic transformation

This paper shows how the choice of representation substantially affects the generalization performance of connectionistnetworks. The starting point is Chalmers' simulations involving structure-sensitive processing. Chalmers argued that a connectionist network could handle structure sensitive processing without the use of syntactically structured representations. He trained a connectionist architecture to encode/decode distributed representations for simple sentences. These distributed representations were then holistically transformed such that active sentences were transformed into their passive counterpart. However, he noted that the recursive auto-associative memory (RAAM), which was used to encode and decode distributed representations for the structures, exhibited only a limited ability to generalize when trained to encode/decode a randomly selected sample of the total corpus. When the RAAM was trained to encode/decode all sentences, and a separate transformation network was trained to make some active-passive transformations of the RAAMencoded sentences, the transformation network demonstrated perfect generalization on the remaining test sentences. It is argued here that the main reason for the limited generalization is not the ability of the RAAM architecture per se, but the choice of representation for the tokens used. This paper shows that 100% generalization can be achieved for Chalmers' original set up (i.e. using only 30% of the total corpus for training). The key to this success is to use distributed representations for the tokens (capturing different characteristics for differentclasses of tokens, e.g. verbs or nouns).     

Extraction,Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks

Recurrent neural networks readily process, learn and generate temporal sequences. In addition, they have been shown to have impressive computational power. Recurrent neural networks can be trained with symbolic string examples encoded as temporal sequences to behave like sequential finite slate recognizers. We discuss methods for extracting, inserting and refining symbolic grammatical rules for recurrent networks. This paper discusses various issues: how rules are inserted into recurrent networks, how they affect training and generalization, and how those rules can be checked and corrected. The capability of exchanging information between a symbolic representation (grammatical rules)and a connectionist representation (trained weights) has interesting implications. After partially known rules are inserted, recurrent networks can be trained to preserve inserted rules that were correct and to correct through training inserted rules that were ‘incorrec’—rules inconsistent with the training data.     

Behavioural Aspects of Combining Backpropagation Learning and Self-organizing Maps

Backpropagation learning (BP) is known for its serious limitations in generalizing knowledge from certain types of learning material. In this paper, we describe a new learning algorithm, BP-SOM, which overcomes some of these limitations as is shown by its application to four benchmark tasks. BP-SOM is a combination of a multi-layered feedforward network (MFN) trained with BP and Kohonen's self-organizing maps (SOMs). During the learning process, hidden-unit activations of the MFN are presented as learning vectors to SOMs trained in parallel. The SOM information is used when updating the connection weights of the MFN in addition to standard error backpropagation. The effect of the augmented error signal is that, during learning, clusters of hiddenunit activation patterns of instances associated with the same class tend to become highly similar. In a number of experiments, BP-SOM is shown (i) to improve generalization performance (i.e. avoid overfitting); (ii) to increase the amount of hidden units that can be pruned without loss of generalization performance and (iii) to provide a means for automatic rule extraction from trained networks. The results are compared with results achieved by two other learning algorithms for MFNs: conventional BP and BP augmented with weight decay. From the experiments and the comparisons, we conclude that the hybrid BP-SOM architecture, in which supervised and unsupervised and learning co-operate in finding adequate hidden-layer representations, successfully combines the advantages of supervised and unsupervised learning.     

Connectionism,Processing, Memory

computing devices such as Turing machines resolve the dilemma between the necessary finitude of effective procedures and the potential infinity of a function's domain by distinguishing between a finite-state processing part, defined over finitely many representation types, and a memory sufficiently large to contain representation tokens for any of the function's arguments and values. Connectionist networks have been shown to be (at least) Turing-equivalent if provided with infinitely many nodes or infinite-precision activation values and weights. Physical computation, however, is necessarily finite.

The notion of a processing-memory system is introduced to discuss physical computing systems. Constitutive for a processing-memory system is that its causal structure supports the functional distinction between processing part and memory necessary for employing a type-token distinction for representations, which in turn allows for representations to be the objects of computational manipulation. Moreover, the processing part realized by such systems provides a criterion of identity for the function computed as well as helps to define competence and performance of a processing-memory system.

Networks, on the other hand, collapse the functional distinction between processing part and memory. Since preservation of this distinction is necessary for employing a type-token distinction for representation, connectionist information processing does not consist in the computational manipulation of representations. Moreover, since we no longer have a criterion of identity for the function processed other than the behaviour of the network itself, we are left without a competence-performance distinction for connectionist networks,     

联想记忆网络及其在刀具磨损量监测中的应用

文章介绍了联想记忆网络的基本概念、组成特点及其在刀具磨损监测中的应用,详细分析了一种格构联想记忆网络-B样条模糊神经网络的结构和算法.研究表明,应用B样条模糊神经网络构造的刀具磨损量监测系统,与BP型前馈神经网络相比,具有训练时间短,拟合精度高,局部推广能力强等特点,有较高的工程应用推广价值.     

Effects of Collinearity on Combining Neural Networks

Collinearity or linear dependency among a number of estimators may pose a serious problem when combining these estimators. The corresponding outputs of a number of neural networks NNs , which are trained to approximate the same quantity or quantities , may be highly correlated. Thus, the estimation of the optimal weights for combining such networks may be subjected to the harmful effects of collinearity, which results in a final model with inferior generalizations ability compared with the individual networks. In this paper, we investigate the harmful effects of collinearity on the estimation of the optimal weights for combining a number on NNs. We discuss an approach for selecting the component networks in order to improve the generalization ability of the combined model. Our experimental results demonstrate significant improvements in the generalization ability of a combined model as a result of the proper selection of the component networks. The approximation accuracy of the combined model is compared with two common alternatives: the apparent best network or the simple average of the corresponding outputs of the networks.     

Visual Schemas in Neural Networks for Object Recognition and Scene Analysis

VISOR is a large connectionist system that shows how visual schemas can be learned, represented and used through mechanisms natural to neural networks. Processing in VISOR is based on cooperation, competition, and parallel bottom-up and top-down activation of schema representations. VISOR is robust against noise and variations in the inputs and parameters. It can indicate the confidence of its analysis, pay attention to important minor differences, and use context to recognize ambiguous objects. Experiments also suggest that the representation and learning are stable, and behavior is consistent with human processes such as priming, perceptual reversal and circular reaction in learning. The schema mechanisms of VISOR can serve as a starting point for building robust high-level vision systems, and perhaps for schema-based motor control and natural language processing systems as well.     

A Network for Encoding,Decoding and Translating Locative Prepositions

In general, the spatial relationship between two objects can be roughly expressed by a locative expression of the form ‘noun-preposition-noun’. The mapping from preposition to meaning is context sensitive; for example, the preposition ‘on’ expresses different spatial relationships in the phrases ‘house on lake’ and ‘plate on table’. While very accurate selection of the proper sense of a preposition depends on the broad context, the immediate context (the two nouns) can often lend enough information to make a reasonable judgement. Back-propagation is used to train a feed-forward network to associate locative prepositions with semantic representations using several contexts (noun pairs). After training, the network can produce a core meaning for each preposition; this prototype meaning can be altered by the influence of the nouns, even for pairs which are not included in the training set. A toy example of machine translation of prepositions is presented in which two networks are trained, one using English prepositions, the other using German prepositions. After successful training of the two networks, attempts were made at translating from one language to the other.     

High temperature deformation behavior of Ti−6Al−4V alloy with and equiaxed microstructure: a neural networks analysis

The hot deformation behavior of Ti−6Al−4V alloy with an equiaxed microstructure was investigated by means of Artificial Neural Networks (ANN). The flow stress data for the ANN model training was obtained from compression tests performed on a thermo-mechanical simulator over a wide range of temperature (from 700°C to 1100°C) with strain rates of 0.0001 s⁻¹ to 100 s⁻¹ and true strains of 0.1 to 0.6. It was found that the trained neural network could reliably predict flow stress for unseen data. Workability was evaluated by means of processing maps with respect to strain, strain rate, and temperature. Processing maps were constructed at different strains by utilizing the flow stress predicted by the model at finer intervals of strain rates and temperatures. The specimen failures at various instances were predicted and confirmed by experiments. The results establish that artificial neural networks can be effectively used for generating a more reliable processing map for industrial applications. A graphical user interface was designed for ease of use of the model.     

Selecting an artificial neural network for efficient modeling and accurate simulation of the milling process

In this paper, two supervised neural networks are used to estimate the forces developed during milling. These two Artificial Neural Networks (ANNs) are compared based on a cost function that relates the size of the training data to the accuracy of the model. Training experiments are screened based on design of experiments. Verification experiments are conducted to evaluate these two models. It is shown that the Radial Basis Network model is superior in this particular case. Orthogonal design and specifically equally spaced dimensioning showed to be a good way to select the training experiments.     

Speaker Normalization and Model Selection of Combined Neural Networks

This paper introduces bootstrap error estimation for automatic tuning of parameters in combined networks, applied as front-end preprocessors for a speech recognition system based on hidden Markov models. The method is evaluated on a large-vocabulary (10 000 words) continuous speech recognition task. Bootstrap estimates of minimum mean squared error allow selection of speaker normalization models improving recognition performance. The procedure allows a flexible strategy for dealing with inter-speaker variability without requiring an additional validation set. Recognition results are compared for linear, generalized radial basis functions and multi-layer perceptron network architectures.