Stack-like and queue-like dynamics in recurrent neural networks

What dynamics do simple recurrent networks (SRNs) develop to represent stack-like and queue-like memories? SRNs have been widely used as models in cognitive science. However, they are interesting in their own right as non-symbolic computing devices from the viewpoints of analogue computing and dynamical systems theory. In this paper, SRNs are trained on two prototypical formal languages with recursive structures that need stack-like or queue-like memories for processing, respectively. The evolved dynamics are analysed, then interpreted in terms of simple dynamical systems, and the different ease with which SRNs aquire them is related to the properties of these simple dynamical systems. Within the dynamical systems framework, it is concluded that the stack-like language is simpler than the queue-like language, without making use of arguments from symbolic computation theory.     

Context-free and context-sensitive dynamics in recurrent neural networks

Continuous-valued recurrent neural networks can learn mechanisms for processing context-free languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown that qualitatively similar dynamics with similar constraints hold for aⁿbⁿcⁿ , a context-sensitive language. The additional difficulty with aⁿbⁿcⁿ , compared with the context-free language aⁿbⁿ , consists of 'counting up' and 'counting down' letters simultaneously. The network solution is to oscillate in two principal dimensions, one for counting up and one for counting down. This study focuses on the dynamics employed by the sequential cascaded network, in contrast to the simple recurrent network, and the use of backpropagation through time. Found solutions generalize well beyond training data, however, learning is not reliable. The contribution of this study lies in demonstrating how the dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages (traditionally thought of as requiring additional computation resources). This continuity of mechanism between language classes contributes to our understanding of neural networks in modelling language learning and processing.     

Connectionist Finite State Natural Language Processing

Unless one is prepared to argue that existing, ‘classical’formal language and automata theory, together with the natural language linguistics built on them, are fundamentally mistaken about the nature of language, any viable connectionist natural language processing (NLP) model will have to be characterizable, at least approximately, by some generative grammar or by an automaton of the corresponding class. An obvious way of ensuring that a connectionist NLP device is so characterizable is to specify it in classical terms and then to implement it in an artificial neural network, and that is what this paper does. It adopts the deterministic pushdown transducer (DPDT) as an adequate formal model for general NLP and shows how a simple recurrent network (SRN) can be trained to implement a finite state transducer (FST) which simulates the DPDT. A computer simulation of a parser for a small fragment of English is used to study the properties of the model. The conclusion is that such SRN implementation results in a device which is broadly consistent with its classical specification, but also has emergent properties relative to that specification which are desirable in an NLP device.     

非线性对象快速神经网络辨识方法

本文使用改进的神经网络模型结构与算法来辨识未知非线性系统，具有辨识精度高，速度快的特点。该方法简单有效，为设计非线性对象控制器提供了一条思路，从而摆脱了用线性模型近似被控对象的粗略做法。算法中，学习率采用随误差变化率而改变的做法减小了学习率选取的盲目性，加速了网络训练过程。     

Comparison of optimal motion planning algorithms for intelligent control of robotic part micro-assembly task

Two intelligent motion (or path) planning algorithms, one based on a neural network and the other based on a fuzzy coordinator, to mate a part with an assembly hole or a receptacle (target) without a jamming related to a robotic quasi-static part micro-assembly task are introduced. These algorithms are then compared by experiment results and several criteria. In the first algorithm, a neural network control strategy with a fuzzy entropy measure for avoiding jamming during the part micro-assembly is presented. An entropy function, which is a useful measure of the variability and the information in terms of uncertainty, is introduced to measure its overall performance of a task execution related to the part micro-assembly task. In the second algorithm, a learning control strategy with a fuzzy coordinator to minimize entropy and eliminate unneeded events in the plan related to avoiding jamming is described. Fuzzy set theory is introduced to address the uncertainty associated with the part micro-assembly procedure. The degree of uncertainty associated with the part micro-assembly is used as an optimality criterion, e.g. minimum fuzzy entropy, for a specific task execution. It is shown that the machine organizer using a sensor system can intelligently determine an optimal control value, based on explicit performance criteria. The algorithms utilize knowledge processing functions such as machine reasoning, planning, inferencing, learning, and decision-making. The results show the effectiveness of the proposed approaches. The proposed techniques are applicable to a wide range of robotic tasks including pick and place operations, motion planning, and part mating with various shaped parts.     

Interactive natural language acquisition in a multi-modal recurrent neural architecture

ABSTRACT

For the complex human brain that enables us to communicate in natural language, we gathered good understandings of principles underlying language acquisition and processing, knowledge about sociocultural conditions, and insights into activity patterns in the brain. However, we were not yet able to understand the behavioural and mechanistic characteristics for natural language and how mechanisms in the brain allow to acquire and process language. In bridging the insights from behavioural psychology and neuroscience, the goal of this paper is to contribute a computational understanding of appropriate characteristics that favour language acquisition. Accordingly, we provide concepts and refinements in cognitive modelling regarding principles and mechanisms in the brain and propose a neurocognitively plausible model for embodied language acquisition from real-world interaction of a humanoid robot with its environment. In particular, the architecture consists of a continuous time recurrent neural network, where parts have different leakage characteristics and thus operate on multiple timescales for every modality and the association of the higher level nodes of all modalities into cell assemblies. The model is capable of learning language production grounded in both, temporal dynamic somatosensation and vision, and features hierarchical concept abstraction, concept decomposition, multi-modal integration, and self-organisation of latent representations.     

基于人工神经网络的金属土壤腐蚀预测方法

将神经网络用于金属土壤腐蚀研究，利用神经网络的学习特征和高度的非线性特征，以土壤理化性能，腐蚀时间，Ａ３钢在土腐蚀试验１，２，８个月的腐蚀数据作为网络训练样本，对土壤中埋片２４个月的Ａ３钢腐蚀速率进行预测，并对结果进行了分析。     

Competitive Learning and its Application in Adaptive Vision for Autonomous Mobile Robots

The task of providing robust vision for autonomous mobile robots is a complex signal processing problem which cannot be solved using traditional deterministic computing techniques. In this article we investigate four unsupervised neural learning algorithms, known collectively as competitive learning, in order to assess both their theoretical operation and their ability to learn to represent a basic robotic vision task. This task involves the ability of a modest robotic system to identify the components of basic motion and to generalize upon that learned knowledge to classify correctly novel visual experiences. This investigation shows that standard competitive learning and the DeSieno version of frequency-sensitive competitive learning (FSCL) are unsuitable for solving this problem. Soft competitive learning, while capable of producing an appropriate solution, is too computationally expensive in its present form to be used under the constraints of this application. However, the Krishnamurthy version of FSCL is found to be both computationally efficient and capable of reliably learning a suitable solution to the motion identification problem both in simulated tests and in actual hardware-based experiments.     

复合正交柔性神经网络及其应用

针对目前神经网络所存在的不足,提出一种带参数的单极性Sigmoid函数的柔性复合正交神经网络,并给出相应的参数学习算法,这种柔性复合正交神经网络不仅扩大了网络辨识模型的能力与学习适应性,而且算法简单,学习收敛速度快,有线性,非线性逼近精度高等优异特性。以模型辨识作为应用实例,仿真结果表明,其算法是有效的,柔性神经网络能提高正交神经网络的性能。     

An Approach to Combining Explanation-based and Neural Learning Algorithms

Machine learning is an area where both symbolic and neural approaches to artificial intelligence have been heavily investigated. However, there has been little research into the synergies achievable by combining these two learning paradigms. A hybrid system that combines the symbolically-oriented explanation-based learning paradigm with the neural backpropagation algorithm is described. In the presented EBL-ANN algorithm, the initial neural network configuration is determined by the generalized explanation of the solution to a specific classification task. This approach overcomes problems that arise when using imperfect theories to build explanations and addresses the problem of choosing a good initial neural network configuration. Empirical results show that the hybrid system more accurately learns a concept than the explanation-based system by itself and learns faster and generalizes better than the neural learning system by itself.     

基于Bi-GRU模型的航空发动机外部液压管路故障诊断研究

针对航空液压管路故障信号含有噪声干扰导致管路故障识别困难的问题，提出一种基于双向门控循环单元(Bi-GRU)的深度学习液压管路故障诊断方法。由Bi-GRU神经网络模型综合液压管路数据进行时序特征提取，基于同一含噪声的液压管路振动实测数据，输入到Bi-GRU、GRU、RNN、SVM、BPNN等5种故障诊断模型中进行训练。最后，为了进一步展示Bi-GRU模型对于航空液压管路不同故障类型特征的学习能力，利用t-SNE降维算法进行液压管路特征可视化。结果表明：基于Bi-GRU航空故障诊断方法能达到9960%的准确性，明显优于GRU等其他4种神经网络模型，Bi-GRU模型在含有噪声的液压管路数据上具备更出色的特征提取能力，可有效地提取出液压管路故障数据特征，从而实现了液压管路故障的智能化识别。     

灰色神经网络在高速铣削表面质量预测模型中的应用

高速铣削加工技术在制造业中应用广泛,为了快捷、全面地获取其表面质量的加工工艺特性,设计了L9(34)多因素正交试验,在此基础上结合灰色理论小样本、简单与神经网络非线性超强的特性,建立了灰色神经组合模型,试验证明该模型具有精确的拟合精度和理想的泛化能力,为高速铣削加工工艺的仿真研究提供了一条新的途径。     

Chebyshev神经网络的改进及其应用

针对目前Chebyshev神经网络所存在的不足，提出一种改进的Chebyshev神经网络，它使用多输入多输出神经网络结构与使用改进的Chebyshev正交多项式。因此改进的神经网络不仅扩大了网络辨识模型的能力与学习适应性，而且算法简单，学习收敛速度快，有线性、非线性逼近精度高等优异特性。文中给出两个应用实例，仿真结果表明是有效的。     

The classification game: combining supervised learning and language evolution

We study the emergence of shared representations in a population of agents engaged in a supervised classification task, using a model called the classification game. We connect languages with tasks by treating the agents’ classification hypothesis space as an information channel. We show that by learning through the classification game, agents can implicitly perform complexity regularisation, which improves generalisation. Improved generalisation also means that the languages that emerge are well adapted to the given task. The improved language-task fit springs from the interplay of two opposing forces: the dynamics of collective learning impose a preference for simple representations, while the intricacy of the classification task imposes a pressure towards representations that are more complex. The push–pull of these two forces results in the emergence of a shared representation that is simple but not too simple. Our agents use artificial neural networks to solve the classification tasks they face, and a simple counting algorithm to learn a language as a form-meaning mapping. We present several experiments to demonstrate that both compositional and holistic languages can emerge in our system. We also demonstrate that the agents avoid overfitting on noisy data, and can learn some very difficult tasks through interaction, which they are unable to learn individually. Further, when the agents use simple recurrent networks to solve temporal classification tasks, we see the emergence of a rudimentary grammar, which does not have to be explicitly learned.     

A spiking half-cognitive model for classification

This paper describes a spiking neural network that learns classes. Following a classic Psychological task, the model learns some types of classes better than other types, so the net is a spiking cognitive model of classification. A simulated neural system, derived from an existing model, learns natural kinds, but is unable to form sufficient attractor states for all of the types of classes. An extension of the model, using a combination of singleton and triplets of input features, learns all of the types. The models make use of a principled mechanism for spontaneous firing, and a compensatory Hebbian learning rule. Combined, the mechanisms allow learning to spread to neurons not directly stimulated by the environment. The overall network learns the types of classes in a fashion broadly consistent with the Psychological data. However, the order of speed of learning the types is not entirely consistent with the Psychological data, but may be consistent with one of two Psychological systems a given person possesses. A Psychological test of this hypothesis is proposed.     

Visual Schemas in Neural Networks for Object Recognition and Scene Analysis

VISOR is a large connectionist system that shows how visual schemas can be learned, represented and used through mechanisms natural to neural networks. Processing in VISOR is based on cooperation, competition, and parallel bottom-up and top-down activation of schema representations. VISOR is robust against noise and variations in the inputs and parameters. It can indicate the confidence of its analysis, pay attention to important minor differences, and use context to recognize ambiguous objects. Experiments also suggest that the representation and learning are stable, and behavior is consistent with human processes such as priming, perceptual reversal and circular reaction in learning. The schema mechanisms of VISOR can serve as a starting point for building robust high-level vision systems, and perhaps for schema-based motor control and natural language processing systems as well.     

Symbols Language Neural Networks Performance

An implementation of non-regular symbol manipulation with neural networks is presented. In particular, it is shown how a context-free language can be produced with neural networks. The rules of the language are stored as patterns in an attractor neural network. Another such network is used as a working memory, which can be enlarged without changing the production system itself. As a result, the competence of symbol manipulation with neural networks equals that of classical non-regular production systems. In actual behaviour (performance), however, there are differences between the systems, which shows the importance of implementation in the generation of rule-like behaviour.     

STUDY ON PROPERTY PREDICTION FOR SEALING ALLOYS

ＳＴＵＤＹＯＮＰＲＯＰＥＲＴＹＰＲＥＤＩＣＴＩＯＮＦＯＲＳＥＡＬＩＮＧＡＬＬＯＹＳ￥Ｚ．Ｎ．Ｘｉａ;Ｓ．Ｇ．Ｌａｉ;Ｙ．Ｚ．ＳｕｎａｎｄＹ．Ｗ．Ｌｕ（ＤｅｐａｒｔｍｅｎｔｏｆＭａｔｅｒｉａｌｓＳｃｉｅｎｃｅａｎｄＥｎｇｉｎｅｅｒｉｎｇ,Ｔｓｉｎｇｈｕ...     

Retroactive interference in neural networks and in humans: The effect of pattern-based learning

Catastrophic interference is addressed as a problem that arises from pattern-based learning algorithms. As such, it is not limited to artificial neural networks but can be demonstrated in human subjects in so far as they use a pattern-based learning strategy. The experiment tests retroactive interference in humans learning lists of consonant-vowel-consonant nonsense syllable pairs. Results show significantly more interference for subjects learning patterned lists than subjects learning arbitrarily paired lists. To examine how different learning strategies depend on the structure of the learning task, a mixture-of-experts neural network model is presented. The results show how these strategies may interact to give rise to the results seen in the human data.