On fuzzy associative memory with multiple-rule storage capacity

Kosko's fuzzy associative memory (FAM) is the very first neural network model for implementing fuzzy systems. Despite its success in various applications, the model suffers from very low storage capacity, i.e., one rule per FAM matrix. A lot of hardware and computations are usually required to implement the model and, hence, it is limited to applications with small fuzzy rule-base. In this paper, the inherent property for storing multiple rules in a FAM matrix is identified. A theorem for perfect recalls of all the stored rules is established and based upon which the hardware and computation requirements of the FAM model can be reduced significantly. Furthermore, we have shown that when the FAM model is generalized to the one with max-bounded-product composition, single matrix implementation is possible if the rule-base is a set of semi-overlapped fuzzy rules. Rule modification schemes are also developed and the inference performance of the established high capacity models is reported through a numerical example     

Memory capacity and information capacity of sparsely encoded associative memory with replacement units

We introduce sparse encoding into the autoassociative memory model with replacement units. Utilizing computer simulation, we search the optimal number of replacement units in two terms: the memory capacity and the information capacity of the network. We show that the optimal number of replacement units to maximize the memory capacity and the information capacity decreases as the firing ratio decreases, and that the difference in the memory capacity between sparse encoding and non-sparse encoding becomes small as the number of replacement units increases.     

Dynamics of selective recall in an associative memory model with one-to-many associations.

The dynamics of selective recall in an associative memory model are analyzed in the scenario of one-to-many association. The present model, which can deal with one-to-many association, consists of a heteroassociative network and an autoassociative network. In the heteroassociative network, a mixture of associative items in one-to-many association is recalled by a key item. In the autoassociative network, the selective recall of one of the associative items is examined by providing a seed of a target item either to the heteroassociative network (Model 1) or to the autoassociative network (Model 2). We show that the critical similarity of Model 2 is not sensitive to the change in the dimension ratio of key vectors to associative vectors, and it has smaller critical similarity than Model 1 for a large initial overlap. On the other hand, we show that Model 1 has smaller critical similarity for a small initial overlap. We also show that unreachable equilibrium states exist in the proposed model.     

A dynamic associative memory system by adopting an amygdala model

Although several kinds of computational associative memory models and emotion models have been proposed since the last century, the interaction between memory and emotion is almost always neglected in these conventional models. This study constructs a dynamic memory system, named the amygdala-hippocampus model, which intends to realize dynamic auto-association and the mutual association of time-series patterns more naturally by adopting an emotional factor, i.e., the functional model of the amygdala given by Morén and Balkenius. The output of the amygdala is designed to control the recollection state of multiple chaotic neural networks (MCNN) in CA3 of the hippocampus-neocortex model proposed in our early work. The efficiency of the proposed association system is verified by computer simulation using several benchmark time-series patterns. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Storage of linguistic information in a continuous classifying associative memory

In this article, we analyze the use of the continuous classifying associative memory (CCLAM) to store linguistic information. Freedom in the choice of the functions that control the operation of the CCLAM equip this memory with the capacity to adapt to different information storage and recovery needs. We begin with the problem of storing linguistic terms by memorizing the patterns formed by the degrees of compatibility with these terms. After that, the problem of storing linguistic rules is discussed. Let us remark that in these cases not a single CCLAM is used, but rather a set of them connected in suitable structured ways. © 2002 Wiley Periodicals, Inc.     

A vector model of associative memory with clipped synapses

The influence of a clipping procedure on the properties of vector associative memory is investigated. The analysis is performed for the particular case of a phase model of a parametric neural network with 2q-state neurons. The critical network size N _c is found. It is shown that, for small network sizes (N < N _c ), the clipping leads to an increase of the storage capacity and enhances the network ability to retrieve strongly distorted patterns. Clipping of bigger networks (N > N _c ) leads to a deterioration of the recognition ability and reduces the storage capacity. Boris Vladimirovich Kryzhanovsky was born in 1950 in Yasnaya Polyana in the Tula region of Russia and graduated (with an M.Sc.) from Yerevan State University in 1971. He received his Ph.D. (Optics) in 1981 and his D.Sc. (Laser Physics) in 1991. At the present time, he is the director of the Center for Optical Neural Technologies of the Scientific Research Institute for Systems Analysis of the Russian Academy of Sciences. His research interests include neural networks. He is a corresponding member of the Russian Academy of Sciences and the author of over 200 research publications. Vladimir Mikhailovich Kryzhanovsky was born in 1984 in Kirovakan, Armenia and graduated (with an M.Sc.) from the Moscow Engineering Physics Institute in 2007. At the present time, he is a junior research assistant at the Center for Optical Neural Technologies of the Scientific Research Institute for Systems Analysis of the Russian Academy of Sciences. His research interests include Neural Networks, and he is the author of over 20 research publications. Dina Igorevna Simkina was born 1981 in Buinaksk in Dagestan, Russia and graduated (with an M.Sc.) from Dagestan State University in 2003. At the present time, she is a junior research assistant at the Center for Optical Neural Technologies of the Scientific Research Institute for Systems Analysis of the Russian Academy of Sciences. Her research interests include neural networks, and she is the author of over 20 research publications.     

A competitive associative memory model and its dynamics

Conventional associative memory networks perform "noncompetitive recognition" or "competitive recognition in distance". In this paper a "competitive recognition" associative memory model is introduced which simulates the competitive persistence of biological species. Unlike most of the conventional networks, the proposed model takes only the prototype patterns as its equilibrium points, so that the spurious points are effectively excluded. Furthermore, it is shown that, as the competitive parameters vary, the network has a unique stable equilibrium point corresponding to the winner competitive parameter and, in this case, the unique stable equilibrium state can be recalled from any initial key.     

Maximum memory capacity on neural networks with short-term synaptic depression and facilitation

In this work, we study, analytically and employing Monte Carlo simulations, the influence of the competition between several activity-dependent synaptic processes, such as short-term synaptic facilitation and depression, on the maximum memory storage capacity in a neural network. In contrast to the case of synaptic depression, which drastically reduces the capacity of the network to store and retrieve "static" activity patterns, synaptic facilitation enhances the storage capacity in different contexts. In particular, we found optimal values of the relevant synaptic parameters (such as the neurotransmitter release probability or the characteristic facilitation time constant) for which the storage capacity can be maximal and similar to the one obtained with static synapses, that is, without activity-dependent processes. We conclude that depressing synapses with a certain level of facilitation allow recovering the good retrieval properties of networks with static synapses while maintaining the nonlinear characteristics of dynamic synapses, convenient for information processing and coding.     

Part 3: Brain science, information science and associative memory model

I review recent progress on the associative memory model, which is a kind of neural network model. First, I introduce this model and a mathematical theory called statistical neurodynamics describing its properties. Next, I discuss an associative memory model with hierarchically correlated memory patterns. Initially, in this model, the state approaches a mixed state that is a superposition of memory patterns. After that, it diverges from the mixed state, and finally converges to a memory pattern. I show that this retrieval dynamics can qualitatively replicate the temporal dynamics of face-responsive neurons in the inferior temporal cortex, which is considered to be the final stage of visual perception in the brain. Finally, I show an unexpected link between associative memory and mobile phones (CDMA). The mathematical structure of the CDMA multi-user detection problem resembles that of the associative memory model. It enables us to apply a theoretical framework of the associative memory model to CDMA.     

An associative memory-based learning model with an efficient hardware implementation in FPGA

In this paper we propose a learning model based on a short- and long-term memory and a ranking mechanism which manages the transition of reference vectors between the two memories. Furthermore, an optimization algorithm is used to adjust the reference vectors components as well as their distribution, continuously. Comparing to other learning models like neural networks, the main advantage of the proposed model is that a pre-training phase is unnecessary and it has a hardware-friendly structure which makes it implementable by an efficient LSI architecture without requiring a large amount of resources. A prototype system is implemented on an FPGA platform and tested with real data of handwritten and printed English characters delivering satisfactory classification results.     

The rise and fall of memory in a model of synaptic integration

Plasticity-inducing stimuli must typically be presented many times before synaptic plasticity is expressed, perhaps because induction signals gradually accumulate before overt strength changes occur. We consider memory dynamics in a mathematical model with synapses that integrate plasticity induction signals before expressing plasticity. We find that the memory trace initially rises before reaching a maximum and then falling. The memory signal dissociates into separate oblivescence and reminiscence components, with reminiscence initially dominating recall. In radical contrast, related but nonintegrative models exhibit only a highly problematic oblivescence. Synaptic integration mechanisms possess natural timescales, depending on the statistics of the induction signals. Together with neuromodulation, these timescales may therefore also begin to provide a natural account of the well-known spacing effect in the transition to late-phase plasticity. Finally, we propose experiments that could distinguish between integrative and nonintegrative synapses. Such experiments should further elucidate the synaptic signal processing mechanisms postulated by our model.     

Pattern storage and associative memory in quasi-neural networks

Theoretical estimates and experimental data are given for the binary-pattern storage capacity of a quasi-neural network with threshold logic units. The theory confirms results obtained by Hopfield in 1982 for patterns with 50% density of active elements.     

Stability and oscillations of an autotroph-herbivore model with time delay

In the present study a mathematical model of a non-interactive type of autotrophherbivore system with discrete time delay due to gestation is proposed. The amount of autotroph biomass consumed by the herbivore is assumed to follow a Holling type-II function. We have derived the conditions for asymptotic stability and switching to instability of the steady state. The length of the delay preserving the stability has also been derived. Finally, the conditions for instability and bifurcation results have been derived for the linearized model. Phase portraits of the original nonlinear model have been simulated and the results have been interpreted ecologically.     

Performance analysis of the bidirectional associative memory and animproved model from the matched-filtering viewpoint

This paper discusses the bidirectional associative memory (BAM) model from the matched-filtering viewpoint and offers it a new interpretation. Our attention is focused on the problem of stability and attractivity of equilibrium states. Several sufficient and/or necessary conditions are presented. To improve the BAM performance, an exponential function is used to enhance the correlations between the binary vectors of the retrieval key and that of the stored pattern similar to the key. The modified model is shown to be asymptotically stable. Theoretical analysis and simulation results demonstrate that the modified model performs much better than the original BAM in terms of memory capacity and error correction capability.     

Optimality model of unsupervised spike-timing-dependent plasticity: synaptic memory and weight distribution

We studied the hypothesis that synaptic dynamics is controlled by three basic principles: (1) synapses adapt their weights so that neurons can effectively transmit information, (2) homeostatic processes stabilize the mean firing rate of the postsynaptic neuron, and (3) weak synapses adapt more slowly than strong ones, while maintenance of strong synapses is costly. Our results show that a synaptic update rule derived from these principles shares features, with spike-timing-dependent plasticity, is sensitive to correlations in the input and is useful for synaptic memory. Moreover, input selectivity (sharply tuned receptive fields) of postsynaptic neurons develops only if stimuli with strong features are presented. Sharply tuned neurons can coexist with unselective ones, and the distribution of synaptic weights can be unimodal or bimodal. The formulation of synaptic dynamics through an optimality criterion provides a simple graphical argument for the stability of synapses, necessary for synaptic memory.     

Dynamics analysis and analog associative memory of networks with LT neurons

The additive recurrent network structure of linear threshold neurons represents a class of biologically-motivated models, where nonsaturating transfer functions are necessary for representing neuronal activities, such as that of cortical neurons. This paper extends the existing results of dynamics analysis of such linear threshold networks by establishing new and milder conditions for boundedness and asymptotical stability, while allowing for multistability. As a condition for asymptotical stability, it is found that boundedness does not require a deterministic matrix to be symmetric or possess positive off-diagonal entries. The conditions put forward an explicit way to design and analyze such networks. Based on the established theory, an alternate approach to study such networks is through permitted and forbidden sets. An application of the linear threshold (LT) network is analog associative memory, for which a simple design method describing the associative memory is suggested in this paper. The proposed design method is similar to a generalized Hebbian approach, but with distinctions of additional network parameters for normalization, excitation and inhibition, both on a global and local scale. The computational abilities of the network are dependent on its nonlinear dynamics, which in turn is reliant upon the sparsity of the memory vectors.     

Discrete-time model for an adaptive optics system with input delay

The standard adaptive optics (AO) system can be viewed as a sampled-data feedback system with a continuous-time disturbance (the incident wavefront from the observed object) and discrete-time measurement noise. A common measure of performance of AO systems is the time average of the pupil variance of the residual wavefront. This performance can be related to that of a discrete-time system obtained by lifting the incident and residual wavefronts. This article derives the corresponding discrete-time model and the computation of the AO system residual variance based on that model. The predicted variance of a single mode of an AO system is shown to be the same as that obtained via simulation.     

Synthesis of a nonrecurrent associative memory model based on anonlinear transformation in the spectral domain

A new nonrecurrent associative memory model is proposed. This model is composed of a nonlinear transformation in the spectral domain followed by the association. The Moore-Penrose pseudoinverse is employed to obtain the least squares optimal solution. Computer simulations are done to evaluate the performance of the model. The simulations use one-dimensional speech signals and two-dimensional head/shoulder images. Comparison of the proposed model with the classical optimal linear associative memory and an optimal nonlinear associative memory is presented.     

Dynamics of an ecological model with impulsive control strategy and distributed time delay

In this paper, using the theories and methods of ecology and ordinary differential equations, an ecological model with an impulsive control strategy and a distributed time delay is defined. Using the theory of the impulsive equation, small-amplitude perturbations, and comparative techniques, a condition is identified which guarantees the global asymptotic stability of the prey-(x) and predator-(y) eradication periodic solution. It is proved that the system is permanent. Furthermore, the influences of impulsive perturbations on the inherent oscillation are studied numerically, an oscillation which exhibits rich dynamics including period-halving bifurcation, chaotic narrow or wide windows, and chaotic crises. Computation of the largest Lyapunov exponent confirms the chaotic dynamic behavior of the model. All these results may be useful for study of the dynamic complexity of ecosystems.