A new neural associative memory model

We present a new associative memory model based on the Hamming memory, but where the winner-take-all network part is replaced by a layer of nodes with somewhat complex node functions. This new memory can produce output vectors with individual “don't know” bits. the simulations demonstrate that this memory model works appropriately. © 1992 John Wiley & Sons, Inc.     

A competitive associative memory model and its dynamics

Conventional associative memory networks perform "noncompetitive recognition" or "competitive recognition in distance". In this paper a "competitive recognition" associative memory model is introduced which simulates the competitive persistence of biological species. Unlike most of the conventional networks, the proposed model takes only the prototype patterns as its equilibrium points, so that the spurious points are effectively excluded. Furthermore, it is shown that, as the competitive parameters vary, the network has a unique stable equilibrium point corresponding to the winner competitive parameter and, in this case, the unique stable equilibrium state can be recalled from any initial key.     

A dynamic associative memory system by adopting an amygdala model

Although several kinds of computational associative memory models and emotion models have been proposed since the last century, the interaction between memory and emotion is almost always neglected in these conventional models. This study constructs a dynamic memory system, named the amygdala-hippocampus model, which intends to realize dynamic auto-association and the mutual association of time-series patterns more naturally by adopting an emotional factor, i.e., the functional model of the amygdala given by Morén and Balkenius. The output of the amygdala is designed to control the recollection state of multiple chaotic neural networks (MCNN) in CA3 of the hippocampus-neocortex model proposed in our early work. The efficiency of the proposed association system is verified by computer simulation using several benchmark time-series patterns. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

A feedforward bidirectional associative memory

In contrast to conventional feedback bidirectional associative memory (BAM) network models, a feedforward BAM network is developed based on a one-shot design algorithm of O(p(2)(n+m)) computational complexity, where p is the number of prototype pairs and n, m are the dimensions of the input/output bipolar vectors. The feedforward BAM is an n-p-m three-layer network of McCulloch-Pitts neurons with storage capacity 2(min{m,n}) and guaranteed perfect bidirectional recall. The overall network design procedure is fully scalable in the sense that any number p=/<2(min{m,n}) of bidirectional associations can be implemented. The prototype patterns may be arbitrarily correlated. With respect to inference performance, it is shown that the Hamming attractive radius of each prototype reaches the maximum possible value. Simulation studies and comparisons illustrate and support these theoretical developments.     

Associative memory with dynamic synapses

We have examined a role of dynamic synapses in the stochastic Hopfield-like network behavior. Our results demonstrate an appearance of a novel phase characterized by quick transitions from one memory state to another. The network is able to retrieve memorized patterns corresponding to classical ferromagnetic states but switches between memorized patterns with an intermittent type of behavior. This phenomenon might reflect the flexibility of real neural systems and their readiness to receive and respond to novel and changing external stimuli.     

Dynamics of selective recall in an associative memory model with one-to-many associations.

The dynamics of selective recall in an associative memory model are analyzed in the scenario of one-to-many association. The present model, which can deal with one-to-many association, consists of a heteroassociative network and an autoassociative network. In the heteroassociative network, a mixture of associative items in one-to-many association is recalled by a key item. In the autoassociative network, the selective recall of one of the associative items is examined by providing a seed of a target item either to the heteroassociative network (Model 1) or to the autoassociative network (Model 2). We show that the critical similarity of Model 2 is not sensitive to the change in the dimension ratio of key vectors to associative vectors, and it has smaller critical similarity than Model 1 for a large initial overlap. On the other hand, we show that Model 1 has smaller critical similarity for a small initial overlap. We also show that unreachable equilibrium states exist in the proposed model.     

A property of neural networks of associative memory with replacing units

A memory capacity exists for artificial neural networks of associative memory. The addition of new memories beyond the capacity overloads the network system and makes all learned memories irretrievable (catastrophic forgetting) unless there is a provision for forgetting old memories. This article describes a property of associative memory networks in which a number of units are replaced when networks learn. In our network, every time the network learns a new item or pattern, a number of units are erased and the same number of units are added. It is shown that the memory capacity of the network depends on the number of replaced units, and that there exists a optimal number of replaced units in which the memory capacity is maximized. The optimal number of replaced units is small, and seems to be independent of the network size. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

Oscillatory neural associative memories with synapses based on memristor bridges

An approach to the implementation of electronic associative memories with tunable weights based on the resistor bridges containing memristors—a bidirectional associative memory (BAM) and an associative memory based on the Hopfield network—is proposed. These memories we implement as a networks of coupled phase oscillators. The conditions for the use of the operational amplifier in a comparator mode for implementing the step activation function are determined. It is shown how to use the CMOS transistors switches to control the memristance value. The experiments using LTSPICE models show that for the reference binary images with size 3 × 3 the proposed networks converges to the reference images (and, accordingly, to their inversion) with a random uniform distribution of binary pixel values of the input images. In all experiments we have no error states in spite of the number of reference patterns exceeds the classical estimations for traditional BAM and Hopfield networks.     

Neural associative memory with optimal Bayesian learning

Neural associative memories are perceptron-like single-layer networks with fast synaptic learning typically storing discrete associations between pairs of neural activity patterns. Previous work optimized the memory capacity for various models of synaptic learning: linear Hopfield-type rules, the Willshaw model employing binary synapses, or the BCPNN rule of Lansner and Ekeberg, for example. Here I show that all of these previous models are limit cases of a general optimal model where synaptic learning is determined by probabilistic Bayesian considerations. Asymptotically, for large networks and very sparse neuron activity, the Bayesian model becomes identical to an inhibitory implementation of the Willshaw and BCPNN-type models. For less sparse patterns, the Bayesian model becomes identical to Hopfield-type networks employing the covariance rule. For intermediate sparseness or finite networks, the optimal Bayesian learning rule differs from the previous models and can significantly improve memory performance. I also provide a unified analytical framework to determine memory capacity at a given output noise level that links approaches based on mutual information, Hamming distance, and signal-to-noise ratio.     

Operation of an associative memory

This paper describes the operation of an associative memory (LYAM) governed by only ordinary differential equations, useful for pattern clustering. Several computer simulations illustrate its operation as an unsupervised classifier, vector quantizer, and content-addressable memory.     

A 16-Kbit Θ-search associative memory

The design of a high-capacity Θ-search associative memory (Θ∈{<,>,⩽,⩾,=≠}) is presented. PSPICE simulation and layouts show that the proposed Θ-search associative memory chip consisting of 256 words, each 64-b long, can fit on a 13.5-mm×9.5-mm chip. It can perform maskable Θ-search operations over its contents in 110 ns     

On fuzzy associative memory with multiple-rule storage capacity

Kosko's fuzzy associative memory (FAM) is the very first neural network model for implementing fuzzy systems. Despite its success in various applications, the model suffers from very low storage capacity, i.e., one rule per FAM matrix. A lot of hardware and computations are usually required to implement the model and, hence, it is limited to applications with small fuzzy rule-base. In this paper, the inherent property for storing multiple rules in a FAM matrix is identified. A theorem for perfect recalls of all the stored rules is established and based upon which the hardware and computation requirements of the FAM model can be reduced significantly. Furthermore, we have shown that when the FAM model is generalized to the one with max-bounded-product composition, single matrix implementation is possible if the rule-base is a set of semi-overlapped fuzzy rules. Rule modification schemes are also developed and the inference performance of the established high capacity models is reported through a numerical example     

A robust associative watermarking technique based on vector quantization

In this paper, the concepts of vector quantization (VQ) and association rules in data mining are employed to propose a robust watermarking technique. Unlike ordinary or traditional watermarking techniques, our approach hides association rules of the watermark, instead of the whole watermark; in other words, the embedded information is the association rules of the watermark. First, VQ encoding is performed on the original image and watermark to generate the index tables, and from which association rules are further mined. Subsequently, by embedding the association rules of the watermark into the association rules of the original image, the purpose for watermarking is accomplished. Finally, VQ decoding technique is applied to reconstruct the watermarked image from the watermarked index table. Experimental results show that our proposed method achieves effective resistance against several image processings such as blurring, sharpening, adding in Gaussian noise, cropping, and JPEG lossy compression. Moreover, the embedding capacity is also significantly increased, so any a complex watermark image is still acceptable in this method.     

CMOS current-mode neural associative memory design with on-chiplearning

Based on the Grossberg mathematical model called the outstar, a modular neural net with on-chip learning and memory is designed and analyzed. The outstar is the minimal anatomy that can interpret the classical conditioning or associative memory. It can also be served as a general-purpose pattern learning device. To realize the outstar, CMOS (complimentary metal-oxide semiconductor) current-mode analog dividers are developed to implement the special memory called the ratio-type memory. Furthermore, a CMOS current-mode analog multiplier is used to implement the correlation. The implemented CMOS outstar can on-chip store the relative ratio values of the trained weights for a long time. It can also be modularized to construct general neural nets. HSPICE (a circuit simulator of Meta Software, Inc.) simulation results of the CMOS outstar circuits as associative memory and pattern learner have successfully verified their functions. The measured results of the fabricated CMOS outstar circuits have also successfully confirmed the ratio memory and on-chip learning capability of the circuits. Furthermore, it has been shown that the storage time of the ratio memory can be as long as five minutes without refreshment. Also the outstar can enhance the contrast of the stored pattern within a long period. This makes the outstar circuits quite feasible in many applications.     

Storage capacity diverges with synaptic efficiency in an associative memory model with synaptic delay and pruning

It is known that storage capacity per synapse increases by synaptic pruning in the case of a correlation-type associative memory model. However, the storage capacity of the entire network then decreases. To overcome this difficulty, we propose decreasing the connectivity while keeping the total number of synapses constant by introducing delayed synapses. In this paper, a discrete synchronous-type model with both delayed synapses and their prunings is discussed as a concrete example of the proposal. First, we explain the Yanai-Kim theory by employing statistical neurodynamics. This theory involves macrodynamical equations for the dynamics of a network with serial delay elements. Next, considering the translational symmetry of the explained equations, we rederive macroscopic steady-state equations of the model by using the discrete Fourier transformation. The storage capacities are analyzed quantitatively. Furthermore, two types of synaptic prunings are treated analytically: random pruning and systematic pruning. As a result, it becomes clear that in both prunings, the storage capacity increases as the length of delay increases and the connectivity of the synapses decreases when the total number of synapses is constant. Moreover, an interesting fact becomes clear: the storage capacity asymptotically approaches 2//spl pi/ due to random pruning. In contrast, the storage capacity diverges in proportion to the logarithm of the length of delay by systematic pruning and the proportion constant is 4//spl pi/. These results theoretically support the significance of pruning following an overgrowth of synapses in the brain and may suggest that the brain prefers to store dynamic attractors such as sequences and limit cycles rather than equilibrium states.     

Navigation system including associative memory

A new adaptive linear robot control system for a robot work cell that can visually track and intercept stationary and moving objects undergoing arbitrary motion anywhere along its predicted trajectory within the robot's workspace is presented in this paper. The proposed system was designed by integrating a stationary monocular CCD camera with off-the-shelf frame grabber and an industrial robot operation into a single application on the MATLAB platform. A combination of the model based object recognition technique and a learning vector quantization network is used for classifying stationary objects without overlapping. The optical flow technique and the MADALINE network are used for determining the target trajectory and generating the predicted robot trajectory based on visual servoing, respectively. The necessity of determining a model of the robot, camera, all the stationary and moving objects, and environment is eliminated. The location and image features of these objects need not be preprogrammed, marked and known before, and any change in a task is possible without changing the robot program. After the learning process on the robot, it is shown that the KUKA robot is capable of tracking and intercepting both stationary and moving objects at an optimal rendezvous point on the conveyor accurately in real-time.     

An analog feedback associative memory

A method for the storage of analog vectors, i.e., vectors whose components are real-valued, is developed for the Hopfield continuous-time network. An important requirement is that each memory vector has to be an asymptotically stable (i.e. attractive) equilibrium of the network. Some of the limitations imposed by the continuous Hopfield model on the set of vectors that can be stored are pointed out. These limitations can be relieved by choosing a network containing visible as well as hidden units. An architecture consisting of several hidden layers and a visible layer, connected in a circular fashion, is considered. It is proved that the two-layer case is guaranteed to store any number of given analog vectors provided their number does not exceed 1 + the number of neurons in the hidden layer. A learning algorithm that correctly adjusts the locations of the equilibria and guarantees their asymptotic stability is developed. Simulation results confirm the effectiveness of the approach.     

Complex-valued multistate neural associative memory

A model of a multivalued associative memory is presented. This memory has the form of a fully connected attractor neural network composed of multistate complex-valued neurons. Such a network is able to perform the task of storing and recalling gray-scale images. It is also shown that the complex-valued fully connected neural network may be considered as a generalization of a Hopfield network containing real-valued neurons. A computational energy function is introduced and evaluated in order to prove network stability for asynchronous dynamics. Storage capacity as related to the number of accessible neuron states is also estimated.     

Adaptive competitive self-organizing associative memory

This work presents the design of an adaptive competitive self-organizing associative memory (ACSAM) system for use in classification and recognition of pattern information. Volterra and Lotka's models of interacting species in biology motivated the ACSAM model; a model based on a system of nonlinear ordinary differential equations (ODEs). Self-organizing behavior is modeled for unsupervised neural networks employing the concept of interacting/competing species in biology. In this model, self-organizing properties can be implicitly coded within the systems trajectory structure using only ODEs. Among the features of this continuous-time system are: 1) the dynamic behavior is well-understood and characterized; 2) the desired fixed points are the only asymptotically stable states of the system; 3) the trajectories of ACSAM derived from the weight activities of the gradient system have no periodic or homoclinic orbits; and 4) the heteroclinic orbits that exist between equilibrium states are structurally unstable and can be removed by small perturbations.