Spatio–Temporal Adaptation in the Unsupervised Development of Networked Visual Neurons

There have been many computational models mimicking the visual cortex that are based on spatial adaptations of unsupervised neural networks. In this paper, we present a new model called neuronal cluster which includes spatial as well as temporal weights in its unified adaptation scheme. The “in-place” nature of the model is based on two biologically plausible learning rules, Hebbian rule and lateral inhibition. We present the mathematical demonstration that the temporal weights are derived from the delay in lateral inhibition. By training with the natural videos, this model can develop spatio–temporal features such as orientation selective cells, motion sensitive cells, and spatio–temporal complex cells. The unified nature of the adaption scheme allows us to construct a multilayered and task-independent attention selection network which uses the same learning rule for edge, motion, and color detection, and we can use this network to engage in attention selection in both static and dynamic scenes.      

Segmented-Memory Recurrent Neural Networks

Conventional recurrent neural networks (RNNs) have difficulties in learning long-term dependencies. To tackle this problem, we propose an architecture called segmented-memory recurrent neural network (SMRNN). A symbolic sequence is broken into segments and then presented as inputs to the SMRNN one symbol per cycle. The SMRNN uses separate internal states to store symbol-level context, as well as segment-level context. The symbol-level context is updated for each symbol presented for input. The segment-level context is updated after each segment. The SMRNN is trained using an extended real-time recurrent learning algorithm. We test the performance of SMRNN on the information latching problem, the “two-sequence problem” and the problem of protein secondary structure (PSS) prediction. Our implementation results indicate that SMRNN performs better on long-term dependency problems than conventional RNNs. Besides, we also theoretically analyze how the segmented memory of SMRNN helps learning long-term temporal dependencies and study the impact of the segment length.      

A Modified Backpropagation Learning Algorithm With Added Emotional Coefficients

Much of the research work into artificial intelligence (AI) has been focusing on exploring various potential applications of intelligent systems with successful results in most cases. In our attempts to model human intelligence by mimicking the brain structure and function, we overlook an important aspect in human learning and decision making: the emotional factor. While it currently sounds impossible to have “machines with emotions,” it is quite conceivable to artificially simulate some emotions in machine learning. This paper presents a modified backpropagation (BP) learning algorithm, namely, the emotional backpropagation (EmBP) learning algorithm. The new algorithm has additional emotional weights that are updated using two additional emotional parameters: anxiety and confidence. The proposed “emotional” neural network will be implemented to a facial recognition problem, and the results will be compared to a similar application using a conventional neural network. Experimental results show that the addition of the two novel emotional parameters improves the performance of the neural network yielding higher recognition rates and faster recognition time.      

K-Means Clustering Versus Validation Measures: A Data-Distribution Perspective

K-means is a well-known and widely used partitional clustering method. While there are considerable research efforts to characterize the key features of the K-means clustering algorithm, further investigation is needed to understand how data distributions can have impact on the performance of K-means clustering. To that end, in this paper, we provide a formal and organized study of the effect of skewed data distributions on K-means clustering. Along this line, we first formally illustrate that K-means tends to produce clusters of relatively uniform size, even if input data have varied “true” cluster sizes. In addition, we show that some clustering validation measures, such as the entropy measure, may not capture this uniform effect and provide misleading information on the clustering performance. Viewed in this light, we provide the coefficient of variation (CV) as a necessary criterion to validate the clustering results. Our findings reveal that K-means tends to produce clusters in which the variations of cluster sizes, as measured by CV, are in a range of about 0.3–1.0. Specifically, for data sets with large variation in “true” cluster sizes (e.g., $ hbox{CV} ≫ 1.0$), K-means reduces variation in resultant cluster sizes to less than 1.0. In contrast, for data sets with small variation in “true” cluster sizes (e.g., $hbox{CV} ≪ 0.3$), K-means increases variation in resultant cluster sizes to greater than 0.3. In other words, for the earlier two cases, K-means produces the clustering results which are away from the “true” cluster distributions.      

Large Developing Receptive Fields Using a Distributed and Locally Reprogrammable Address–Event Receiver

A distributed and locally reprogrammable address–event receiver has been designed, in which incoming address–events are monitored simultaneously by all synapses, allowing for arbitrarily large axonal fan-out without reducing channel capacity. Synapses can change the address of their presynaptic neuron, allowing the distributed implementation of a biologically realistic learning rule, with both synapse formation and elimination (synaptic rewiring). Probabilistic synapse formation leads to topographic map development, made possible by a cross-chip current-mode calculation of Euclidean distance. As well as synaptic plasticity in rewiring, synapses change weights using a competitive Hebbian learning rule (spike-timing-dependent plasticity). The weight plasticity allows receptive fields to be modified based on spatio–temporal correlations in the inputs, and the rewiring plasticity allows these modifications to become embedded in the network topology.      

A view-based neurocomputational system for relational map-making and navigation in visual environments

Artificial navigation systems stand to benefit greatly from learning maps of visual environments, but traditional map-making techniques are inadequate in several respects. This paper describes an adaptive, view-based, relational map-making system for navigating within a 3D environment defined by a spatially distributed set of visual landmarks. Inspired by an analogy to learning aspect graphs of 3D objects, the system comprises two neurocomputational architectures that emulate cognitive mapping in the rat hippocampus. The first architecture performs unsupervised place learning by combining the “What” with the “Where”, namely through conjunctions of landmark identity, pose, and egocentric gaze direction within a local, restricted sensory view of the environment. The second associatively learns action consequences by incorporating the “When”, namely through conjunctions of learned places and coarsely coded robot motions. Together, these networks form a map reminiscent of a partially observable Markov decision process, and consequently provide an ideal neural substrate for prediction, environment recognition, route planning, and exploration. Preliminary results from real-time implementations on a mobile robot called MAVIN (the Mobile Adaptive VIsual Navigator) demonstrate the potential for these capabilities.     

Knowledge-based artificial neural networks

Hybrid learning methods use theoretical knowledge of a domain and a set of classified examples to develop a method for accurately classifying examples not seen during training. The challenge of hybrid learning systems is to use the information provided by one source of information to offset information missing from the other source. By so doing, a hybrid learning system should learn more effectively than systems that use only one of the information sources. KBANN (Knowledge-Based Artificial Neural Networks) is a hybrid learning system built on top of connectionist learning techniques. It maps problem-specific “domain theories”, represented in propositional logic, into neural networks and then refines this reformulated knowledge using backpropagation. KBANN is evaluated by extensive empirical tests on two problems from molecular biology. Among other results, these tests show that the networks created by KBANN generalize better than a wide variety of learning systems, as well as several techniques proposed by biologists.     

Real-time computing without stable states: a new framework for neural computation based on perturbations

A key challenge for neural modeling is to explain how a continuous stream of multimodal input from a rapidly changing environment can be processed by stereotypical recurrent circuits of integrate-and-fire neurons in real time. We propose a new computational model for real-time computing on time-varying input that provides an alternative to paradigms based on Turing machines or attractor neural networks. It does not require a task-dependent construction of neural circuits. Instead, it is based on principles of high-dimensional dynamical systems in combination with statistical learning theory and can be implemented on generic evolved or found recurrent circuitry. It is shown that the inherent transient dynamics of the high-dimensional dynamical system formed by a sufficiently large and heterogeneous neural circuit may serve as universal analog fading memory. Readout neurons can learn to extract in real time from the current state of such recurrent neural circuit information about current and past inputs that may be needed for diverse tasks. Stable internal states are not required for giving a stable output, since transient internal states can be transformed by readout neurons into stable target outputs due to the high dimensionality of the dynamical system. Our approach is based on a rigorous computational model, the liquid state machine, that, unlike Turing machines, does not require sequential transitions between well-defined discrete internal states. It is supported, as the Turing machine is, by rigorous mathematical results that predict universal computational power under idealized conditions, but for the biologically more realistic scenario of real-time processing of time-varying inputs. Our approach provides new perspectives for the interpretation of neural coding, the design of experiments and data analysis in neurophysiology, and the solution of problems in robotics and neurotechnology.     

Dynamics of Learning in Multilayer Perceptrons Near Singularities

The dynamical behavior of learning is known to be very slow for the multilayer perceptron, being often trapped in the “plateau.” It has been recently understood that this is due to the singularity in the parameter space of perceptrons, in which trajectories of learning are drawn. The space is Riemannian from the point of view of information geometry and contains singular regions where the Riemannian metric or the Fisher information matrix degenerates. This paper analyzes the dynamics of learning in a neighborhood of the singular regions when the true teacher machine lies at the singularity. We give explicit asymptotic analytical solutions (trajectories) both for the standard gradient (SGD) and natural gradient (NGD) methods. It is clearly shown, in the case of the SGD method, that the plateau phenomenon appears in a neighborhood of the critical regions, where the dynamical behavior is extremely slow. The analysis of the NGD method is much more difficult, because the inverse of the Fisher information matrix diverges. We conquer the difficulty by introducing the “blow-down” technique used in algebraic geometry. The NGD method works efficiently, and the state converges directly to the true parameters very quickly while it staggers in the case of the SGD method. The analytical results are compared with computer simulations, showing good agreement. The effects of singularities on learning are thus qualitatively clarified for both standard and NGD methods.      

Dynamical synapses enhance neural information processing: gracefulness, accuracy, and mobility

Experimental data have revealed that neuronal connection efficacy exhibits two forms of short-term plasticity: short-term depression (STD) and short-term facilitation (STF). They have time constants residing between fast neural signaling and rapid learning and may serve as substrates for neural systems manipulating temporal information on relevant timescales. This study investigates the impact of STD and STF on the dynamics of continuous attractor neural networks and their potential roles in neural information processing. We find that STD endows the network with slow-decaying plateau behaviors: the network that is initially being stimulated to an active state decays to a silent state very slowly on the timescale of STD rather than on that of neuralsignaling. This provides a mechanism for neural systems to hold sensory memory easily and shut off persistent activities gracefully. With STF, we find that the network can hold a memory trace of external inputs in the facilitated neuronal interactions, which provides a way to stabilize the network response to noisy inputs, leading to improved accuracy in population decoding. Furthermore, we find that STD increases the mobility of the network states. The increased mobility enhances the tracking performance of the network in response to time-varying stimuli, leading to anticipative neural responses. In general, we find that STD and STP tend to have opposite effects on network dynamics and complementary computational advantages, suggesting that the brain may employ a strategy of weighting them differentially depending on the computational purpose.     

Comments on “The Extreme Learning Machine”

This comment letter points out that the essence of the “extreme learning machine (ELM)” recently appeared has been proposed earlier by Broomhead and Lowe and Pao , and discussed by other authors. Hence, it is not necessary to introduce a new name “ELM.”      

Law, learning and representation

In machine learning terms, reasoning in legal cases can be compared to a lazy learning approach in which courts defer deciding how to generalize beyond the prior cases until the facts of a new case are observed. The HYPO family of systems implements a “lazy” approach since they defer making arguments how to decide a problem until the programs have positioned a new problem with respect to similar past cases. In a kind of “reflective adjustment”, they fit the new problem into a patchwork of past case decisions, comparing cases in order to reason about the legal significance of the relevant similarities and differences. Empirical evidence from diverse experiments shows that for purposes of teaching legal argumentation and performing legal information retrieval, HYPO-style systems' lazy learning approach and implementation of aspects of reflective adjustment can be very effective.     

Partially‐observable stochastic hybrid systems (poshss) state estimation and optimal control

This paper discusses the state estimation and optimal control problem of a class of partially‐observable stochastic hybrid systems (POSHS). The POSHS has interacting continuous and discrete dynamics with uncertainties. The continuous dynamics are given by a Markov‐jump linear system and the discrete dynamics are defined by a Markov chain whose transition probabilities are dependent on the continuous state via guard conditions. The only information available to the controller are noisy measurements of the continuous state. To solve the optimal control problem, a separable control scheme is applied: the controller estimates the continuous and discrete states of the POSHS using noisy measurements and computes the optimal control input from the state estimates. Since computing both optimal state estimates and optimal control inputs are intractable, this paper proposes computationally efficient algorithms to solve this problem numerically. The proposed hybrid estimation algorithm is able to handle state‐dependent Markov transitions and compute Gaussian‐ mixture distributions as the state estimates. With the computed state estimates, a reinforcement learning algorithm defined on a function space is proposed. This approach is based on Monte Carlo sampling and integration on a function space containing all the probability distributions of the hybrid state estimates. Finally, the proposed algorithm is tested via numerical simulations.     

On Utilizing Association and Interaction Concepts for Enhancing Microaggregation in Secure Statistical Databases

This paper presents a possibly pioneering endeavor to tackle the Microaggregation Techniques (MATs) in secure statistical databases by resorting to the principles of associative neural networks (NNs). The prior art has improved the available solutions to the MAT by incorporating proximity information, and this approach is done by recursively reducing the size of the data set by excluding points that are farthest from the centroid and points that are closest to these farthest points. Thus, although the method is extremely effective, arguably, it uses only the proximity information while ignoring the mutual interaction between the records. In this paper, we argue that interrecord relationships can be quantified in terms of the following two entities: 1) their “association” and 2) their “interaction.” This case means that records that are not necessarily close to each other may still be “grouped,” because their mutual interaction, which is quantified by invoking transitive-closure-like operations on the latter entity, could be significant, as suggested by the theoretically sound principles of NNs. By repeatedly invoking the interrecord associations and interactions, the records are grouped into sizes of cardinality “$k$,” where $k$ is the security parameter in the algorithm. Our experimental results, which are done on artificial data and benchmark real-life data sets, demonstrate that the newly proposed method is superior to the state of the art not only based on the Information Loss (IL) perspective but also when it concerns a criterion that involves a combination of the IL and the Disclosure Risk (DR).      

Fractals in neurodynamics: From hybrid dynamical systems point of view

Recently, hybrid dynamical systems have attracted considerable attention in the automatic control domain. In this article, a theory for recurrent neural networks is presented from a hybrid dynamical systems point of view. The hybrid dynamical system is defined by a continuous dynamical system discretely switched by external temporal inputs. The theory suggests that the dynamics of continuous-time recurrent neural networks, which are stochastically excited by external temporal inputs, are generally characterized by a set of continuous trajectories with a fractal-like structure in hyper-cylindrical phase space. This work was presented, in part, at the 7th International Symposium on Artificial Life and Robotics, Oita, Japan, January 16–18, 2002     

H2 control strategy for Na ion channels on the neural cellular membrane

The channel gating process of neural cells is the first step of neural information transmission. We have proposed a kinetic model for state transitions for a sodium (Na) ion gating channel under H2 control. The channel state consisted of an open state, three closed but activated states, and four inactivated but not closed states. This modeling was based strictly on molecular biological observations. Three charged amino acid helixes of the specific subunits of the Na channel hole act as activating gates. Another helix of the subunit having membrane voltagesensing properties behaves as an inactivating particle that invades the Na channel hole after membrane depolarization. This particle blocks the free movements of the three activating gates and inactivates the Na channel gating function. In total there are eight channel states, which consist of four inactivated states, three closed states, and one open state. We expressed the transitions among these states by eight linear differential equations using the law of conservation. For the control principle, the channel system is always exposed to a biological mimetic that is a false transmitter and competes for the channel sites with Na ions. Hence, we regarded such biological agencies as noises in the system that disturb the effective transmission of information, i.e., rapid transitions through the channel gating systems. The physiological Na gating is understood to minimize influences from the disturbing noises on the transition of the channels states, and we have proposed the H2 control principle. The computed results of temporal changes in the amount of channel species per unit membrane area showed rapid changes and then termination. This behavior was strongly dependent on the membrane potential. Our modeling could describe the rapid excitation and resetting of the Na ion channel gating function of the neural system. These results strongly reflect the digital nature of the neural system. The present investigation could be used to evaluate the function of neural systems that minimizes the influences of noises on the information transmission process by the transitions of the Na ion channel gating state.     

A distributed approach for fault detection and diagnosis based on Time Petri Nets

This paper proposes an algorithm for the model based design of a distributed protocol for fault detection and diagnosis for very large systems. The overall process is modeled as different Time Petri Net (TPN) models (each one modeling a local process) that interact with each other via guarded transitions that becomes enabled only when certain conditions (expressed as predicates over the marking of some places) are satisfied (the guard is true). In order to use this broad class of time DES models for fault detection and diagnosis we derive in this paper the timing analysis of the TPN models with guarded transitions. In this paper we also extend the modeling capability of the faults calling some transitions faulty when operations they represent take more or less time than a prescribed time interval corresponding to their normal execution. We consider here that different local agents receive local observation as well as messages from neighboring agents. Each agent estimates the state of the part of the overall process for which it has model and from which it observes events by reconciling observations with model based predictions. We design algorithms that use limited information exchange between agents and that can quickly decide “questions” about “whether and where a fault occurred?” and “whether or not some components of the local processes have operated correctly?”. The algorithms we derive allow each local agent to generate a preliminary diagnosis prior to any communication and we show that after communicating the agents we design recover the global diagnosis that a centralized agent would have derived. The algorithms are component oriented leading to efficiency in computation.     

“Chaotic relaxation” in concurrently asynchronous neural networks

In this paper we analyze a fundamental issue which directly impacts the scalability of current theoretical neural network models to applicative embodiments, in both software as well as hardware. This pertains to the inherent and unavoidable concurrent asynchronicity of emerging fine-grained computational ensembles and the consequent chaotic manifestations in the absence of proper conditioning. The latter concern is particularly significant since the computational inertia of neural networks in general and our dynamical learning formalisms manifests itself substantially, only in massively parallel hardward—optical, VLSI or opto-electronic. We introduce a mathematical framework for systematically reconditioning additive-type models and derive a neuro-operator, based on the chaotic relaxation paradigm whose resulting dynamics are neither “concurrently” synchronous nor “sequentially” asynchronous. Necessary and sufficient conditions guaranteeing concurrent asynchronous convergence are established in terms of contracting operators. Lyapunov exponents are also computed to characterize the network dynamics and to ensure that throughput-limiting “emergent computational chaos” behavior in models reconditioned with concurrently asynchronous algorithms was eliminated.