On the computational power of circuits of spiking neurons

Complex real-time computations on multi-modal time-varying input streams are carried out by generic cortical microcircuits. Obstacles for the development of adequate theoretical models that could explain the seemingly universal power of cortical microcircuits for real-time computing are the complexity and diversity of their computational units (neurons and synapses), as well as the traditional emphasis on offline computing in almost all theoretical approaches towards neural computation. In this article, we initiate a rigorous mathematical analysis of the real-time computing capabilities of a new generation of models for neural computation, liquid state machines, that can be implemented with—in fact benefit from—diverse computational units. Hence, realistic models for cortical microcircuits represent special instances of such liquid state machines, without any need to simplify or homogenize their diverse computational units. We present proofs of two theorems about the potential computational power of such models for real-time computing, both on analog input streams and for spike trains as inputs.     

Sequential optimal design of neurophysiology experiments

Adaptively optimizing experiments has the potential to significantly reduce the number of trials needed to build parametric statistical models of neural systems. However, application of adaptive methods to neurophysiology has been limited by severe computational challenges. Since most neurons are high-dimensional systems, optimizing neurophysiology experiments requires computing high-dimensional integrations and optimizations in real time. Here we present a fast algorithm for choosing the most informative stimulus by maximizing the mutual information between the data and the unknown parameters of a generalized linear model (GLM) that we want to fit to the neuron's activity. We rely on important log concavity and asymptotic normality properties of the posterior to facilitate the required computations. Our algorithm requires only low-rank matrix manipulations and a two-dimensional search to choose the optimal stimulus. The average running time of these operations scales quadratically with the dimensionality of the GLM, making real-time adaptive experimental design feasible even for high-dimensional stimulus and parameter spaces. For example, we require roughly 10 milliseconds on a desktop computer to optimize a 100-dimensional stimulus. Despite using some approximations to make the algorithm efficient, our algorithm asymptotically decreases the uncertainty about the model parameters at a rate equal to the maximum rate predicted by an asymptotic analysis. Simulation results show that picking stimuli by maximizing the mutual information can speed up convergence to the optimal values of the parameters by an order of magnitude compared to using random (nonadaptive) stimuli. Finally, applying our design procedure to real neurophysiology experiments requires addressing the nonstationarities that we would expect to see in neural responses; our algorithm can efficiently handle both fast adaptation due to spike history effects and slow, nonsystematic drifts in a neuron's activity.     

Neural systems as nonlinear filters

Experimental data show that biological synapses behave quite differently from the symbolic synapses in all common artificial neural network models. Biological synapses are dynamic; their "weight" changes on a short timescale by several hundred percent in dependence of the past input to the synapse. In this article we address the question how this inherent synaptic dynamics (which should not be confused with long term learning) affects the computational power of a neural network. In particular, we analyze computations on temporal and spatiotemporal patterns, and we give a complete mathematical characterization of all filters that can be approximated by feedforward neural networks with dynamic synapses. It turns out that even with just a single hidden layer, such networks can approximate a very rich class of nonlinear filters: all filters that can be characterized by Volterra series. This result is robust with regard to various changes in the model for synaptic dynamics. Our characterization result provides for all nonlinear filters that are approximable by Volterra series a new complexity hierarchy related to the cost of implementing such filters in neural systems.     

Real-time computing without stable states: a new framework for neural computation based on perturbations

A key challenge for neural modeling is to explain how a continuous stream of multimodal input from a rapidly changing environment can be processed by stereotypical recurrent circuits of integrate-and-fire neurons in real time. We propose a new computational model for real-time computing on time-varying input that provides an alternative to paradigms based on Turing machines or attractor neural networks. It does not require a task-dependent construction of neural circuits. Instead, it is based on principles of high-dimensional dynamical systems in combination with statistical learning theory and can be implemented on generic evolved or found recurrent circuitry. It is shown that the inherent transient dynamics of the high-dimensional dynamical system formed by a sufficiently large and heterogeneous neural circuit may serve as universal analog fading memory. Readout neurons can learn to extract in real time from the current state of such recurrent neural circuit information about current and past inputs that may be needed for diverse tasks. Stable internal states are not required for giving a stable output, since transient internal states can be transformed by readout neurons into stable target outputs due to the high dimensionality of the dynamical system. Our approach is based on a rigorous computational model, the liquid state machine, that, unlike Turing machines, does not require sequential transitions between well-defined discrete internal states. It is supported, as the Turing machine is, by rigorous mathematical results that predict universal computational power under idealized conditions, but for the biologically more realistic scenario of real-time processing of time-varying inputs. Our approach provides new perspectives for the interpretation of neural coding, the design of experiments and data analysis in neurophysiology, and the solution of problems in robotics and neurotechnology.     

Computation on programmable graphics hardware

GPUs have evolved into powerful and flexible streaming processors with fully programmable floating-point pipelines and tremendous aggregate computational power and memory bandwidth. With these advances, modern GPUs can now perform more functions than the specific graphics computations for which they were designed. This article describes approaches to using GPU processing power to accelerate traditionally CPU-based tasks. We discuss some important characteristics of algorithms that make them good candidates for GPU acceleration. We discuss a specific GPU image-processing application that is a common postprocess for many physically based rendering systems.     

Computation by Time

Over the last years, the amount of research performed in the field of spiking neural networks has been growing steadily. Spiking neurons are modeled to approximate the complex dynamic behavior of biological neurons. They communicate via discrete impulses called spikes with the actual information being encoded in the timing of these spikes. As already pointed out by Maass in his paper on the third generation of neural network models, this renders time a central factor for neural computation. In this paper, we investigate at different levels of granularity how absolute time and relative timing enable new ways of biologically inspired neural information processing. At the lowest level of single spiking neurons, we give an overview of coding schemes and learning techniques which rely on precisely timed neural spikes. A high-level perspective is provided in the second part of the paper which focuses on the role of time at the network level. The third aspect of time considered in this work is related to the interfacing of neural networks with real-time systems. In this context, we discuss how the concepts of computation by time can be implemented in computer simulations and on specialized neuromorphic hardware. The contributions of this paper are twofold: first, we show how the exact modeling of time in spiking neural networks serves as an important basis for powerful computation based on neurobiological principles. Second, by presenting a range of diverse learning techniques, we prove the biologically plausible applicability of spiking neural networks to real world problems like pattern recognition and path planning.     

Which model to use for cortical spiking neurons?

We discuss the biological plausibility and computational efficiency of some of the most useful models of spiking and bursting neurons. We compare their applicability to large-scale simulations of cortical neural networks.     

Fine-Grained Parallelism in Computational Mathematics

In the paper, a systematic discussion is made of state of the art theory and applications of fine-grained (massive) parallelism, which is permanently being adopted in computational mathematics. All known models of fine-grained computations (cellular automaton, neural and cellular neural networks, statistical automata, etc.) are represented in terms of a unique formalism, the so-called parallel substitution algorithm (PSA), which made it possible, on the one hand, to highlight common properties of the models and, on the other hand, to demonstrate expressiveness capabilities of the PSA. Theoretical and experimental results of studies on applications of fine-grained algorithms to modeling of spatial dynamics of reaction–diffusion and molecular processes are presented. Promising prospects of their application are substantiated both for the creation of special processors designed for the implementation of the algorithms and for the implementation of the algorithms on multiprocessor systems.     

Parallel algorithms for model reduction of discrete-time systems

Computing reduced-order models of controlled dynamical systems is of fundamental importance in many analysis and synthesis problems in systems and control theory. Algorithmic aspects of model reduction methods based on state-space truncation for linear discrete-time systems are addressed here. In contrast to the often-used approach of applying methods for continuous-time systems to discrete-time models employing a bilinear transformation, we devise special algorithms for discrete-time systems. Usually, this is more reliable and efficient. All methods discussed require in an initial stage the computation of the Gramians of the system. Using an accelerated fixed-point iteration for computing the full-rank factors of the Gramians yields some favorable computational aspects, particularly for non-minimal systems. The computations only require efficient implementations of basic linear algebra operations readily available on modern computer architectures. We discuss aspects of the parallel implementation of these methods and show the performance and scalability on distributed memory computers. Our approach enables users to deal with very complex systems using relatively cheap infrastructure, as, for example, a local PC or workstation network.     

Neural network learning with generalized-mean based neuron model

The advances in biophysics of computations and neurocomputing models have brought the foreground importance of dendritic structure of neuron. These structures are assumed as basic computational units of the neuron, capable of realizing the various mathematical operations. The well structured higher order neurons have shown improved computational power and generalization ability. However, these models are difficult to train because of a combinatorial explosion of higher order terms as the number of inputs to the neuron increases. In this paper we present a neural network using new neuron architecture i.e., generalized mean neuron (GMN) model. This neuron model consists of an aggregation function which is based on the generalized mean of all the inputs applied to it. The resulting neuron model has the same number of parameters with improved computational power as the existing multilayer perceptron (MLP) model. The capability of this model has been tested on the classification and time series prediction problems.     

Modeling and prediction with a class of time delay dynamic neural networks

In this paper, we propose a time delay dynamic neural network (TDDNN) to track and predict a chaotic time series systems. The application of artificial neural networks to dynamical systems has been constrained by the non-dynamical nature of popular network architectures. Many of the drawbacks caused by the algebraic structures can be overcome with TDDNNs. TDDNNs have time delay elements in their states. This approach provides the natural properties of physical systems. The minimization of a quadratic performance index is considered for trajectory tracking applications. Gradient computations are presented based on adjoint sensitivity analysis. The computational complexity is significantly less than direct method, but it requires a backward integration capability. We used Levenberg–Marquardt parameter updating method.     

Dynamic state and parameter estimation applied to neuromorphic systems

Neuroscientists often propose detailed computational models to probe the properties of the neural systems they study. With the advent of neuromorphic engineering, there is an increasing number of hardware electronic analogs of biological neural systems being proposed as well. However, for both biological and hardware systems, it is often difficult to estimate the parameters of the model so that they are meaningful to the experimental system under study, especially when these models involve a large number of states and parameters that cannot be simultaneously measured. We have developed a procedure to solve this problem in the context of interacting neural populations using a recently developed dynamic state and parameter estimation (DSPE) technique. This technique uses synchronization as a tool for dynamically coupling experimentally measured data to its corresponding model to determine its parameters and internal state variables. Typically experimental data are obtained from the biological neural system and the model is simulated in software; here we show that this technique is also efficient in validating proposed network models for neuromorphic spike-based very large-scale integration (VLSI) chips and that it is able to systematically extract network parameters such as synaptic weights, time constants, and other variables that are not accessible by direct observation. Our results suggest that this method can become a very useful tool for model-based identification and configuration of neuromorphic multichip VLSI systems.     

On the phase-space dynamics of systems of spiking neurons. I: model and experiments

We investigate the phase-space dynamics of a general model of local systems of biological neurons in order to deduce the salient dynamical characteristics of such systems. In this article, we present a detailed exposition of an abstract dynamical system that models systems of biological neurons. The abstract system is based on a limited set of realistic assumptions and thus accommodates a wide range of neuronal models. Simulation results are presented for several instantiations of the abstract system, each modeling a typical neocortical column to a different degree of accuracy. The results demonstrate that the dynamics of the systems are generally consistent with that observed in neurophysiological experiments. They reveal that the qualitative behavior of the class of systems can be classified into three distinct categories: quiescence, intense periodic activity resembling a state of seizure, and sustained chaos over the range of intrinsic activity typically associated with normal operational conditions in the neocortex. We discuss basic ramifications of this result with regard to the computational nature of neocortical neuronal systems.     

SpikeNET: an event-driven simulation package for modelling large networks of spiking neurons

Many biological neural network models face the problem of scalability because of the limited computational power of today's computers. Thus, it is difficult to assess the efficiency of these models to solve complex problems such as image processing. Here, we describe how this problem can be tackled using event-driven computation. Only the neurons that emit a discharge are processed and, as long as the average spike discharge rate is low, millions of neurons and billions of connections can be modelled. We describe the underlying computation and implementation of such a mechanism in SpikeNET, our neural network simulation package. The type of model one can build is not only biologically compliant, it is also computationally efficient as 400 000 synaptic weights can be propagated per second on a standard desktop computer. In addition, for large networks, we can set very small time steps (< 0.01 ms) without significantly increasing the computation time. As an example, this method is applied to solve complex cognitive tasks such as face recognition in natural images.     

Neural networks for shortest path computation and routing incomputer networks

The application of neural networks to the optimum routing problem in packet-switched computer networks, where the goal is to minimize the network-wide average time delay, is addressed. Under appropriate assumptions, the optimum routing algorithm relies heavily on shortest path computations that have to be carried out in real time. For this purpose an efficient neural network shortest path algorithm that is an improved version of previously suggested Hopfield models is proposed. The general principles involved in the design of the proposed neural network are discussed in detail. Its computational power is demonstrated through computer simulations. One of the main features of the proposed model is that it will enable the routing algorithm to be implemented in real time and also to be adaptive to changes in link costs and network topology.     

Autonomous learning with complex dynamics

Traditionally, associative memory models are based on point attractor dynamics, where a memory state corresponds to a stationary point in state space. However, biological neural systems seem to display a rich and complex dynamics whose function is still largely unknown. We use a neural network model of the olfactory cortex to investigate the functional significance of such dynamics, in particular with regard to learning and associative memory. the model uses simple network units, corresponding to populations of neurons connected according to the structure of the olfactory cortex. All essential dynamical properties of this system are reproduced by the model, especially oscillations at two separate frequency bands and aperiodic behavior similar to chaos. By introducing neuromodulatory control of gain and connection weight strengths, the dynamics can change dramatically, in accordance with the effects of acetylcholine, a neuromodulator known to be involved in attention and learning in animals. With computer simulations we show that these effects can be used for improving associative memory performance by reducing recall time and increasing fidelity. the system is able to learn and recall continuously as the input changes, mimicking a real world situation of an artificial or biological system in a changing environment. © 1995 John Wiley & Sons, Inc.     

Isolated word recognition with the Liquid State Machine: a case study

The Liquid State Machine (LSM) is a recently developed computational model with interesting properties. It can be used for pattern classification, function approximation and other complex tasks. Contrary to most common computational models, the LSM does not require information to be stored in some stable state of the system: the inherent dynamics of the system are used by a memoryless readout function to compute the output. In this paper we present a case study of the performance of the Liquid State Machine based on a recurrent spiking neural network by applying it to a well known and well studied problem: speech recognition of isolated digits. We evaluate different ways of coding the speech into spike trains. In its optimal configuration, the performance of the LSM approximates that of a state-of-the-art recognition system. Another interesting conclusion is the fact that the biologically most realistic encoding performs far better than more conventional methods.     

Analysing neurobiological models using communicating automata

Two important issues in computational modelling in cognitive neuroscience are: first, how to formally describe neuronal networks (i.e. biologically plausible models of the central nervous system), and second, how to analyse complex models, in particular, their dynamics and capacity to learn. We make progress towards these goals by presenting a communicating automata perspective on neuronal networks. Specifically, we describe neuronal networks and their biological mechanisms using Data-rich Communicating Automata, which extend classic automata theory with rich data types and communication. We use two case studies to illustrate our approach. In the first case study, we model a number of learning frameworks, which vary in respect of their biological detail, for instance the Backpropagation (BP) and the Generalized Recirculation (GeneRec) learning algorithms. We then used the SPIN model checker to investigate a number of behavioral properties of the neural learning algorithms. SPIN is a well-known model checker for reactive distributed systems, which has been successfully applied to many non-trivial problems. The verification results show that the biologically plausible GeneRec learning is less stable than BP learning. In the second case study, we presented a large scale (cognitive-level) neuronal network, which models an attentional spotlight mechanism in the visual system. A set of properties of this model was verified using Uppaal, a popular real-time model checker. The results show that the asynchronous processing supported by concurrency theory is not only a more biologically plausible way to model neural systems, but also provides a better performance in cognitive modelling of the brain than conventional artificial neural networks that use synchronous updates. Finally, we compared our approach with several other related theories that apply formal methods to cognitive modelling. In addition, the practical implications of the approach are discussed in the context of neuronal network based controllers.