Transmission of population-coded information

As neural activity is transmitted through the nervous system, neuronal noise degrades the encoded information and limits performance. It is therefore important to know how information loss can be prevented. We study this question in the context of neural population codes. Using Fisher information, we show how information loss in a layered network depends on the connectivity between the layers. We introduce an algorithm, reminiscent of the water filling algorithm for Shannon information that minimizes the loss. The optimal connection profile has a center-surround structure with a spatial extent closely matching the neurons' tuning curves. In addition, we show how the optimal connectivity depends on the correlation structure of the trial-to-trial variability in the neuronal responses. Our results explain how optimal communication of population codes requires the center-surround architectures found in the nervous system and provide explicit predictions on the connectivity parameters.     

Hebbian learning of recurrent connections: a geometrical perspective

We show how a Hopfield network with modifiable recurrent connections undergoing slow Hebbian learning can extract the underlying geometry of an input space. First, we use a slow and fast analysis to derive an averaged system whose dynamics derives from an energy function and therefore always converges to equilibrium points. The equilibria reflect the correlation structure of the inputs, a global object extracted through local recurrent interactions only. Second, we use numerical methods to illustrate how learning extracts the hidden geometrical structure of the inputs. Indeed, multidimensional scaling methods make it possible to project the final connectivity matrix onto a Euclidean distance matrix in a high-dimensional space, with the neurons labeled by spatial position within this space. The resulting network structure turns out to be roughly convolutional. The residual of the projection defines the nonconvolutional part of the connectivity, which is minimized in the process. Finally, we show how restricting the dimension of the space where the neurons live gives rise to patterns similar to cortical maps. We motivate this using an energy efficiency argument based on wire length minimization. Finally, we show how this approach leads to the emergence of ocular dominance or orientation columns in primary visual cortex via the self-organization of recurrent rather than feedforward connections. In addition, we establish that the nonconvolutional (or long-range) connectivity is patchy and is co-aligned in the case of orientation learning.     

Optimal short-term population coding: when Fisher information fails

Efficient coding has been proposed as a first principle explaining neuronal response properties in the central nervous system. The shape of optimal codes, however, strongly depends on the natural limitations of the particular physical system. Here we investigate how optimal neuronal encoding strategies are influenced by the finite number of neurons N (place constraint), the limited decoding time window length T (time constraint), the maximum neuronal firing rate f(max) (power constraint), and the maximal average rate (f)(max) (energy constraint). While Fisher information provides a general lower bound for the mean squared error of unbiased signal reconstruction, its use to characterize the coding precision is limited. Analyzing simple examples, we illustrate some typical pitfalls and thereby show that Fisher information provides a valid measure for the precision of a code only if the dynamic range (f(min)T, f(max)T) is sufficiently large. In particular, we demonstrate that the optimal width of gaussian tuning curves depends on the available decoding time T. Within the broader class of unimodal tuning functions, it turns out that the shape of a Fisher-optimal coding scheme is not unique. We solve this ambiguity by taking the minimum mean square error into account, which leads to flat tuning curves. The tuning width, however, remains to be determined by energy constraints rather than by the principle of efficient coding.     

基于脑功能网络和共空间模式分析脑电情绪识别

传统脑网络的情绪分类将聚类系数、平均最短路径等拓扑属性作为分类特征。针对这些属性易受网络连接阈值和特征选择的影响,难以完全表征不同情绪状态下的网络空间拓扑结构差异的问题,提出了一种基于脑网络和共空间模式的脑电情绪识别方法(EEG emotion classification based on common spatial patterns of brain networks topology,EEC-CSP-BNT)。该算法基于互信息在各个子频段内计算电极间的功能连接矩阵,同时利用共空间模式(common spatial pattern,CSP)分析学习空间滤波器,构建分类特征,最后通过分类器(如Fisher线性判别、支持向量机、K最近邻)实现基于脑电的情绪分类。基于DEAP和SEED数据集的实验结果表明,相比于脑网络拓扑属性,EEC-CSP-BNT能有效提取脑网络拓扑结构的分类信息,提高脑电情绪识别性能。     

On the implementation of frontier-to-root tree automata inrecursive neural networks

We explore the node complexity of recursive neural network implementations of frontier-to-root tree automata (FRA). Specifically, we show that an FRAO (Mealy version) with m states, l input-output labels, and maximum rank N can be implemented by a recursive neural network with O( radical(log l+log m)lm(N)/log l+N log m) units and four computational layers, i.e., without counting the input layer. A lower bound is derived which is tight when no restrictions are placed on the number of layers. Moreover, we present a construction with three computational layers having node complexity of O((log l+log m) radicallm (N)) and O((log l+log m)lm(N)) connections. A construction with two computational layers is given that implements any given FRAO with a node complexity of O(lm(N)) and O((log l+log m)lm(N)) connections. As a corollary we also get a new upper bound for the implementation of finite-state automata into recurrent neural networks with three computational layers.     

Sufficient conditions for error backflow convergence in dynamical recurrent neural networks

This article extends previous analysis of the gradient decay to a class of discrete-time fully recurrent networks, called dynamical recurrent neural networks, obtained by modeling synapses as finite impulse response (FIR) filters instead of multiplicative scalars. Using elementary matrix manipulations, we provide an upper bound on the norm of the weight matrix, ensuring that the gradient vector, when propagated in a reverse manner in time through the error-propagation network, decays exponentially to zero. This bound applies to all recurrent FIR architecture proposals, as well as fixed-point recurrent networks, regardless of delay and connectivity. In addition, we show that the computational overhead of the learning algorithm can be reduced drastically by taking advantage of the exponential decay of the gradient.     

Time-Space Tradeoffs for Undirected Graph Traversal by Graph Automata

We investigate time-space tradeoffs for traversing undirected graphs, using a variety of structured models that are all variants of Cook and Rackoff's “Jumping Automata for Graphs.” Our strongest tradeoff is a quadratic lower bound on the product of time and space for graph traversal. For example, achieving linear time requires linear space, implying that depth-first search is optimal. Since our bound in fact applies to nondeterministic algorithms fornonconnectivity, it also implies that closure under complementation of nondeterministic space-bounded complexity classes is achieved only at the expense of increased time. To demonstrate that these structured models are realistic, we also investigate their power. In addition to admitting well known algorithms such as depth-first search and random walk, we show that one simple variant of this model is nearly as powerful as a Turing machine. Specifically, for general undirected graph problems, it can simulate a Turing machine with only a constant factor increase in space and a polynomial factor increase in time.     

一种面向目标的有向传感器网络连通覆盖算法

针对传统的有向传感器网络目标覆盖算法只考虑网络覆盖率而不能保证网络连通性的问题,利用目标点部署圆内覆盖最多邻居目标点的候选节点集合和整数线性规划(ILP)模型设计了一种面向目标的连通覆盖算法(CTA)。该算法通过建立目标部署圆内覆盖最多邻居目标点的候选节点集合对随机部署的节点进行初步调度,在此基础上,通过ILP模型找出实现目标检测,并保证整个网络连通性的最少节点数和最佳位置的节点集合。仿真实验表明:CTA在保证目标覆盖率的前提下,不仅极大地降低了网络部署成本,而且保证了网络连通性。     

Implications of neuronal diversity on population coding

In many cortical and subcortical areas, neurons are known to modulate their average firing rate in response to certain external stimulus features. It is widely believed that information about the stimulus features is coded by a weighted average of the neural responses. Recent theoretical studies have shown that the information capacity of such a coding scheme is very limited in the presence of the experimentally observed pairwise correlations. However, central to the analysis of these studies was the assumption of a homogeneous population of neurons. Experimental findings show a considerable measure of heterogeneity in the response properties of different neurons. In this study, we investigate the effect of neuronal heterogeneity on the information capacity of a correlated population of neurons. We show that information capacity of a heterogeneous network is not limited by the correlated noise, but scales linearly with the number of cells in the population. This information cannot be extracted by the population vector readout, whose accuracy is greatly suppressed by the correlated noise. On the other hand, we show that an optimal linear readout that takes into account the neuronal heterogeneity can extract most of this information. We study analytically the nature of the dependence of the optimal linear readout weights on the neuronal diversity. We show that simple online learning can generate readout weights with the appropriate dependence on the neuronal diversity, thereby yielding efficient readout.     

Memory capacity for sequences in a recurrent network with biological constraints

The CA3 region of the hippocampus is a recurrent neural network that is essential for the storage and replay of sequences of patterns that represent behavioral events. Here we present a theoretical framework to calculate a sparsely connected network's capacity to store such sequences. As in CA3, only a limited subset of neurons in the network is active at any one time, pattern retrieval is subject to error, and the resources for plasticity are limited. Our analysis combines an analytical mean field approach, stochastic dynamics, and cellular simulations of a time-discrete McCulloch-Pitts network with binary synapses. To maximize the number of sequences that can be stored in the network, we concurrently optimize the number of active neurons, that is, pattern size, and the firing threshold. We find that for one-step associations (i.e., minimal sequences), the optimal pattern size is inversely proportional to the mean connectivity c, whereas the optimal firing threshold is independent of the connectivity. If the number of synapses per neuron is fixed, the maximum number P of stored sequences in a sufficiently large, nonmodular network is independent of its number N of cells. On the other hand, if the number of synapses scales as the network size to the power of 3/2, the number of sequences P is proportional to N. In other words, sequential memory is scalable. Furthermore, we find that there is an optimal ratio r between silent and nonsilent synapses at which the storage capacity alpha = P//[c(1 + r)N] assumes a maximum. For long sequences, the capacity of sequential memory is about one order of magnitude below the capacity for minimal sequences, but otherwise behaves similar to the case of minimal sequences. In a biologically inspired scenario, the information content per synapse is far below theoretical optimality, suggesting that the brain trades off error tolerance against information content in encoding sequential memories.     

基于最优连通功率控制的WSNs跨层路由优化算法

针对非连通区域节点空洞效应和热点区域节点间通信干扰导致的路由服务质量（QoS）下降问题,提出了一种基于最优连通功率控制的无线传感器网络（WSNs）跨层路由优化算法。算法采用自适应最优连通功率控制策略,在避免路由空洞产生和保证网络连通性条件下,降低热点区域节点数据转发竞争干扰;通过位置信息、剩余能量和干扰等级的跨层信息交互,动态选取最优转发节点,提高网络整体性能。仿真实验表明：算法能够提高路由（QoS）、优化网络生命周期和降低热点区域通信干扰。     

超立方体网络的3路结构连通度及子结构连通度

针对以超立方体网络为蓝本的多处理机系统的可靠性和容错能力的精准度量问题，结合多处理机系统遭受计算机病毒攻击时常常发生结构性故障的特点，研究了n维超立方体网络的结构连通性和子结构连通性评价问题。首先，使用构造n维超立方体网络的3路结构割的方法得到其3路结构连通度的一个上界；然后，使用构造n维超立方体网络的3路子结构集的等价变换或约简变换的方法，得到其3路结构子连通度的一个下界；最后，利用任意网络的3路结构连通度不小于3路子结构连通度的性质，证实了超立方体网络的3路结构连通度和子结构连通度均为该超立方体网络维数的一半。这一结果表明，在3路结构故障模型下，破坏敌方以超立方体网络为底层拓扑的多处理系统至少需要攻击该系统中维数一半的3路结构或子结构。     

The high-conductance state of cortical networks

We studied the dynamics of large networks of spiking neurons with conductance-based (nonlinear) synapses and compared them to networks with current-based (linear) synapses. For systems with sparse and inhibition-dominated recurrent connectivity, weak external inputs induced asynchronous irregular firing at low rates. Membrane potentials fluctuated a few millivolts below threshold, and membrane conductances were increased by a factor 2 to 5 with respect to the resting state. This combination of parameters characterizes the ongoing spiking activity typically recorded in the cortex in vivo. Many aspects of the asynchronous irregular state in conductance-based networks could be sufficiently well characterized with a simple numerical mean field approach. In particular, it correctly predicted an intriguing property of conductance-based networks that does not appear to be shared by current-based models: they exhibit states of low-rate asynchronous irregular activity that persist for some period of time even in the absence of external inputs and without cortical pacemakers. Simulations of larger networks (up to 350,000 neurons) demonstrated that the survival time of self-sustained activity increases exponentially with network size.     

Noise tolerance of attractor and feedforward memory models

In short-term memory networks, transient stimuli are represented by patterns of neural activity that persist long after stimulus offset. Here, we compare the performance of two prominent classes of memory networks, feedback-based attractor networks and feedforward networks, in conveying information about the amplitude of a briefly presented stimulus in the presence of gaussian noise. Using Fisher information as a metric of memory performance, we find that the optimal form of network architecture depends strongly on assumptions about the forms of nonlinearities in the network. For purely linear networks, we find that feedforward networks outperform attractor networks because noise is continually removed from feedforward networks when signals exit the network; as a result, feedforward networks can amplify signals they receive faster than noise accumulates over time. By contrast, attractor networks must operate in a signal-attenuating regime to avoid the buildup of noise. However, if the amplification of signals is limited by a finite dynamic range of neuronal responses or if noise is reset at the time of signal arrival, as suggested by recent experiments, we find that attractor networks can outperform feedforward ones. Under a simple model in which neurons have a finite dynamic range, we find that the optimal attractor networks are forgetful if there is no mechanism for noise reduction with signal arrival but nonforgetful (perfect integrators) in the presence of a strong reset mechanism. Furthermore, we find that the maximal Fisher information for the feedforward and attractor networks exhibits power law decay as a function of time and scales linearly with the number of neurons. These results highlight prominent factors that lead to trade-offs in the memory performance of networks with different architectures and constraints, and suggest conditions under which attractor or feedforward networks may be best suited to storing information about previous stimuli.     

A context layered locally recurrent neural network for dynamic system identification

This paper puts forward a novel recurrent neural network (RNN), referred to as the context layered locally recurrent neural network (CLLRNN) for dynamic system identification. The CLLRNN is a dynamic neural network which appears in effective in the input–output identification of both linear and nonlinear dynamic systems. The CLLRNN is composed of one input layer, one or more hidden layers, one output layer, and also one context layer improving the ability of the network to capture the linear characteristics of the system being identified. Dynamic memory is provided by means of feedback connections from nodes in the first hidden layer to nodes in the context layer and in case of being two or more hidden layers, from nodes in a hidden layer to nodes in the preceding hidden layer. In addition to feedback connections, there are self-recurrent connections in all nodes of the context and hidden layers. A dynamic backpropagation algorithm with adaptive learning rate is derived to train the CLLRNN. To demonstrate the superior properties of the proposed architecture, it is applied to identify not only linear but also nonlinear dynamic systems. The efficiency of the proposed architecture is demonstrated by comparing the results to some existing recurrent networks and design configurations. In addition, performance of the CLLRNN is analyzed through an experimental application to a dc motor connected to a load to show practicability and effectiveness of the proposed neural network. Results of the experimental application are presented to make a quantitative comparison with an existing recurrent network in the literature.     

Bounds of the incremental gain for discrete-time recurrent neural networks

As a nonlinear system, a recurrent neural network generally has an incremental gain different from its induced norm. While most of the previous research efforts were focused on the latter, this paper presents a method to compute an effective upper bound of the former for a class of discrete-time recurrent neural networks, which is not only applied to systems with arbitrary inputs but also extended to systems with small-norm inputs. The upper bound is computed by simple optimizations subject to linear matrix inequalities (LMIs). To demonstrate the wide connections of our results to problems in control, the servomechanism is then studied, where a feedforward neural network is designed to control the output of a recurrent neural network to track a set of trajectories. This problem can be converted into the synthesis of feedforward-feedback gains such that the incremental gain of a certain system is minimized. An algorithm to perform such a synthesis is proposed and illustrated with a numerical example.     

Constructive learning of recurrent neural networks: limitations ofrecurrent cascade correlation and a simple solution

It is often difficult to predict the optimal neural network size for a particular application. Constructive or destructive methods that add or subtract neurons, layers, connections, etc. might offer a solution to this problem. We prove that one method, recurrent cascade correlation, due to its topology, has fundamental limitations in representation and thus in its learning capabilities. It cannot represent with monotone (i.e., sigmoid) and hard-threshold activation functions certain finite state automata. We give a "preliminary" approach on how to get around these limitations by devising a simple constructive training method that adds neurons during training while still preserving the powerful fully-recurrent structure. We illustrate this approach by simulations which learn many examples of regular grammars that the recurrent cascade correlation method is unable to learn.     

Population coding and decoding in a neural field: a computational study

This study uses a neural field model to investigate computational aspects of population coding and decoding when the stimulus is a single variable. A general prototype model for the encoding process is proposed, in which neural responses are correlated, with strength specified by a gaussian function of their difference in preferred stimuli. Based on the model, we study the effect of correlation on the Fisher information, compare the performances of three decoding methods that differ in the amount of encoding information being used, and investigate the implementation of the three methods by using a recurrent network. This study not only rediscovers main results in existing literatures in a unified way, but also reveals important new features, especially when the neural correlation is strong. As the neural correlation of firing becomes larger, the Fisher information decreases drastically. We confirm that as the width of correlation increases, the Fisher information saturates and no longer increases in proportion to the number of neurons. However, we prove that as the width increases further--wider than (sqrt)2 times the effective width of the turning function--the Fisher information increases again, and it increases without limit in proportion to the number of neurons. Furthermore, we clarify the asymptotic efficiency of the maximum likelihood inference (MLI) type of decoding methods for correlated neural signals. It shows that when the correlation covers a nonlocal range of population (excepting the uniform correlation and when the noise is extremely small), the MLI type of method, whose decoding error satisfies the Cauchy-type distribution, is not asymptotically efficient. This implies that the variance is no longer adequate to measure decoding accuracy.     

Optimal state estimation for stochastic systems: an informationtheoretic approach

In this paper, we examine the problem of optimal state estimation or filtering in stochastic systems using an approach based on information theoretic measures. In this setting, the traditional minimum mean-square measure is compared with information theoretic measures, Kalman filtering theory is reexamined, and some new interpretations are offered. We show that for a linear Gaussian system, the Kalman filter is the optimal filter not only for the mean-square error measure, but for several information theoretic measures which are introduced in this work. For nonlinear systems, these same measures generally are in conflict with each other, and the feedback control policy has a dual role with regard to regulation and estimation. For linear stochastic systems with general noise processes, a lower bound on the achievable mutual information between the estimation error and the observation are derived. The properties of an optimal (probing) control law and the associated optimal filter, which achieve this lower bound, and their relationships are investigated. It is shown that for a linear stochastic system with an affine linear filter for the homogeneous system, under some reachability and observability conditions, zero mutual information between estimation error and observations can be achieved only when the system is Gaussian