Dynamic stochastic synapses as computational units

In most neural network models, synapses are treated as static weights that change only with the slow time scales of learning. It is well known, however, that synapses are highly dynamic and show use-dependent plasticity over a wide range of time scales. Moreover, synaptic transmission is an inherently stochastic process: a spike arriving at a presynaptic terminal triggers the release of a vesicle of neurotransmitter from a release site with a probability that can be much less than one. We consider a simple model for dynamic stochastic synapses that can easily be integrated into common models for networks of integrate-and-fire neurons (spiking neurons). The parameters of this model have direct interpretations in terms of synaptic physiology. We investigate the consequences of the model for computing with individual spikes and demonstrate through rigorous theoretical results that the computational power of the network is increased through the use of dynamic synapses.     

Adaptive optimal control without weight transport

Many neural control systems are at least roughly optimized, but how is optimal control learned? There are algorithms for this purpose, but in their current forms, they are not suited for biological neural networks because they rely on a type of communication that is not available in the brain, namely, weight transport-transmitting the strengths, or "weights," of individual synapses to other synapses and neurons. Here we show how optimal control can be learned without weight transport. Our method involves a set of simple mechanisms that can compensate for the absence of weight transport in the brain and so may be useful for neural computation generally.     

一种非线性可自学习的联想记忆神经网络

醉庆生物神经突触特性的基础上,提出了非线性神经突触神经元的概念,并以此为根据构造了一种可自学习的联想记忆神经网络模型。这种模型可以按照Hebb规则进行学习,学习机制由网络本身完成。在此模型中,由于非线性权重的引入,使此神经网络模型能以简单的结构实现网络的自学习功能。文中对网络的记忆容量和此种网络在以特定的学习方式学习后与Hopfield网络的等效性方面进行了讨论。试验表明,此种网络模型结构是有效的。     

Optimality model of unsupervised spike-timing-dependent plasticity: synaptic memory and weight distribution

We studied the hypothesis that synaptic dynamics is controlled by three basic principles: (1) synapses adapt their weights so that neurons can effectively transmit information, (2) homeostatic processes stabilize the mean firing rate of the postsynaptic neuron, and (3) weak synapses adapt more slowly than strong ones, while maintenance of strong synapses is costly. Our results show that a synaptic update rule derived from these principles shares features, with spike-timing-dependent plasticity, is sensitive to correlations in the input and is useful for synaptic memory. Moreover, input selectivity (sharply tuned receptive fields) of postsynaptic neurons develops only if stimuli with strong features are presented. Sharply tuned neurons can coexist with unselective ones, and the distribution of synaptic weights can be unimodal or bimodal. The formulation of synaptic dynamics through an optimality criterion provides a simple graphical argument for the stability of synapses, necessary for synaptic memory.     

Stochastic synchrony of chaos in a pulse-coupled neural network with both chemical and electrical synapses among inhibitory neurons

The synchronous firing of neurons in a pulse-coupled neural network composed of excitatory and inhibitory neurons is analyzed. The neurons are connected by both chemical synapses and electrical synapses among the inhibitory neurons. When electrical synapses are introduced, periodically synchronized firing as well as chaotically synchronized firing is widely observed. Moreover, we find stochastic synchrony where the ensemble-averaged dynamics shows synchronization in the network but each neuron has a low firing rate and the firing of the neurons seems to be stochastic. Stochastic synchrony of chaos corresponding to a chaotic attractor is also found.     

Learning only when necessary: better memories of correlated patterns in networks with bounded synapses

Learning in a neuronal network is often thought of as a linear superposition of synaptic modifications induced by individual stimuli. However, since biological synapses are naturally bounded, a linear superposition would cause fast forgetting of previously acquired memories. Here we show that this forgetting can be avoided by introducing additional constraints on the synaptic and neural dynamics. We consider Hebbian plasticity of excitatory synapses. A synapse is modified only if the postsynaptic response does not match the desired output. With this learning rule, the original memory performances with unbounded weights are regained, provided that (1) there is some global inhibition, (2) the learning rate is small, and (3) the neurons can discriminate small differences in the total synaptic input (e.g., by making the neuronal threshold small compared to the total postsynaptic input). We prove in the form of a generalized perceptron convergence theorem that under these constraints, a neuron learns to classify any linearly separable set of patterns, including a wide class of highly correlated random patterns. During the learning process, excitation becomes roughly balanced by inhibition, and the neuron classifies the patterns on the basis of small differences around this balance. The fact that synapses saturate has the additional benefit that nonlinearly separable patterns, such as similar patterns with contradicting outputs, eventually generate a subthreshold response, and therefore silence neurons that cannot provide any information.     

Analog CMOS implementation of a multilayer perceptron withnonlinear synapses

A neurocomputer based on a high-density analog integrated circuit developed in a 3 mum CMOS technology has been built. The 1.6 mmx2.4 mm chip contains 18 neurons and 161 synapses in three layers, and provides 16 inputs and 4 outputs. The weights are stored on storage capacitors of the synapses. A formalization of the error back-propagation algorithm which allows the use of very small nonlinear synapses is shown. The influence of offset voltages in the synapses on the circuit performance is analyzed. Some experimental results are reported and discussed.     

基于权重分摊的LeNet-5卷积神经网络防御策略

随着神经网络在自动驾驶、医疗诊断等关键领域的应用不断深入,如何确保神经网络的鲁棒性和安全性已成为当前研究的热点和挑战。在对抗攻击、数据中毒攻击、后门攻击等众多攻击方式中,随机翻转攻击是一种对安全性影响极大的攻击,其通过改变模型内部的权重参数来攻击网络,以降低网络性能。为应对此攻击方式,研究了一种基于权重分摊的防御策略。通过计算和分析权重的梯度来确定关键神经元,并为这些神经元添加冗余结构,使错误的权重最终被稀释,以提高模型的容错能力。为了验证这一防御策略,以LeNet-5模型为实验对象进行实验。实验表明,在相同的攻击条件下,经过防御后的模型相较于原始LeNet-5模型,容错精度提升了6.5%,相较于Inception-LeNet-5模型在全连接层上容错精度提升了1.9%。     

The combined effects of inhibitory and electrical synapses in synchrony

Recent experimental results have shown that GABAergic interneurons in the central nervous system are frequently connected via electrical synapses. Hence, depending on the area or the subpopulation, interneurons interact via inhibitory synapses or electrical synapses alone or via both types of interactions. The theoretical work presented here addresses the significance of these different modes of interactions for the interneuron networks dynamics. We consider the simplest system in which this issue can be investigated in models or in experiments: a pair of neurons, interacting via electrical synapses, inhibitory synapses, or both, and activated by the injection of a noisy external current. Assuming that the couplings and the noise are weak, we derive an analytical expression relating the cross-correlation (CC) of the activity of the two neurons to the phase response function of the neurons. When electrical and inhibitory interactions are not too strong, they combine their effect in a linear manner. In this regime, the effect of electrical and inhibitory interactions when combined can be deduced knowing the effects of each of the interactions separately. As a consequence, depending on intrinsic neuronal properties, electrical and inhibitory synapses may cooperate, both promoting synchrony, or may compete, with one promoting synchrony while the other impedes it. In contrast, for sufficiently strong couplings, the two types of synapses combine in a nonlinear fashion. Remarkably, we find that in this regime, combining electrical synapses with inhibition amplifies synchrony, whereas electrical synapses alone would desynchronize the activity of the neurons. We apply our theory to predict how the shape of the CC of two neurons changes as a function of ionic channel conductances, focusing on the effect of persistent sodium conductance, of the firing rate of the neurons and the nature and the strength of their interactions. These predictions may be tested using dynamic clamp techniques.     

A model for fast analog computation based on unreliable synapses

We investigate through theoretical analysis and computer simulations the consequences of unreliable synapses for fast analog computations in networks of spiking neurons, with analog variables encoded by the current firing activities of pools of spiking neurons. Our results suggest a possible functional role for the well-established unreliability of synaptic transmission on the network level. We also investigate computations on time series and Hebbian learning in this context of space-rate coding in networks of spiking neurons with unreliable synapses.     

基于改进GA-Elman的无线智能传播损耗预测方法

在5G移动通信系统商用落地的背景下,设计准确、高效的信道估计方法对无线网络性能优化具有重要意义。基于改进GA-Elman算法,提出一种新的无线智能传播损耗预测方法。对Elman神经网络中的连接权值、阈值和隐藏神经元进行实数编码,在隐藏神经元编码中加入二进制控制基因,同时利用自适应遗传算法对权值、阈值和隐藏神经元数量进行优化,解决网络易陷入局部极小值和神经元数目难以确定的问题,从而提高预测性能。仿真结果表明,与仅优化连接权值及阈值的GA-Elman神经网络和标准Elman神经网络相比,该方法具有较高的预测精度。     

Dynamics of strongly-coupled spiking neurons

We present a dynamical theory of integrate-and-fire neurons with strong synaptic coupling. We show how phase-locked states that are stable in the weak coupling regime can destabilize as the coupling is increased, leading to states characterized by spatiotemporal variations in the interspike intervals (ISIs). The dynamics is compared with that of a corresponding network of analog neurons in which the outputs of the neurons are taken to be mean firing rates. A fundamental result is that for slow interactions, there is good agreement between the two models (on an appropriately defined timescale). Various examples of desynchronization in the strong coupling regime are presented. First, a globally coupled network of identical neurons with strong inhibitory coupling is shown to exhibit oscillator death in which some of the neurons suppress the activity of others. However, the stability of the synchronous state persists for very large networks and fast synapses. Second, an asymmetric network with a mixture of excitation and inhibition is shown to exhibit periodic bursting patterns. Finally, a one-dimensional network of neurons with long-range interactions is shown to desynchronize to a state with a spatially periodic pattern of mean firing rates across the network. This is modulated by deterministic fluctuations of the instantaneous firing rate whose size is an increasing function of the speed of synaptic response.     

基于Temporal rule的忆阻神经网络电路

忆阻器是一种动态特性的电阻,其阻值可以根据外场的变化而变化,并且在外场撤掉后能够保持原来的阻值,具有类似于生物神经突触连接强度的特性,可以用来存储突触权值。在此基础上,为了实现基于Temporal rule对IRIS数据集识别学习的功能,建立了以桥式忆阻器为突触的神经网络SPICE仿真电路。采用单个脉冲的编码方式,脉冲的时刻代表着数据信息,该神经网络电路由48个脉冲输入端口、144个突触、3个输出端口组成。基于Temporal rule学习规则对突触的权值修改,通过仿真该神经网络电路对IRIS数据集的分类正确率最高能达到93.33%,表明了此神经系统结构设计在类脑脉冲神经网络中的可用性。     

A population density approach that facilitates large-scale modeling of neural networks: extension to slow inhibitory synapses

A previously developed method for efficiently simulating complex networks of integrate-and-fire neurons was specialized to the case in which the neurons have fast unitary postsynaptic conductances. However, inhibitory synaptic conductances are often slower than excitatory ones for cortical neurons, and this difference can have a profound effect on network dynamics that cannot be captured with neurons that have only fast synapses. We thus extend the model to include slow inhibitory synapses. In this model, neurons are grouped into large populations of similar neurons. For each population, we calculate the evolution of a probability density function (PDF), which describes the distribution of neurons over state-space. The population firing rate is given by the flux of probability across the threshold voltage for firing an action potential. In the case of fast synaptic conductances, the PDF was one-dimensional, as the state of a neuron was completely determined by its transmembrane voltage. An exact extension to slow inhibitory synapses increases the dimension of the PDF to two or three, as the state of a neuron now includes the state of its inhibitory synaptic conductance. However, by assuming that the expected value of a neuron's inhibitory conductance is independent of its voltage, we derive a reduction to a one-dimensional PDF and avoid increasing the computational complexity of the problem. We demonstrate that although this assumption is not strictly valid, the results of the reduced model are surprisingly accurate.     

Learning real-world stimuli in a neural network with spike-driven synaptic dynamics

We present a model of spike-driven synaptic plasticity inspired by experimental observations and motivated by the desire to build an electronic hardware device that can learn to classify complex stimuli in a semisupervised fashion. During training, patterns of activity are sequentially imposed on the input neurons, and an additional instructor signal drives the output neurons toward the desired activity. The network is made of integrate-and-fire neurons with constant leak and a floor. The synapses are bistable, and they are modified by the arrival of presynaptic spikes. The sign of the change is determined by both the depolarization and the state of a variable that integrates the postsynaptic action potentials. Following the training phase, the instructor signal is removed, and the output neurons are driven purely by the activity of the input neurons weighted by the plastic synapses. In the absence of stimulation, the synapses preserve their internal state indefinitely. Memories are also very robust to the disruptive action of spontaneous activity. A network of 2000 input neurons is shown to be able to classify correctly a large number (thousands) of highly overlapping patterns (300 classes of preprocessed Latex characters, 30 patterns per class, and a subset of the NIST characters data set) and to generalize with performances that are better than or comparable to those of artificial neural networks. Finally we show that the synaptic dynamics is compatible with many of the experimental observations on the induction of long-term modifications (spike-timing-dependent plasticity and its dependence on both the postsynaptic depolarization and the frequency of pre- and postsynaptic neurons).     

Efficient event-driven simulation of large networks of spiking neurons and dynamical synapses

A simulation procedure is described for making feasible large-scale simulations of recurrent neural networks of spiking neurons and plastic synapses. The procedure is applicable if the dynamic variables of both neurons and synapses evolve deterministically between any two successive spikes. Spikes introduce jumps in these variables, and since spike trains are typically noisy, spikes introduce stochasticity into both dynamics. Since all events in the simulation are guided by the arrival of spikes, at neurons or synapses, we name this procedure event-driven. The procedure is described in detail, and its logic and performance are compared with conventional (synchronous) simulations. The main impact of the new approach is a drastic reduction of the computational load incurred upon introduction of dynamic synaptic efficacies, which vary organically as a function of the activities of the pre- and postsynaptic neurons. In fact, the computational load per neuron in the presence of the synaptic dynamics grows linearly with the number of neurons and is only about 6% more than the load with fixed synapses. Even the latter is handled quite efficiently by the algorithm. We illustrate the operation of the algorithm in a specific case with integrate-and-fire neurons and specific spike-driven synaptic dynamics. Both dynamical elements have been found to be naturally implementable in VLSI. This network is simulated to show the effects on the synaptic structure of the presentation of stimuli, as well as the stability of the generated matrix to the neural activity it induces.     

Image preprocessing with dynamic synapses

Different algorithms suitable for a specific class of picture were developed for image processing. We will represent the filtering capability of a spiking neural network based on dynamic synapses. For this intention we chose an x-ray image of the human coronary trees and another noisy image. In other words the task at hand is to show how accurately such a network is able to store various aspects (object/background) of stimulus in the variables which describe dynamic of synaptic response. The behavior of these synapses influences the effective connection in the network in a short time-scale. Such a network has a low activity and a balanced behavior. Dynamic synapses are able to adjust their behavior by fast changing stimuli. These synapses retain the information in the variables, such as potential and time.     

When response variability increases neural network robustness to synaptic noise

Cortical sensory neurons are known to be highly variable, in the sense that responses evoked by identical stimuli often change dramatically from trial to trial. The origin of this variability is uncertain, but it is usually interpreted as detrimental noise that reduces the computational accuracy of neural circuits. Here we investigate the possibility that such response variability might in fact be beneficial, because it may partially compensate for a decrease in accuracy due to stochastic changes in the synaptic strengths of a network. We study the interplay between two kinds of noise, response (or neuronal) noise and synaptic noise, by analyzing their joint influence on the accuracy of neural networks trained to perform various tasks. We find an interesting, generic interaction: when fluctuations in the synaptic connections are proportional to their strengths (multiplicative noise), a certain amount of response noise in the input neurons can significantly improve network performance, compared to the same network without response noise. Performance is enhanced because response noise and multiplicative synaptic noise are in some ways equivalent. So if the algorithm used to find the optimal synaptic weights can take into account the variability of the model neurons, it can also take into account the variability of the synapses. Thus, the connection patterns generated with response noise are typically more resistant to synaptic degradation than those obtained without response noise. As a consequence of this interplay, if multiplicative synaptic noise is present, it is better to have response noise in the network than not to have it. These results are demonstrated analytically for the most basic network consisting of two input neurons and one output neuron performing a simple classification task, but computer simulations show that the phenomenon persists in a wide range of architectures, including recurrent (attractor) networks and sensorimotor networks that perform coordinate transformations. The results suggest that response variability could play an important dynamic role in networks that continuously learn.     

增量型极限学习机改进算法

增量型极限学习机（incremental extreme learning machine,I-ELM）在训练过程中,由于输入权值及隐层神经元阈值的随机获取,造成部分隐层神经元的输出权值过小,使其对网络输出贡献小,从而成为无效神经元．这个问题不但使网络变得更加复杂,而且降低了网络的稳定性．针对此问题,本文提出了一种给I-ELM隐层输出加上偏置的改进方法（即Ⅱ-ELM）,并分析证明了该偏置的存在性．最后对I-ELM方法在分类和回归问题上进行仿真对比,验证Ⅱ-ELM的有效性.     

A multi-start algorithm to design a multi-class classifier for a multi-criteria ABC inventory classification problem

In this paper we deal with the problem of designing a classifier able to learn the classification of existing units in inventory and then use it to classify new units according to their attributes in a multi-criteria ABC inventory classification environment. To solve this problem we design a multi-start constructive algorithm to train a discrete artificial neural network using a randomized greedy strategy to add neurons to the network hidden layer. The process of weights’ searching for the neurons to be added is based on solving linear programming formulations. The computational experiments show that the proposed algorithm is much more efficient when the dual formulations are used to find the weights of the network neurons and that the obtained classifier has good levels of generalization accuracy. In addition, the proposed algorithm can be straight applied to other multi-class classification problems with more than three classes.