On-chip template training system and image processing applications using iterative annealing on ACE16k chip

Cellular neural networks proved to be a useful parallel computing system for image processing applications. Cellular neural networks (CNNs) constitute a class of recurrent and locally coupled arrays of identical cells. The connectivity among the cells is determined by a set of parameters called templates. CNN templates are the key parameters to perform a desired task. One of the challenging problems in designing templates is to find the optimal template that functions appropriately for the solution of the intended problem. In this paper, we have implemented the Iterative Annealing Optimization Method on the analog CNN chip to find an optimum template by training a randomly selected initial template. We have been able to show that the proposed system is efficient to find the suitable template for some specific image processing applications.     

针对卷积神经网络流量分类器的对抗样本攻击防御

随着深度学习的兴起,深度神经网络被成功应用于多种领域,但研究表明深度神经网络容易遭到对抗样本的恶意攻击.作为深度神经网络之一的卷积神经网络(CNN)目前也被成功应用于网络流量的分类问题,因此同样会遭遇对抗样本的攻击.为提高CNN网络流量分类器防御对抗样本的攻击,本文首先提出批次对抗训练方法,利用训练过程反向传播误差的特...     

Neural network-based colonoscopic diagnosis using on-line learning and differential evolution

In this paper, on-line training of neural networks is investigated in the context of computer-assisted colonoscopic diagnosis. A memory-based adaptation of the learning rate for the on-line back-propagation (BP) is proposed and used to seed an on-line evolution process that applies a differential evolution (DE) strategy to (re-) adapt the neural network to modified environmental conditions. Our approach looks at on-line training from the perspective of tracking the changing location of an approximate solution of a pattern-based, and thus, dynamically changing, error function. The proposed hybrid strategy is compared with other standard training methods that have traditionally been used for training neural networks off-line. Results in interpreting colonoscopy images and frames of video sequences are promising and suggest that networks trained with this strategy detect malignant regions of interest with accuracy.     

Advances in feedforward neural networks: demystifying knowledgeacquiring black boxes

We survey research of recent years on the supervised training of feedforward neural networks. The goal is to expose how the networks work, how to engineer them so they can learn data with less extraneous noise, how to train them efficiently, and how to assure that the training is valid. The scope covers gradient descent and polynomial line search, from backpropagation through conjugate gradients and quasi Newton methods. There is a consensus among researchers that adaptive step gains (learning rates) can stabilize and accelerate convergence and that a good starting weight set improves both the training speed and the learning quality. The training problem includes both the design of a network function and the fitting of the function to a set of input and output data points by computing a set of coefficient weights. The form of the function can be adjusted by adjoining new neurons and pruning existing ones and setting other parameters such as biases and exponential rates. Our exposition reveals several useful results that are readily implementable     

Backpropagation for Fully Connected Cascade Networks

The fully connected cascade (FCC) networks are a recently proposed class of neural networks where each layer has only one neuron and each neuron is connected with all the neurons in its previous layers. In this paper we derive and describe in detail an efficient backpropagation algorithm (named BPFCC) for computing the gradient for FCC networks. Actually, the backpropagation in BPFCC is an elaborately designed process for computing the derivative amplification coefficients, which are essential for gradient computation. The average time complexity for computing an entry of the gradient is O(1). BPFCC needs to be called by training algorithms to do any useful work, and we wrote a program FCCNET for that purpose. Currently, FCCNET uses the Levenberg–Marquardt algorithm to train FCC networks, and the loss function for classification is designed based on a nonlinear extension of logistic regression. For two-class classification, we derive a Gauss–Newton-like approximation for the Hessian of the loss function, and when the number of classes is more than two, numerical approximation of the Hessian is used. Experimental results confirm the efficiency of BPFCC, and the validity of the companion techniques.     

A fast and scalable architecture to run convolutional neural networks in low density FPGAs

Deep learning and, in particular, convolutional neural networks (CNN) achieve very good results on several computer vision applications like security and surveillance, where image and video analysis are required. These networks are quite demanding in terms of computation and memory and therefore are usually implemented in high-performance computing platforms or devices. Running CNNs in embedded platforms or devices with low computational and memory resources requires a careful optimization of system architectures and algorithms to obtain very efficient designs. In this context, Field Programmable Gate Arrays (FPGA) can achieve this efficiency since the programmable hardware fabric can be tailored for each specific network. In this paper, a very efficient configurable architecture for CNN inference targeting any density FPGAs is described. The architecture considers fixed-point arithmetic and image batch to reduce computational, memory and memory bandwidth requirements without compromising network accuracy. The developed architecture supports the execution of large CNNs in any FPGA devices including those with small on-chip memory size and logic resources. With the proposed architecture, it is possible to infer an image in AlexNet in 4.3 ms in a ZYNQ7020 and 1.2 ms in a ZYNQ7045.     

Convergence of gradient method for a fully recurrent neural network

Recurrent neural networks have been successfully used for analysis and prediction of temporal sequences. This paper is concerned with the convergence of a gradient-descent learning algorithm for training a fully recurrent neural network. In literature, stochastic process theory has been used to establish some convergence results of probability nature for the on-line gradient training algorithm, based on the assumption that a very large number of (or infinitely many in theory) training samples of the temporal sequences are available. In this paper, we consider the case that only a limited number of training samples of the temporal sequences are available such that the stochastic treatment of the problem is no longer appropriate. Instead, we use an off-line gradient training algorithm for the fully recurrent neural network, and we accordingly prove some convergence results of deterministic nature. The monotonicity of the error function in the iteration is also guaranteed. A numerical example is given to support the theoretical findings.     

基于星形互连网络的并行快速傅立叶变换算法

星形互连网络是一种易于实现大规模并行计算的互连网络拓扑结构。利用星形互连网络的递归可分解性的多样性，提出了一种基于星形互连网络的并行快速傅立叶变换算法的实现方法。该方法能够有效地减少计算过程中处理器结点之间的通信开销。提出的星图结点和数据的映射应运 及实现并行FFT的思想可推广到线性方程组求解、矩阵乘法等其它并行算法在星形互连网络上的实现。     

Real-time penetration state monitoring using convolutional neural network for laser welding of tailor rolled blanks

In this paper, an innovative monitoring system capable of diagnosing the penetration state during the laser welding process is introduced, which consists of two main blocks: a coaxial visual monitoring platform and a penetration state diagnosis unit. The platform can capture coaxial images of the interaction zone during the laser welding through a partially transmitting mirror and a high-speed camera. An image dataset representing four welding states was created for training and validation. The unit mainly consists of an embedded power-efficient computing TX2 and image processing algorithms based on a convolution neural network (CNN). Experiment results show that the platform can stably capture state-of-the-art welding images. The CNN used for a diagnosis of the penetration state is optimized using an optimal network structure and hyperparameters, applying a super-Gaussian function to initialize the weights of the convolutional layer. Its latency on TX2 is less than 2 ms, satisfying the real-time requirement. During the real laser welding of tailor-rolled blanks, a penetration state diagnosis with an accuracy of 94.6 % can be achieved even if the illumination changes significantly. The similar accuracy between the validating set and a real laser welding demonstrates that the proposed monitoring system has strong robustness. The precision and recall ratios of the CNN are higher than those of other methods such as a histogram of oriented gradients and local binary pattern.     

基于自适应遗传算法的细胞神经网络模板设计

细胞神经网络具有能够高速并行计算,易于硬件实现等特点,使其广泛应用于图像处理边缘提取、字符识别等诸多领域。细胞神经网络要正确实现不同的图像处理功能的关键在于模板参数的设计。提出一种基于自适应遗传算法求解模板参数的方法,一方面,通过对交叉概率和变异概率的改进以及遗传算子的设计,克服了基于简单遗传算法设计模板时算法容易早熟的不足;另一方面,采用准精确惩罚函数来设计适应度函数．降低了算法的运算量,提高了算法的收敛速度。给出了实例仿真结果,表明该方法的有效性。     

基于FPGA的量化CNN加速系统设计

深度卷积神经网络（CNN）模型中卷积层和全连接层包含大量卷积操作,导致网络规模、参数量和计算量大幅增加,部署于CPU/GPU平台时存在并行计算性能差和不适用于移动设备环境的问题,需要对卷积参数做量化处理并结合硬件进行加速设计。现场可编程门阵列（FPGA）可满足CNN并行计算和低功耗的需求,并具有高度的灵活性,因此,基于FPGA设计CNN量化方法及其加速系统。提出一种通用的动态定点量化方法,同时对网络的各个层级进行不同精度的量化,以减少网络准确率损失和网络参数的存储需求。在此基础上,针对量化后的CNN设计专用加速器及其片上系统,加速网络的前向推理计算。使用ImageNet ILSVRC2012数据集,基于VGG-16与ResNet-50网络对所设计的量化方法和加速系统进行性能验证。实验结果显示,量化后VGG-16与ResNet-50的网络规模仅为原来的13.8%和24.8%,而Top-1准确率损失均在1%以内,表明量化方法效果显著,同时,加速系统在运行VGG-16时,加速效果优于其他3种FPGA实现的加速系统,峰值性能达到614.4 GOPs,最高提升4.5倍,能耗比达到113.99 GOPs/W,最高提升4.7倍。     

一种基于分布式编码的同步梯度下降算法

基于数据并行化的异步随机梯度下降（ASGD）算法由于需要在分布式计算节点之间频繁交换梯度数据,从而影响算法执行效率。提出基于分布式编码的同步随机梯度下降（SSGD）算法,利用计算任务的冗余分发策略对每个节点的中间结果传输时间进行量化以减少单一批次训练时间,并通过数据传输编码策略的分组数据交换模式降低节点间的数据通信总量。实验结果表明,当配置合适的超参数时,与SSGD和ASGD算法相比,该算法在深度神经网络和卷积神经网络分布式训练中平均减少了53.97%、26.89%和39.11%、26.37%的训练时间,从而证明其能有效降低分布式集群的通信负载并保证神经网络的训练精确度。     

Applications of multi-agent slime mould computing

The giant single-celled slime mould Physarum polycephalum has inspired rapid developments in unconventional computing substrates since the start of this century. This is primarily due to its simple component parts and the distributed nature of the ‘computation’ which it approximates during its growth, foraging and adaptation to a changing environment. Slime mould functions as a living embodied computational material which can be influenced (or programmed) by the placement of external stimuli. The goal of exploiting this material behaviour for unconventional computation led to the development of a multi-agent approach to the approximation of slime mould behaviour. The basis of the model is a simple dynamical pattern formation mechanism which exhibits self-organised formation and subsequent adaptation of collective transport networks. The system exhibits emergent properties such as relaxation and minimisation and it can be considered as a virtual computing material, influenced by the external application of spatial concentration gradients. In this paper we give an overview of this multi-agent approach to unconventional computing. We describe its computational mechanisms and different generic application domains, together with concrete example applications of material computation. We examine the potential exploitation of the approach for computational geometry, path planning, combinatorial optimisation, data smoothing and statistical applications.     

道路网络中的连续最近邻查询

为了减少连续最近邻查询中计算K个最近邻的次数和减小算法需要的存储空间,提出一种道路网络中求连续最近邻的方法。给出分点的计算方法及连续最近邻查询算法,对算法的正确性、可终止性进行证明,并分析算法复杂度。与相关算法进行实验比较,得出该算法更适合于对象频繁发生变化的实际网络。     

A generative image fusion approach based on supervised deep convolution network driven by weighted gradient flow

In recent times, convolution neural networks (CNNs) have been utilized to generate desired images benefiting from the layered features. However, few studies have focused on integrating these features gained from multiple sources to obtain a high-quality image. In this paper, we propose a generative fusion approach using a supervised CNN framework with analysis and synthesis modules. According to it, the salient feature maps obtained from the analysis module are integrated to yield output generation by iteratively back-propagating gradients. Furthermore, a differential fusion strategy based on weighted gradient flow is embedded into the end-to-end fusion procedure. To transfer previous network configurations to current fusion tasks, the proposed network is fine-tuned according to the pretrained network such as VGG16, VGG19 and ResNet50. The experimental results indicate superior evaluations of the proposed approach compared with other state-of-the-art schemes in various fusion scenes, and also verify that the CNN features are adaptable and expressive to be aligned to generate fused images.     

基于中间层监督卷积神经网络的图像超分辨率重建

目的 基于学习的图像超分辨率重建方法已成为近年来图像超分辨率重建研究的热点。针对基于卷积神经网络的图像超分辨率重建（SRCNN）方法网络层少、感受野小、泛化能力差等缺陷,提出了基于中间层监督卷积神经网络的图像超分辨率重建方法,以进一步提高图像重建的质量。方法 设计了具有中间层监督的卷积神经网络结构,该网络共有16层卷积层,其中第7层为中间监督层;定义了监督层误差函数和重建误差函数,用于改善深层卷积神经网络梯度消失现象。训练网络时包括图像预处理、特征提取和图像重建3个步骤,采用不同尺度因子（2、3、4）模糊的低分辨率图像交叉训练网络,以适应对不同模糊程度的图像重建;使用卷积操作提取图像特征时将参数pad设置为1,提高了对图像和特征图的边缘信息利用;利用残差学习完成高分辨率图像重建。结果 在Set5和Set14数据集上进行了实验,并和双三次插值、A+、SelfEx和SRCNN等方法的结果进行比较。在主观视觉评价方面,本文方法重建图像的清晰度和边缘锐度更好。客观评价方面,本文方法的峰值信噪比（PSNR）平均分别提高了2.26 dB、0.28 dB、0.28 dB和0.15 dB,使用训练好的网络模型重建图像耗用的时间不及SRCNN方法的一半。结论 实验结果表明,本文方法获得了更好的主观视觉评价和客观量化评价,提升了图像超分辨率重建质量,泛化能力好,而且图像重建过程耗时更短,可用于自然场景图像的超分辨率重建。     

改进卷积神经网络在互感器故障诊断中的应用

低压电流互感器作为电网中的关键设备,已经得到广泛使用。低压电流互感器故障诊断的在线检定也显得十分重要。提出了一种改进的全局平均池化的一维卷积神经网络（1DCNN-SVM）故障诊断模型应用于低压电流互感器在线检定。该方法改进了传统卷积神经网络（CNN）模型的结构,引入全局平均池化而不是全连接网络结构,并在测试阶段使用支持向量机（SVM）替代Softmax函数。通过进行实验分析,将所提的方法与传统的CNN进行实验对比,实验结果表明所提方法在训练时间、测试时间以及模型的测试精度等方面的表现都比传统的CNN结构模型要好。     

一种基于数据并行的过程神经网络训练算法

针对过程神经网络时空聚合运算机制复杂、学习周期长的问题,提出了一种基于数据并行的过程神经网络训练算法。该方法基于梯度下降的批处理训练方式,应用MPI并行模式进行算法设计,在局域网内实现多台计算机的机群并行计算。文中给出了基于数据并行的过程神经网络训练算法和实现机制,对不同规模的训练函数样本集和进程数进行了对比实验,并对加速比、并行效率等算法性质进行了分析。实验结果表明,根据网络和样本规模适当选取并行粒度,算法可较大提高过程神经网络的训练效率。     

Morphological associative memories

The theory of artificial neural networks has been successfully applied to a wide variety of pattern recognition problems. In this theory, the first step in computing the next state of a neuron or in performing the next layer neural network computation involves the linear operation of multiplying neural values by their synaptic strengths and adding the results. A nonlinear activation function usually follows the linear operation in order to provide for nonlinearity of the network and set the next state of the neuron. In this paper we introduce a novel class of artificial neural networks, called morphological neural networks, in which the operations of multiplication and addition are replaced by addition and maximum (or minimum), respectively. By taking the maximum (or minimum) of sums instead of the sum of products, morphological network computation is nonlinear before possible application of a nonlinear activation function. As a consequence, the properties of morphological neural networks are drastically different than those of traditional neural network models. The main emphasis of the research presented here is on morphological associative memories. We examine the computing and storage capabilities of morphological associative memories and discuss differences between morphological models and traditional semilinear models such as the Hopfield net.     

基于细胞神经网络的图像复原新方法

针对图像复原方法普遍运算量大的问题,提出了一种利用细胞神经网络进行图像复原的新方法,并首先提出了易于硬件实现的基于边缘方向判据的正则化复原方法;然后通过细胞神经网络的能量函数设计合适的网络参数来对该正则化函数进行细胞神经网络实现。仿真结果表明,该新方法是有效的,复原效果优于有约束的最小二乘复原法和已有的细胞神经网络图像复原法,而且由于细胞神经网络的并行性和硬件易实现性,使该新方法可以实时进行图像复原。