Nonmonotone Levenberg�CMarquardt training of recurrent neural architectures for processing symbolic sequences

In this paper, we present nonmonotone variants of the Levenberg–Marquardt (LM) method for training recurrent neural networks (RNNs). These methods inherit the benefits of previously developed LM with momentum algorithms and are equipped with nonmonotone criteria, allowing temporal increase in training errors, and an adaptive scheme for tuning the size of the nonmonotone slide window. The proposed algorithms are applied to training RNNs of various sizes and architectures in symbolic sequence-processing problems. Experiments show that the proposed nonmonotone learning algorithms train more effectively RNNs for sequence processing than the original monotone methods.     

A Hybrid Projection-based and Radial Basis Function Architecture: Initial Values and Global Optimisation

We introduce a mechanism for constructing and training a hybrid architecture of projection-based units and radial basis functions. In particular, we introduce an optimisation scheme which includes several steps and assures a convergence to a useful solution. During network architecture construction and training, it is determined whether a unit should be removed or replaced. The resulting architecture often has a smaller number of units compared with competing architectures. A specific overfitting resulting from shrinkage of the RBF radii is addressed by introducing a penalty on small radii. Classification and regression results are demonstrated on various benchmark data sets and compared with several variants of RBF networks [1,2]. A striking performance improvement is achieved on the vowel data set [3]. Received: 03 November 2000, Received in revised form: 25 October 2001, Accepted: 04 January 2002     

Distributed computing methodology for training neural networks in an image-guided diagnostic application

Distributed computing is a process through which a set of computers connected by a network is used collectively to solve a single problem. In this paper, we propose a distributed computing methodology for training neural networks for the detection of lesions in colonoscopy. Our approach is based on partitioning the training set across multiple processors using a parallel virtual machine. In this way, interconnected computers of varied architectures can be used for the distributed evaluation of the error function and gradient values, and, thus, training neural networks utilizing various learning methods. The proposed methodology has large granularity and low synchronization, and has been implemented and tested. Our results indicate that the parallel virtual machine implementation of the training algorithms developed leads to considerable speedup, especially when large network architectures and training sets are used.     

Evolving recurrent perceptrons for time-series modeling

Evolutionary programming, a systematic multi-agent stochastic search technique, is used to generate recurrent perceptrons (nonlinear IIR filters). A hybrid optimization scheme is proposed that embeds a single-agent stochastic search technique, the method of Solis and Wets, into the evolutionary programming paradigm. The proposed hybrid optimization approach is further augmented by "blending" randomly selected parent vectors to create additional offspring. The first part of this work investigates the performance of the suggested hybrid stochastic search method. After demonstration on the Bohachevsky and Rosenbrock response surfaces, the hybrid stochastic optimization approach is applied in determining both the model order and the coefficients of recurrent perceptron time-series models. An information criterion is used to evaluate each recurrent perceptron structure as a candidate solution. It is speculated that the stochastic training method implemented in this study for training recurrent perceptrons can be used to train perceptron networks that have radically recurrent architectures.     

基于深度残差网络图像分类算法研究综述

近年来,由于计算机技术的飞速迅猛发展,特别是硬件条件的改善,计算能力不断提高,深层神经网络训练的时间大大缩短,深度残差网络也迅速成为一个新的研究热点.深度残差网络作为一种极深的网络架构,在精度和收敛等方面都展现出了很好的特性.研究者们深入研究其本质并在此基础上提出了很多关于深度残差网络的改进,如宽残差网络,金字塔型残差网络,密集型残差网络,注意力残差网络等等.本文从残差网络的设计出发,分析了不同残差单元的构造方式,介绍了深度残差网络不同的变体.从不同的角度比较了不同网络之间的差异以及这些网络架构在常用图像分类数据集上的性能表现.最后我们对于这些网络进行了总结,并讨论了未来深度残差网络在图像分类领域的一些研究方向.     

On the game-theoretic analysis of distributed generative adversarial networks

In this paper, a distributed method is proposed for training multiple generative adversarial networks (GANs) with private data sets via a game-theoretic approach. To facilitate the requirement of privacy protection, distributed training algorithms offer a promising solution to learn global models without sample exchanges. Existing studies have mainly concentrated on training neural networks using pure cooperation strategies, which are not suitable for GANs. This paper develops a new framework for distributed GANs, where two groups of discriminators and generators are involved in a zero-sum game. Under connected graphs, such a framework is reformulated as a constrained minmax optimisation problem. Then, a fully distributed training algorithm is proposed without exchanging any private data samples. The convergence of the proposed algorithm is established via advanced consensus and optimisation techniques. Simulation studies are presented to validate the effectiveness of the proposed framework and algorithm.     

Hybrid of genetic algorithm and local search to solve MAX-SAT problem using nVidia CUDA framework

General Purpose computing over Graphical Processing Units (GPGPUs) is a huge shift of paradigm in parallel computing that promises a dramatic increase in performance. But GPGPUs also bring an unprecedented level of complexity in algorithmic design and software development. In this paper we describe the challenges and design choices involved in parallelizing a hybrid of Genetic Algorithm (GA) and Local Search (LS) to solve MAXimum SATisfiability (MAX-SAT) problem on a state-of-the-art nVidia Tesla GPU using nVidia Compute Unified Device Architecture (CUDA). MAX-SAT is a problem of practical importance and is often solved by employing metaheuristics based search methods like GAs and hybrid of GA with LS. Almost all the parallel GAs (pGAs) designed in the last two decades were designed for either clusters or MPPs. Unfortunately, very little research is done on the implementation of such algorithms over commodity graphics hardware. GAs in their simple form are not suitable for implementation over the Single Instruction Multiple Thread (SIMT) architecture of a GPU, and the same is the case with conventional LS algorithms. In this paper we explore different genetic operators that can be used for an efficient implementation of GAs over nVidia GPUs. We also design and introduce new techniques/operators for an efficient implementation of GAs and LS over such architectures. We use nVidia Tesla C1060 to perform several numerical tests and performance measurements and show that in the best case we obtain a speedup of 25×. We also discuss the effects of different optimization techniques on the overall execution time.     

Neural network method: delay and system of delay differential equations

In the present article, delay and system of delay differential equations are treated using feed-forward artificial neural networks. We have solved multiple problems using neural network architectures with different depths. The neural networks are trained using the extreme learning machine algorithm for the satisfaction of delay differential equations and associated initial/boundary conditions. Further, numerical rates of convergence of the proposed algorithm are reported based on variation of error in the obtained solution for different number of training points. Emphasis is on analysing whether deeper network architectures trained with extreme learning machine algorithm can perform better than shallow network architectures for approximating the solutions of delay differential equations.

     

A unified mathematical form for removing neurons based on orthogonal projection and crosswise propagation

It is a common practice to adjust the number of hidden neurons in training, and the removal of neurons in neural networks plays an indispensable role in this architecture manipulation. In this paper, a succinct and unified mathematical form is upgraded to the generic case for removing neurons based on orthogonal projection and crosswise propagation in a feedforward layer with different architectures of neural networks, and further developed for several neural networks with different architectures. For a trained neural network, the method is divided into three stages. In the first stage, the output vectors of the feedforward observation layer are classified to clusters. In the second stage, the orthogonal projection is performed to locate a neuron whose output vector can be approximated by the other output vectors in the same cluster with the least information loss. In the third stage, the previous located neuron is removed and the crosswise propagation is implemented in each cluster. On accomplishment of the three stages, the neural network with the pruned architecture is retrained. If the number of clusters is one, the method is degenerated into its special case with only one neuron being removed. The applications to different architectures of neural networks with an extension to the support vector machine are exemplified. The methodology supports in theory large-scale applications of neural networks in the real world. In addition, with minor modifications, the unified method is instructive in pruning other networks as far as they have similar network structure to the ones in this paper. It is concluded that the unified pruning method in this paper equips us an effective and powerful tool to simplify the architecture in neural networks.     

感知器在语言模型训练中的应用

感知器（perceptron）是神经网络模型中的一种,它可以通过监督学习（supervised learning）的方法建立模式识别的能力．将感知器应用到语言模型的训练中,实现了感知器的两种不同训练规则以及多种特征权值计算方法,讨论了不同的训练参数对训练效果的影响．在训练之前,使用了一种基于经验风险最小化（empirical risk minimization,ERM）的特征选择算法确定特征集合．感知器训练之后的语言模型在日文假名到汉字（kana-kanji）的转换中进行评估．通过实验对比了感知器的两种训练规则以及变形算法的性能,同时发现通过感知器训练的模型比传统模型（N-gram）在性能上有了很大的提高,使相对错误率下降了15％～20％．     

Decision support system for water distribution systems based on neural networks and graphs theory for leakage detection

This paper presents an efficient and effective decision support system (DSS) for operational monitoring and control of water distribution systems based on a three layer General Fuzzy Min–Max Neural Network (GFMMNN) and graph theory. The operational monitoring and control involves detection of pipe leakages. The training data for the GFMMNN is obtained through simulation of leakages in a water network for a 24 h operational period. The training data generation scheme includes a simulator algorithm based on loop corrective flows equations, a Least Squares (LS) loop flows state estimator and a Confidence Limit Analysis (CLA) algorithm for uncertainty quantification entitled Error Maximization (EM) algorithm. These three numerical algorithms for modeling and simulation of water networks are based on loop corrective flows equations and graph theory. It is shown that the detection of leakages based on the training and testing of the GFMMNN with patterns of variation of nodal consumptions with or without confidence limits produces better recognition rates in comparison to the training based on patterns of nodal heads and pipe flows state estimates with or without confidence limits. It produces also comparable recognition rates to the original recognition system trained with patterns of data obtained with the LS nodal heads state estimator while being computationally superior by requiring a single architecture of the GFMMNN type and using a small number of pattern recognition hyperbox fuzzy sets built by the same GFMMNN architecture. In this case the GFMMNN relies on the ability of the LS loop flows state estimator of making full use of the pressure/nodal heads measurements existent in a water network.     

简单高效耦合策略的粒子群混合算法

建立了评判耦合策略优劣的定量分析方法,发现了现有带中间启动局部搜索(local search,LS)的粒子群混合算法的不足,进而提出一种简单高效的耦合策略.基于该策略,在全局性能优异的综合学习粒子群(comprehensive learning particle swarm optimizer,CLPSO)算法中引入具有快速收敛性能的传统LS方法,提出了带LS的CLPSO混合算法(CLPSO hybrid algorithm with LS,CLPSO-LS).以10维、30维和50维的11个标准函数,对基于不同LS方法的4种混合算法的性能进行大量测试.结果表明,4种CLPSO-LS混合算法的性能均优于CLPSO算法,验证了混合算法的有效性.其中,基于BFGS拟牛顿方法的混合算法的综合性能最优.最后,与8种先进粒子群算法的对比,结果表明CLPSO-LS混合算法作为一种改进CLPSO算法,其性能优于包括已有CLPSO改进算法在内的对比算法,进一步验证了其优越性.     

Training Winner-Take-All Simultaneous Recurrent Neural Networks

The winner-take-all (WTA) network is useful in database management, very large scale integration (VLSI) design, and digital processing. The synthesis procedure of WTA on single-layer fully connected architecture with sigmoid transfer function is still not fully explored. We discuss the use of simultaneous recurrent networks (SRNs) trained by Kalman filter algorithms for the task of finding the maximum among N numbers. The simulation demonstrates the effectiveness of our training approach under conditions of a shared-weight SRN architecture. A more general SRN also succeeds in solving a real classification application on car engine data     

基于自适应多目标进化CNN的图像分割方法

卷积神经网络已经成为强大的分割模型,但通常为手动设计,这需要大量时间并且可能导致庞大而复杂的网络.人们对自动设计能够准确分割特定领域图像的高效网络架构越来越感兴趣,然而大部分方法或者没有考虑构建更加灵活的网络架构,或者没有考虑多个目标优化模型.鉴于此,提出一种称为AdaMo-ECNAS的自适应多目标进化卷积神经架构搜索算法,用于特定领域的图像分割,在进化过程中考虑多个性能指标并通过优化模型的多目标适应特定的数据集. AdaMo-ECNAS可以构建灵活多变的预测分割模型,其网络架构和超参数通过基于多目标进化的算法找到,算法基于自适应PBI实现3个目标进化问题,即提升预测分割的F₁-score、最大限度减少计算成本以及最大限度挖掘额外训练潜能.将AdaMo-ECNAS在两个真实数据集上进行评估,结果表明所提出方法与其他先进算法相比具有较高的竞争性,甚至是超越的.     

Scalable network-on-chip architecture for configurable neural networks

Providing highly flexible connectivity is a major architectural challenge for hardware implementation of reconfigurable neural networks. We perform an analytical evaluation and comparison of different configurable interconnect architectures (mesh NoC, tree, shared bus and point-to-point) emulating variants of two neural network topologies (having full and random configurable connectivity). We derive analytical expressions and asymptotic limits for performance (in terms of bandwidth) and cost (in terms of area and power) of the interconnect architectures considering three communication methods (unicast, multicast and broadcast). It is shown that multicast mesh NoC provides the highest performance/cost ratio and consequently it is the most suitable interconnect architecture for configurable neural network implementation. Routing table size requirements and their impact on scalability were analyzed. Modular hierarchical architecture based on multicast mesh NoC is proposed to allow large scale neural networks emulation. Simulation results successfully validate the analytical models and the asymptotic behavior of the network as a function of its size.     

Mesh-connected trees: a bridge between grids and meshes of trees

The grid and the mesh of trees (or MOT) are among the best-known parallel architectures in the literature. Both of them enjoy efficient VLSI layouts, simplicity of topology, and a large number of parallel algorithms that can efficiently execute on them. One drawback of these architectures is that algorithms that perform best on one of them do not perform very well on the other. Thus there is a gap between the algorithmic capabilities of these two architectures. We propose a new class of parallel architectures, called the mesh-connected trees (or MCT) that can execute grid algorithms as efficiently as the grid, and MOT algorithms as efficiently as the MOT, up to a constant amount of slowdown. In particular, the MCT topology contains the MOT as a subgraph and emulates the grid via embedding with dilation 3 and congestion two. This significant amount of computational versatility offered by the MCT comes at no additional VLSI area cost over these earlier networks. Many topological, routing, and embedding properties analyzed here suggest that the MCT architecture is also a serious competitor for the hypercube. In fact, while the MCT is much simpler and cheaper than the hypercube, for all the algorithms we developed, the running time complexity on the MCT matches those of well known hypercube algorithms. We also present an interesting variant of the MCT architecture that admits both the MOT and the torus as its subgraphs. While most of the discussion in this paper is focused on the MCT architecture itself, these analyses can be easily extended to the variant of the MCT presented here     

Neurocontrol of nonlinear dynamical systems with Kalman filtertrained recurrent networks

Although the potential of the powerful mapping and representational capabilities of recurrent network architectures is generally recognized by the neural network research community, recurrent neural networks have not been widely used for the control of nonlinear dynamical systems, possibly due to the relative ineffectiveness of simple gradient descent training algorithms. Developments in the use of parameter-based extended Kalman filter algorithms for training recurrent networks may provide a mechanism by which these architectures will prove to be of practical value. This paper presents a decoupled extended Kalman filter (DEKF) algorithm for training of recurrent networks with special emphasis on application to control problems. We demonstrate in simulation the application of the DEKF algorithm to a series of example control problems ranging from the well-known cart-pole and bioreactor benchmark problems to an automotive subsystem, engine idle speed control. These simulations suggest that recurrent controller networks trained by Kalman filter methods can combine the traditional features of state-space controllers and observers in a homogeneous architecture for nonlinear dynamical systems, while simultaneously exhibiting less sensitivity than do purely feedforward controller networks to changes in plant parameters and measurement noise.     

Integrating scattering feature maps with convolutional neural networks for Malayalam handwritten character recognition

Convolutional neural network (CNN)-based deep learning architectures are the state-of-the-art in image-based pattern recognition applications. The receptive filter fields in convolutional layers are learned from training data patterns automatically during classifier learning. There are number of well-defined, well-studied and proven filters in the literature that can extract informative content from the input patterns. This paper focuses on utilizing scattering transform-based wavelet filters as the first-layer convolutional filters in CNN architecture. The scattering networks are generated by a series of scattering transform operations. The scattering coefficients generated in first few layers are effective in capturing the dominant energy contained in the input data patterns. The present work aims at replacing the first-layer convolutional feature maps in CNN architecture with scattering feature maps. This architecture is equivalent to utilizing scattering wavelet filters as the first-layer receptive fields in CNN architecture. The proposed hybrid CNN architecture experiments the Malayalam handwritten character recognition which is one of the challenging multi-class classification problems. The initial studies confirm that the proposed hybrid CNN architecture based on scattering feature maps could perform better than the equivalent self-learning architecture of CNN on handwritten character recognition problems.     

On Enforced Convergence of ACO and its Implementation on the Reconfigurable Mesh Architecture Using Size Reduction Tasks

In this paper we show that size reduction tasks can be used for executing iterative randomized metaheuristics on runtime reconfigurable architectures so that an improved throughput and better solution qualities are obtained compared to conventional architectures that do not allow runtime reconfiguration. In particular, the problem of executing ant colony optimization (ACO) algorithms on a dynamically reconfigurable mesh architecture is studied. It is shown how ACO can be implemented such that the convergence behavior of the algorithm can be used to dynamically reduce the size of the submesh that is needed for execution. Furthermore we propose a method to enforce the convergence of ACO leading to a faster reduction process. This increases the throughput of ACO algorithms on runtime reconfigurable meshes. The increased throughput is used for repeated runs of ACO algorithms on a given set of problem instances which significantly improves the obtained solution quality.     

Image Segmentation Using Markov Random Field Model in Fully Parallel Cellular Network Architectures

Markovian approaches to early vision processes need a huge amount of computing power. These algorithms can usually be implemented on parallel computing structures. Herein, we show that the Markovian labeling approach can be implemented in fully parallel cellular network architectures, using simple functions and data representations. This makes possible to implement our model in parallel imaging VLSI chips.As an example, we have developed a simplified statistical image segmentation algorithm for the Cellular Neural/Nonlinear Networks Universal Machine (CNN-UM), which is a new image processing tool, containing thousands of cells with analog dynamics, local memories and processing units. The Modified Metropolis Dynamics (MMD) optimization method can be implemented into the raw analog architecture of the CNN-UM. We can introduce the whole pseudo-stochastic segmentation process in the CNN architecture using 8 memories/cell. We use simple arithmetic functions (addition, multiplication), equality-test between neighboring pixels and very simple nonlinear output functions (step, jigsaw). With this architecture, the proposed VLSI CNN chip can execute a pseudo-stochastic relaxation algorithm of about 100 iterations in about 100 μs.In the suggested solution the segmentation is unsupervised, where a pixel-level statistical estimation model is used. We have tested different monogrid and multigrid architectures.In our CNN-UM model several complex preprocessing steps can be involved, such as texture-classification or anisotropic diffusion. With these preprocessing steps, our fully parallel cellular system may work as a high-level image segmentation machine, using only simple functions based on the close-neighborhood of a pixel.