Parallel nonlinear optimization techniques for training neural networks

In this paper, we propose the use of parallel quasi-Newton (QN) optimization techniques to improve the rate of convergence of the training process for neural networks. The parallel algorithms are developed by using the self-scaling quasi-Newton (SSQN) methods. At the beginning of each iteration, a set of parallel search directions is generated. Each of these directions is selectively chosen from a representative class of QN methods. Inexact line searches are then carried out to estimate the minimum point along each search direction. The proposed parallel algorithms are tested over a set of nine benchmark problems. Computational results show that the proposed algorithms outperform other existing methods, which are evaluated over the same set of test problems.     

Local cluster neural net: Architecture, training and applications

This paper describes the structure, training and computational abilities of the local cluster (LC) artificial neural net architecture. LC nets are a special class of multilayer perceptrons that use sigmoid functions to generate localised functions. LC nets train as fast as radial basis functions nets and are more general. They are well suited for both, multi-dimensional function approximation and discrete classification. The LC net is the result of our search for a widely applicable neural net architecture suitable for low-cost hardware realisation. The LC net seem particularly well suited for analog VLSI realisation of small-size, low-power, fully parallel neural net chip for real time control applications.     

Parallel growing and training of neural networks using outputparallelism

In order to find an appropriate architecture for a large-scale real-world application automatically and efficiently, a natural method is to divide the original problem into a set of subproblems. In this paper, we propose a simple neural-network task decomposition method based on output parallelism. By using this method, a problem can be divided flexibly into several subproblems as chosen, each of which is composed of the whole input vector and a fraction of the output vector. Each module (for one subproblem) is responsible for producing a fraction of the output vector of the original problem. The hidden structure for the original problem's output units are decoupled. These modules can be grown and trained in parallel on parallel processing elements. Incorporated with a constructive learning algorithm, our method does not require excessive computation and any prior knowledge concerning decomposition. The feasibility of output parallelism is analyzed and proved. Some benchmarks are implemented to test the validity of this method. Their results show that this method can reduce computational time, increase learning speed and improve generalization accuracy for both classification and regression problems.     

Parallel evolutionary training algorithms for “hardware-friendly” neural networks

In this paper, Parallel Evolutionary Algorithms for integer weightneural network training are presented. To this end, each processoris assigned a subpopulation of potential solutions. Thesubpopulations are independently evolved in parallel andoccasional migration is employed to allow cooperation betweenthem. The proposed algorithms are applied to train neural networksusing threshold activation functions and weight values confined toa narrow band of integers. We constrain the weights and biases inthe range [–3, 3], thus they can be represented by just 3 bits.Such neural networks are better suited for hardware implementationthan the real weight ones. These algorithms have been designedkeeping in mind that the resulting integer weights require lessbits to be stored and the digital arithmetic operations betweenthem are easier to be implemented in hardware. Another advantageof the proposed evolutionary strategies is that they are capableof continuing the training process ``on-chip', if needed. Ourintention is to present results of parallel evolutionaryalgorithms on this difficult task. Based on the application of theproposed class of methods on classical neural network problems,our experience is that these methods are effective and reliable.     

A fast neural net training algorithm and its application to speech classification

This paper describes a fast training algorithm for feedforward neural nets, as applied to a two-layer neural network to classify segments of speech as voiced, unvoiced, or silence. The speech classification method is based on five features computed for each speech segment and used as input to the network. The network weights are trained using a new fast training algorithm which minimizes the total least squares error between the actual output of the network and the corresponding desired output. The iterative training algorithm uses a quasi-Newtonian error-minimization method and employs a positive-definite approximation of the Hessian matrix to quickly converge to a locally optimal set of weights. Convergence is fast, with a local minimum typically reached within ten iterations; in terms of convergence speed, the algorithm compares favorably with other training techniques. When used for voiced-unvoiced-silence classification of speech frames, the network performance compares favorably with current approaches. Moreover, the approach used has the advantage of requiring no assumption of a particular probability distribution for the input features.     

Parallel memetic algorithm for training recurrent neural networks for the energy efficiency problem

In our state-of-the-art study, we improve neural network-based models for predicting energy consumption in buildings by parallelizing the CHC adaptive search algorithm. We compared the sequential implementation of the evolutionary algorithm with the new parallel version to obtain predictors and found that this new version of our software tool halved the execution time of the sequential version. New predictors based on various classes of neural networks have been developed and the obtained results support the validity of the proposed approaches with an average improvement of 75% of the average execution time in relation to previous sequential implementations.     

基于PVM的微机网络并行计算及其应用

文章叙述了在微机网络条件下通过 ＬＩＮＵＸ＋ＰＶＭ构造并行处理环境的途径和方法,通过实例探讨了微机网络并行计算中一些技术问题及解决办法,最后给出了微机网络并行计算需要进一步研究的课题。     

k-winners-take-all neural net with Theta(1) time complexity.

In this article we present a k-winners-take-all (k-WTA) neural net that is established based on the concept of the constant time sorting machine by Hsu and Wang. It fits some specific applications, such as real-time processing, since its Theta(1) time complexity is independent to the problem size. The proposed k-WTA neural net produces the solution in constant time while the Hopfield network requires a relatively long transient to converge to the solution from some initial states.     

Parallel consensual neural networks

A new type of a neural-network architecture, the parallel consensual neural network (PCNN), is introduced and applied in classification/data fusion of multisource remote sensing and geographic data. The PCNN architecture is based on statistical consensus theory and involves using stage neural networks with transformed input data. The input data are transformed several times and the different transformed data are used as if they were independent inputs. The independent inputs are first classified using the stage neural networks. The output responses from the stage networks are then weighted and combined to make a consensual decision. In this paper, optimization methods are used in order to weight the outputs from the stage networks. Two approaches are proposed to compute the data transforms for the PCNN, one for binary data and another for analog data. The analog approach uses wavelet packets. The experimental results obtained with the proposed approach show that the PCNN outperforms both a conjugate-gradient backpropagation neural network and conventional statistical methods in terms of overall classification accuracy of test data.     

基于Petri网建模的并行粒子群算法

粒子群优化算法,起源于鸟群行为的研究,是一种基于群智能的进化计算技术,通过粒子之间的协作与竞争以实现对多维复杂空间的高效搜索。提出了基于Petri网的并行粒子群算法,并采用经典测试函数验证算法的有效性。测试结果表明,算法能很好地控制粒子群优化过程中的早熟问题,并能够较好地得到群落全局最优解。     

Strategies for improving neural net generalisation

We address the problem of training multilayer perceptrons to instantiate a target function. In particular, we explore the accuracy of the trained network on a test set of previously unseen patterns — the generalisation ability of the trained network. We systematically evaluate alternative strategies designed to improve the generalisation performance. The basic idea is to generate a diverse set of networks, each of which is designed to be an implementation of the target function. We then have a set of trained, alternative versions — a version set. The goal is to achieve useful diversity within this set, and thus generate potential for improved generalisation performance of the set as a wholewhen compared to the performance of any individual version. We define this notion of useful diversity, we define a metric for it, we explore a number of ways of generating it, and we present the results of an empirical study of a number of strategies for exploiting it to achieve maximum generalisation performance. The strategies encompass statistical measures as well as a selectornet approach which proves to be particularly promising. The selector net is a form of metanet that operates in conjunction with a version set.     

A framework for neural net specification

A notation for the specification of neural nets is proposed. The aim is to produce a simple mathematical framework for use in specifying neural nets essentially by defining their transfer functions and connections. Nets are specified as interacting processing elements (nodes), communicating via instant links. Dynamics and adaptation are defined at the processing elements themselves, and all interaction is explicitly specified by directed arcs. Specifications can be built up hierarchically by turning a specification into a generator for a node, or they can be developed top-down. The use of the system is illustrated     

基于神经网络的注塑模KBE系统开发

针对现有注塑模具设计CAD系统存在的缺点,提出基于神经网络的注塑模KBE系统开发思想。阐述了模具设计的特点、KBE的内涵和特点以及神经网络的原理和在模具设计KBE上的应用,设计了基于神经网络的摩托车塑件模具设计KBE的基本系统框架,并分配了模具设计KBE系统中神经网络算法的榆入、输出以及隐层各神经元,为注塑模具设计KBE提供了系统构造基础研究。     

FPN并行反向推理算法研究

通过对Petri网模型和专家系统推理方法的研究,建立了模糊Petri网（FPN）推理模型。在此基础上提出了专家系统的FPN反向推理算法。最后通过实例对算法进行了检验,结果表明该算法具有解决复杂问题专家系统的并行推理能力,推理效率高,推理过程简单,容易实现。     

Multilevel neural net adaptive models using the metagraph approach

The paper considers adaptive models enabling real-time processing of data flows. The drawbacks of current algorithms are examined. A method that combines advantages of deep learning, self-organizing neural nets and the metagraph approach is offered for designing adaptive models. A part of the method is realized, data clustering experiments are carried out and experimental results are analyzed.     

Artificial neural net applications in telecommunication systems

The artificial neural net development has had something of a renaissance in the last decade with an impressive range of application areas. From the viewpoint of telecommunication networks and systems, an increasing number of studies can be observed in recent literature dealing with proposed applications of neural nets in telecommunication environments, such as connection admission control in broadband networks, the control of high-speed interconnection networks, channel allocation in cellular mobile systems, adaptive routing, etc. These proposed applications largely use three main neural net classes: feed-forward nets with backpropagation learning, Hopfield feedback nets, and selforganising neural nets. In this paper, we first give an overview of neural net classes and their main properties, and then present a review of applications in telecommunication systems, where attention is devoted to numerical aspects such as the convergence property and learning speed of the proposed neural nets.     

An improved Voronoi-diagram-based neural net for patternclassification

We propose a novel two-layer neural network to answer a point query in R(n) which is partitioned into polyhedral regions; such a task solves among others nearest neighbor clustering. As in previous approaches to the problem, our design is based on the use of Voronoi diagrams. However, our approach results in substantial reduction of the number of neurons, completely eliminating the second layer, at the price of requiring only two additional clock steps. In addition, the design process is also simplified while retaining the main advantage of the approach, namely its ability to furnish precise values for the number of neurons and the connection weights necessitating neither trial and error type iterations nor ad hoc parameters.     

基于粗糙集的容错神经网络故障诊断系统

粗糙集和神经网络在故障诊断中都得到了广泛的应用,但两者都有其局限性,同时在许多方面有其互补性,融合粗糙集和神经网络各自的优势,建立了粗糙集——客错神经网络故障诊断系统。利用粗糙集对原始数据进行简约,导出最简诊断规则,根据选择的冗余约简和最简诊断规则建立粗糙集——容错神经网络故障诊断系统。以滚动轴承故障诊断为例,仿真结果表明系统提高了故障诊断准确率和诊断速度,消除了故障诊断中的误报和漏报现象。     

时间Petri网与GA-PSO算法相结合的并行测试

摘 要：并行测试任务调度方案在自动测试系统中一直是尚未解决的难题。本文基于Petri网理论的基础,建立了并行测试的时间Petri网模型,并且首次将GA-PSO算法引入到时间Petri网的变迁序列的寻找过程中,快速地求得了最优调度方案。仿真结果表明,本算法能够以较大的收敛概率快速地收敛,最终得到最优变迁序列。