Incremental communication for multilayer neural networks: erroranalysis

Artificial neural networks (ANNs) involve a large amount of internode communications. To reduce the communication cost as well as the time of learning process in ANNs, we earlier proposed (1995) an incremental internode communication method. In the incremental communication method, instead of communicating the full magnitude of the output value of a node, only the increment or decrement to its previous value is sent to a communication link. In this paper, the effects of the limited precision incremental communication method on the convergence behavior and performance of multilayer neural networks are investigated. The nonlinear aspects of representing the incremental values with reduced (limited) precision for the commonly used error backpropagation training algorithm are analyzed. It is shown that the nonlinear effect of small perturbations in the input(s)/output of a node does not cause instability. The analysis is supported by simulation studies of two problems. The simulation results demonstrate that the limited precision errors are bounded and do not seriously affect the convergence of multilayer neural networks.     

Incremental communication for adaptive resonance theory networks

We have proposed earlier the incremental internode communication method to reduce the communication cost as well as the time of the learning process in artificial neural networks (ANNs). In this paper, the limited precision incremental communication method is applied to a class of recurrent neural networks, the adaptive resonance theory 2 (ART2) networks. Simulation studies are carried out to examine the effects of the incremental communication method on the convergence behavior of ART2 networks. We have found that 7-13-b precision is sufficient to obtain almost the same results as those with full (32-b) precision conventional communication. A theoretical error analysis is also carried out to analyze the effects of the limited precision incremental communication. The simulation and analytical results show that the limited precision errors are bounded and do not seriously degrade the convergence of ART2 networks. Therefore, the incremental communication can be incorporated in parallel and special-purpose very large scale integration (VLSI) implementations of the ART2 networks.     

Incremental growth in modular neural networks

This paper outlines an algorithm for incrementally growing Artificial Neural Networks. The algorithm allows the network to expand by adding new sub-networks or modules to an existing structure; the modules are trained using an Evolutionary Algorithm. Only the latest module added to the network is trained, the previous structure remains fixed. The algorithm allows information from different data domains to be integrated into the network and because the search space in each iteration is small, large and complex networks with a modular structure can emerge naturally. The paper describes an application of the algorithm to a legged robot and discusses its biological inspiration.     

A novel activation function for multilayer feed-forward neural networks

Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee’s movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.     

Classification error of multilayer perceptron neural networks

In subject classification, artificial neural networks (ANNS) are efficient and objective classification methods. Thus, they have been successfully applied to the numerous classification fields. Sometimes, however, classifications do not match the real world, and are subjected to errors. These problems are caused by the nature of ANNS. We discuss these on multilayer perceptron neural networks. By studying of these problems, it helps us to have a better understanding on its classification.     

Learning in certainty-factor-based multilayer neural networks forclassification

The computational framework of rule-based neural networks inherits from the neural network and the inference engine of an expert system. In one approach, the network activation function is based on the certainty factor (CF) model of MYCIN-like systems. In this paper, it is shown theoretically that the neural network using the CF-based activation function requires relatively small sample sizes for correct generalization. This result is also confirmed by empirical studies in several independent domains.     

A performance model for multilayer neural networks in linear arrays

An analytical model is presented for assessing the performance of multilayer neural networks implemented in linear arrays. Metrics to assess latency, throughput rate, and computational and input-output bandwidth are developed. These metrics demonstrate a rich and complex interaction between the performance of the hardware and the number and relative dimensions of the layers in a network. Practical illustration of the use of these metrics is demonstrated for a two-hidden-layer network     

The effects of quantization on multilayer neural networks

The effect of weight quantization in multilayer neural networks is discussed. A method is derived by which one can predict the performance degradation at the output given the properties of the network and number of bits of quantization. Predictions from this method are evaluated against simulation results. An algorithm to decrease the noise at the output is presented and the results are compared with those above.     

Robust local stability of multilayer recurrent neural networks

We derive a condition for robust local stability of multilayer recurrent neural networks with two hidden layers. The stability condition follows from linking theories about linearization, robustness analysis of linear systems under nonlinear perturbation, and matrix inequalities. A characterization of the basin of attraction of the origin is given in terms of the level set of a quadratic Lyapunov function. Similar to the NL(q) theory, the local stability is imposed around the origin and the apparent basin of attraction is made large by applying the criterion, while the proven basin of attraction is relatively small due to conservatism of the criterion. Modification of the dynamic backpropagation by the new stability condition is discussed and illustrated by simulation examples.     

Using recurrent neural networks for adaptive communication channelequalization

Nonlinear adaptive filters based on a variety of neural network models have been used successfully for system identification and noise-cancellation in a wide class of applications. An important problem in data communications is that of channel equalization, i.e., the removal of interferences introduced by linear or nonlinear message corrupting mechanisms, so that the originally transmitted symbols can be recovered correctly at the receiver. In this paper we introduce an adaptive recurrent neural network (RNN) based equalizer whose small size and high performance makes it suitable for high-speed channel equalization. We propose RNN based structures for both trained adaptation and blind equalization, and we evaluate their performance via extensive simulations for a variety of signal modulations and communication channel models. It is shown that the RNN equalizers have comparable performance with traditional linear filter based equalizers when the channel interferences are relatively mild, and that they outperform them by several orders of magnitude when either the channel's transfer function has spectral nulls or severe nonlinear distortion is present. In addition, the small-size RNN equalizers, being essentially generalized IIR filters, are shown to outperform multilayer perceptron equalizers of larger computational complexity in linear and nonlinear channel equalization cases.     

Recursive dynamic node creation in multilayer neural networks

The derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer backpropagation neural networks are presented. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The proposed approach allows optimal dynamic node creation in the sense that the mean-squared error is minimized for each new topology. The effectiveness of the algorithm is demonstrated on several benchmark problems (the multiplexer and the decoder problems) as well as a real world application for detection and classification of buried dielectric anomalies using a microwave sensor.     

基于卷积计算的多层脉冲神经网络的监督学习

针对脉冲神经元基于精确定时的多脉冲编码信息的特点,提出了一种基于卷积计算的多层脉冲神经网络监督学习的新算法。该算法应用核函数的卷积计算将离散的脉冲序列转换为连续函数,在多层前馈脉冲神经网络结构中,使用梯度下降的方法得到基于核函数卷积表示的学习规则,并用来调整神经元连接的突触权值。在实验部分,首先验证了该算法学习脉冲序列的效果,然后应用该算法对Iris数据集进行分类。结果显示,该算法能够实现脉冲序列复杂时空模式的学习,对非线性模式分类问题具有较高的分类正确率。     

Evaluation of the Simplex method for training simple multilayer neural networks

A learning algorithm based on the modified Simplex method is proposed for training multilayer neural networks. This algorithm is tested for neural modelling of experimental results obtained during cross-flow filtration tests. The Simplex method is compared to standard back-propagation. Simpler to implement, Simplex has allowed us to achieve better results over four different databases with lower calculation times. The Simplex algorithm is therefore of interest compared to the classical learning techniques for simple neural structures.     

Improvise approach for respiratory pathologies classification with multilayer convolutional neural networks

Multimedia Tools and Applications - Every respiratory-related checkup includes audio samples collected from the individual, collected through different tools (sonograph, stethoscope). This audio is...     

Clearly defined architectures of neural networks and multilayer perceptron

Neural networks with clearly defined architecture differ in the fact that they make it possible to determine the structure of neural network (number of neurons, layers, connections) on the basis of initial parameters of recognition problem. For these networks, the value of weights determined also analytically. In this paper, we consider the problem of networks with clearly defined architecture transformation into the classical schemes of multilayer perceptron architectures. Such possibility may allow us to combine the advantages of neural networks with clearly defined architecture with the capabilities of multilayer perceptron, that eventually may enable us to speed up and simplify the process of creating and training a neural network.     

A 12-transistor PFM demodulator for analog neural networks communication

Pulse frequency modulation (PFM) provides robust long-distance communication and event-driven access to the communication channel. A PFM demodulator small and simple enough to equip every cell of a large analog neural network is analyzed. The circuit can demodulate PFM signals with pulse rates as low as 1 kHz using the switched-capacitor technique. Measurement results from integrated demodulators are presented.     

A successive projection method for binary pattern recognition with multilayer feedforward neural networks

Error back-propagation (BP) is one of the most popular ideas used in learning algorithms for multilayer neural networks. In BP algorithms, there are two types of learning schemes, online learning and batch learning. The online BP has been applied to various problems in practice, because of its simplicity of implementation. However, efficient implementation of the online BP usually requires an ad hoc rule for determining the learning rate of the algorithm. In this paper, we propose a new learning algorithm called SPM, which is derived from the successive projection method for solving a system of nonlinear inequalities. Although SPM can be regarded as a modification of online BP, the former algorithm determines the learning rate (step-size) adoptively based on the output for each input pattern. SPM may also be considered a modification of the globally guided back-propagation (GGBP) proposed by Tang and Koehler. Although no theoretical proof of the convergence for SPM is given, some simulation results on pattern classification problems indicate that SPM is more effective and robust than the standard online BP and GGBP     

Routing in broadband communication networks using neural computations

This paper will briefly review some of the published work on applying Artificial Neural Networks (ANN) to routing in Asynchronous Transfer Mode (ATM) networks and critically appraise some of the conclusions which have been drawn from these studies. The paper will go on to explain the design and implementation of a simulation of a neural network array superimposed on a simulated ATM network. The results of routing experiments, which were compared with decisions made using Shortest Path First (SPF) routing, will be discussed and the need for appropriate comparison metrics explained. The paper will end with a review of what needs to be considered when the applicability of ANNs to ATM network control is being assessed and the required functionality of simulation tools which allow these assessments to be made.     

Predicting the helpfulness of online reviews using multilayer perceptron neural networks

With the great development of e-commerce, users can create and publish a wealth of product information through electronic communities. It is difficult, however, for manufacturers to discover the best reviews and to determine the true underlying quality of a product due to the sheer volume of reviews available for a single product. The goal of this paper is to develop models for predicting the helpfulness of reviews, providing a tool that finds the most helpful reviews of a given product. This study intends to propose HPNN (a helpfulness prediction model using a neural network), which uses a back-propagation multilayer perceptron neural network (BPN) model to predict the level of review helpfulness using the determinants of product data, the review characteristics, and the textual characteristics of reviews. The prediction accuracy of HPNN was better than that of a linear regression analysis in terms of the mean-squared error. HPNN can suggest better determinants which have a greater effect on the degree of helpfulness. The results of this study will identify helpful online reviews and will effectively assist in the design of review sites.     

Nonlinear blind equalization schemes using complex-valued multilayer feedforward neural networks.

Among the useful blind equalization algorithms, stochastic-gradient iterative equalization schemes are based on minimizing a nonconvex and nonlinear cost function. However, as they use a linear FIR filter with a convex decision region, their residual estimation error is high. In the paper, four nonlinear blind equalization schemes that employ a complex-valued multilayer perceptron instead of the linear filter are proposed and their learning algorithms are derived. After the important properties that a suitable complex-valued activation function must possess are discussed, a new complex-valued activation function is developed for the proposed schemes to deal with QAM signals of any constellation sizes. It has been further proven that by the nonlinear transformation of the proposed function, the correlation coefficient between the real and imaginary parts of input data decreases when they are jointly Gaussian random variables. Last, the effectiveness of the proposed schemes is verified in terms of initial convergence speed and MSE in the steady state. In particular, even without carrier phase tracking procedure, the proposed schemes correct an arbitrary phase rotation caused by channel distortion.