Analysis of quantization effects on high-order function neural networks

In this paper we investigate the combined effects of quantization and clipping on high-order function neural networks (HOFNN). Statistical models are used to analyze the effects of quantization in a digital implementation. We analyze the performance degradation caused as a function of the number of fixed-point and floating-point quantization bits under the assumption of different probability distributions for the quantized variables, and then compare the training performance between situations with and without weight clipping. We establish and analyze the relationships for a true nonlinear neuron between inputs and outputs bit resolution, training and quantization methods, network order and performance degradation, all based on statistical models, and for on-chip and off-chip training. Our experimental simulation results verify the presented theoretical analysis.

Vector quantization of neural networks

The problem of vector quantizing the parameters of a neural network is addressed, followed by a discussion of different algorithms applicable for quantizer design. Optimal, as well as several suboptimal quantization schemes are described. Simulations involving nonlinear prediction of speech signals are presented to compare the performance of different quantization techniques. Performance evaluation conducted uncover the tradeoffs in implementational complexity. Among the three examined suboptimal quantization schemes, it is shown that the multistage quantizer offers the best tradeoff between complexity and performance.     

Incremental communication for multilayer neural networks

A new method of inter-neuron communication called incremental communication is presented. In the incremental communication method, instead of communicating the whole value of a variable, only the increment or decrement of its previous value is sent on a communication link. The incremental value may be either a fixed-point or a floating-point value. Multilayer feedforward network architecture is used to illustrate the effectiveness of the proposed communication scheme. The method is applied to three different learning problems and the effect of the precision of incremental input-output values of the neurons on the convergence behavior is examined. It is shown through simulation that for some problems even four-bit precision in fixed- and/or floating-point representations is sufficient for the network to converge. With 8-12 bit precisions almost the same results are obtained as that with the conventional communication using 32-bit precision. The proposed method of communication can lead to significant savings in the intercommunication cost for implementations of artificial neural networks on parallel computers as well as the interconnection cost of direct hardware realizations. The method can be incorporated into most of the current learning algorithms in which inter-neuron communications are required. Moreover, it can be used along with the other limited precision strategies for representation of variables suggested in literature.     

Classification error of multilayer perceptron neural networks

In subject classification, artificial neural networks (ANNS) are efficient and objective classification methods. Thus, they have been successfully applied to the numerous classification fields. Sometimes, however, classifications do not match the real world, and are subjected to errors. These problems are caused by the nature of ANNS. We discuss these on multilayer perceptron neural networks. By studying of these problems, it helps us to have a better understanding on its classification.     

Learning in certainty-factor-based multilayer neural networks forclassification

The computational framework of rule-based neural networks inherits from the neural network and the inference engine of an expert system. In one approach, the network activation function is based on the certainty factor (CF) model of MYCIN-like systems. In this paper, it is shown theoretically that the neural network using the CF-based activation function requires relatively small sample sizes for correct generalization. This result is also confirmed by empirical studies in several independent domains.     

Robust local stability of multilayer recurrent neural networks

We derive a condition for robust local stability of multilayer recurrent neural networks with two hidden layers. The stability condition follows from linking theories about linearization, robustness analysis of linear systems under nonlinear perturbation, and matrix inequalities. A characterization of the basin of attraction of the origin is given in terms of the level set of a quadratic Lyapunov function. Similar to the NL(q) theory, the local stability is imposed around the origin and the apparent basin of attraction is made large by applying the criterion, while the proven basin of attraction is relatively small due to conservatism of the criterion. Modification of the dynamic backpropagation by the new stability condition is discussed and illustrated by simulation examples.     

Recursive dynamic node creation in multilayer neural networks

The derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer backpropagation neural networks are presented. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The proposed approach allows optimal dynamic node creation in the sense that the mean-squared error is minimized for each new topology. The effectiveness of the algorithm is demonstrated on several benchmark problems (the multiplexer and the decoder problems) as well as a real world application for detection and classification of buried dielectric anomalies using a microwave sensor.     

Incremental communication for multilayer neural networks: erroranalysis

Artificial neural networks (ANNs) involve a large amount of internode communications. To reduce the communication cost as well as the time of learning process in ANNs, we earlier proposed (1995) an incremental internode communication method. In the incremental communication method, instead of communicating the full magnitude of the output value of a node, only the increment or decrement to its previous value is sent to a communication link. In this paper, the effects of the limited precision incremental communication method on the convergence behavior and performance of multilayer neural networks are investigated. The nonlinear aspects of representing the incremental values with reduced (limited) precision for the commonly used error backpropagation training algorithm are analyzed. It is shown that the nonlinear effect of small perturbations in the input(s)/output of a node does not cause instability. The analysis is supported by simulation studies of two problems. The simulation results demonstrate that the limited precision errors are bounded and do not seriously affect the convergence of multilayer neural networks.     

Clearly defined architectures of neural networks and multilayer perceptron

Neural networks with clearly defined architecture differ in the fact that they make it possible to determine the structure of neural network (number of neurons, layers, connections) on the basis of initial parameters of recognition problem. For these networks, the value of weights determined also analytically. In this paper, we consider the problem of networks with clearly defined architecture transformation into the classical schemes of multilayer perceptron architectures. Such possibility may allow us to combine the advantages of neural networks with clearly defined architecture with the capabilities of multilayer perceptron, that eventually may enable us to speed up and simplify the process of creating and training a neural network.     

Comments on “The effects of quantization on multilayer neuralnetworks” [and reply]

The commenters point out and correct the errors in the derivations of the equations appeared in the above paper by Dundar and Rose (ibid., vol.6 (1995)). In reply, Dundar and Rose agree with their corrections and state that the errors are relatively small and should not affect the conclusions.     

A novel activation function for multilayer feed-forward neural networks

Traditional activation functions such as hyperbolic tangent and logistic sigmoid have seen frequent use historically in artificial neural networks. However, nowadays, in practice, they have fallen out of favor, undoubtedly due to the gap in performance observed in recognition and classification tasks when compared to their well-known counterparts such as rectified linear or maxout. In this paper, we introduce a simple, new type of activation function for multilayer feed-forward architectures. Unlike other approaches where new activation functions have been designed by discarding many of the mainstays of traditional activation function design, our proposed function relies on them and therefore shares most of the properties found in traditional activation functions. Nevertheless, our activation function differs from traditional activation functions on two major points: its asymptote and global extremum. Defining a function which enjoys the property of having a global maximum and minimum, turned out to be critical during our design-process since we believe it is one of the main reasons behind the gap observed in performance between traditional activation functions and their recently introduced counterparts. We evaluate the effectiveness of the proposed activation function on four commonly used datasets, namely, MNIST, CIFAR-10, CIFAR-100, and the Pang and Lee’s movie review. Experimental results demonstrate that the proposed function can effectively be applied across various datasets where our accuracy, given the same network topology, is competitive with the state-of-the-art. In particular, the proposed activation function outperforms the state-of-the-art methods on the MNIST dataset.     

Drill flank wear estimation using supervised vector quantization neural networks

Drill wear detection and prognosis is one of the most important considerations in reducing the cost of rework and scrap and to optimize tool utilization in hole making industry. This study presents the development and implementation of two supervised vector quantization neural networks for estimating the flank-land wear size of a twist drill. The two algorithms are; the learning vector quantization (LVQ) and the fuzzy learning vector quantization (FLVQ). The input features to the neural networks were extracted from the vibration signals using power spectral analysis and continuous wavelet transform techniques. Training and testing were performed under a variety of speeds and feeds in the dry drilling of steel plates. It was found that the FLVQ is more efficient in assessing the flank wear size than the LVQ. The experimental procedure for acquiring vibration data and extracting features in the time-frequency domain using the wavelet transform is detailed. Experimental results demonstrated that the proposed neural network algorithms were effective in estimating the size of the drill flank wear.     

A performance model for multilayer neural networks in linear arrays

An analytical model is presented for assessing the performance of multilayer neural networks implemented in linear arrays. Metrics to assess latency, throughput rate, and computational and input-output bandwidth are developed. These metrics demonstrate a rich and complex interaction between the performance of the hardware and the number and relative dimensions of the layers in a network. Practical illustration of the use of these metrics is demonstrated for a two-hidden-layer network     

Approximation capabilities of multilayer fuzzy neural networks on the set of fuzzy-valued functions

This paper first introduces a piecewise linear interpolation method for fuzzy-valued functions. Based on this, we present a concrete approximation procedure to show the capability of four-layer regular fuzzy neural networks to perform approximation on the set of all d_p continuous fuzzy-valued functions. This approach can also be used to approximate d_∞ continuous fuzzy-valued functions. An example is given to illustrate the approximation procedure.     

Predicting the helpfulness of online reviews using multilayer perceptron neural networks

With the great development of e-commerce, users can create and publish a wealth of product information through electronic communities. It is difficult, however, for manufacturers to discover the best reviews and to determine the true underlying quality of a product due to the sheer volume of reviews available for a single product. The goal of this paper is to develop models for predicting the helpfulness of reviews, providing a tool that finds the most helpful reviews of a given product. This study intends to propose HPNN (a helpfulness prediction model using a neural network), which uses a back-propagation multilayer perceptron neural network (BPN) model to predict the level of review helpfulness using the determinants of product data, the review characteristics, and the textual characteristics of reviews. The prediction accuracy of HPNN was better than that of a linear regression analysis in terms of the mean-squared error. HPNN can suggest better determinants which have a greater effect on the degree of helpfulness. The results of this study will identify helpful online reviews and will effectively assist in the design of review sites.     

Evaluation of the Simplex method for training simple multilayer neural networks

A learning algorithm based on the modified Simplex method is proposed for training multilayer neural networks. This algorithm is tested for neural modelling of experimental results obtained during cross-flow filtration tests. The Simplex method is compared to standard back-propagation. Simpler to implement, Simplex has allowed us to achieve better results over four different databases with lower calculation times. The Simplex algorithm is therefore of interest compared to the classical learning techniques for simple neural structures.     

Analysis of the effects of quantization in multilayer neuralnetworks using a statistical model

A statistical quantization model is used to analyze the effects of quantization when digital techniques are used to implement a real-valued feedforward multilayer neural network. In this process, a parameter called the effective nonlinearity coefficient, which is important in the studying of quantization effects, is introduced. General statistical formulations of the performance degradation of the neural network caused by quantization are developed as functions of the quantization parameters. The formulations predict that the network's performance degradation gets worse when the number of bits is decreased; that a change of the number of hidden units in a layer has no effect on the degradation; that for a constant effective nonlinearity coefficient and number of bits, an increase in the number of layers leads to worse performance degradation; and the number of bits in successive layers can be reduced if the neurons of the lower layer are nonlinear     

Improvise approach for respiratory pathologies classification with multilayer convolutional neural networks

Multimedia Tools and Applications - Every respiratory-related checkup includes audio samples collected from the individual, collected through different tools (sonograph, stethoscope). This audio is...     

Nonlinear blind equalization schemes using complex-valued multilayer feedforward neural networks.

Among the useful blind equalization algorithms, stochastic-gradient iterative equalization schemes are based on minimizing a nonconvex and nonlinear cost function. However, as they use a linear FIR filter with a convex decision region, their residual estimation error is high. In the paper, four nonlinear blind equalization schemes that employ a complex-valued multilayer perceptron instead of the linear filter are proposed and their learning algorithms are derived. After the important properties that a suitable complex-valued activation function must possess are discussed, a new complex-valued activation function is developed for the proposed schemes to deal with QAM signals of any constellation sizes. It has been further proven that by the nonlinear transformation of the proposed function, the correlation coefficient between the real and imaginary parts of input data decreases when they are jointly Gaussian random variables. Last, the effectiveness of the proposed schemes is verified in terms of initial convergence speed and MSE in the steady state. In particular, even without carrier phase tracking procedure, the proposed schemes correct an arbitrary phase rotation caused by channel distortion.     

Machine fusion to enhance the topology preservation of vector quantization artificial neural networks

Artificial neural networks techniques have been successfully applied in vector quantization (VQ) encoding. The objective of VQ is to statistically preserve the topological relationships existing in a data set and to project the data to a lattice of lower dimensions, for visualization, compression, storage, or transmission purposes. However, one of the major drawbacks in the application of artificial neural networks is the difficulty to properly specify the structure of the lattice that best preserves the topology of the data. To overcome this problem, in this paper we introduce merging algorithms for machine-fusion, boosting-fusion-based and hybrid-fusion ensembles of SOM, NG and GSOM networks. In these ensembles not the output signals of the base learners are combined, but their architectures are properly merged. We empirically show the quality and robustness of the topological representation of our proposed algorithm using both synthetic and real benchmarks datasets.