Sensitivity analysis of multilayer perceptron to input and weightperturbations

An important issue in the design and implementation of a neural network is the sensitivity of its output to input and weight perturbations. In this paper, we discuss the sensitivity of the most popular and general feedforward neural networks-multilayer perceptron (MLP). The sensitivity is defined as the mathematical expectation of the output errors of the MLP due to input and weight perturbations with respect to all input and weight values in a given continuous interval. The sensitivity for a single neuron is discussed first and an analytical expression that is a function of the absolute values of input and weight perturbations is approximately derived. Then an algorithm is given to compute the sensitivity for the entire MLP. As intuitively expected, the sensitivity increases with input and weight perturbations, but the increase has an upper bound that is determined by the structural configuration of the MLP, namely the number of neurons per layer and the number of layers. There exists an optimal value for the number of neurons in a layer, which yields the highest sensitivity value. The effect caused by the number of layers is quite unexpected. The sensitivity of a neural network may decrease at first and then almost keeps constant while the number increases.     

Sensitivity analysis of multilayer perceptron with differentiableactivation functions

In a neural network, many different sets of connection weights can approximately realize an input-output mapping. The sensitivity of the neural network varies depending on the set of weights. For the selection of weights with lower sensitivity or for estimating output perturbations in the implementation, it is important to measure the sensitivity for the weights. A sensitivity depending on the weight set in a single-output multilayer perceptron (MLP) with differentiable activation functions is proposed. Formulas are derived to compute the sensitivity arising from additive/multiplicative weight perturbations or input perturbations for a specific input pattern. The concept of sensitivity is extended so that it can be applied to any input patterns. A few sensitivity measures for the multiple output MLP are suggested. For the verification of the validity of the proposed sensitivities, computer simulations have been performed, resulting in good agreement between theoretical and simulation outcomes for small weight perturbations.     

Computation of madalines' sensitivity to input and weight perturbations

The sensitivity of a neural network's output to its input and weight perturbations is an important measure for evaluating the network's performance. In this letter, we propose an approach to quantify the sensitivity of Madalines. The sensitivity is defined as the probability of output deviation due to input and weight perturbations with respect to overall input patterns. Based on the structural characteristics of Madalines, a bottom-up strategy is followed, along which the sensitivity of single neurons, that is, Adalines, is considered first and then the sensitivity of the entire Madaline network. By means of probability theory, an analytical formula is derived for the calculation of Adalines' sensitivity, and an algorithm is designed for the computation of Madalines' sensitivity. Computer simulations are run to verify the effectiveness of the formula and algorithm. The simulation results are in good agreement with the theoretical results.     

Using function approximation to analyze the sensitivity of MLP withantisymmetric squashing activation function

Sensitivity analysis on a neural network is mainly investigated after the network has been designed and trained. Very few have considered this as a critical issue prior to network design. Piche's statistical method (1992, 1995) is useful for multilayer perceptron (MLP) design, but too severe limitations are imposed on both input and weight perturbations. This paper attempts to generalize Piche's method by deriving an universal expression of MLP sensitivity for antisymmetric squashing activation functions, without any restriction on input and output perturbations. Experimental results which are based on, a three-layer MLP with 30 nodes per layer agree closely with our theoretical investigations. The effects of the network design parameters such as the number of layers, the number of neurons per layer, and the chosen activation function are analyzed, and they provide useful information for network design decision-making. Based on the sensitivity analysis of MLP, we present a network design method for a given application to determine the network structure and estimate the permitted weight range for network training.     

Achieving a More Robust Neural Network Model for Control of a MR Damper by Signal Sensitivity Analysis

Most neural network models can work accurately on their trained samples, but when encountering noise, there could be significant errors if the trained neural network is not robust enough to resist the noise. Sensitivity to perturbation in the control signal due to noise is very important for the prediction of an output signal. The goal of this paper is to provide a methodology of signal sensitivity analysis in order to enable the selection of an ideal Multi-Layer Perception (MLP) neural network model from a group of MLP models with different parameters, i.e. to get a highly accurate and robust model for control problems. This paper proposes a signal sensitivity which depends upon the variance of the output error due to noise in the input signals of a single output MLP with differentiable activation functions. On the assumption that noise arises from additive/multiplicative perturbations, the signal sensitivity of the MLP model can be easily calculated, and a method of lowering the sensitivity of the MLP model is proposed. A control system of a magnetorheological (MR) fluid damper, which is a relatively new type of device that shows the future promise for the control of vibration, is modelled by MLP. A large number of simulations on the MR damper’s MLP model show that a much better model is selected using the proposed method.     

A sensitivity-based approach for pruning architecture of Madalines

Architecture design is a very important issue in neural network research. One popular way to find proper size of a network is to prune an oversize trained network to a smaller one while keeping established performance. This paper presents a sensitivity-based approach to prune hidden Adalines from a Madaline with causing as little as possible performance loss and thus easy compensating for the loss. The approach is novel in setting up a relevance measure, by means of an Adalines’ sensitivity measure, to locate the least relevant Adaline in a Madaline. The sensitivity measure is the probability of an Adaline’s output inversions due to input variation with respect to overall input patterns, and the relevance measure is defined as the multiplication of the Adaline’s sensitivity value by the summation of the absolute value of the Adaline’s outgoing weights. Based on the relevance measure, a pruning algorithm can be simply programmed, which iteratively prunes an Adaline with the least relevance value from hidden layer of a given Madaline and then conducts some compensations until no more Adalines can be removed under a given performance requirement. The effectiveness of the pruning approach is verified by some experimental results.     

A hybrid neural network for input that is both categorical and quantitative

The data on which a MLP (multilayer perceptron) is normally trained to approximate a continuous function may include inputs that are categorical in addition to the numeric or quantitative inputs. Examples of categorical variables are gender, race, and so on. An approach examined in this article is to train a hybrid network consisting of a MLP and an encoder with multiple output units; that is, a separate output unit for each of the various combinations of values of the categorical variables. Input to the feed forward subnetwork of the hybrid network is then restricted to truly numerical quantities. A MLP with connection matrices that multiply input values and sigmoid functions that further transform values represents a continuous mapping in all input variables. A MLP therefore requires that all inputs correspond to numeric, continuously valued variables and represents a continuous function in all input variables. A categorical variable, on the other hand, produces a discontinuous relationship between an input variable and the output. The way that this problem is often dealt with is to replace the categorical values by numeric ones and treat them as if they were continuously valued. However there is no meaningful correspondence between the continuous quantities generated this way and the original categorical values. The basic difficulty with using these variables is that they define a metric for the categories that may not be reasonable. This suggests that the categorical inputs should be segregated from the continuous inputs as explained above. Results show that the method utilizing a hybrid network and separating numerical from quantitative input, as discussed here, is quite effective. © 2004 Wiley Periodicals, Inc. Int J Int Syst 19: 979–1001, 2004.     

Efficient detection of spurious inputs for improving the robustness of MLP networks in practical applications

The problem of the rejection of patterns not belonging to identified training classes is investigated with respect to Multilayer Perceptron Networks (MLP). The reason for the inherent unreliability of the standard MLP in this respect is explained, and some mechanisms for the enhancement of its rejection performance are considered. Two network configurations are presented as candidates for a more reliable structure, and are compared to the so-called negative training approach. The first configuration is an MLP which uses a Gaussian as its activation function, and the second is an MLP with direct connections from the input to the output layer of the network. The networks are examined and evaluated both through the technique of network inversion, and through practical experiments in a pattern classification application. Finally, the model of Radial Basis Function (RBF) networks is also considered in this respect, and its performance is compared to that obtained with the other networks described.     

Conventional modeling of the multilayer perceptron using polynomialbasis functions

A technique for modeling the multilayer perceptron (MLP) neural network, in which input and hidden units are represented by polynomial basis functions (PBFs), is presented. The MLP output is expressed as a linear combination of the PBFs and can therefore be expressed as a polynomial function of its inputs. Thus, the MLP is isomorphic to conventional polynomial discriminant classifiers or Volterra filters. The modeling technique was successfully applied to several trained MLP networks.     

Use of some sensitivity criteria for choosing networks with good generalization ability

In most applications of the multilayer perceptron (MLP) the main objective is to maximize the generalization ability of the network. We show that this ability is related to the sensitivity of the output of the MLP to small input changes. Several criteria have been proposed for the evaluation of the sensitivity. We propose a new index and present a way for improving these sensitivity criteria. Some numerical experiments allow a first comparison of the efficiencies of these criteria.     

Two-Phase Construction of Multilayer Perceptrons Using Information Theory

This brief presents a two-phase construction approach for pruning both input and hidden units of multilayer perceptrons (MLPs) based on mutual information (MI). First, all features of input vectors are ranked according to their relevance to target outputs through a forward strategy. The salient input units of an MLP are thus determined according to the order of the ranking result and by considering their contributions to the network's performance. Then, the irrelevant features of input vectors can be identified and eliminated. Second, the redundant hidden units are removed from the trained MLP one after another according to a novel relevance measure. Compared with its related work, the proposed strategy exhibits better performance. Moreover, experimental results show that the proposed method is comparable or even superior to support vector machine (SVM) and support vector regression (SVR). Finally, the advantages of the MI-based method are investigated in comparison with the sensitivity analysis (SA)-based method.     

Determination of Aircraft Orientation for a Vision-Based System Using Artificial Neural Networks

An algorithm for real-time estimation of 3-D orientation of an aircraft, given its monocular, binary image from an arbitrary viewing direction is presented. This being an inverse problem, we attempt to provide an approximate but a fast solution using the artificial neural network technique. A set of spatial moments (scale, translation, and planar rotation invariant) is used as features to characterize different views of the aircraft, which corresponds to the feature space representation of the aircraft. A new neural network topology is suggested in order to solve the resulting functional approximation problem for the input (feature vector)-output (viewing direction) relationship. The feature space is partitioned into a number of subsets using a Kohonen clustering algorithm to express the complex relationship into a number of simpler ones. Separate multi-layer perceptrons (MLP) are then trained to capture the functional relations that exist between each class of feature vectors and the corresponding target orientation. This approach is shown to give better results when compared to those obtained with a single MLP trained for the entire feature space.     

An inference network for bidirectional approximate reasoning basedon an equality measure

An inference network is proposed as a tool for bidirectional approximate reasoning. The inference network can be designed directly from the given fuzzy data (knowledge). If a fuzzy input is given for the inference network, then the network renders a reasonable fuzzy output after performing approximate reasoning based on an equality measure. Conversely, due to the bidirectional structure, the network can yield its corresponding reasonable fuzzy input for a given fuzzy output. This property makes it possible to perform forward and backward reasoning in the knowledge base system     

Application of neural networks to detection of medical fraud

A multi-layer perceptron (MLP) network was trained to classify the practice profiles of a sample of medical general practitioners who had been classified by expert consultants into four classes ranging from having normal to having abnormal profiles. This method follows the two-class neural network classification of medical practice profiles developed at the Health Insurance Commission in 1990. A technique based on the probabilistic interpretation of the output of the neural network was used to see if it improved the performance of the MLP given the extent of noise (i.e. inconsistencies) in the experts' classifications. Kohonen's Self-Organising Map was also applied to analyse the consultants' classifications and it was found that an approach which combined the four classes into two was a more appropriate way to represent the classification data. The MLP network was then retrained using a two-class classification and a high agreement rate between the classifications of the MLP and the classifications of consultants was achieved.     

Applying undistorted neural network sensitivity analysis in iris plant classification and construction productivity prediction

The present research focuses on the development and applications of a sensitivity analysis technique on multi-layer perceptron (MLP) neural networks (NN), which eliminates distortions on the sensitivity measures due to dissimilar input ranges with different units of measure for input features of both continuous and symbolic types in NNs practical engineering applications. The effect of randomly splitting the dataset into training and testing sets on the stability of a MLP networks sensitivity is also observed and discussed. The IRIS-UCI dataset and a real concreting productivity dataset serve as case studies to illustrate the validity of the undistorted sensitivity measure proposed. The results of the two case studies lead to the conclusion that the sensitivity measures accounting for the relevant input range for each input feature are more accurate and effective for revealing the relevance of each input feature and identifying less significant ones for potential feature reduction on the model. The MLP NN model obtained in such a way can give not only high prediction accuracy, but also valid sensitivity measures on its input features, and hence can be deployed as a predictive tool for supporting the decision process on new scenarios within the engineering problem domain.     

Multilayer perceptrons: approximation order and necessary number of hidden units.

This paper considers the approximation of sufficiently smooth multivariable functions with a multilayer perceptron (MLP). For a given approximation order, explicit formulas for the necessary number of hidden units and its distributions to the hidden layers of the MLP are derived. These formulas depend only on the number of input variables and on the desired approximation order. The concept of approximation order encompasses Kolmogorov-Gabor polynomials or discrete Volterra series, which are widely used in static and dynamic models of nonlinear systems. The results are obtained by considering structural properties of the Taylor polynomials of the function in question and of the MLP function.     

A channel equalizer using reduced decision feedback Chebyshev functional link artificial neural networks

In this paper, a reduced decision feedback Chebyshev functional link artificial neural network (RDF-CFLANN) is proposed for the design of a nonlinear channel equalizer. An RDF-CFLANN structure uses functional expansion utilities to nonlinearly transform its input signals into the output space. In most MLP structures, one or more hidden layers are needed to nonlinearly map the input signals to the output signal space. Therefore, the complexity of the RDF-CFLANN structure is generally much lower than that of an MLP structure. In addition, the required amount of computing at the training mode can also be reduced. The comparisons of the mean squared error (MSE) and the average transmission bit error rate (BER) among RDF-CFLANN, DF-CFLANN and CFLANN are presented in this paper. Simulation results demonstrate that RDF-CFLANN presents the best performance among the three structures.     

An analytical framework for local feedforward networks

Interference in neural networks occurs when learning in one area of the input space causes unlearning in another area. Networks that are less susceptible to interference are referred to as spatially local networks. To obtain a better understanding of these properties, a theoretical framework, consisting of a measure of interference and a measure of network localization, is developed. These measures incorporate not only the network weights and architecture but also the learning algorithm. Using this framework to analyze sigmoidal, multilayer perceptron (MLP) networks that employ the backpropagation learning algorithm on the quadratic cost function, we address a familiar misconception that single-hidden-layer sigmoidal networks are inherently nonlocal by demonstrating that given a sufficiently large number of adjustable weights, single-hidden-layer sigmoidal MLPs exist that are arbitrarily local and retain the ability to approximate any continuous function on a compact domain.     

An adaptive neuro-fuzzy system for automatic image segmentation andedge detection

An autoadaptive neuro-fuzzy segmentation and edge detection architecture is presented. The system consists of a multilayer perceptron (MLP)-like network that performs image segmentation by adaptive thresholding of the input image using labels automatically pre-selected by a fuzzy clustering technique. The proposed architecture is feedforward, but unlike the conventional MLP the learning is unsupervised. The output status of the network is described as a fuzzy set. Fuzzy entropy is used as a measure of the error of the segmentation system as well as a criterion for determining potential edge pixels. The proposed system is capable to perform automatic multilevel segmentation of images, based solely on information contained by the image itself. No a priori assumptions whatsoever are made about the image (type, features, contents, stochastic model, etc.). Such an "universal" algorithm is most useful for applications that are supposed to work with different (and possibly initially unknown) types of images. The proposed system can be readily employed, "as is," or as a basic building block by a more sophisticated and/or application-specific image segmentation algorithm. By monitoring the fuzzy entropy relaxation process, the system is able to detect edge pixels     

Spiking神经元输入脉冲扰动敏感性研究

脉冲神经网络是一种基于生物的网络模型,它的输入输出为具有时间特性的脉冲序列,其运行机制相比其他传统人工神经网络更加接近于生物神经网络。神经元之间通过脉冲序列传递信息,这些信息通过脉冲的激发时间编码能够更有效地发挥网络的学习性能。脉冲神经元的时间特性导致了其工作机制较为复杂,而spiking神经元的敏感性反映了当神经元输入发生扰动时输出的spike的变化情况,可以作为研究神经元内部工作机制的工具。不同于传统的神经网络,spiking神经元敏感性定义为输出脉冲的变化时刻个数与运行时间长度的比值,能直接反映出输入扰动对输出的影响程度。通过对不同形式的输入扰动敏感性的分析,可以看出spiking神经元的敏感性较为复杂,当全体突触发生扰动时,神经元为定值,而当部分突触发生扰动时,不同突触的扰动会导致不同大小的神经元敏感性。