A cooperative constructive method for neural networks for pattern recognition

In this paper, we propose a new constructive method, based on cooperative coevolution, for designing automatically the structure of a neural network for classification. Our approach is based on a modular construction of the neural network by means of a cooperative evolutionary process. This process benefits from the advantages of coevolutionary computation as well as the advantages of constructive methods. The proposed methodology can be easily extended to work with almost any kind of classifier.The evaluation of each module that constitutes the network is made using a multiobjective method. So, each new module can be evaluated in a comprehensive way, considering different aspects, such as performance, complexity, or degree of cooperation with the previous modules of the network. In this way, the method has the advantage of considering not only the performance of the networks, but also other features.The method is tested on 40 classification problems from the UCI machine learning repository with very good performance. The method is thoroughly compared with two other constructive methods, cascade correlation and GMDH networks, and other classification methods, namely, SVM, C4.5, and k nearest-neighbours, and an ensemble of neural networks constructed using four different methods.     

Node splitting: A constructive algorithm for feed-forward neural networks

A constructive algorithm is proposed for feed-forward neural networks which uses node-splitting in the hidden layers to build large networks from smaller ones. The small network forms an approximate model of a set of training data, and the split creates a larger, more powerful network which is initialised with the approximate solution already found. The insufficiency of the smaller network in modelling the system which generated the data leads to oscillation in those hidden nodes whose weight vectors cover regions in the input space where more detail is required in the model. These nodes are identified and split in two using principal component analysis, allowing the new nodes to cover the two main modes of the oscillating vector. Nodes are selected for splitting using principal component analysis on the oscillating weight vectors, or by examining the Hessian matrix of second derivatives of the network error with respect to the weights.     

A PDP constructive algorithm for system modelling

A new constructive algorithm is presented for building neural networks that learn to reproduce output temporal sequences based on one or several input sequences. This algorithm builds a network for the task of system modelling, dealing with continuous variables in the discrete time domain. The constructive scheme makes it user independent. The network's structure consists of an ordinary set and a classification set, so it is a hybrid network like that of Stokbro et al. [6], but with a binary classification. The networks can easily be interpreted, so the learned representation can be transferred to a human engineer, unlike many other network models. This allows for a better understanding of the system structure than just its simulation. This constructive algorithm limits the network complexity automatically, hence preserving extrapolation capabilities. Examples with real data from three totally different sources show good performance and allow for a promising line of research.     

Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Online learning algorithms have been preferred in many applications due to their ability to learn by the sequentially arriving data. One of the effective algorithms recently proposed for training single hidden-layer feedforward neural networks (SLFNs) is online sequential extreme learning machine (OS-ELM), which can learn data one-by-one or chunk-by-chunk at fixed or varying sizes. It is based on the ideas of extreme learning machine (ELM), in which the input weights and hidden layer biases are randomly chosen and then the output weights are determined by the pseudo-inverse operation. The learning speed of this algorithm is extremely high. However, it is not good to yield generalization models for noisy data and is difficult to initialize parameters in order to avoid singular and ill-posed problems. In this paper, we propose an improvement of OS-ELM based on the bi-objective optimization approach. It tries to minimize the empirical error and obtain small norm of network weight vector. Singular and ill-posed problems can be overcome by using the Tikhonov regularization. This approach is also able to learn data one-by-one or chunk-by-chunk. Experimental results show the better generalization performance of the proposed approach on benchmark datasets.     

A transformation strategy for implementing distributed, multilayer feedforward neural networks: Backpropagation transformation

Most artificial neural networks (ANNs) have a fixed topology during learning, and often suffer from a number of shortcomings as a result. Variations of ANNs that use dynamic topologies have shown ability to overcome many of these problems. This paper introduces location-independent transformations (LITs) as a general strategy for implementing distributed feed forward networks that use dynamic topologies (dynamic ANNs) efficiently in parallel hardware. A LIT creates a set of location-independent nodes, where each node computes its part of the network output independent of other nodes, using local information. This type of transformation allows efficient support for adding and deleting nodes dynamically during learning. In particular, this paper presents a LIT that supports both the standard (static) multilayer backpropagation network, and backpropagation with dynamic extensions. The complexity of both learning and execution algorithms is O(q(Nlog M)) for a single pattern, where q is the number of weight layers in the original network, N the number of nodes in the widest node layer in the original network, and M is the number of nodes in the transformed network (which is linear in the number hidden nodes in the original network). This paper extends previous work with 2-weight-layer backpropagation networks.     

Numeric sensitivity analysis applied to feedforward neural networks

During the last 10 years different interpretative methods for analysing the effect or importance of input variables on the output of a feedforward neural network have been proposed. These methods can be grouped into two sets: analysis based on the magnitude of weights; and sensitivity analysis. However, as described throughout this study, these methods present a series of limitations. We have defined and validated a new method, called Numeric Sensitivity Analysis (NSA), that overcomes these limitations, proving to be the procedure that, in general terms, best describes the effect or importance of the input variables on the output, independently of the nature (quantitative or discrete) of the variables included. The interpretative methods used in this study are implemented in the software program Sensitivity Neural Network 1.0, created by our team.     

Recognition of hand-written rotated digits by neural networks

In this paper, an optical character recognition system for hand-written rotated digits in land registry maps is presented. It is based on a neural network and trained by a constructive learning rule, the Hyperbox Perception Cascade (HPC). The HPC classifier can design complex, possibly nonconvex, disjoint, and bounded decision regions and treat the rejection problems of outliers and unanticipated patterns, which would otherwise tend to be classified positively in an incorrect class. We use shape features and a novel approach to select the most promising features to attain a low generalization error. The numerous experiments show that a subset of 24 of the 46 features obtains a good classifier with a high rate of correct classification and a low rate of rejection.     

A new robust training algorithm for a class of single-hidden layer feedforward neural networks

A robust training algorithm for a class of single-hidden layer feedforward neural networks (SLFNs) with linear nodes and an input tapped-delay-line memory is developed in this paper. It is seen that, in order to remove the effects of the input disturbances and reduce both the structural and empirical risks of the SLFN, the input weights of the SLFN are assigned such that the hidden layer of the SLFN performs as a pre-processor, and the output weights are then trained to minimize the weighted sum of the output error squares as well as the weighted sum of the output weight squares. The performance of an SLFN-based signal classifier trained with the proposed robust algorithm is studied in the simulation section to show the effectiveness and efficiency of the new scheme.     

A fuzzy neighborhood-based training algorithm for feedforward neural networks

In this work we present a new hybrid algorithm for feedforward neural networks, which combines unsupervised and supervised learning. In this approach, we use a Kohonen algorithm with a fuzzy neighborhood for training the weights of the hidden layers and gradient descent method for training the weights of the output layer. The goal of this method is to assist the existing variable learning rate algorithms. Simulation results show the effectiveness of the proposed algorithm compared with other well-known learning methods.     

A fast neural net training algorithm and its application to speech classification

This paper describes a fast training algorithm for feedforward neural nets, as applied to a two-layer neural network to classify segments of speech as voiced, unvoiced, or silence. The speech classification method is based on five features computed for each speech segment and used as input to the network. The network weights are trained using a new fast training algorithm which minimizes the total least squares error between the actual output of the network and the corresponding desired output. The iterative training algorithm uses a quasi-Newtonian error-minimization method and employs a positive-definite approximation of the Hessian matrix to quickly converge to a locally optimal set of weights. Convergence is fast, with a local minimum typically reached within ten iterations; in terms of convergence speed, the algorithm compares favorably with other training techniques. When used for voiced-unvoiced-silence classification of speech frames, the network performance compares favorably with current approaches. Moreover, the approach used has the advantage of requiring no assumption of a particular probability distribution for the input features.     

有理式二阶前馈型神经网络

文章提出了二阶有理式多层前馈神经网络的数学模型。有理式多层神经网络的思想来源于函数逼近理论中的有理式逼近。有理式前馈神经网络模型是传统前俯神经网络模型的推广，能有效地求解函数逼近问题。文章给出了有理式多层神经网络的学习算法，即误差反传播学习算法。就计算复杂度而言，有理式神经网络的学习算法与传统的多层神经网络反传播算法是同阶的。文章还给出了函数逼近和模式识别两个应用实例，实验结果说明二阶有理式多层神经网络在解决传统的问题上是有效的。     

A relaxation algorithm for estimating the domain of validity of feedforward neural networks

We expand on a recent paper by Courrieu which introduces three algorithms for determining the distance between any point and the interpolation domain associated with a feedforward neural network. This has been shown to have a significant relation with the network's generalization capability. A further neural-like relaxation algorithm is presented here, which is proven to naturally solve the problem originally posed by Courrieu. The algorithm is based on a powerful result developed in the context of Markov chain theory, and turns out to be a special case of a more general relaxation model which has long become a standard technique in the machine vision domain. Some experiments are presented which confirm the validity of the proposed approach.     

A mesh optimization algorithm based on neural networks

We have developed a mesh simplification method called GNG3D which is able to produce high quality approximations of polygonal models. This method consists of two distinct phases: an optimization phase and a reconstruction phase. The optimization phase is developed by applying an extension algorithm of the growing neural gas model, which constitutes an unsupervised incremental clustering algorithm. The primary goal of this phase is to obtain a simplified set of vertices representing the best approximation of the original 3D object. In the reconstruction phase we use the information provided by the optimization algorithm to reconstruct the faces obtaining the optimized mesh as a result. We study the model theoretically, analyzing its main components, and experimentally, using for this purpose some 3D objects with different topologies. To evaluate the quality of approximations produced by the method proposed in this paper, three existing error measurements are used. The ability of the model to establish the number of vertices of the final simplified mesh is demonstrated in the examples.     

A neural networks approach to minority game

The minority game (MG) comes from the so-called “El Farol bar” problem by W.B. Arthur. The underlying idea is competition for limited resources and it can be applied to different fields such as: stock markets, alternative roads between two locations and in general problems in which the players in the “minority” win. Players in this game use a window of the global history for making their decisions, we propose a neural networks approach with learning algorithms in order to determine players strategies. We use three different algorithms to generate the sequence of minority decisions and consider the prediction power of a neural network that uses the Hebbian algorithm. The case of sequences randomly generated is also studied. Research supported by Local Project 2004–2006 (EX 40%) Università di Foggia. A. Sfrecola is a researcher financially supported by Dipartimento di Scienze Economiche, Matematiche e Statistiche, Università degli Studi di Foggia, Foggia, Italy.     

A training algorithm for binary feedforward neural networks

The authors present a new training algorithm to be used on a four-layer perceptron-type feedforward neural network for the generation of binary-to-binary mappings. This algorithm is called the Boolean-like training algorithm (BLTA) and is derived from original principles of Boolean algebra followed by selected extensions. The algorithm can be implemented on analog hardware, using a four-layer binary feedforward neural network (BFNN). The BLTA does not constitute a traditional circuit building technique. Indeed, the rules which govern the BLTA allow for generalization of data in the face of incompletely specified Boolean functions. When compared with techniques which employ descent methods, training times are greatly reduced in the case of the BLTA. Also, when the BFNN is used in conjunction with A/D converters, the applicability of the present algorithm can be extended to accept real-valued inputs.     

A practical overview of neural networks

This paper overviews the myths and misconceptions that have surrounded neural networks in recent years. Focusing on backpropagation and the Hopfield network, we discuss the problems that have plagued practical application of these techniques, and review some of the recent progress made. Both real and perceived inadequacies of backpropagation are discussed, as well as the need for an understanding of statistics and of the problem domain in order to apply and assess the neural network properly. We consider alternatives or variants to backpropagation, which overcome some of its real limitations. The Hopfield network's poor performance on the traveling salesman problem in combinatorial optimization has colored its reception by engineers; we describe both new research in this area and promising results in other practical optimization applications. Overall, it is hoped, this paper will aid in a more balanced understanding of neural networks. They seem worthy of consideration in many applications, but they do not deserve the status of a panacea – nor are they as fraught with problems as would now seem to be implied.     

Modeling of oxygen mass transfer in the presence of oxygen-vectors using neural networks developed by differential evolution algorithm

The search capabilities of the Differential Evolution (DE) algorithm – a global optimization technique – make it suitable for finding both the architecture and the best internal parameters of a neural network, usually determined by the training phase. In this paper, two variants of the DE algorithm (classical DE and self-adaptive mechanism) were used to obtain the best neural networks in two distinct cases: for prediction and classification problems. Oxygen mass transfer in stirred bioreactors is modeled with neural networks developed with the DE algorithm, based on the consideration that the oxygen constitutes one of the decisive factors of cultivated microorganism growth and can play an important role in the scale-up and economy of aerobic biosynthesis systems. The coefficient of mass transfer oxygen is related to the viscosity, superficial speed of air, specific power, and oxygen-vector volumetric fraction (being predicted as function of these parameters) using stacked neural networks. On the other hand, simple neural networks are designed with DE in order to classify the values of the mass transfer coefficient oxygen into different classes. Satisfactory results are obtained in both cases, proving that the neural network based modeling is an appropriate technique and the DE algorithm is able to lead to the near-optimal neural network topology.     

A general backpropagation algorithm for feedforward neural networks learning

A general backpropagation algorithm is proposed for feedforward neural network learning with time varying inputs. The Lyapunov function approach is used to rigorously analyze the convergence of weights, with the use of the algorithm, toward minima of the error function. Sufficient conditions to guarantee the convergence of weights for time varying inputs are derived. It is shown that most commonly used backpropagation learning algorithms are special cases of the developed general algorithm.     

Interpolation functions of feedforward neural networks

Mathematical essence and structures of the feedforward neural networks are investigated in this paper. The interpolation mechanisms of the feedforward neural networks are explored. For example, the well-known result, namely, that a neural network is an universal approximator, can be concluded naturally from the interpolative representations. Finally, the learning algorithms of the feedforward neural networks are discussed.     

Multi-class pattern classification using neural networks

Multi-class pattern classification has many applications including text document classification, speech recognition, object recognition, etc. Multi-class pattern classification using neural networks is not a trivial extension from two-class neural networks. This paper presents a comprehensive and competitive study in multi-class neural learning with focuses on issues including neural network architecture, encoding schemes, training methodology and training time complexity. Our study includes multi-class pattern classification using either a system of multiple neural networks or a single neural network, and modeling pattern classes using one-against-all, one-against-one, one-against-higher-order, and P-against-Q. We also discuss implementations of these approaches and analyze training time complexity associated with each approach. We evaluate six different neural network system architectures for multi-class pattern classification along the dimensions of imbalanced data, large number of pattern classes, large vs. small training data through experiments conducted on well-known benchmark data.