Improving learning and generalization capabilities of the C-Mantec constructive neural network algorithm

C-Mantec neural network constructive algorithm Ortega (C-Mantec neural network algorithm implementation on MATLAB. https://github.com/IvanGGomez/CmantecPaco, 2015) creates very compact architectures with generalization capabilities similar to feed-forward networks trained by the well-known back-propagation algorithm. Nevertheless, constructive algorithms suffer much from the problem of overfitting, and thus, in this work the learning procedure is first analyzed for networks created by this algorithm with the aim of trying to understand the training dynamics that will permit optimization possibilities. Secondly, several optimization strategies are analyzed for the position of class separating hyperplanes, and the results analyzed on a set of public domain benchmark data sets. The results indicate that with these modifications a small increase in prediction accuracy of C-Mantec can be obtained but in general this was not better when compared to a standard support vector machine, except in some cases when a mixed strategy is used.

     

Improving the generalization performance of RBF neural networks using a linear regression technique

In this paper we present a method for improving the generalization performance of a radial basis function (RBF) neural network. The method uses a statistical linear regression technique which is based on the orthogonal least squares (OLS) algorithm. We first discuss a modified way to determine the center and width of the hidden layer neurons. Then, substituting a QR algorithm for the traditional Gram–Schmidt algorithm, we find the connected weight of the hidden layer neurons. Cross-validation is utilized to determine the stop training criterion. The generalization performance of the network is further improved using a bootstrap technique. Finally, the solution method is used to solve a simulation and a real problem. The results demonstrate the improved generalization performance of our algorithm over the existing methods.     

Computational capabilities of recurrent NARX neural networks

Recently, fully connected recurrent neural networks have been proven to be computationally rich-at least as powerful as Turing machines. This work focuses on another network which is popular in control applications and has been found to be very effective at learning a variety of problems. These networks are based upon Nonlinear AutoRegressive models with eXogenous Inputs (NARX models), and are therefore called NARX networks. As opposed to other recurrent networks, NARX networks have a limited feedback which comes only from the output neuron rather than from hidden states. They are formalized by y(t)=Psi(u(t-n(u)), ..., u(t-1), u(t), y(t-n(y)), ..., y(t-1)) where u(t) and y(t) represent input and output of the network at time t, n(u) and n(y) are the input and output order, and the function Psi is the mapping performed by a Multilayer Perceptron. We constructively prove that the NARX networks with a finite number of parameters are computationally as strong as fully connected recurrent networks and thus Turing machines. We conclude that in theory one can use the NARX models, rather than conventional recurrent networks without any computational loss even though their feedback is limited. Furthermore, these results raise the issue of what amount of feedback or recurrence is necessary for any network to be Turing equivalent and what restrictions on feedback limit computational power.     

Input-to-state stabilization of dynamic neural networks

As a continuation of their previous published results, in this paper the authors propose a new methodology, for input-to-state stabilization of a dynamic neural network. This approach is developed on the basis of the recent introduced inverse optimal control technique for nonlinear control. An example illustrates the applicability of the proposed approach.     

Bayes statistical behavior and valid generalization of patternclassifying neural networks

It is demonstrated both theoretically and experimentally that, under appropriate assumptions, a neural network pattern classifier implemented with a supervised learning algorithm generates the empirical Bayes rule that is optimal against the empirical distribution of the training sample. It is also shown that, for a sufficiently large sample size, asymptotic equivalence of the network-generated rule to the theoretical Bayes optimal rule against the true distribution governing the occurrence of data follows immediately from the law of large numbers. It is proposed that a Bayes statistical decision approach leads naturally to a probabilistic definition of the valid generalization which a neural network can be expected to generate from a finite training sample.     

On the partitioning capabilities of feedforward neural networks with sigmoid nodes

In this letter, the capabilities of feedforward neural networks (FNNs) on the realization and approximation of functions of the form g: R(l) --> A, which partition the R(l) space into polyhedral sets, each one being assigned to one out of the c classes of A, are investigated. More specifically, a constructive proof is given for the fact that FNNs consisting of nodes having sigmoid output functions are capable of approximating any function g with arbitrary accuracy. Also, the capabilities of FNNs consisting of nodes having the hard limiter as output function are reviewed. In both cases, the two-class as well as the multiclass cases are considered.     

Issues on representational capabilities of artificial neural networks and their implementation

This work is an organized review on the representational capabilities of artificial neural networks and the questions that arise in their implementation. It covers the Kolmogorov's superposition theorem and different statements regarding how it could be related to the representational power of neural networks. Generalization capability of neural networks is then considered and methods of improving this capability are discussed. Some theorems and statements concerning the bound on the number of hidden layers, form of the activation function, and time complexity of training of neural networks are other subjects of this article. © 1995 John Wiley & Sons, Inc.     

Almost everywhere approximation capabilities of double Mellin approximate identity neural networks

The research works in approximation theory of artificial neural networks is still far from completion. To fill a gap in this issue, this study focuses on the almost everywhere approximation capabilities of single-hidden-layer feedforward double Mellin approximate identity neural networks. First, the notion of double Mellin approximate identity is introduced. Using this notion, an auxiliary theorem is proved. The auxiliary theorem provides a connection between a class of double Mellin convolution linear operators and the notion of almost everywhere convergence. This theorem is applied to prove a main theorem. The proof of the main theorem is based on the notion of epsilon-net. The main theorem shows almost everywhere approximation capability of single-hidden-layer feedforward double Mellin approximate identity neural networks in the space of almost everywhere continuous bivariate functions on \( \mathbb {R}_{+}^{2} \). Moreover, similar results are obtained in the spaces of almost everywhere Lebesgue integrable bivariate functions on \( \mathbb {R}_{+}^{2} \).     

Variational Bayes solution of linear neural networks and its generalization performance

It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and has been attracting attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications; nevertheless, it has not yet been theoretically shown. In this letter, through analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then we theoretically clarify its free energy, generalization error, and training error. Comparing them with those of the ML estimation and the Bayes estimation, we discuss the advantage of the VB approach. We also show that unlike in the Bayes estimation, the free energy and the generalization error are less simply related with each other and that in typical cases, the VB free energy well approximates the Bayes one, while the VB generalization error significantly differs from the Bayes one.     

Energy function analysis of dynamic programming neural networks

All analytical examination of the energy function associated with a dynamic programming neural network is presented. The analysis is carried out in two steps. First, the locations and numbers of the minimum states for different components of the energy function are investigated in the extreme cases. A clearer insight into the energy function can be gained through the minimum states of different components. Secondly, the locations of the minimum states of the energy function using different parameter values are derived. It is shown that the minimum states can reside in regions which are regarded as valid solutions with certain conditions. Examples and simulation results are given to justify the validity of the theories developed.     

Enhancing the generalization ability of neural networks through controlling the hidden layers

In this paper we proposed two new variants of backpropagation algorithm. The common point of these two new algorithms is that the outputs of nodes in the hidden layers are controlled with the aim to solve the moving target problem and the distributed weights problem. One algorithm (AlgoRobust) is not so insensitive to the noises in the data, the second one (AlgoGS) is through using Gauss–Schmidt algorithm to determine in each epoch which weight should be updated, while the other weights are kept unchanged in this epoch. In this way a better generalization can be obtained. Some theoretical explanations are also provided. In addition, simulation comparisons are made between Gaussian regularizer, optimal brain damage (OBD) and the proposed algorithms. Simulation results confirm that the new proposed algorithms perform better than that of Gaussian regularizer, and the first algorithm AlgoRobust performs better than the second algorithm AlgoGS in the noisy data. On the other hand AlgoGS performs better than the AlgoRobust on the data without noise and the final structure obtained by two new algorithms is comparable to that obtained by using OBD.     

Improving reinforcement learning in stochastic RAM-based neural networks

RAM-based neural networks are designed to be efficiently implemented in hardware. The desire to retain this property influences the training algorithms used, and has led to the use of reinforcement (reward-penalty) learning. An analysis of the reinforcement algorithm applied to RAM-based nodes has shown the ease with which unlearning can occur. An amended algorithm is proposed which demonstrates improved learning performance compared to previously published reinforcement regimes.     

Approximation capabilities of multilayer fuzzy neural networks on the set of fuzzy-valued functions

This paper first introduces a piecewise linear interpolation method for fuzzy-valued functions. Based on this, we present a concrete approximation procedure to show the capability of four-layer regular fuzzy neural networks to perform approximation on the set of all d_p continuous fuzzy-valued functions. This approach can also be used to approximate d_∞ continuous fuzzy-valued functions. An example is given to illustrate the approximation procedure.     

Recursive dynamic node creation in multilayer neural networks

The derivations of a novel approach for simultaneous recursive weight adaptation and node creation in multilayer backpropagation neural networks are presented. The method uses time and order update formulations in the orthogonal projection method to derive a recursive weight updating procedure for the training process of the neural network and a recursive node creation algorithm for weight adjustment of a layer with added nodes during the training process. The proposed approach allows optimal dynamic node creation in the sense that the mean-squared error is minimized for each new topology. The effectiveness of the algorithm is demonstrated on several benchmark problems (the multiplexer and the decoder problems) as well as a real world application for detection and classification of buried dielectric anomalies using a microwave sensor.     

Trajectory generation and modulation using dynamic neural networks

Generation of desired trajectory behavior using neural networks involves a particularly challenging spatio-temporal learning problem. This paper introduces a novel solution, i.e., designing a dynamic system whose terminal behavior emulates a prespecified spatio-temporal pattern independently of its initial conditions. The proposed solution uses a dynamic neural network (DNN), a hybrid architecture that employs a recurrent neural network (RNN) in cascade with a nonrecurrent neural network (NRNN). The RNN generates a simple limit cycle, which the NRNN reshapes into the desired trajectory. This architecture is simple to train. A systematic synthesis procedure based on the design of relay control systems is developed for configuring an RNN that can produce a limit cycle of elementary complexity. It is further shown that a cascade arrangement of this RNN and an appropriately trained NRNN can emulate any desired trajectory behavior irrespective of its complexity. An interesting solution to the trajectory modulation problem, i.e., online modulation of the generated trajectories using external inputs, is also presented. Results of several experiments are included to demonstrate the capabilities and performance of the DNN in handling trajectory generation and modulation problems.     

Static and dynamic preprocessing methods in neural networks

Preprocessing is recognized as an important tool in modeling, particularly when the data or underlying physical process involves complex nonlinear dynamical interactions. This paper will give a review of preprocessing methods used in linear and nonlinear models. The problem of static preprocessing will be considered first, where no dependence on time between the input vectors is assumed. Then, dynamic preprocessing methods which involve the modification of time-dependent input values before they are used in the linear or nonlinear models will be considered. Furthermore, the problem of an insufficient number of input vectors is considered. It is shown that one way in which this problem can be overcome is by expanding the weight vector in terms of the available input vectors. Finally, a new problem which involves both cases of: (1) transformation of input vectors; and (2) insufficient number of input vectors is considered. It is shown how a combination of the techniques used to solve the individual problems can be combined to solve this composite problem. Some open issues in this type of preprocessing methods are discussed.     

Diagonal recurrent neural networks for dynamic systems control

A new neural paradigm called diagonal recurrent neural network (DRNN) is presented. The architecture of DRNN is a modified model of the fully connected recurrent neural network with one hidden layer, and the hidden layer comprises self-recurrent neurons. Two DRNN's are utilized in a control system, one as an identifier called diagonal recurrent neuroidentifier (DRNI) and the other as a controller called diagonal recurrent neurocontroller (DRNC). A controlled plant is identified by the DRNI, which then provides the sensitivity information of the plant to the DRNC. A generalized dynamic backpropagation algorithm (DBP) is developed and used to train both DRNC and DRNI. Due to the recurrence, the DRNN can capture the dynamic behavior of a system. To guarantee convergence and for faster learning, an approach that uses adaptive learning rates is developed by introducing a Lyapunov function. Convergence theorems for the adaptive backpropagation algorithms are developed for both DRNI and DRNC. The proposed DRNN paradigm is applied to numerical problems and the simulation results are included.     

Improving optimization of convolutional neural networks through parameter fine-tuning

Neural Computing and Applications - In recent years, convolutional neural networks have achieved state-of-the-art performance in a number of computer vision problems such as image classification....     

Fuzzy generalization of the counter-propagation neural network: a family of soft competitive basis function neural networks

 Based on combining neural network (NN) with fuzzy logical system (FLS), a new family of three-layer feedforward networks, called soft-competition basis function neural networks (SCBFs), is proposed under the framework of the counter-propagation (CP) network. The hidden layer of SCBFs is designed as competitive layer with soft competitive strategy. The output function of their hidden neuron is defined as basis function taking the form of fuzzy membership function. SCBFs possess the ability of functional approximation. They are fuzzy generalization of the CP network and functionally equivalent to TS-model of fuzzy logical system. Therefore, they can be regard as either a NN or a FLS. Their learning algorithms are also discussed in this paper. Finally, some experiments are given to test the performance of SCBFs.