Improvement of RBF neural networks using Fuzzy-OSD algorithm in an online radar pulse classification system

In this paper a new methodology for training radial basis function (RBF) neural networks is introduced and examined. This novel approach, called Fuzzy-OSD, could be used in applications, which need real-time capabilities for retraining neural networks. The proposed method uses fuzzy clustering in order to improve the functionality of the Optimum Steepest Descent (OSD) learning algorithm. This improvement is due to initialization of RBF units more precisely using fuzzy C-Means clustering algorithm that results in producing better and the same network response in different retraining attempts. In addition, adjusting RBF units in the network with great accuracy will result in better performance in fewer train iterations, which is essential when fast retraining of the network is needed, especially in the real-time systems. We employed this new method in an online radar pulse classification system, which needs quick retraining of the network once new unseen emitters detected. Having compared result of applying the new algorithm and Three-Phase OSD method to benchmark problems from Proben1 database and also using them in our system, we achieved improvement in the results as presented in this paper.     

Global optimization models in data networks: a case study

In this paper we present a methodology for a rural and semi-urban data network placement. In order to optimally place the network and to ensure that the network is realistic and viable we address four key issues, namely, the demographic and socio-economic issues, geographical estimation, optimization of the network placement and financial optimization. A digital representation of the map of the region where the network has to be placed is used. A continuous optimization algorithm is applied to optimally place the backbone rings, and a combinatorial optimization algorithm is applied to obtain the optimal rollout order for the network. Mathematical formulations for both the optimization problems are presented. Optimal financial indicators are obtained.     

On-line training of recurrent neural networks with continuoustopology adaptation

This paper presents an online procedure for training dynamic neural networks with input-output recurrences whose topology is continuously adjusted to the complexity of the target system dynamics. This is accomplished by changing the number of the elements of the network hidden layer whenever the existing topology cannot capture the dynamics presented by the new data. The training mechanism is based on the suitably altered extended Kalman filter (EKF) algorithm which is simultaneously used for the network parameter adjustment and for its state estimation. The network consists of a single hidden layer with Gaussian radial basis functions (GRBF), and a linear output layer. The choice of the GRBF is induced by the requirements of the online learning. The latter implies the network architecture which permits only local influence of the new data point in order not to forget the previously learned dynamics. The continuous topology adaptation is implemented in our algorithm to avoid memory and computational problems of using a regular grid of GRBF'S which covers the network input space. Furthermore, we show that the resulting parameter increase can be handled "smoothly" without interfering with the already acquired information. If the target system dynamics are changing over time, we show that a suitable forgetting factor can be used to "unlearn" the no longer-relevant dynamics. The quality of the recurrent network training algorithm is demonstrated on the identification of nonlinear dynamic systems.     

Divide-and-conquer learning and modular perceptron networks

A novel modular perceptron network (MPN) and divide-and-conquer learning (DCL) schemes for the design of modular neural networks are proposed. When a training process in a multilayer perceptron falls into a local minimum or stalls in a flat region, the proposed DCL scheme is applied to divide the current training data region into two easier to be learned regions. The learning process continues when a self-growing perceptron network and its initial weight estimation are constructed for one of the newly partitioned regions. Another partitioned region will resume the training process on the original perceptron network. Data region partitioning, weight estimating and learning are iteratively repeated until all the training data are completely learned by the MPN. We evaluated and compared the proposed MPN with several representative neural networks on the two-spirals problem and real-world dataset. The MPN achieved better weight learning performance by requiring much less data presentations during the network training phases, and better generalization performance, and less processing time during the retrieving phase.     

Pruning recurrent neural networks for improved generalizationperformance

Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.     

Design of neural networks for fast convergence and accuracy:dynamics and control

A procedure for the design and training of artificial neural networks, used for rapid and efficient controls and dynamics design and analysis for flexible space systems, has been developed. Artificial neural networks are employed, such that once property trained, they provide a means of evaluating the impact of design changes rapidly. Specifically, two-layer feedforward neural networks are designed to approximate the functional relationship between the component/spacecraft design changes and measures of its performance or nonlinear dynamics of the system/components. A training algorithm, based on statistical sampling theory is presented, which guarantees that the trained networks provide a designer-specified degree of accuracy in mapping the functional relationship. Within each iteration of this statistical-based algorithm, a sequential design algorithm is used for the design and training of the feedforward network to provide rapid convergence to the network goals. Here, at each sequence a new network is trained to minimize the error of previous network. The proposed method should work for applications wherein an arbitrary large source of training data can be generated. Two numerical examples are performed on a spacecraft application in order to demonstrate the feasibility of the proposed approach.     

Fuzzy neural networks for classification and detection of anomalies

A new learning algorithm for the Simpson fuzzy min-max neural network is presented. It overcomes some undesired properties of the Simpson model. Our new algorithm improves the network performance; the classification result does not depend on the presentation order of the patterns in the training set, and at each step, the classification error in the training set cannot increase. The new neural model is particularly useful in classification problems. Tests were executed on three different classification problems: 1) with two-dimensional synthetic data; 2) with realistic data generated by a simulator to find anomalies in the cooling system of a blast furnace; and 3) with real data for industrial diagnosis. The experiments were made following some recent evaluation criteria known in the literature and by using Microsoft Visual C++ development environment on personal computers.     

A computation-efficient on-line training algorithm for neurofuzzy networks

Neurofuzzy networks are often used to model linear or nonlinear processes, as they can provide some insights into the underlying processes and can be trained using experimental data. As the training of the networks involves intensive computation, it is often performed off line. However, it is well known that neurofuzzy networks trained off line may not be able to cope successully with time-varying processes. To overcome this problem, the weights of the networks are trained on line. In this paper, an on-line training algorithm with a computation time that is linear in the number of weights is derived by making full use of the local change property of neurofuzzy networks. It is shown that the estimated weights converge to that obtained from the least-squares method, and that the range of the input domain can be extended without retraining the network. Furthermore, it has a better ability in tracking time-varying systems than the recursive least-squares method, since in the proposed algorithm a positive definite submatrix is added to the relevant part of the covariance matrix. The performance of the proposed algorithm is illustrated by simulation examples and compared with that obtained using the recursive least-squares method.     

On self-organizing algorithms and networks for class-separability features.

We describe self-organizing learning algorithms and associated neural networks to extract features that are effective for preserving class separability. As a first step, an adaptive algorithm for the computation of Q(-1/2) (where Q is the correlation or covariance matrix of a random vector sequence) is described. Convergence of this algorithm with probability one is proven by using stochastic approximation theory, and a single-layer linear network architecture for this algorithm is described, which we call the Q(-1/2) network. Using this network, we describe feature extraction architectures for: 1) unimodal and multicluster Gaussian data in the multiclass case; 2) multivariate linear discriminant analysis (LDA) in the multiclass case; and 3) Bhattacharyya distance measure for the two-class case. The LDA and Bhattacharyya distance features are extracted by concatenating the Q (-1/2) network with a principal component analysis network, and the two-layer network is proven to converge with probability one. Every network discussed in the study considers a flow or sequence of inputs for training. Numerical studies on the performance of the networks for multiclass random data are presented.     

An Online Learning Approach: A Methodology for Time Varying Applications

In this paper, a new procedure to continuously adjust weights in a multi-layered neural network is proposed. The network is initially trained by using a traditional backpropagation algorithm. After this first step, a non-linear programming technique is used to properly calculate the new weights sets online. This methodology is tailored to be used in time varying (non-stationary) models, eliminating the necessity for retraining. Numerical results for a controlled experiment and for real data are presented.     

An adaptable neural-network model for recursive nonlinear traffic prediction and modeling of MPEG video sources

Multimedia services and especially digital video is expected to be the major traffic component transmitted over communication networks [such as internet protocol (IP)-based networks]. For this reason, traffic characterization and modeling of such services are required for an efficient network operation. The generated models can be used as traffic rate predictors, during the network operation phase (online traffic modeling), or as video generators for estimating the network resources, during the network design phase (offline traffic modeling). In this paper, an adaptable neural-network architecture is proposed covering both cases. The scheme is based on an efficient recursive weight estimation algorithm, which adapts the network response to current conditions. In particular, the algorithm updates the network weights so that 1) the network output, after the adaptation, is approximately equal to current bit rates (current traffic statistics) and 2) a minimal degradation over the obtained network knowledge is provided. It can be shown that the proposed adaptable neural-network architecture simulates a recursive nonlinear autoregressive model (RNAR) similar to the notation used in the linear case. The algorithm presents low computational complexity and high efficiency in tracking traffic rates in contrast to conventional retraining schemes. Furthermore, for the problem of offline traffic modeling, a novel correlation mechanism is proposed for capturing the burstness of the actual MPEG video traffic. The performance of the model is evaluated using several real-life MPEG coded video sources of long duration and compared with other linear/nonlinear techniques used for both cases. The results indicate that the proposed adaptable neural-network architecture presents better performance than other examined techniques.     

Learning nonlinear multiregression networks based on evolutionary computation

This paper describes a novel knowledge discovery and data mining framework dealing with nonlinear interactions among domain attributes. Our network-based model provides an effective and efficient reasoning procedure to perform prediction and decision making. Unlike many existing paradigms based on linear models, the attribute relationship in our framework is represented by nonlinear nonnegative multiregressions based on the Choquet integral. This kind of multiregression is able to model a rich set of nonlinear interactions directly. Our framework involves two layers. The outer layer is a network structure consisting of network elements as its components, while the inner layer is concerned with a particular network element modeled by Choquet integrals. We develop a fast double optimization algorithm (FDOA) for learning the multiregression coefficients of a single network element. Using this local learning component and multiregression-residual-cost evolutionary programming (MRCEP), we propose a global learning algorithm, called MRCEP-FDOA, for discovering the network structures and their elements from databases. We have conducted a series of experiments to assess the effectiveness of our algorithm and investigate the performance under different parameter combinations, as well as sizes of the training data sets. The empirical results demonstrate that our framework can successfully discover the target network structure and the regression coefficients.     

Using random weights to train multilayer networks of hard-limitingunits

A gradient descent algorithm suitable for training multilayer feedforward networks of processing units with hard-limiting output functions is presented. The conventional backpropagation algorithm cannot be applied in this case because the required derivatives are not available. However, if the network weights are random variables with smooth distribution functions, the probability of a hard-limiting unit taking one of its two possible values is a continuously differentiable function. In the paper, this is used to develop an algorithm similar to backpropagation, but for the hard-limiting case. It is shown that the computational framework of this algorithm is similar to standard backpropagation, but there is an additional computational expense involved in the estimation of gradients. Upper bounds on this estimation penalty are given. Two examples which indicate that, when this algorithm is used to train networks of hard-limiting units, its performance is similar to that of conventional backpropagation applied to networks of units with sigmoidal characteristics are presented.     

Design of a multi-hop dynamic consensus algorithm over wireless sensor networks

In the last few years great attention has been concentrated on the consensus algorithm in a network of agents. A consensus problem in which the agreement value is a distributed estimation of some non-constant quantity of interest is referred to as a dynamic consensus. In practical applications an effective network architecture to support sensing and communication between agents is based on a Wireless Sensor Network (WSN). This paper deals with the design of a fast dynamic consensus algorithm when it is implemented over the WSN. A sufficient stability condition of the dynamic consensus algorithm in the presence of heterogeneous time delays affecting communication through the multi hops of the WSN is introduced and used for consensus algorithm gain design. Moreover, the algorithm implementation by the standard AODV routing protocol is discussed and the best parameter setting to reduce the effect of packet collision phenomena on the performance of the consensus algorithm is indicated. Several trade-offs between network parameter setting, sensor node technology selection and application performance have to be taken into account by the designer in the implementation of the dynamic consensus algorithm. A representative simulation based design procedure is presented to validate through realistic simulation experiments the proposed design approach.     

Neural networks on the basis of the sample method

This paper describes the procedure of creating a neural network on the basis of the “etalon” (sample) method. The algorithm for the calculation of the weight table for the neural network connections is adduced and doesn’t need network training. The technology of the optimum creation of the neural network layers is described. The possibility is pointed out to use such an approach for the recognition of curves and symbolic images. An algorithm for the optimal selection of samples from a tested data set is adduced.     

Regularized variational Bayesian learning of echo state networks with delay&sum readout

In this work, a variational Bayesian framework for efficient training of echo state networks (ESNs) with automatic regularization and delay&sum (D&S) readout adaptation is proposed. The algorithm uses a classical batch learning of ESNs. By treating the network echo states as fixed basis functions parameterized with delay parameters, we propose a variational Bayesian ESN training scheme. The variational approach allows for a seamless combination of sparse Bayesian learning ideas and a variational Bayesian space-alternating generalized expectation-maximization (VB-SAGE) algorithm for estimating parameters of superimposed signals. While the former method realizes automatic regularization of ESNs, which also determines which echo states and input signals are relevant for "explaining" the desired signal, the latter method provides a basis for joint estimation of D&S readout parameters. The proposed training algorithm can naturally be extended to ESNs with fixed filter neurons. It also generalizes the recently proposed expectation-maximization-based D&S readout adaptation method. The proposed algorithm was tested on synthetic data prediction tasks as well as on dynamic handwritten character recognition.     

Online stabilization of block-diagonal recurrent neural networks

Deals with a discrete-time recurrent neural network (DTRNN) with a block-diagonal feedback weight matrix, called the block-diagonal recurrent neural network (BDRNN), that allows a simplified approach to online training and to address network and training stability issues. The structure of the BDRNN is exploited to modify the conventional backpropagation through time (BPTT) algorithm. To reduce its storage requirement by a numerically stable method of recomputing the network state variables. The network and training stability is addressed by exploiting the BDRNN structure to directly monitor and maintain stability during weight updates by developing a functional measure of system stability that augments the cost function being minimized. Simulation results are presented to demonstrate the performance of the BDRNN architecture, its training algorithm, and the stabilization method.     

A sequential multi-category classifier using radial basis function networks

This paper presents a new sequential multi-category classifier using radial basis function (SMC-RBF) network for real-world classification problems. The classification algorithm processes the training data one by one and builds the RBF network starting with zero hidden neuron. The growth criterion uses the misclassification error, the approximation error to the true decision boundary and a distance measure between the current sample and the nearest neuron belonging to the same class. SMC-RBF uses the hinge loss function (instead of the mean square loss function) for a more accurate estimate of the posterior probability. For network parameter updates, a decoupled extended Kalman filter is used to reduce the computational overhead. Performance of the proposed algorithm is evaluated using three benchmark problems, viz., image segmentation, vehicle and glass from the UCI machine learning repository. In addition, performance comparison has also been done on two real-world problems in the areas of remote sensing and bio-informatics. The performance of the proposed SMC-RBF classifier is also compared with the other RBF sequential learning algorithms like MRAN, GAP-RBFN, OS-ELM and the well-known batch classification algorithm SVM. The results indicate that SMC-RBF produces a higher classification accuracy with a more compact network. Also, the study indicates that using a function approximation algorithm for classification problems may not work well when the classes are not well separated and the training data is not uniformly distributed among the classes.     

Efficacy of utilizing a hybrid algorithmic method in enhancing the functionality of multi-instance multi-label radial basis function neural networks

The facts show that multi-instance multi-label (MIML) learning plays a pivotal role in Artificial Intelligence studies. Evidently, the MIML learning introduces a framework in which data is described by a bag of instances associated with a set of labels. In this framework, the modeling of the connection is the challenging problem for MIML. The RBF neural network can explain the complex relations between the instances and labels in the MIMLRBF. The parameters estimation of the RBF network is a difficult task. In this paper, the computational convergence and the modeling accuracy of the RBF network has been improved. The present study aimed to investigate the impact of a novel hybrid algorithm consisting of Gases Brownian Motion optimization (GBMO) algorithm and the gradient based fast converging parameter estimation method on multi-instance multi-label learning. In the current study, a hybrid algorithm was developed to estimate the RBF neural network parameters (the weights, widths and centers of the hidden units) simultaneously. The algorithm uses the robustness of the GBMO to search the parameter space and the efficiency of the gradient. For this purpose, two real-world MIML tasks and a Corel dataset were utilized within a two-step experimental design. In the first step, the GBMO algorithm was used to determine the widths and centers of the network nodes. In the second step, for each molecule with fixed inputs and number of hidden nodes, the parameters were optimized by a structured nonlinear parameter optimization method (SNPOM). The findings demonstrated the superior performance of the hybrid algorithmic method. Additionally, the results for training and testing the dataset revealed that the hybrid method enhances RBF network learning more efficiently in comparison with other conventional RBF approaches. The results obtain better modeling accuracy than some other algorithms.     

Knowledge discovery by inductive neural networks

A new neural network model for inducing symbolic knowledge from empirical data is presented. This model capitalizes on the fact that the certainty factor-based activation function can improve the network generalization performance from a limited amount of training data. The formal properties of the procedure for extracting symbolic knowledge from such a trained neural network are investigated. In the domain of molecular genetics, a case study demonstrated that the described learning system effectively discovered the prior domain knowledge with some degree of refinement. Also, in cross-validation experiments, the system outperformed C4.5, a commonly used rule learning system