Investigating data preprocessing methods for circuit complexity models

Preprocessing the data is an important step while creating neural network (NN) applications because this step usually has a significant effect on the prediction performance of the model. This paper compares different data processing strategies for NNs for prediction of Boolean function complexity (BFC). We compare NNs’ predictive capabilities with (1) no preprocessing (2) scaling the values in different curves based on every curve’s own peak and then normalizing to [0, 1] range (3) applying z-score to values in all curves and then normalizing to [0, 1] range, and (4) logarithmically scaling all curves and then normalizing to [0, 1] range. The efficiency of these methods was measured by comparing RMS errors in NN-made BFC predictions for numerous ISCAS benchmark circuits. Logarithmic preprocessing method resulted in the best prediction statistics as compared to other techniques.     

An evaluation of quantum neural networks in the detection of epileptic seizures in the neonatal electroencephalogram

This paper presents the results of an experimental study that evaluated the ability of quantum neural networks (QNNs) to capture and quantify uncertainty in data and compared their performance with that of conventional feedforward neural networks (FFNNs). In this work, QNNs and FFNNs were trained to classify short segments of epileptic seizures in neonatal EEG. The experiments revealed significant differences between the internal representations created by trained QNNs and FFNNs from sample information provided by the training data. The results of this experimental study also confirmed that the responses of trained QNNs are more reliable indicators of uncertainty in the input data compared with the responses of trained FFNNs.     

Multiobjective hybrid optimization and training of recurrent neural networks.

The application of neural networks to solve a problem involves tasks with a high computational cost until a suitable network is found, and these tasks mainly involve the selection of the network topology and the training step. We usually select the network structure by means of a trial-and-error procedure, and we then train the network. In the case of recurrent neural networks (RNNs), the lack of suitable training algorithms sometimes hampers these procedures due to vanishing gradient problems. This paper addresses the simultaneous training and topology optimization of RNNs using multiobjective hybrid procedures. The proposal is based on the SPEA2 and NSGA2 algorithms for making hybrid methods using the Baldwinian hybridization strategy. We also study the effects of the selection of the objectives, crossover, and mutation in the diversity during evolution. The proposals are tested in the experimental section to train and optimize the networks in the competition on artificial time-series (CATS) benchmark.     

Short‐term electric power load forecasting based on cosine radial basis function neural networks: An experimental evaluation

This article presents the results of a study aimed at the development of a system for short‐term electric power load forecasting. This was attempted by training feedforward neural networks (FFNNs) and cosine radial basis function (RBF) neural networks to predict future power demand based on past power load data and weather conditions. This study indicates that both neural network models exhibit comparable performance when tested on the training data but cosine RBF neural networks generalize better since they outperform considerably FFNNs when tested on the testing data. © 2005 Wiley Periodicals, Inc. Int J Int Syst 20: 591–605, 2005.     

Analysis and Modeling of Naturalness in Handwritten Characters

In this paper, we define the naturalness of handwritten characters as being the difference between the strokes of the handwritten characters and the archetypal fonts on which they are based. With this definition, we mathematically analyze the relationship between the font and its naturalness using canonical correlation analysis (CCA), multiple linear regression analysis, feedforward neural networks (FFNNs) with sliding windows, and recurrent neural networks (RNNs). This analysis reveals that certain properties of font character strokes do not have a linear relationship with their naturalness. In turn, this suggests that nonlinear techniques should be used to model the naturalness, and in our investigations, we find that an RNN with a recurrent output layer performs the best among four linear and nonlinear models. These results indicate that it is possible to model naturalness, defined in our study as the difference between handwritten and archetypal font characters but more generally as the difference between the behavior of a natural system and a corresponding basic system, and that naturalness learning is a promising approach for generating handwritten characters.     

循环神经网络研究综述

循环神经网络是神经网络序列模型的主要实现形式,近几年得到迅速发展,其是机器翻译、机器问题回答、序列视频分析的标准处理手段,也是对于手写体自动合成、语音处理和图像生成等问题的主流建模手段.鉴于此,循环神经网络的各分支按照网络结构进行详细分类,大致分为3大类:一是衍生循环神经网络,这类网络是基于基本RNNs模型的结构衍生变体,即对RNNs的内部结构进行修改;二是组合循环神经网络,这类网络将其他一些经典的网络模型或结构与第一类衍生循环神经网络进行组合,得到更好的模型效果,是一种非常有效的手段;三是混合循环神经网络,这类网络模型既有不同网络模型的组合,又在RNNs内部结构上进行修改,是同属于前两类网络分类的结构.为了更加深入地理解循环神经网络,进一步介绍与循环神经网络经常混为一谈的递归神经网络结构以及递归神经网络与循环神经网络的区别和联系.在详略描述上述模型的应用背景、网络结构以及模型变种后,对各个模型的特点进行总结和比较,并对循环神经网络模型进行展望和总结.     

Hybridization of the probabilistic neural networks with feed-forward neural networks for forecasting

Feed-forward neural networks (FFNNs) are among the most important neural networks that can be applied to a wide range of forecasting problems with a high degree of accuracy. Several large-scale forecasting competitions with a large number of commonly used time series forecasting models conclude that combining forecasts from more than one model often leads to improved performance, especially when the models in the ensemble are quite different. In the literature, several hybrid models have been proposed by combining different time series models together. In this paper, in contrast of the traditional hybrid models, a novel hybridization of the feed-forward neural networks (FFNNs) is proposed using the probabilistic neural networks (PNNs) in order to yield more accurate results than traditional feed-forward neural networks. In the proposed model, the estimated values of the FFNN models are modified based on the distinguished trend of their residuals and optimum step length, which are respectively yield from a probabilistic neural network and a mathematical programming model. Empirical results with three well-known real data sets indicate that the proposed model can be an effective way in order to construct a more accurate hybrid model than FFNN models. Therefore, it can be applied as an appropriate alternative model for forecasting tasks, especially when higher forecasting accuracy is needed.     

Binary Decision Diagrams and neural networks

This paper describes a neural network approach that gives an estimation method for the space complexity of Binary Decision Diagrams (BDDs). A model has been developed to predict the complexity of digital circuits. The formal core of the developed neural network model (NNM) is a unique matrix for the complexity estimation over a set of BDDs derived from Boolean logic expressions with a given number of variables and Sum of Products (SOP) terms. Experimental results show good correlation between the theoretical results and those predicted by the NNM, which will give insights to the complexity of Very Large Scale Integration (VLSI)/Computer Aided Design (CAD) designs. The proposed model is capable of predicting the maximum BDD complexity (MaxBC) and the number of product terms (NPT) in the Boolean function that corresponds to the minimum BDD complexity (MinBC). This model provides an alternative way to predict the complexity of digital VLSI circuits.

Stability analysis of discrete-time recurrent neural networks based on standard neural network models

In order to conveniently analyze the stability of various discrete-time recurrent neural networks (RNNs), including bidirectional associative memory, Hopfield, cellular neural network, Cohen-Grossberg neural network, and recurrent multiplayer perceptrons, etc., the novel neural network model, named standard neural network model (SNNM) is advanced to describe this class of discrete-time RNNs. The SNNM is the interconnection of a linear dynamic system and a bounded static nonlinear operator. By combining Lyapunov functional with S-Procedure, some useful criteria of global asymptotic stability for the discrete-time SNNMs are derived, whose conditions are formulated as linear matrix inequalities. Most delayed (or non-delayed) RNNs can be transformed into the SNNMs to be stability analyzed in a unified way. Some application examples of the SNNMs to the stability analysis of the discrete-time RNNs shows that the SNNMs make the stability conditions of the RNNs easily verified.     

Removal of bird-contaminated wind profiler data based on neural networks

This paper presents the results of a study that relied on trainable neural network classifiers to identify and remove bird-contaminated data from wind measurements recorded by a 1290-MHz wind profiler. A wind profiler is a Doppler radar system measuring the three-dimensional wind field. Migrating birds crossing the radar beam can lead to erroneous wind observations. Bird removal was performed by training conventional feedforward neural networks (FFNNs) and quantum neural networks (QNNs) to identify and remove bird-contaminated data recorded by a 1290-MHz wind profiler. A series of experiments evaluated several sets of input features extracted from wind profiler data, various FFNNs and QNNs of different sizes, and criteria employed for identifying birds in wind profiler data.     

The copula echo state network

Echo state networks (ESNs) constitute a novel approach to recurrent neural network (RNN) training, with an RNN (the reservoir) being generated randomly, and only a readout being trained using a simple, computationally efficient algorithm. ESNs have greatly facilitated the practical application of RNNs, outperforming classical approaches on a number of benchmark tasks. This paper studies the formulation of a class of copula-based semiparametric models for sequential data modeling, characterized by nonparametric marginal distributions modeled by postulating suitable echo state networks, and parametric copula functions that help capture all the scale-free temporal dependence of the modeled processes. We provide a simple algorithm for the data-driven estimation of the marginal distribution and the copula parameters of our model under the maximum-likelihood framework. We exhibit the merits of our approach by considering a number of applications; as we show, our method offers a significant enhancement in the dynamical data modeling capabilities of ESNs, without significant compromises in the algorithm's computational efficiency.     

Improving the Fuzzy Min-Max neural network with a K-nearest hyperbox expansion rule for pattern classification

An improved Fuzzy Min-Max (FMM) neural network with a K-nearest hyperbox expansion rule is proposed in this paper. The aim is to reduce the FMM network complexity for undertaking pattern classification tasks. In the proposed model, a useful modification to overcome a number of identified limitations of the original FMM network and to improve its classification performance is derived. In particular, the K-nearest hyperbox expansion rule is formulated to reduce the network complexity by avoiding the creation of too many small hyperboxes within the vicinity of the winning hyperbox during the FMM learning stage. The effectiveness of the proposed model is evaluated using a number of benchmark data sets. The results compare favorably with those from various FMM variants and other existing classifiers.     

Delayed standard neural network models for control systems.

In order to conveniently analyze the stability of recurrent neural networks (RNNs) and successfully synthesize the controllers for nonlinear systems, similar to the nominal model in linear robust control theory, the novel neural network model, named delayed standard neural network model (DSNNM) is presented, which is the interconnection of a linear dynamic system and a bounded static delayed (or nondelayed) nonlinear operator. By combining a number of different Lyapunov functionals with S-procedure, some useful criteria of global asymptotic stability and global exponential stability for the continuous-time DSNNMs (CDSNNMs) and discrete-time DSNNMs (DDSNNMs) are derived, whose conditions are formulated as linear matrix inequalities (LMIs). Based on the stability analysis, some state-feedback control laws for the DSNNM with input and output are designed to stabilize the closed-loop systems. Most RNNs and neurocontrol nonlinear systems with (or without) time delays can be transformed into the DSNNMs to be stability-analyzed or stabilization-synthesized in a unified way. In this paper, the DSNNMs are applied to analyzing the stability of the continuous-time and discrete-time RNNs with or without time delays, and synthesizing the state-feedback controllers for the chaotic neural-network-system and discrete-time nonlinear system. It turns out that the DSNNM makes the stability conditions of the RNNs easily verified, and provides a new idea for the synthesis of the controllers for the nonlinear systems.     

Quantum neural networks (QNNs): inherently fuzzy feedforward neuralnetworks

This paper introduces quantum neural networks (QNNs), a class of feedforward neural networks (FFNNs) inherently capable of estimating the structure of a feature space in the form of fuzzy sets. The hidden units of these networks develop quantized representations of the sample information provided by the training data set in various graded levels of certainty. Unlike other approaches attempting to merge fuzzy logic and neural networks, QNNs can be used in pattern classification problems without any restricting assumptions such as the availability of a priori knowledge or desired membership profile, convexity of classes, a limited number of classes, etc. Experimental results presented here show that QNNs are capable of recognizing structures in data, a property that conventional FFNNs with sigmoidal hidden units lack.     

Extracting rules from trained neural networks

Presents an algorithm for extracting rules from trained neural networks. The algorithm is a decompositional approach which can be applied to any neural network whose output function is monotone such as a sigmoid function. Therefore, the algorithm can be applied to multilayer neural networks, recurrent neural networks and so on. It does not depend on training algorithms, and its computational complexity is polynomial. The basic idea is that the units of neural networks are approximated by Boolean functions. But the computational complexity of the approximation is exponential, and so a polynomial algorithm is presented. The author has applied the algorithm to several problems to extract understandable and accurate rules. The paper shows the results for the votes data, mushroom data, and others. The algorithm is extended to the continuous domain, where extracted rules are continuous Boolean functions. Roughly speaking, the representation by continuous Boolean functions means the representation using conjunction, disjunction, direct proportion, and reverse proportion. This paper shows the results for iris data.     

A Neural-Network-Based Model for the Dynamic Simulation of the Tire/Suspension System While Traversing Road Irregularities

This paper deals with the simulation of the tire/suspension dynamics by using recurrent neural networks (RNNs). RNNs are derived from the multilayer feedforward neural networks, by adding feedback connections between output and input layers. The optimal network architecture derives from a parametric analysis based on the optimal tradeoff between network accuracy and size. The neural network can be trained with experimental data obtained in the laboratory from simulated road profiles (cleats). The results obtained from the neural network demonstrate good agreement with the experimental results over a wide range of operation conditions. The NN model can be effectively applied as a part of vehicle system model to accurately predict elastic bushings and tire dynamics behavior. Although the neural network model, as a black-box model, does not provide a good insight of the physical behavior of the tire/suspension system, it is a useful tool for assessing vehicle ride and noise, vibration, harshness (NVH) performance due to its good computational efficiency and accuracy.      

Prediction of Hydroforming Characteristics using Random Neural Networks

In recent years, hydroforming has become the topic of a lot of active research. Researchers have been looking for better procedures and prediction tools to improve the quality of the product and reduce the prototyping cost. Similar to any other metal forming process, hydroforming leads to non-homogeneous plastic deformations of the workpiece. In this paper, a model is developed to predict the amount of deformation caused by hydroforming using random neural networks (RNNs). RNNs learn the behavior of a system from the provided input/output data in a manner similar to the way the human brain does. This is different from the usual connectionist neural network (NN) models which are based on simple functional analyses. Experimental data is collected and used in training as well as testing the RNNs. The RNN models have feedforward architectures and use a generalized learning algorithm in the training process. Multi-layer RNNs with as few as six neurons were used to capture the nonlinear correlations between the input and output data collected from an experimental setup. The RNN models were able to predict the center deflection, the thickness variation, as well as the deformed shape of circular plate specimens with good accuracy. Received: February 2004 / Accepted: September 2005     

Two Neural Network Construction Methods

Two low complexity methods for neural network construction, that are applicable to various neural network models, are introduced and evaluated for high order perceptrons. The methods are based on a Boolean approximation of real-valued data. This approximation is used to construct an initial neural network topology which is subsequently trained on the original (real-valued) data. The methods are evaluated for their effectiveness in reducing the network size and increasing the network's generalization capabilities in comparison to fully connected high order perceptrons.     

Boolean Factor Analysis by Attractor Neural Network

A common problem encountered in disciplines such as statistics, data analysis, signal processing, textual data representation, and neural network research, is finding a suitable representation of the data in the lower dimension space. One of the principles used for this reason is a factor analysis. In this paper, we show that Hebbian learning and a Hopfield-like neural network could be used for a natural procedure for Boolean factor analysis. To ensure efficient Boolean factor analysis, we propose our original modification not only of Hopfield network architecture but also its dynamics as well. In this paper, we describe neural network implementation of the Boolean factor analysis method. We show the advantages of our Hopfield-like network modification step by step on artificially generated data. At the end, we show the efficiency of the method on artificial data containing a known list of factors. Our approach has the advantage of being able to analyze very large data sets while preserving the nature of the data     

Modeling plasma surface modification of textile fabrics using artificial neural networks

In this paper, a neural network approach is used to understand the effects of fabric features and plasma processing parameters on fabric surface wetting properties. In this approach, fourteen features characterizing woven structures and two plasma parameters are taken as input variables, and the water contact angle cosine and the capillarity height of woven fabrics as output variables. In order to reduce the complexity of the model and effectively learn the network structure from a small number of data, a fuzzy logic based method is used for selecting the most relevant parameters which are taken as input variables of the reduced neural network models. With these relevant parameters, we can effectively control the plasma treatment by selecting the most appropriate fabric materials. Two techniques are used for improving the generalization capability of neural networks: (i) early stopping and (ii) Bayesian regularization. A methodology for optimizing such models is described. The learning abilities and prediction capabilities of the neural net models are compared in terms of different statistical performance criteria. Moreover, a connection weight method is used to determine the relative importance of each input variable in the networks. The obtained results show that neural network models could predict the process performance with reasonable accuracy. However, the neural model trained using Bayesian regularization provides the best results. Thus, it can be concluded that Bayesian network promises to be a valuable quantitative tool to evaluate, understand, and predict woven fabric surface modification by atmospheric air-plasma treatment.