A taxonomy for wavelet neural networks applied to nonlinear modelling

This article presents a novel classification of wavelet neural networks based on the orthogonality/non-orthogonality of neurons and the type of nonlinearity employed. On the basis of this classification different network types are studied and their characteristics illustrated by means of simple one-dimensional nonlinear examples. For multidimensional problems, which are affected by the curse of dimensionality, the idea of spherical wavelet functions is considered. The behaviour of these networks is also studied for modelling of a low-dimension map.     

Artificial neural networks applied to odor classification for chemical compounds

Three-layer artificial neural networks (ANN) with back-propagation of error were used to classify the odors of chemical compounds. The network's architecture and parameters were optimized, and an empirical rule for dynamically adjusting the learning rate of network was put forward. It was found that the network gave the maximum recognition rate of 100% and converged quickly by using the plural semiconductor gas sensors' response data in combination with the molecular structure codes of odorants. The network's predictive ability for untrained samples was also satisfactory.     

CPBUM neural networks for modeling with outliers and noise

In this study, CPBUM neural networks with annealing robust learning algorithm (ARLA) are proposed to improve the problems of conventional neural networks for modeling with outliers and noise. In general, the obtained training data in the real applications maybe contain the outliers and noise. Although the CPBUM neural networks have fast convergent speed, these are difficult to deal with outliers and noise. Hence, the robust property must be enhanced for the CPBUM neural networks. Additionally, the ARLA can be overcome the problems of initialization and cut-off points in the traditional robust learning algorithm and deal with the model with outliers and noise. In this study, the ARLA is used as the learning algorithm to adjust the weights of the CPBUM neural networks. It tunes out that the CPBUM neural networks with the ARLA have fast convergent speed and robust against outliers and noise than the conventional neural networks with robust mechanism. Simulation results are provided to show the validity and applicability of the proposed neural networks.     

Numeric sensitivity analysis applied to feedforward neural networks

During the last 10 years different interpretative methods for analysing the effect or importance of input variables on the output of a feedforward neural network have been proposed. These methods can be grouped into two sets: analysis based on the magnitude of weights; and sensitivity analysis. However, as described throughout this study, these methods present a series of limitations. We have defined and validated a new method, called Numeric Sensitivity Analysis (NSA), that overcomes these limitations, proving to be the procedure that, in general terms, best describes the effect or importance of the input variables on the output, independently of the nature (quantitative or discrete) of the variables included. The interpretative methods used in this study are implemented in the software program Sensitivity Neural Network 1.0, created by our team.     

Back-propagation neural networks for modeling complex systems

In complex engineering systems, empirical relationships are often employed to estimate design parameters and engineering properties. A complex domain is characterized by a number of interacting factors and their relationships are, in general, not precisely known. In addition, the data associated with these parameters are usually incomplete or erroneous (noisy). The development of these empirical relationships is a formidable task requiring sophisticated modeling techniques as well as human intuition and experience. This paper demonstrates the use of back-propagation neural networks to alleviate this problem. Backpropagation neural networks are a product of artificial intelligence research. First, an overview of the neural network methodology is presented. This is followed by some practical guidelines for implementing back-propagation neural networks. Two examples are then presented to demonstrate the potential of this approach for capturing nonlinear interactions between variables in complex engineering systems.     

On the dynamical modeling with neural fuzzy networks.

In the literature, researchers have introduced delay feedback (or recurrent) networks and claimed that those networks could accurately model dynamical systems without knowing their system orders. In this paper, we have studied those delay feedback networks and also proposed a better version of delay feedback neural-fuzzy networks, called additive delay feedback neural-fuzzy networks (ADFNFN). From our simulations for various examples, it is clearly evident that ADFNFN can have the best modeling accuracy among those existing delay feedback networks. Nevertheless, we also showed by examples that those delay feedback networks can only reach the accuracy of nonlinear autoregressive with exogenous inputs (NARX) models with order two, and that the number of delays in delay feedback networks plays the same role as the order in NARX models.     

Artificial neural networks applied to fetal monitoring in labour

Birth asphyxia can result in death or permanent brain damage. To prevent it, the fetal heart rate (FHR) is recorded in labour on a paper strip. In clinical practice, the complicated FHR patterns are assessed by eye, which is error-prone, inconsistent and unreliable. Objective alternatives are needed and thus we investigated the applicability of feed-forward artificial neural networks (ANNs) for FHR analysis. Six FHR features were extracted and combined with six clinical parameters to form a feature space of 12 dimensions. The feature space was reduced to six dimensions by principal component analysis. Subsequently, a network committee of ten ANNs was trained with the data of 124 patients (a balanced set of 62 adverse, coded 1, and 62 normal outcomes, coded 0). The ANN committee was tested on another balanced set of 252 patients obtaining misclassification rate of 36%. Finally, the committee was tested on a large dataset of 7,568 patients (non-balanced). As the committee output continuously increased from 0 to 1, there was a consistent growth of the adverse outcome rate (from 0.26 to 5.3%) and the low umbilical pH rate (from 2.6 to 16.7%.) Based on this correlation between the committee output and the risk of compromise, we concluded that ANNs can be successfully applied to FHR monitoring in labour. However, extensive further work is necessary, for which we outline our plans. To our knowledge, this is the first time that an automated method for FHR diagnostic analysis has been tested on a database of this size.     

Techniques for biased data distributions and variable classification with neural networks applied to otoneurological data

It is frequently useful and advantageous to investigate not only the classification efficacy of neural networks, but also the reasons for misclassification and relations between input variables and output classes. We have developed novel techniques to disentangle these dilemmas: a network structure and learning strategy for biased output class distributions, a method to measure the classification information incorporated in variables and variable groups, and methods to express properties learned by a network from its structure. We tested these techniques with otoneurological data from the conjunction with vertiginous diseases that we have explored in our previous neural network studies.     

Lower estimation of approximation rate for neural networks

Let SFd and Πψ,n,d = { nj=1bjψ(ωj·x+θj) :bj,θj∈R,ωj∈Rd} be the set of periodic and Lebesgue’s square-integrable functions and the set of feedforward neural network (FNN) functions, respectively. Denote by dist (SF d, Πψ,n,d) the deviation of the set SF d from the set Πψ,n,d. A main purpose of this paper is to estimate the deviation. In particular, based on the Fourier transforms and the theory of approximation, a lower estimation for dist (SFd, Πψ,n,d) is proved. That is, dist(SF d, Πψ,n,d) (nlogC2n)1/2 . T...     

Trend time-series modeling and forecasting with neural networks.

Despite its great importance, there has been no general consensus on how to model the trends in time-series data. Compared to traditional approaches, neural networks (NNs) have shown some promise in time-series forecasting. This paper investigates how to best model trend time series using NNs. Four different strategies (raw data, raw data with time index, detrending, and differencing) are used to model various trend patterns (linear, nonlinear, deterministic, stochastic, and breaking trend). We find that with NNs differencing often gives meritorious results regardless of the underlying data generating processes (DGPs). This finding is also confirmed by the real gross national product (GNP) series.     

A grammar-based GP approach applied to the design of deep neural networks

Genetic Programming and Evolvable Machines - Deep Learning has been very successful in automating the feature engineering process, widely applied for various tasks, such as speech recognition,...     

Compression of recurrent neural networks for efficient language modeling

Recurrent neural networks have proved to be an effective method for statistical language modeling. However, in practice their memory and run-time complexity are usually too large to be implemented in real-time offline mobile applications. In this paper we consider several compression techniques for recurrent neural networks including Long–Short Term Memory models. We make particular attention to the high-dimensional output problem caused by the very large vocabulary size. We focus on effective compression methods in the context of their exploitation on devices: pruning, quantization, and matrix decomposition approaches (low-rank factorization and tensor train decomposition, in particular). For each model we investigate the trade-off between its size, suitability for fast inference and perplexity. We propose a general pipeline for applying the most suitable methods to compress recurrent neural networks for language modeling. It has been shown in the experimental study with the Penn Treebank (PTB) dataset that the most efficient results in terms of speed and compression–perplexity balance are obtained by matrix decomposition techniques.     

Artificial neural networks for modeling in reaction process systems

This work is focused on developing a more efficient computational scheme for estimation of process reaction rates based on NN models. Two scenarios are considered: (1) the kinetics coefficients of the process are completely known and the process states are partly known (measured); (2) the kinetics coefficients and the states of the process are partly known. The contribution of the paper is twofold. From one side we formulate a hybrid (ANN and mechanistic) model that outperforms the traditional reaction rate estimation approaches. From other side, a new procedure for NN supervised training is proposed when target outputs are not available. The two scenarios are successfully tested for two benchmark problems, estimation of the precipitation rate of calcium phosphate and estimation of sugar crystallization growth rate.     

Hybridization of the probabilistic neural networks with feed-forward neural networks for forecasting

Feed-forward neural networks (FFNNs) are among the most important neural networks that can be applied to a wide range of forecasting problems with a high degree of accuracy. Several large-scale forecasting competitions with a large number of commonly used time series forecasting models conclude that combining forecasts from more than one model often leads to improved performance, especially when the models in the ensemble are quite different. In the literature, several hybrid models have been proposed by combining different time series models together. In this paper, in contrast of the traditional hybrid models, a novel hybridization of the feed-forward neural networks (FFNNs) is proposed using the probabilistic neural networks (PNNs) in order to yield more accurate results than traditional feed-forward neural networks. In the proposed model, the estimated values of the FFNN models are modified based on the distinguished trend of their residuals and optimum step length, which are respectively yield from a probabilistic neural network and a mathematical programming model. Empirical results with three well-known real data sets indicate that the proposed model can be an effective way in order to construct a more accurate hybrid model than FFNN models. Therefore, it can be applied as an appropriate alternative model for forecasting tasks, especially when higher forecasting accuracy is needed.     

Exponential convergence rate estimation for neutral BAM neural networks with mixed time-delays

This paper is concerned with the exponential stability analysis problem for a class of neutral bidirectional associative memory neural networks with mixed time-delays, where discrete, distributed and neutral delays are involved. By utilizing the delay decomposition approach and an appropriately constructed Lyapunov–Krasovskii functional, some novel delay-dependent and decay rate-dependent criteria for the exponential stability of the considered neural networks are derived and presented in terms of linear matrix inequalities. Furthermore, the maximum allowable decay rate can be estimated based on the obtained results. Three numerical examples are given to demonstrate the effectiveness of the proposed method.     

Public transportation trip flow modeling with generalized regression neural networks

Artificial neural networks (ANNs) are one of the recently explored advanced technologies, which show promise in the area of transportation engineering. The presented study comprised the employment of this seldom used ANN method, generalized regression neural network (GRNN), in comparison to both a frequently applied neural network training algorithm, feed-forward back-propagation (FFBP), and a stochastic model of auto-regressive structure for the purpose of forecasting daily trip flows, which is an essential component in demand analysis. The study is carried out under the motivation of knowing that modeling daily trips for available transportation modes will facilitate the arrangement for effective public infrastructure investments and the cited papers in the literature did not make use of and handle any comparison with GRNN method. The ANN predictions are found to be quite close to the observations as reflected in the selected performance criteria. The selected stochastic model performance is quite poor compared with ANN results. It is seen that the GRNN did not provide negative forecasts in contrast to FFBP applications. Besides, the local minima problem faced by FFBP algorithm is not encountered in GRNNs.     

On fuzzy modeling using fuzzy neural networks with theback-propagation algorithm

A fuzzy modeling method using fuzzy neural networks with the backpropagation algorithm is presented. The method can identify the fuzzy model of a nonlinear system automatically. The feasibility of the method is examined using simple numerical data.     

Large-scale user modeling with recurrent neural networks for music discovery on multiple time scales

The amount of content on online music streaming platforms is immense, and most users only access a tiny fraction of this content. Recommender systems are the application of choice to open up the collection to these users. Collaborative filtering has the disadvantage that it relies on explicit ratings, which are often unavailable, and generally disregards the temporal nature of music consumption. On the other hand, item co-occurrence algorithms, such as the recently introduced word2vec-based recommenders, are typically left without an effective user representation. In this paper, we present a new approach to model users through recurrent neural networks by sequentially processing consumed items, represented by any type of embeddings and other context features. This way we obtain semantically rich user representations, which capture a user’s musical taste over time. Our experimental analysis on large-scale user data shows that our model can be used to predict future songs a user will likely listen to, both in the short and long term.