A methodology for building neural networks models from empirical engineering data

Neural networks (NN) are general tools for modeling functional relationships in engineering. They are used to model the behavior of products and the properties of processes. Nevertheless, their use is often ad hoc. This paper provides a sound basis for using NN as tools for modeling functional relationships implicit in empirical engineering data. First, a clear definition of a modeling task is given, followed by reviewing the theoretical modeling capabilities of NN and NN model estimation. Subsequently, a procedure for using NN in engineering practice is described and illustrated with an example of modeling marine propeller behavior. Particular attention is devoted to better estimation of model quality, insight into the influence of measurement errors on model quality, and the use of advanced methods such as stacked generalization and ensemble modeling to further improve model quality. Using a new method of ensemble of SG(k-NN), one could improve the quality of models even if they are close to being optimal.     

Gompertz software reliability model: Estimation algorithm and empirical validation

Gompertz curve has been used to estimate the number of residual faults in testing phases of software development, especially by Japanese software development companies. Since the Gompertz curve is a deterministic function, the curve cannot be applied to estimating software reliability which is the probability that software system does not fail in a prefixed time period. In this article, we propose a stochastic model called the Gompertz software reliability model based on non-homogeneous Poisson processes. The proposed model can be derived from the statistical theory of extreme-value, and has a similar asymptotic property to the deterministic Gompertz curve. Also, we develop an EM algorithm to determine the model parameters effectively. In numerical examples with software failure data observed in real software development projects, we evaluate performance of the Gompertz software reliability model in terms of reliability assessment and failure prediction.     

Developing empirical models from observational data using artificial neural networks

Computer-integrated manufacturing requires models of manufacturing processes to be implemented on the computer. Process models are required for designing adaptive control systems and selecting optimal parameters during process planning. Mechanistic models developed from the principles of machining science are useful for implementing on a computer. However, in spite of the progress being made in mechanistic process modeling, accurate models are not yet available for many manufacturing processes. Empirical models derived from experimental data still play a major role in manufacturing process modeling. Generally, statistical regression techniques are used for developing such models. However, these techniques suffer from several disadvantages. The structure (the significant terms) of the regression model needs to be decided a priori. These techniques cannot be used for incrementally improving models as new data becomes available. This limitation is particularly crucial in light of the advances in sensor technology that allow economical on-line collection of manufacturing data. In this paper, we explore the use of artificial neural networks (ANN) for developing empirical models from experimental data for a machining process. These models are compared with polynomial regression models to assess the applicability of ANN as a model-building tool for computer-integrated manufacturing.Operated for the United States Department of Energy under contract No. DE-AC04-76-DP00613.     

A novel method for learning policies from variable constraint data

Many everyday human skills can be framed in terms of performing some task subject to constraints imposed by the environment. Constraints are usually unobservable and frequently change between contexts. In this paper, we present a novel approach for learning (unconstrained) control policies from movement data, where observations come from movements under different constraints. As a key ingredient, we introduce a small but highly effective modification to the standard risk functional, allowing us to make a meaningful comparison between the estimated policy and constrained observations. We demonstrate our approach on systems of varying complexity, including kinematic data from the ASIMO humanoid robot with 27 degrees of freedom, and present results for learning from human demonstration.     

A learning method of immune multi-agent neural networks

Many different learning algorithms for neural networks have been developed, with advantages offered in terms of network structure, initial values of some parameters, learning speed, and learning accuracy. If we train the networks only on good examples, without noise and shortage, the neural network can be trained to classify, with reasonable accuracy, target patterns or random patterns, but not both. To solve this problem, we propose a learning method of immune multi-agent neural networks (IMANNs), which have agents of macrophages, B-cells and T-cells. Each agent employs a different type of neural network. Because the agents work cooperatively and competitively, IMANNs can automatically classify the training dataset into some subclasses. In this paper, two types of IMANNs are described and their classification capabilities are compared. In order to verify the effectiveness of our proposed method, we used two datasets: the dataset of the MONKs problem (as a traditional classification problem) and a dataset from a medical diagnosis problem (hepatobiliary disorders).     

Iterative algorithm of wavelet network learning from nonuniform data

The learning algorithm based on multiresolution analysis (LAMA) is a powerful tool for wavelet networks. It has many advantages over other algorithms, but it seldom does well in the learning of nonuniform data. A new algorithm is proposed to solve this problem, which develops from the learning algorithm based on sampling theory (LAST). From the good concentration of wavelet energy, we discuss the approximation capacity of wavelet network in the local domain when the training data are not dense enough. From this discussion, the new algorithm is realized by the iterative application of LAST. The corresponding theorems based on the sampling theory are also proposed to prove the rationality of new algorithm. In the simulation, we compare the performance of new algorithm with that of LAMA and LAST. The results show that our new algorithm has as many advantages as LAMA and LAST, does better in the learning of nonuniform data and has high approximation accuracy.     

A fast algorithm for learning a ranking function from large-scale data sets

We consider the problem of learning the ranking function that maximizes a generalization of the Wilcoxon-Mann-Whitney statistic on the training data. Relying on an $\epsilon$-accurate approximation for the error-function, we reduce the computational complexity of each iteration of a conjugate gradient algorithm for learning ranking functions from O(m2) to O(m2), where m is the number of training samples. Experiments on public benchmarks for ordinal regression and collaborative filtering indicate that the proposed algorithm is as accurate as the best available methods in terms of ranking accuracy, when the algorithms are trained on the same data. However, since it is several orders of magnitude faster than the current state-of-the-art approaches, it is able to leverage much larger training datasets.     

A fast learning algorithm for training feedforward neural networks

This paper deals with a fast and computationally simple Successive Over-relaxation Resilient Backpropagation (SORRPROP) learning algorithm which has been developed by modifying the Resilient Backpropagation (RPROP) algorithm. It uses latest computed values of weights between the hidden and output layers to update remaining weights. The modification does not add any extra computation in RPROP algorithm and maintains its computational simplicity. Classification and regression simulations examples have been used to compare the performance. From the test results for the examples undertaken it is concluded that SORRPROP has small convergence times and better performance in comparison to other first-order learning algorithms.     

A general backpropagation algorithm for feedforward neural networks learning

A general backpropagation algorithm is proposed for feedforward neural network learning with time varying inputs. The Lyapunov function approach is used to rigorously analyze the convergence of weights, with the use of the algorithm, toward minima of the error function. Sufficient conditions to guarantee the convergence of weights for time varying inputs are derived. It is shown that most commonly used backpropagation learning algorithms are special cases of the developed general algorithm.     

Method of probabilistic inference from learning data in Bayesian networks

Bayesian networks (BN) are a powerful tool for various data-mining systems. The available methods of probabilistic inference from learning data have shortcomings such as high computation complexity and cumulative error. This is due to a partial loss of information in transition from empiric information to conditional probability tables. The paper presents a new simple and exact algorithm for probabilistic inference in BN from learning data. __________ Translated from Kibernetika i Sistemnyi Analiz, No. 3, pp. 93–99, May–June 2007.     

A conjugate gradient learning algorithm for recurrent neural networks

The real-time recurrent learning (RTRL) algorithm, which is originally proposed for training recurrent neural networks, requires a large number of iterations for convergence because a small learning rate should be used. While an obvious solution to this problem is to use a large learning rate, this could result in undesirable convergence characteristics. This paper attempts to improve the convergence capability and convergence characteristics of the RTRL algorithm by incorporating conjugate gradient computation into its learning procedure. The resulting algorithm, referred to as the conjugate gradient recurrent learning (CGRL) algorithm, is applied to train fully connected recurrent neural networks to simulate a second-order low-pass filter and to predict the chaotic intensity pulsations of NH₃ laser. Results show that the CGRL algorithm exhibits substantial improvement in convergence (in terms of the reduction in mean squared error per epoch) as compared to the RTRL and batch mode RTRL algorithms.     

A globally convergent learning algorithm for PCA neural networks

Principal component analysis (PCA) by neural networks is one of the most frequently used feature extracting methods. To process huge data sets, many learning algorithms based on neural networks for PCA have been proposed. However, traditional algorithms are not globally convergent. In this paper, a new PCA learning algorithm based on cascade recursive least square (CRLS) neural network is proposed. This algorithm can guarantee the network weight vector converges to an eigenvector associated with the largest eigenvalue of the input covariance matrix globally. A rigorous mathematical proof is given. Simulation results show the effectiveness of the algorithm.     

A neural networks algorithm for data path synthesis

This paper presents a deterministic parallel algorithm to solve the data path allocation problem in high-level synthesis. The algorithm is driven by a motion equation that determines the neurons firing conditions based on the modified Hopfield neural network model of computation. The method formulates the allocation problem using the clique partitioning problem, an NP-complete problem, and handles multicycle functional units as well as structural pipelining. The algorithm has a running time complexity of O(1) for a circuit with n operations and c shared resources. A sequential simulator was implemented on a Linux Pentium PC under X-Windows. Several benchmark examples have been implemented and favorable design comparisons to other synthesis systems are reported.     

A machine learning method for extracting symbolic knowledge from recurrent neural networks

Neural networks do not readily provide an explanation of the knowledge stored in their weights as part of their information processing. Until recently, neural networks were considered to be black boxes, with the knowledge stored in their weights not readily accessible. Since then, research has resulted in a number of algorithms for extracting knowledge in symbolic form from trained neural networks. This article addresses the extraction of knowledge in symbolic form from recurrent neural networks trained to behave like deterministic finite-state automata (DFAs). To date, methods used to extract knowledge from such networks have relied on the hypothesis that networks' states tend to cluster and that clusters of network states correspond to DFA states. The computational complexity of such a cluster analysis has led to heuristics that either limit the number of clusters that may form during training or limit the exploration of the space of hidden recurrent state neurons. These limitations, while necessary, may lead to decreased fidelity, in which the extracted knowledge may not model the true behavior of a trained network, perhaps not even for the training set. The method proposed here uses a polynomial time, symbolic learning algorithm to infer DFAs solely from the observation of a trained network's input-output behavior. Thus, this method has the potential to increase the fidelity of the extracted knowledge.     

A simulation-based study of TCP dynamics over HFC networks

New broadband access technologies such as hybrid fiber coaxial (HFC) are likely to provide fast and cost effective support to a variety of applications including video on demand (VoD), inter-active computer games, and Internet-type applications such as Web browsing, ftp, e-mail, and telephony. Since most of these applications use TCP as the transport layer protocol, the key to their efficiency largely depends on TCP protocol performance.We investigate the performance of TCP in terms of effective throughput in an HFC network environment using different load conditions and network buffer sizes. We find that TCP experiences low throughput as a result of the well-known problem of ACK compression. An algorithm that controls ACK spacing is introduced to improve TCP performance.     

A method for selecting similar learning data in the prediction of time series using neural networks

This paper explores a method of improving the predictive performance by the multi-layer feedforward neural network in time series predicting. For the similar data selective learning method, we propose a method of weighting the distance by a power function of correlation coefficients for the time series (CSDS method). The results of numerical experiments show that with the case of a time series whose nature is rather choppy or chaotic, using the CSDS method appropriately is considerably effective to improve the predictive performance and its performance is considerably better than that by the previously proposed other methods.     

A heavy-tailed empirical Bayes method for replicated microarray data

DNA microarray has been recognized as being an important tool for studying the expression of thousands of genes simultaneously. These experiments allow us to compare two different samples of cDNA obtained under different conditions. A novel method for the analysis of replicated microarray experiments based upon the modelling of gene expression distribution as a mixture of α-stable distributions is presented. Some features of the distribution of gene expression, such as Pareto tails and the fact that the variance of any given array increases concomitantly with an increase in the number of genes studied, suggest the possibility of modelling gene expression distribution on the basis of α-stable density. The proposed methodology uses very well known properties of α-stable distribution, such as the scale mixture of normals. A Bayesian log-posterior odds is calculated, which allows us to decide whether a gene is expressed differentially or not. The proposed methodology is illustrated using simulated and experimental data and the results are compared with other existing statistical approaches. The proposed heavy-tail model improves the performance of other distributions and is easily applicable to microarray gene data, specially if the dataset contains outliers or presents high variance between replicates.