Creating efficient nonlinear neural network process models that allow model interpretation

The KBANN (knowledge-based artificial neural networks) approach uses neural networks to refine knowledge that can be written in the form of simple propositional rules. This idea is extended by presenting the MANNIDENT (multivariable artificial neural network identification) algorithm by which the mathematical equations of linear process models determine the topology and initial weights of a network, which is further trained using backpropagation. This method is applied to the task of modelling a non-isothermal CSTR in which a first-order exothermic reaction is occurring. This method produces statistically significant gains in accuracy over both a standard neural network approach and a linear model. Furthermore, using the approximate linear model to initialize the weights of the network produces statistically less variation in accuracy. By structuring the neural network according to the approximate linear model, the model can be readily interpreted.     

Improvement on Higher-Order Neural Networks for Invariant Object Recognition

The higher order neural network(HONN) was proved to be able to realize invariant object recognition. By taking the relationship between input units into account, HONN's are superior to other neural models in invariant pattern recognition. However, there are two main problems preventing HONN's from practical applications. One is the combinatorial increase of weight number, that is, as input size increases, the number of weights in a HONN increases exponentially. The other problem is sensitivity to distortion and noise. In this paper, we described a method, in which by modifying the constraints imposed on the weights in HONN's, the performance of a HONN with respect to distortion can be improved considerably.     

A combination of linear and nonlinear activation functions in neural networks for modeling a de-superheater

This paper deals with modeling a power plant component with mild nonlinear characteristics using a modified neural network structure. The hidden layer of the proposed neural network has a combination of neurons with linear and nonlinear activation functions. This approach is particularly suitable for nonlinear system with a low grade of nonlinearity, which can not be modeled satisfactorily by neural networks with purely nonlinear hidden layers or by the method of least square of errors (the ideal modeling method of linear systems). In this approach, two channels are installed in a hidden layer of the neural network to cover both linear and nonlinear behavior of systems. If the nonlinear characteristics of the system (i.e. de-superheater) are not negligible, then the nonlinear channel of the neural network is activated; that is, after training, the connections in nonlinear channel get considerable weights. The approach was applied to a de-superheater of a 325 MW power generating plant. The actual plant response, obtained from field experiments, is compared with the response of the proposed model and the responses of linear and neuro-fuzzy models as well as a neural network with purely nonlinear hidden layer. A better accuracy is observed using the proposed approach.     

基于分布式神经网络递推预报误差算法的非线性系统建模

采用基于递推预报误差算法的分布式神经网络 结构建立非线性系统模型．子神经网络模型及其连接权值均采用递推预报误差方法来进行训 练,将所有子网络融合得到的分布式神经网络模型在模型精确性和鲁棒性方面有显著地增加 ．该方法较好地应用于复杂非线性动态系统的建模．     

Linear programming models for traffic engineering in 100% survivable networks under combined IS–IS/OSPF and MPLS-TE

This paper concerns the problem of minimizing the maximum link utilization of IP telecommunication networks under the joint use of traditional IGP routing protocols, such as IS–IS and OSPF, and the more sophisticated MPLS-TE technology. It is shown that the problem of choosing the optimal routing, both under working conditions and under single link failure scenarios, can be cast as a linear program of reasonable size. The proposed model is validated by a computational experimentation performed on synthetic and real networks: the obtained results show that the new approach considerably reduces the maximum link utilization of the network with respect to simply optimizing the IGP weights, at the cost of adding a limited number of label switched paths (LSPs). Optimizing the set of IGP weights within the overall approach further improves performances. The computational time needed to solve the models matches well with real-time requirements, and makes it possible to consider network design problems.     

Parametric system identification using neural networks

Neural networks are used in many applications such as image recognition, classification, control and system identification. However, the parameters of the identified system are embedded within the neural network architecture and are not identified explicitly. In this paper, a mathematical relationship between the network weights and the transfer function parameters is derived. Furthermore, an easy-to-follow algorithm that can estimate the transfer function models for multi-layer feedforward neural networks is proposed. These estimated models provide an insight into the system dynamics, where information such as time response, frequency response, and pole/zero locations can be calculated and analyzed. In order to validate the suitability and accuracy of the proposed algorithm, four different simulation examples are provided and analyzed for three-layer neural network models.     

Heuristic configuration of single hidden-layer feed-forward neural networks

For optimum statistical classification and generalization with single hidden-layer neural network models, two tasks must be performed: (a) learning the best set of weights for a network of k hidden units and (b) determining k, the best complexity fit. We contrast two approaches to construction of neural network classifiers: (a) standard back-propagation as applied to a series of single hidden-layer feed-forward nerual networks with differing number of hidden units and (b) a heuristic cascade-correlation approach that quickly and dynamically configures the hidden units in a network and learns the best set of weights for it. Four real-world applications are considered. On these examples, the back-propagation approach yielded somewhat better results, but with far greater computation times. The best complexity fit, k, for both approaches were quite similar. This suggests a hybrid approach to constructing single hidden-layer feed-forward neural network classifiers in which the number of hidden units is determined by cascade-correlation and the weights are learned by back-propagation.     

Target Recognition Based on Directional Filter Banks and Higher-Order Neural Networks

Park, Sang-il, Smith, Mark J. T., and Mersereau, Russell M., Target Recognition Based on Directional Filter Banks and Higher-Order Neural Networks, Digital Signal Processing10 (2000), 297–308.A new approach for the classification of SAR targets is presented here, which combines maximally decimated directional filter banks with higher-order neural networks (HONNs). HONNs are neural networks that can achieve performance similar to that of standard multilayered neural networks, but without the hidden layer. Their performance can be made invariant to geometric transformations of the input imagery in the design process, while their computational complexity can be reduced by employing a preprocessor to reduce the dimensionality, such as coarse coding. Most past image classifiers using HONNs have been designed for carefully thresholded binary images. However, generating useful binary representations that can be used as inputs can be difficult for modalities such as SAR. As an alternative, we use a novel HONN implementation that accepts gray-level input pixels using directional filter banks. In order to do this, a new modified tree-structured directional filter bank structure in a very computationally efficient form is introduced. The performance of the proposed approach is demonstrated and compared in imagery taken from the public MSTAR database. The new approach is shown to be effective in enhancing the discrimination power of the HONN inputs, leading to significantly improved performance.     

Modeling and sensitivity analysis of neural networks

This paper investigates the use of neural networks for the identification of linear time invariant dynamical systems. Two classes of networks, namely the multilayer feedforward network and the recurrent network with linear neurons, are studied. A notation based on Kronecker product and vector-valued function of matrix is introduced for neural models. It permits to write a feedforward network as a one step ahead predictor used in parameter estimation. A special attention is devoted to system theory interpretation of neural models. Sensitivity analysis can be formulated using derivatives based on the above-mentioned notation.     

Recovering networks from distance data

A fully probabilistic approach to reconstructing Gaussian graphical models from distance data is presented. The main idea is to extend the usual central Wishart model in traditional methods to using a likelihood depending only on pairwise distances, thus being independent of geometric assumptions about the underlying Euclidean space. This extension has two advantages: the model becomes invariant against potential bias terms in the measurements, and can be used in situations which on input use a kernel- or distance matrix, without requiring direct access to the underlying vectors. The latter aspect opens up a huge new application field for Gaussian graphical models, as network reconstruction is now possible from any Mercer kernel, be it on graphs, strings, probabilities or more complex objects. We combine this likelihood with a suitable prior to enable Bayesian network inference. We present an efficient MCMC sampler for this model and discuss the estimation of module networks. Experiments depict the high quality and usefulness of the inferred networks.     

Monotonicity Properties for Multi-Class Queueing Systems

We study multi-dimensional stochastic processes that arise in queueing models used in the performance evaluation of wired and wireless networks. The evolution of the stochastic process is determined by the scheduling policy used in the associated queueing network. For general arrival and service processes, we give sufficient conditions in order to compare sample-path wise the workload and the number of users under different policies. This allows us to evaluate the performance of the system under various policies in terms of stability, the mean overall delay and the mean holding cost. We apply the general framework to linear networks, where users of one class require service from several shared resources simultaneously. For the important family of weighted α-fair policies, stability results are derived and monotonicity of the mean holding cost with respect to the fairness parameter α and the relative weights is established. In order to broaden the comparison results, we investigate a heavy-traffic regime and perform numerical experiments. In addition, we study a single-server queue with two user classes, and show that under Discriminatory Processor Sharing (DPS) or Generalized Processor Sharing (GPS) the mean overall sojourn time is monotone with respect to the ratio of the weights. Finally we extend the framework to obtain comparison results that cover the single-server queue with an arbitrary number of classes as well.     

Network capacity analysis for latent attractor computation

Attractor networks have been one of the most successful paradigms in neural computation, and have been used as models of computation in the nervous system. Recently, we proposed a paradigm called 'latent attractors' where attractors embedded in a recurrent network via Hebbian learning are used to channel network response to external input rather than becoming manifest themselves. This allows the network to generate context-sensitive internal codes in complex situations. Latent attractors are particularly helpful in explaining computations within the hippocampus--a brain region of fundamental significance for memory and spatial learning. Latent attractor networks are a special case of associative memory networks. The model studied here consists of a two-layer recurrent network with attractors stored in the recurrent connections using a clipped Hebbian learning rule. The firing in both layers is competitive--K winners take all firing. The number of neurons allowed to fire, K, is smaller than the size of the active set of the stored attractors. The performance of latent attractor networks depends on the number of such attractors that a network can sustain. In this paper, we use signal-to-noise methods developed for standard associative memory networks to do a theoretical and computational analysis of the capacity and dynamics of latent attractor networks. This is an important first step in making latent attractors a viable tool in the repertoire of neural computation. The method developed here leads to numerical estimates of capacity limits and dynamics of latent attractor networks. The technique represents a general approach to analyse standard associative memory networks with competitive firing. The theoretical analysis is based on estimates of the dendritic sum distributions using Gaussian approximation. Because of the competitive firing property, the capacity results are estimated only numerically by iteratively computing the probability of erroneous firings. The analysis contains two cases: the simple case analysis which accounts for the correlations between weights due to shared patterns and the detailed case analysis which includes also the temporal correlations between the network's present and previous state. The latter case predicts better the dynamics of the network state for non-zero initial spurious firing. The theoretical analysis also shows the influence of the main parameters of the model on the storage capacity.     

Fusing Neural Networks Through Space Partitioning and Fuzzy Integration

To improve recognition results, decisions of multiple neural networks can be aggregated into a committee decision. Aggregation weights assigned to neural networks or groups of networks can be the same in the entire data space or can be different (data dependent) in various regions of the space. In this paper, we propose a method for obtaining data dependent aggregation weights. The proposed approach is tested in two aggregation schemes, namely aggregation through neural network selection, and aggregation by the Choquet integral with respect to the -fuzzy measure. The effectiveness of the approach is demonstrated on two artificial and three real data sets.     

Neural Net-Based H ∞ Control for a Class of Nonlinear Systems

A novel neural net-based approach for H _∞ control design of a class of nonlinear continuous-time systems is presented. In the proposed frameworks, the nonlinear system models are approximated by multilayer neural networks. The neural networks are piecewisely interpolated to generate a linear differential inclusion models by which a linear state feedback H _∞ control law can be constructed. It is shown that finding the permissible control gain matrices can be transformed to a standard linear matrix inequality problem and solved using the available computer software. This revised version was published online in June 2006 with corrections to the Cover Date.     

Using single layer networks for discrete, sequential data: An example from Natural Language Processing

Natural Language Processing (NLP) is concerned with processing ordinary, unrestricted text. This work takes a new approach to a traditional NLP task, using neural computing methods. A parser which has been successfully implemented is described. It is a hybrid system, in which neural processors operate within a rule based framework. The neural processing components belong to the class of Generalized Single Layer Networks (GSLN). In general, supervised, feed-forward networks need more than one layer to process data. However, in some cases data can be pre-processed with a non-linear transformation, and then presented in a linearly separable form for subsequent processing by a single layer net. Such networks offer advantages of functional transparency and operational speed. For our parser, the initial stage of processing maps linguistic data onto a higher order representation, which can then be analysed by a single layer network. This transformation is supported by information theoretic analysis. Three different algorithms for the neural component were investigated. Single layer nets can be trained by finding weight adjustments based on (a) factors proportional to the input, as in the Perceptron, (b) factors proportional to the existing weights, and (c) an error minimization method. In our experiments generalization ability varies little; method (b) is used for a prototype parser. This is available via telnet.     

Neural Networks with Image Recognition by Pairs

Neural networks based on metric recognition methods have a strictly determined architecture. Number of neurons, connections, as well as weights and thresholds values are calculated analytically, based on the initial conditions of tasks: number of recognizable classes, number of samples, metric expressions used. This paper discusses the possibility of transforming these networks in order to apply classical learning algorithms to them without using analytical expressions that calculate weight values. In the received network, training is carried out by recognizing images in pairs. This approach simplifies the learning process and easily allows to expand the neural network by adding new images to the recognition task. The advantages of these networks, including such as: (1) network architecture simplicity and transparency; (2) training simplicity and reliability; (3) the possibility of using a large number of images in the recognition problem using a neural network; (4) a consistent increase in the number of recognizable classes without changing the previous values of weights and thresholds.     

基于神经网络的机电伺服系统非线性控制

针对机电伺服系统精确动力学模型难以获取以及系统状态信息的测量易受噪声影响的问题,设计一种基于指令滤波与神经网络相结合的非线性反步控制算法,该算法能够有效地补偿未建模动态和外部扰动对机电伺服系统的影响.首先,引入指令滤波器来获取已知信号的微分估计并处理噪声;其次,利用神经网络估计未知的系统动态,包括未建模的摩擦和外部干扰;然后,神经网络权值的更新律通过梯度下降算法在线实现,没有离线学习阶段;最后,利用李雅普诺夫函数分析方法证明闭环系统的稳定性.为验证所提出算法的有效性,在机电伺服实验平台上进行大量对比实验,实验结果表明,所提出的算法具有良好的控制效果,对系统不确定性和外部干扰具有良好的鲁棒性.     

A rapid learning and dynamic stepwise updating algorithm for flatneural networks and the application to time-series prediction

A fast learning algorithm is proposed to find an optimal weights of the flat neural networks (especially, the functional-link network). Although the flat networks are used for nonlinear function approximation, they can be formulated as linear systems. Thus, the weights of the networks can be solved easily using a linear least-square method. This formulation makes it easier to update the weights instantly for both a new added pattern and a new added enhancement node. A dynamic stepwise updating algorithm is proposed to update the weights of the system on-the-fly. The model is tested on several time-series data including an infrared laser data set, a chaotic time-series, a monthly flour price data set, and a nonlinear system identification problem. The simulation results are compared to existing models in which more complex architectures and more costly training are needed. The results indicate that the proposed model is very attractive to real-time processes.     

An optimized instance based learning algorithm for estimation of compressive strength of concrete

This article proposes an optimized instance-based learning approach for prediction of the compressive strength of high performance concrete based on mix data, such as water to binder ratio, water content, super-plasticizer content, fly ash content, etc. The base algorithm used in this study is the k nearest neighbor algorithm, which is an instance-based machine leaning algorithm. Five different models were developed and analyzed to investigate the effects of the number of neighbors, the distance function and the attribute weights on the performance of the models. For each model a modified version of the differential evolution algorithm was used to find the optimal model parameters. Moreover, two different models based on generalized regression neural network and stepwise regressions were also developed. The performances of the models were evaluated using a set of high strength concrete mix data. The results of this study indicate that the optimized models outperform those derived from the standard k nearest neighbor algorithm, and that the proposed models have a better performance in comparison to generalized regression neural network, stepwise regression and modular neural networks models.     

Discontinuities in recurrent neural networks

This article studies the computational power of various discontinuous real computational models that are based on the classical analog recurrent neural network (ARNN). This ARNN consists of finite number of neurons; each neuron computes a polynomial net function and a sigmoid-like continuous activation function. We introduce arithmetic networks as ARNN augmented with a few simple discontinuous (e.g., threshold or zero test) neurons. We argue that even with weights restricted to polynomial time computable reals, arithmetic networks are able to compute arbitrarily complex recursive functions. We identify many types of neural networks that are at least as powerful as arithmetic nets, some of which are not in fact discontinuous, but they boost other arithmetic operations in the net function (e.g., neurons that can use divisions and polynomial net functions inside sigmoid-like continuous activation functions). These arithmetic networks are equivalent to the Blum-Shub-Smale model, when the latter is restricted to a bounded number of registers. With respect to implementation on digital computers, we show that arithmetic networks with rational weights can be simulated with exponential precision, but even with polynomial-time computable real weights, arithmetic networks are not subject to any fixed precision bounds. This is in contrast with the ARNN that are known to demand precision that is linear in the computation time. When nontrivial periodic functions (e.g., fractional part, sine, tangent) are added to arithmetic networks, the resulting networks are computationally equivalent to a massively parallel machine. Thus, these highly discontinuous networks can solve the presumably intractable class of PSPACE-complete problems in polynomial time.