Learning algorithms for neural networks with the Kalman filters

Based on various approaches, several different learing algorithms have been given in the literature for neural networks. Almost all algorithms have constant learning rates or constant accelerative parameters, though they have been shown to be effective for some practical applications. The learning procedure of neural networks can be regarded as a problem of estimating (or identifying) constant parameters (i.e. connection weights of network) with a nonlinear or linear observation equation. Making use of the Kalman filtering, we derive a new back-propagation algorithm whose learning rate is computed by a time-varying Riccati difference equation. Perceptron-like and correlational learning algorithms are also obtained as special cases. Furthermore, a self-organising algorithm of feature maps is constructed within a similar framework.     

GNSS signals processing via linear and extended Kalman filters

In this paper, a new linear estimator that enables solution of the user position estimation problem has been designed for the case of global navigation satellite system (GNSS) utilization. The proposed recursive linear Kalman filter (LKF) does not need any linearization for nonlinear GNSS measurements and enables accurate estimation of the user coordinates and the distance measuring errors resulting from the time differences between the user and the satellite (clock bias). Comparison of the presented linear and conventional extended Kalman filters for the stationary and moving users' position estimation via GNSS measurements is performed. Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

General two-stage extended Kalman filters

A general two-stage extended Kalman filter (GTSEKF), which extends the linear general two-stage Kalman filter to nonlinear systems, is further proposed. A new nonlinear two-stage transformation is introduced to facilitate achieving this extension. As in the linear one, the GTSEKF is derived mainly by applying the nonlinear two-stage transformation to the well-known extended Kalman filter (EKF), and is shown to be equivalent to the EKF with a decoupled computing structure. A nonlinear filter for estimating constant parameters in dynamic systems is presented to illustrate one application of the proposed GTSEKF. A literature example is also given to demonstrate the correctness and usefulness of the proposed results.     

An improvement of a back propagation algorithm by the extended Kalman filter and demand forecasting by layered neural networks

Recently, Watanabe et al. proposed a back propagation algorithm via the extended Kalman filter, in which the learning rate was time-varying. In their algorithm the weights and biases are treated as independent variables. It is, however, natural that the weights and biases are not always independent, and generally have mutual correlation. In this paper, we improve the back propagation algorithm by considering that there is mutual correlation among the weights and bias directly connected to the unit. Through some numerical examples, our improved learning algorithm is compared with Watanabe et al.'s algorithm in learning ability. Furthermore, we consider demand forecasting as a kind of pattern recognition, and propose a demand forecasting method using layered neural networks with the improved learning algorithm. The effectiveness of this demand forecasting method is also discussed through some simulations.     

An augmented extended Kalman filter algorithm for complex-valued recurrent neural networks

An augmented complex-valued extended Kalman filter (ACEKF) algorithm for the class of nonlinear adaptive filters realized as fully connected recurrent neural networks is introduced. This is achieved based on some recent developments in the so-called augmented complex statistics and the use of general fully complex nonlinear activation functions within the neurons. This makes the ACEKF suitable for processing general complex-valued nonlinear and nonstationary signals and also bivariate signals with strong component correlations. Simulations on benchmark and real-world complex-valued signals support the approach.     

基于扩展卡尔曼神经网络算法估计电池SOC

针对汽车锂电池的荷电状态(SOC)的问题,基于Thevenin电路为等效电路并且应用扩展卡尔曼算法(EKF)结合神经网络算法进行估计。在进行卡尔曼滤波算法估算过程中,需要用到实时的估算模型参数值(最新值),即在不同的SOC下模型的参数不同。传统做法是把SOC与各个参数的关系进行普通的拟合,这种方法在拟合过程中存在较大误差。为了解决这个问题,利用神经网络拟合各个电路模型参数与SOC关系曲线。试验结果表明,与单纯的扩展卡尔曼算法相比,该方法能够准确估计电池剩余电量,误差小于3%。     

Mapping extended Kalman filters onto linear arrays

Techniques for mapping extended Kalman filters onto linear arrays of programmable cells designed for real-time applications are described. First, a general method for mapping a standard (nonsquare root) Kalman filter, where the columns of the covariance matrix are updated in parallel, is introduced. Next, a general method for mapping a factorized (square root) filter, where fast Givens rotations are used to triangularize the prematrix and where rotations of the rows of the prematrix are performed in parallel, is introduced. These mappings are used to implement an extended Kalman filter commonly used in target tracking applications on the Warp computer. The Warp is a commercially available linear array of 10 or more programmable cells connected to an MC68020-based workstation. The Warp implementation of the standard Kalman filter running on 8 Warp cells achieves a measured speedup of 7 over the same filter running on a single cell. The Warp implementation of the factorized filter running on 10 Warp cells achieves a measured speedup of 2     

Nonlinear observers comprising high-gain observers and extended Kalman filters

A full order observer is designed for a class of nonlinear systems that can potentially admit unstable zero dynamics. The structure of the observer is composed of an Extended High Gain Observer (EHGO), for the estimation of the derivatives of the output, augmented with an Extended Kalman Filter (EKF) for the estimation of the states of the internal dynamics. The EHGO is also utilized to estimate a signal that is used as a virtual output to an auxiliary system comprised of the internal dynamics. In the special case of the system being linear in the states of the internal dynamics, we achieve semi-global asymptotic convergence of the estimation error. We demonstrate the efficacy of the observer in two examples; namely, a synchronous generator connected to an infinite bus and a Translating Oscillator with a Rotating Actuator (TORA) system.     

Impact induced composite delamination: state and parameter identification via joint and dual extended Kalman filters

We present and compare the joint and dual variants of the extended Kalman filter for coupled state and parameter identification problems. With reference to nonlinear dynamics of layered composites, we assume that the elastic properties of the laminae are known, whereas the softening constitutive law of the interlaminar phases adopted to simulate delamination needs to be calibrated. Purpose of this study is the identification of the interlaminar model and of the debonding surface(s) on the basis of free-surface measurements only.We show that, in the case of a dominant dilatational wave propagating in the through-the-thickness direction of the laminate, the free-surface displacement can be weakly sensitive to some constitutive parameters, and the relevant model calibration is not performed optimally. As far as delamination detection is concerned, the dual filter performs by far better than the joint filter, particularly in noisy environments.     

Consistent Kalman filters for nonlinear uncertain systems over sensor networks

In this paper, we study how to design filters for nonlinear uncertain systems over sensor networks. We introduce two Kalman-type nonlinear filters in centralized and distributed frameworks. Moreover, the tuning method for the parameters of the filters is established to ensure the consistency, i.e., the mean square error is upper bounded by a known parameter matrix at each time. We apply the consistent filters to the track-to-track association analysis of multi-targets with uncertain dynamics. A novel track-to-track association algorithm is proposed to identify whether two tracks are from the same target. It is proven that the resulting probability of mis-association is lower than the desired threshold. Numerical simulations on track-to-track association are given to show the effectiveness of the methods.     

Diffusion learning algorithms for feedforward neural networks

The problem of training feedforward neural networks is considered. To solve it, new algorithms are proposed. They are based on the asymptotic analysis of the extended Kalman filter (EKF) and on a separable network structure. Linear weights are interpreted as diffusion random variables with zero expectation and a covariance matrix proportional to an arbitrarily large parameter λ. Asymptotic expressions for the EKF are derived as λ→∞. They are called diffusion learning algorithms (DLAs). It is shown that they are robust with respect to the accumulation of rounding errors in contrast to their prototype EKF with a large but finite λ and that, under certain simplifying assumptions, an extreme learning machine (ELM) algorithm can be obtained from a DLA. A numerical example shows that the accuracy of a DLA may be higher than that of an ELM algorithm.     

Two-stage clustering via neural networks

This paper presents a two-stage approach that is effective for performing fast clustering. First, a competitive neural network (CNN) that can harmonize mean squared error and information entropy criteria is employed to exploit the substructure in the input data by identifying the local density centers. A Gravitation neural network (GNN) then takes the locations of these centers as initial weight vectors and undergoes an unsupervised update process to group the centers into clusters. Each node (called gravi-node) in the GNN is associated with a finite attraction radius and would be attracted to a nearby centroid simultaneously during the update process, creating the Gravitation-like behavior without incurring complicated computations. This update process iterates until convergence and the converged centroid corresponds to a cluster. Compared to other clustering methods, the proposed clustering scheme is free of initialization problem and does not need to pre-specify the number of clusters. The two-stage approach is computationally efficient and has great flexibility in implementation. A fully parallel hardware implementation is very possible.     

H∞-learning of layered neural networks

Although the backpropagation (BP) scheme is widely used as a learning algorithm for multilayered neural networks, the learning speed of the BP algorithm to obtain acceptable errors is unsatisfactory in spite of some improvements such as introduction of a momentum factor and an adaptive learning rate in the weight adjustment. To solve this problem, a fast learning algorithm based on the extended Kalman filter (EKF) is presented and fortunately its computational complexity has been reduced by some simplifications. In general, however, the Kalman filtering algorithm is well known to be sensitive to the nature of noises which is generally assumed to be Gaussian. In addition, the H(infinity) theory suggests that the maximum energy gain of the Kalman algorithm from disturbances to the estimation error has no upper bound. Therefore, the EKF-based learning algorithms should be improved to enhance the robustness to variations in the initial values of link weights and thresholds as well as to the nature of noises. The paper proposes H(infinity)-learning as a novel learning rule and to derive new globally and locally optimized learning algorithms based on H (infinity)-learning. Their learning behavior is analyzed from various points of view using computer simulations. The derived algorithms are also compared, in performance and computational cost, with the conventional BP and EKF learning algorithms.     

Performance modeling of load-balancing algorithms using neural networks

The paper presents a new approach that uses neural networks to predict the performance of a number of dynamic decentralized load-balancing strategies. A distributed multicomputer system using distributed load-balancing strategies is represented by a unified analytical queuing model. A large simulation data set is used to train a neural network using the back-propagation learning algorithm based on gradient descent The performance model using the predicted data from the neural network produces the average response time of various load balancing algorithms under various system parameters. The validation and comparison with simulation data show that the neural network is very effective in predicting the performance of dynamic load-balancing algorithms. Our work leads to interesting techniques for designing load balancing schemes (for large distributed systems) that are computationally very expensive to simulate. One of the important findings is that performance is affected least by the number of nodes, and most by the number of links at each node in a large distributed system.     

Application of neural networks to heuristic scheduling algorithms

This paper considers the use of artificial neural networks (ANNs) to model six different heuristic algorithms applied to the n job, m machine real flowshop scheduling problem with the objective of minimizing makespan. The objective is to obtain six ANN models to be used for the prediction of the completion times for each job processed on each machine and to introduce the fuzziness of scheduling information into flowshop scheduling. Fuzzy membership functions are generated for completion, job waiting and machine idle times. Different methods are proposed to obtain the fuzzy parameters. To model the functional relation between the input and output variables, multilayered feedforward networks (MFNs) trained with error backpropagation learning rule are used. The trained network is able to apply the learnt relationship to new problems. In this paper, an implementation alternative to the existing heuristic algorithms is provided. Once the network is trained adequately, it can provide an outcome (solution) faster than conventional iterative methods by its generalizing property. The results obtained from the study can be extended to solve the scheduling problems in the area of manufacturing.     

Learning chaotic attractors by neural networks

An algorithm is introduced that trains a neural network to identify chaotic dynamics from a single measured time series. During training, the algorithm learns to short-term predict the time series. At the same time a criterion, developed by Diks, van Zwet, Takens, and de Goede (1996) is monitored that tests the hypothesis that the reconstructed attractors of model-generated and measured data are the same. Training is stopped when the prediction error is low and the model passes this test. Two other features of the algorithm are (1) the way the state of the system, consisting of delays from the time series, has its dimension reduced by weighted principal component analysis data reduction, and (2) the user-adjustable prediction horizon obtained by "error propagation"-partially propagating prediction errors to the next time step. The algorithm is first applied to data from an experimental-driven chaotic pendulum, of which two of the three state variables are known. This is a comprehensive example that shows how well the Diks test can distinguish between slightly different attractors. Second, the algorithm is applied to the same problem, but now one of the two known state variables is ignored. Finally, we present a model for the laser data from the Santa Fe time-series competition (set A). It is the first model for these data that is not only useful for short-term predictions but also generates time series with similar chaotic characteristics as the measured data.     

用神经网络估计模型误差的预测滤波算法

针对时不变非线性系统,提出一种用神经网络进行模型误差估计的预测滤波算法．该算法用寻优的方法离线获得与当前状态和下一步输出测量相对应的模型误差估值,并作为样本训练神经网络;实际滤波中,用训练好的神经网络进行模型误差估计．该方法与原预测滤波算法相比没有动态过程,不会因为滤波器初始误差太大而振荡或发散,且稳态精度与计算步长无关．通过对一个二阶非线性系统的仿真验证了神经一预测滤波器的优越性。     

Adaptation algorithms for 2-D feedforward neural networks

The generalized weight adaptation algorithms presented by J.G. Kuschewski et al. (1993) and by S.H. Zak and H.J. Sira-Ramirez (1990) are extended for 2-D madaline and 2-D two-layer feedforward neural nets (FNNs).     

Sine-fitting multiharmonic algorithms implemented by artificial neural networks

A new method designed to perform high-accuracy spectral analysis, based on ADALINE artificial neural networks (ANNs), is proposed. The proposed network is able to accurately calculate the fundamental frequency and the harmonic content of an input signal. The method is especially useful in high-precision digital measurement systems in which periodical signals are involved, i.e. digital watt meters. Most of these systems use spectral analysis algorithms as an intermediate step for the computation of the magnitudes of interest. The traditional spectral analysis methods require synchronous sampling, which introduce limitations to the sampling circuitry. Sine-fitting multiharmonics algorithms resolve the hardware limitations concerning the synchronous sampling but have some limitations with regard to the phase of the array of samples. The new implementation of sine-fitting multiharmonics algorithms based on ANN eliminates these limitations.     

Local routing algorithms based on Potts neural networks

A feedback neural approach to static communication routing in asymmetric networks is presented, where a mean field formulation of the Bellman-Ford method for the single unicast problem is used as a common platform for developing algorithms for multiple unicast, multicast and multiple multicast problems. The appealing locality and update philosophy of the Bellman-Ford algorithm is inherited. For all problem types the objective is to minimize a total connection cost, defined as the sum of the individual costs of the involved arcs, subject to capacity constraints. The methods are evaluated for synthetic problem instances by comparing to exact solutions for cases where these are accessible, and else with approximate results from simple heuristics. In general, the quality of the results are better than those of the heuristics. Furthermore, the computational demands are modest, even when the distributed nature of the the approach is not exploited numerically.