Weight Initialization for Simultaneous Recurrent Neural Network Trained with a Fixed-point Learning Algorithm

This letter presents a study of the Simultaneous Recurrent Neural network, an adaptive algorithm, as a nonlinear dynamic system for static optimization. Empirical findings, which were recently reported in the literature, suggest that the Simultaneous Recurrent Neural network offers superior performance for large-scale instances of combinatorial optimization problems in terms of desirable convergence characteristics improved solution quality and computational complexity measures. A theoretical study that encompasses exploration of initialization properties of the Simultaneous Recurrent Neural network dynamics to facilitate application of a fixed-point training algorithm is carried out. Specifically, initialization of the weight matrix entries to induce one or more stable equilibrium points in the state space of the nonlinear network dynamics is investigated and applicable theoretical bounds are derived. A simulation study to confirm the theoretical bounds on initial values of weights is realized. Theoretical findings and correlating simulation study performed suggest that the Simultaneous Recurrent Neural network dynamics possesses desirable stability characteristics as an adaptive recurrent neural network for addressing static optimization problems. This revised version was published online in June 2006 with corrections to the Cover Date.     

一种基于量子粒子群的过程神经元网络学习算法

针对过程神经元网络模型学习参数较多,正交基展开后的BP算法计算复杂、不易收敛等问题,提出了一种基于双链结构的量子粒子群学习算法.该算法用量子比特构成染色体,对于给定过程神经元网络模型,按权值参数的个数确定量子染色体的基因数并完成种群编码,通过量子旋转门和量子非门完成个体的更新与变异.算法中每条染色体携带两条基因链,提高了获得最优解的概率,扩展了对解空间的遍历,从而加速过程神经元网络的优化进程.将经过量子粒子群算法训练的过程神经元网络应用于Mackey-Glass混沌时间序列和太阳黑子预测,仿真结果表明该学习算法不仅收敛速度快,而且寻优能力强.     

Comparison of four gradient-learning algorithms for neural network Wiener models

A gradient-based approach to training neural network Wiener models is presented. Calculation of the gradient or approximate gradient for the series-parallel and parallel Wiener models by the backpropagation, the sensitivity method (SM) and the backpropagation through time (BPTT) is considered in a unified framework. Four different recursive learning algorithms are derived, analysed and compared. For the truncated BPTT, it is shown that the determination of the number of unfolding time steps can be made on the basis of an impulse response function of sensitivity models. Analysis of the computational complexity of these algorithms shows that, contrary to the other recurrent neural network models, computation of the gradient in parallel Wiener models with the sensitivity method or backpropagation through time requires only a little more computational burden than the backpropagation. A simulated data example and a real data example of a laboratory two-tank system are also included to make comparison of different methods and their effectiveness and practical feasibility are shown.     

基于集成神经网络的CSTR状态预测

针对连续搅拌反应釜(CSTR)具有的多重稳态性质,提出使用多个相同拓扑结构的神经网络模块组成的集成神经网络对CSTR的状态进行预测的方法。对集成神经网络的所有网络模块使用多目标粒子群优化算法进行同步训练,使训练结果收敛于参数空间内最优的Pareto面。避免了单一神经网络训练收敛到某一最优点可能产生的过拟和的问题;解决了使用传统训练方法对集成神经网络的子网络进行独立训练时增加学习算法复杂度的问题。对CSTR浓度预测的测试结果证明集成神经网络比同等规模的单一神经网络更适用于CSTR的状态参数预测。     

A stable learning algorithm for block-diagonal recurrent neural networks: application to the analysis of lung sounds.

A novel learning algorithm, the Recurrent Neural Network Constrained Optimization Method (RENNCOM) is suggested in this paper, for training block-diagonal recurrent neural networks. The training task is formulated as a constrained optimization problem, whose objective is twofold: (1) minimization of an error measure, leading to successful approximation of the input/output mapping and (2) optimization of an additional functional, the payoff function, which aims at ensuring network stability throughout the learning process. Having assured the network and training stability conditions, the payoff function is switched to an alternative form with the scope to accelerate learning. Simulation results on a benchmark identification problem demonstrate that, compared to other learning schemes with stabilizing attributes, the RENNCOM algorithm has enhanced qualities, including, improved speed of convergence, accuracy and robustness. The proposed algorithm is also applied to the problem of the analysis of lung sounds. Particularly, a filter based on block-diagonal recurrent neural networks is developed, trained with the RENNCOM method. Extensive experimental results are given and performance comparisons with a series of other models are conducted, underlining the effectiveness of the proposed filter.     

Recursive learning algorithms for training fuzzy recurrent models

A recurrent fuzzy neural network with external feedback, called the Dynamical-Adaptive Fuzzy Neural Network (D-AFNN), is proposed in this article for adapting discrete-time dynamical systems. The fuzzy model is based on the Takagi-Sugeno inference method with polynomial consequent functions. The D-AFNN model is trained to learn the system dynamics using a recursive training algorithm. Both the epochwise and the on-line version of the recursive algorithm are considered. The standard representation of a dynamical system, using the fuzzy recurrent model, is discussed and the computational procedure for the derivation of the output gradients is analytically described. The performance qualities of the D-AFNN model are illustrated with two examples. In the first example, the model is trained to identify a nonlinear plant, while in the second the fuzzy recurrent model implements an IIR adaptive filter for noise attenuation. © 1996 John Wiley & Sons, Inc.     

A neural network based on an inexpensive eight-bit microcontroller

In this paper, a neural network is trained and validated using a low end and inexpensive microcontroller. The well-known backpropagation algorithm is implemented to train a neural network model. Both the training and the validation parts are shown through an alphanumeric liquid crystal display. A chemical process was chosen as a realistic nonlinear system to demonstrate the feasibility, and the performance of the results found using the microcontroller. A comparison was made between the microcontroller and the computer results.     

A machine learning method for extracting symbolic knowledge from recurrent neural networks

Neural networks do not readily provide an explanation of the knowledge stored in their weights as part of their information processing. Until recently, neural networks were considered to be black boxes, with the knowledge stored in their weights not readily accessible. Since then, research has resulted in a number of algorithms for extracting knowledge in symbolic form from trained neural networks. This article addresses the extraction of knowledge in symbolic form from recurrent neural networks trained to behave like deterministic finite-state automata (DFAs). To date, methods used to extract knowledge from such networks have relied on the hypothesis that networks' states tend to cluster and that clusters of network states correspond to DFA states. The computational complexity of such a cluster analysis has led to heuristics that either limit the number of clusters that may form during training or limit the exploration of the space of hidden recurrent state neurons. These limitations, while necessary, may lead to decreased fidelity, in which the extracted knowledge may not model the true behavior of a trained network, perhaps not even for the training set. The method proposed here uses a polynomial time, symbolic learning algorithm to infer DFAs solely from the observation of a trained network's input-output behavior. Thus, this method has the potential to increase the fidelity of the extracted knowledge.     

基于CNN-SIndRNN的恶意TLS流量快速识别方法

传统浅层机器学习方法在识别恶意TLS流量时依赖专家经验且流量表征不足,而现有的深度神经网络检测模型因层次结构复杂导致训练时间过长。提出一种基于CNN-SIndRNN端到端的轻量级恶意加密流量识别方法,使用多层一维卷积神经网络提取流量字节序列局部模式特征,并利用全局最大池化降维以减少计算参数。为增强流量表征,设计一种改进的循环神经网络用于捕获流量字节长距离依赖关系。在此基础上,采用独立循环神经网络IndRNN单元代替传统RNN循环单元,使用切片并行计算结构代替传统RNN的串行计算结构,并将两种类型深度神经网络所提取的特征拼接作为恶意TLS流量表征。在CTU-Maluware-Capure公开数据集上的实验结果表明,该方法在二分类实验上F1值高达0.965 7,在多分类实验上整体准确率为0.848 9,相比BotCatcher模型训练时间与检测时间分别节省了98.47%和98.28%。     

Extended Kalman Filter Training of Neural Networks on a SIMD Parallel Machine

The extended Kalman filter (EKF) algorithm has been shown to be advan- tageous for neural network trainings. However, unlike the backpropagation (BP), many matrix operations are needed for the EKF algorithm and therefore greatly increase the computational complexity. This paper presents a method to do the EKF training on a SIMD parallel machine. We use a multistream decoupled extended Kalman filter (DEKF) training algorithm which can provide efficient use of the parallel resource and more improved trained network weights. From the overall design consideration of the DEKF algorithm and the consideration of maximum usage of the parallel resource, the multistream DEKF training is realized on a MasPar SIMD parallel machine. The performance of the parallel DEKF training algorithm is studied. Comparisons are performed to investigate pattern and batch-form trainings for both EKF and BP training algorithms.     

A robust recurrent simultaneous perturbation stochastic approximation training algorithm for recurrent neural networks

Training of recurrent neural networks (RNNs) introduces considerable computational complexities due to the need for gradient evaluations. How to get fast convergence speed and low computational complexity remains a challenging and open topic. Besides, the transient response of learning process of RNNs is a critical issue, especially for online applications. Conventional RNN training algorithms such as the backpropagation through time and real-time recurrent learning have not adequately satisfied these requirements because they often suffer from slow convergence speed. If a large learning rate is chosen to improve performance, the training process may become unstable in terms of weight divergence. In this paper, a novel training algorithm of RNN, named robust recurrent simultaneous perturbation stochastic approximation (RRSPSA), is developed with a specially designed recurrent hybrid adaptive parameter and adaptive learning rates. RRSPSA is a powerful novel twin-engine simultaneous perturbation stochastic approximation (SPSA) type of RNN training algorithm. It utilizes three specially designed adaptive parameters to maximize training speed for a recurrent training signal while exhibiting certain weight convergence properties with only two objective function measurements as the original SPSA algorithm. The RRSPSA is proved with guaranteed weight convergence and system stability in the sense of Lyapunov function. Computer simulations were carried out to demonstrate applicability of the theoretical results.     

用于过程建模的BP网络的M训练法及其改进

以Sigmoid为传递函数的BP网络在过程系统工程领域已经得到了广泛的应用 ,但是一般的GDR训练算法在极小点附近易发生振荡 ,收敛速度慢。本文提出了人工神经网络M法训练的新途径 ,并且通过不同算例和工业实际数据建模应用证实了M算法的收敛速度大约是GDR算法的 5 - 10倍左右 ,有效地提高了网络训练的速度和训练效率     

Reinforcement and backpropagation training for an optical neural network using self-lensing effects

The optical bench training of an optical feedforward neural network, developed by the authors, is presented. The network uses an optical nonlinear material for neuron processing and a trainable applied optical pattern as the network weights. The nonlinear material, with the applied weight pattern, modulates the phase front of a forward propagating information beam by dynamically altering the index of refraction profile of the material. To verify that the network can be trained in real time, six logic gates were trained using a reinforcement training paradigm. More importantly, to demonstrate optical backpropagation, three gates were trained via optical error backpropagation. The output error is optically backpropagated, detected with a CCD camera, and the weight pattern is updated and stored on a computer. The obtained results lay the ground work for the implementation of multilayer neural networks that are trained using optical error backpropagation and are able to solve more complex problems.     

Design of neural networks for fast convergence and accuracy:dynamics and control

A procedure for the design and training of artificial neural networks, used for rapid and efficient controls and dynamics design and analysis for flexible space systems, has been developed. Artificial neural networks are employed, such that once property trained, they provide a means of evaluating the impact of design changes rapidly. Specifically, two-layer feedforward neural networks are designed to approximate the functional relationship between the component/spacecraft design changes and measures of its performance or nonlinear dynamics of the system/components. A training algorithm, based on statistical sampling theory is presented, which guarantees that the trained networks provide a designer-specified degree of accuracy in mapping the functional relationship. Within each iteration of this statistical-based algorithm, a sequential design algorithm is used for the design and training of the feedforward network to provide rapid convergence to the network goals. Here, at each sequence a new network is trained to minimize the error of previous network. The proposed method should work for applications wherein an arbitrary large source of training data can be generated. Two numerical examples are performed on a spacecraft application in order to demonstrate the feasibility of the proposed approach.     

Online stabilization of block-diagonal recurrent neural networks

Deals with a discrete-time recurrent neural network (DTRNN) with a block-diagonal feedback weight matrix, called the block-diagonal recurrent neural network (BDRNN), that allows a simplified approach to online training and to address network and training stability issues. The structure of the BDRNN is exploited to modify the conventional backpropagation through time (BPTT) algorithm. To reduce its storage requirement by a numerically stable method of recomputing the network state variables. The network and training stability is addressed by exploiting the BDRNN structure to directly monitor and maintain stability during weight updates by developing a functional measure of system stability that augments the cost function being minimized. Simulation results are presented to demonstrate the performance of the BDRNN architecture, its training algorithm, and the stabilization method.     

Extracting rules from trained neural networks

Presents an algorithm for extracting rules from trained neural networks. The algorithm is a decompositional approach which can be applied to any neural network whose output function is monotone such as a sigmoid function. Therefore, the algorithm can be applied to multilayer neural networks, recurrent neural networks and so on. It does not depend on training algorithms, and its computational complexity is polynomial. The basic idea is that the units of neural networks are approximated by Boolean functions. But the computational complexity of the approximation is exponential, and so a polynomial algorithm is presented. The author has applied the algorithm to several problems to extract understandable and accurate rules. The paper shows the results for the votes data, mushroom data, and others. The algorithm is extended to the continuous domain, where extracted rules are continuous Boolean functions. Roughly speaking, the representation by continuous Boolean functions means the representation using conjunction, disjunction, direct proportion, and reverse proportion. This paper shows the results for iris data.     

基本样条循环神经网络及其非线性建模

提出一种新的基于基本样条逼近的循环神经网络,该网络易于训练且收敛速度快。此外为克服定长学习步长训练速度慢的问题,提出一种用于该网络训练的自适应权值更新算法,给出了学习步长的最优估计。该最优学习步长的选择可用于基本样条循环神经网络的训练以及对非线性系统的建模。     

Learning the Regularization Operator for the Optical Flow Problem

The task of displacement estimation for frames of a video sequence is considered. A new convolutional neural network architecture for the optical flow problem is proposed. The method is based on learning the regularization operator for a fast optimization method. The proposed method has low computational complexity and memory footprint at test time. The neural network architecture is based on unrolling iterations of a fast primal-dual method as layers of a convolutional neural network. Iterations of the optimization method are represented as convolutions with filters that are trained on ground truth data by backpropagation. A real-time implementation using graphics processing units is proposed. Experimental results demonstrate an improved quality of the optical flow field as compared to the optimization method based on a fixed regularization operator.     

Use of neural networks for condition rating of jointed concrete pavements

This paper discusses the use of backpropagation neural networks as a management tool for the maintenance of jointed concrete pavement. The backpropagation algorithm is applied to model the condition rating scheme adopted by Oregon State Department of Transportation. The backpropagation technique was successful in accurately capturing the nonlinear characteristics of the condition rating model. A large training set of actual pavement condition cases was used to train the network. The training was terminated when the average training error reached 0.022. A set of 6802 cases was used to test the generalization ability of the system. The trained network was able to accurately determine the correct condition ratings with an average testing error of 0.024. Finally, a statistical hypothesis test was conducted to demonstrate the system's fault-tolerance and generalization properties.     

Evaluation of the performance of backpropagation and radial basis function neural networks in predicting the drill flank wear

This study compares the performance of backpropagation neural network (BPNN) and radial basis function network (RBFN) in predicting the flank wear of high speed steel drill bits for drilling holes on mild steel and copper work pieces. The validation of the methodology is carried out following a series of experiments performed over a wide range of cutting conditions in which the effect of various process parameters, such as drill diameter, feed-rate, spindle speed, etc. on drill wear has been considered. Subsequently, the data, divided suitably into training and testing samples, have been used to effectively train both the backpropagation and radial basis function neural networks, and the individual performance of the two networks is then analyzed. It is observed that the performance of the RBFN fails to match that of the BPNN when the network complexity and the amount of data available are the constraining factors. However, when a simpler training procedure and reduced computational times are required, then RBFN is the preferred choice.