Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.     

Computational models of location-invariant orthographic processing

We trained three topologies of backpropagation neural networks to discriminate 2000 words (lexical representations) presented at different positions of a horizontal letter array. The first topology (zero-deck) contains no hidden layer, the second (one-deck) has a single hidden layer, and for the last topology (two-deck), the task is divided in two subtasks implemented as two stacked neural networks, with explicit word-centred letters as intermediate representations. All topologies successfully simulated two key benchmark phenomena observed in skilled human reading: transposed-letter priming and relative-position priming. However, the two-deck topology most accurately simulated the ability to discriminate words from nonwords, while containing the fewest connection weights. We analysed the internal representations after training. Zero-deck networks implement a letter-based scheme with a position bias to differentiate anagrams. One-deck networks implement a holographic overlap coding in which representations are essentially letter-based and words are linear combinations of letters. Two-deck networks also implement holographic-coding.     

Feedforward Neural Network;Construction Algorithm;Maximum Likelihood Function;Quasi-newton Methods.

A new method for constructing a feedforward neural network is proposed. The method starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that completely recognizes all its input patterns is constructed. The novel idea about this method is that the network is trained to maximize a certain likelihood function and not to minimize the more widely used mean squared error function. We show that when a new hidden unit is added to the network, this likelihood function is guaranteed to increase and this increase ensures the finite termination of the method. We also provide a wide range of numerical results. The method was tested on the n -bit parity problems and the spiral problem. It was able to construct networks having less than n hidden units that solve the n -bit parity problems for n = 4, 5, 6, 7 and 8. The method was also tested on some real-world data and the networks it constructed were shown to be able to predict patterns not in the training set with more than 95% accuracy.     

Actively Searching for an Effective Neural Network Ensemble

A neural network NN ensemble is a very successful technique where the outputs of a set of separately trained NNs are combined to form one unified prediction. An effective ensemble should consist of a set of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well; however, most existing techniques only indirectly address the problem of creating such a set. We present an algorithm called ADDEMUP that uses genetic algorithms to search explicitly for a highly diverse set of accurate trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to create new networks continually, keeping the set of networks that are highly accurate while disagreeing with each other as much as possible. Experiments on four real-world domains show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing ensemble approaches. Experiments also show ADDEMUP is able to incorporate prior knowledge effectively, if available, to improve the quality of its ensemble.     

粗轧过程轧制力BP神经网络预报

利用BP神经网络 ,以某热轧厂粗轧机组数据库中的数据为训练样本 ,采用两种训练方案 ,对粗轧过程轧制力进行预测。BP网络的预报精度 ,既与训练样本的选取有关 ,又与隐层节点的个数以及相对化系数的大小有着密切的联系。以上因素选取得当 ,能够提高网络的预报精度 ,若选取不当 ,则降低网络的预报精度     

A systematic comparison of flat and standard cascade-correlation using a student–teacher network approximation task

Cascade-correlation (cascor) networks grow by recruiting hidden units to adjust their computational power to the task being learned. The standard cascor algorithm recruits each hidden unit on a new layer, creating deep networks. In contrast, the flat cascor variant adds all recruited hidden units on a single hidden layer. Student–teacher network approximation tasks were used to investigate the ability of flat and standard cascor networks to learn the input–output mapping of other, randomly initialized flat and standard cascor networks. For low-complexity approximation tasks, there was no significant performance difference between flat and standard student networks. Contrary to the common belief that standard cascor does not generalize well due to cascading weights creating deep networks, we found that both standard and flat cascor generalized well on problems of varying complexity. On high-complexity tasks, flat cascor networks had fewer connection weights and learned with less computational cost than standard networks did.     

Artificial neural network approach for prediction of stress–strain curve of near β titanium alloy

In the present study, artificial neural network（ANN） approach was used to predict the stress-strain curve of near beta titanium alloy as a function of volume fractions of a and b. This approach is to develop the best possible combination or neural network（NN） to predict the stress-strain curve. In order to achieve this, three different NN architectures（feed-forward back-propagation network,cascade-forward back-propagation network, and layer recurrent network）, three different transfer functions（purelin, Log-Sigmoid, and Tan-Sigmoid）, number of hidden layers（1 and 2）, number of neurons in the hidden layer（s）,and different training algorithms were employed. ANN training modules, the load in terms of strain, and volume fraction of a are the inputs and the stress as an output.ANN system was trained using the prepared training set（a,16 % a, 40 % a, and b stress-strain curves）. After training process, test data were used to check system accuracy. It is observed that feed-forward back-propagation network is the fastest, and Log-Sigmoid transfer function is giving the best results. Finally, layer recurrent NN with a single hidden layer consists of 11 neurons, and Log-Sigmoid transfer function using trainlm as training algorithm is giving good result, and average relative error is1.27 ± 1.45 %. In two hidden layers, layer recurrent NN consists of 7 neurons in each hidden layer with trainrp as the training algorithm having the transfer function of LogSigmoid which gives better results. As a result, the NN is founded successful for the prediction of stress-strain curve of near b titanium alloy.     

Effects of Collinearity on Combining Neural Networks

Collinearity or linear dependency among a number of estimators may pose a serious problem when combining these estimators. The corresponding outputs of a number of neural networks NNs , which are trained to approximate the same quantity or quantities , may be highly correlated. Thus, the estimation of the optimal weights for combining such networks may be subjected to the harmful effects of collinearity, which results in a final model with inferior generalizations ability compared with the individual networks. In this paper, we investigate the harmful effects of collinearity on the estimation of the optimal weights for combining a number on NNs. We discuss an approach for selecting the component networks in order to improve the generalization ability of the combined model. Our experimental results demonstrate significant improvements in the generalization ability of a combined model as a result of the proper selection of the component networks. The approximation accuracy of the combined model is compared with two common alternatives: the apparent best network or the simple average of the corresponding outputs of the networks.     

Connectionism;Backpropagation;Structure;Interpretation;Symbolic Logic

A particular backpropagation network, called a network of value units, was trained to detect problem type and validity of a set of logic problems. This network differs from standard networks in using a Gaussian activation function. After training was successfully completed, jittered density plots were computed for each hidden unit, and used to represent the distribution of activations produced in each hidden unit by the entire training set. The density plots revealed a marked banding. Further analysis revealed that almost all of these bands could be assigned featural interpretations, and played an important role in explaining how the network classified input patterns. These results are discussed in the context of other techniques for analyzing network structure, and in the context of other parallel distributed processing architectures.     

基于堆叠GRU的伺服电机滚动轴承剩余寿命预测

针对基于浅层学习的轴承寿命预测模型非线性学习能力差、预测精度低的问题,提出一种基于堆叠门控循环神经网络(SGRU)的伺服电机滚动轴承剩余寿命预测方法。首先对轴承振动信号进行时域和时频域特征提取,将常用的时域特征参数和经过集合经验模态分解得到的时频域特征参数作为原始特征集,然后采用相似度度量方法选取最能反映轴承退化性能的特征。之后通过堆叠两层GRU隐层来构建一种深层的寿命预测网络,并以训练集的退化特征参数为输入对网络进行训练,不断优化网络参数。最后在FEMTO数据集上与单层长短期记忆网络(LSTM)方法进行对比。结果表明,该方法相比于单层LSTM方法具有更高的预测精度。     

Behavioural Aspects of Combining Backpropagation Learning and Self-organizing Maps

Backpropagation learning (BP) is known for its serious limitations in generalizing knowledge from certain types of learning material. In this paper, we describe a new learning algorithm, BP-SOM, which overcomes some of these limitations as is shown by its application to four benchmark tasks. BP-SOM is a combination of a multi-layered feedforward network (MFN) trained with BP and Kohonen's self-organizing maps (SOMs). During the learning process, hidden-unit activations of the MFN are presented as learning vectors to SOMs trained in parallel. The SOM information is used when updating the connection weights of the MFN in addition to standard error backpropagation. The effect of the augmented error signal is that, during learning, clusters of hiddenunit activation patterns of instances associated with the same class tend to become highly similar. In a number of experiments, BP-SOM is shown (i) to improve generalization performance (i.e. avoid overfitting); (ii) to increase the amount of hidden units that can be pruned without loss of generalization performance and (iii) to provide a means for automatic rule extraction from trained networks. The results are compared with results achieved by two other learning algorithms for MFNs: conventional BP and BP augmented with weight decay. From the experiments and the comparisons, we conclude that the hybrid BP-SOM architecture, in which supervised and unsupervised and learning co-operate in finding adequate hidden-layer representations, successfully combines the advantages of supervised and unsupervised learning.     

基于RF-DGRU-SA的涡扇发动机剩余寿命预测

针对涡扇发动机在退化过程中机制复杂、状态检测数据维数过高等问题,提出一种随机森林（RF）和基于自注意力机制（SA）深度门控循环单元（DGRU）相融合的涡扇发动机剩余使用寿命（RUL）预测方法。利用RF算法确定重要度阈值实现特征筛选。将筛选出的特征输入DGRU-SA模块,通过多层GRU神经网络挖掘出相关特征与目标值之间的隐藏信息,利用SA神经网络为隐藏信息添加不同大小的权重。最后,利用全连接层输出预测结果,采用CMAPSS数据集进行实验验证。结果证明：与传统的多层感知机、卷积神经网络、长短时记忆等方法相比,所提出的融合模型误差更小,具有良好的预测精度和稳定性。     

基于混合特征LLE融合与SVM的质量异常模式识别

针对生产过程中存在的异常模式识别的问题,提出基于LLE融合与支持向量机的质量异常模式识别方法。首先从动态数据流中提取其原始特征、统计特征、几何特征并将其进行混合,形成动态数据流的混合特征,然后利用LLE算法对混合特征进行降维,将降维后的特征集作为MSVM分类器的输入进行训练,同时采用粒子群算法对MSVM分类器进行参数寻优。最后用训练好的模型对动态数据流进行异常模式的识别。并将所提方法与单一类型特征方法、混合特征方法的识别模型进行比较,仿真结果和应用实例表明,所提方法的识别精度较高,可用于生产过程的质量异常模式识别中。     

弱关联冗余环境下的挖掘算法研究

传统网络异构的数据挖掘算法以数据间的关联性为基础进行聚类,当数据集中出现大量冗余数据时,数据间的关联性减弱,使得数据挖掘精确度降低。为解决这个问题,提出一种新的弱关联冗余环境下挖掘算法。该算法先通过数据聚类方法,确定大数据集的原始聚类中心,不断更新聚类中心确保其逼近真实中心,实现大数据集的数据聚类。再对大数据集的弱关联规则进行挖掘,计算弱关联规则下数据间的关联性,采用弱化关联规则方法,挖掘出弱关联冗余环境下的数据。实验结果表明:所提挖掘算法具有较高的挖掘效率和精度,以及较低的复杂度。     

Extraction,Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks

Recurrent neural networks readily process, learn and generate temporal sequences. In addition, they have been shown to have impressive computational power. Recurrent neural networks can be trained with symbolic string examples encoded as temporal sequences to behave like sequential finite slate recognizers. We discuss methods for extracting, inserting and refining symbolic grammatical rules for recurrent networks. This paper discusses various issues: how rules are inserted into recurrent networks, how they affect training and generalization, and how those rules can be checked and corrected. The capability of exchanging information between a symbolic representation (grammatical rules)and a connectionist representation (trained weights) has interesting implications. After partially known rules are inserted, recurrent networks can be trained to preserve inserted rules that were correct and to correct through training inserted rules that were ‘incorrec’—rules inconsistent with the training data.     

Effective training data selection in tool condition monitoring system

When neural networks (NNs) are used to identify tool conditions, the richness and size of training data are crucial. The training data set not only has to cover a wide range of cutting conditions, but also to capture the characteristics of the tool wear process. This data set imposes significant computing burdens, results in a complex identification model, and hampers the feasible application of NNs. In this paper, a training data selection method is proposed, and a systematic procedure is provided to perform this data selection. With this method, the generalization error surface is divided into three regions, and proper sampling factors are chosen for each region to prune the data points from the original training set. The quality of the training set is estimated by performance evaluation through decision making. In this work, SVM is used in the decision making method, and the generalization error is used as the performance evaluation criterion. The tradeoff between the generalization performance and the size of the training set is key to this selection. Experimental results have demonstrated that this selection strategy provides an effective and efficient training set, and the developed model based on this set is fast and reliable for tool condition identification.     

Human Category Learning: Implications for Backpropagation Models

Backpropagation (Rumelhart et al., 1986) was proposed as a general learning algorithm for multi-layer perceptrons. This article demonstrates that a standard version of backprop fails to attend selectively to input dimensions in the same way as humans, suffers catastrophic forgetting of previously learned associations when novel exemplars are trained, and can be overly sensitive to linear category boundaries. Another connectionist model, ALCOVE (Kruschke 1990, 1992), does not suffer those failures. Previous researchers identified these problems; the present article reports quantitative fits of the models to new human learning data. ALCOVE can be functionally approximated by a network that uses linear-sigmoid hidden nodes, like standard backprop. It is argued that models of human category learning should incorporate quasi-local representations and dimensional attention learning, as well as error-driven learning, to address simultaneously all three phenomena.     

基于BP人工神经网络的钢轨交流闪光焊焊接接头质量预测

对刘国东等提出的BP(误差反向传播)神经网络归一化模型进行了改进,得到了适合钢轨交流闪光焊落锤质量预测的BP神经网络归一化模型。基于LabView开发软件编制了高速采集软件。采集了U71Mn钢轨焊接工艺正交试验的焊接电流、焊接电压和动立柱的位移,并从中提取加速烧化前一阶段的闪光率、能量输入、焊接时间和烧化量等质量特征量作为BP神经网络预测模型的输入量。建立了输入层单元数为5、隐含层单元数为14的BP神经网络焊接接头落锤质量的预测模型;以正交设计工艺试验的27个焊接接头中的17个作为训练样本,对预测模型进行训练。以余下的lO个作为检验样本,采用将训练后的预测模型进行预测,预测准确率达到90％。     

面向海量机械故障数据的胶囊网络算法研究

针对卷积神经网络算法在大规模故障数据集检测中出现的故障敏感度低、部分特征丢失等问题,提出一种基于优化胶囊网络算法的机械故障检测方案。胶囊网络算法采用多神经元封装的胶囊体结构设计,且包含多个胶囊层,具有更强的故障数据处理能力和泛化能力;经过squash函数挤压后的胶囊矢量可以更准确地提取和描述故障特征;升维胶囊矢量,基于特征编码和归一化的处理方式,可得到更准确的故障分类结果。实验结果显示:优化胶囊网络算法具有更强的故障特征聚类性能和迭代运算性能,故障集检测精度值高于经典卷积神经网络算法。