Controlling the parallel layer perceptron complexity using a multiobjective learning algorithm

This paper deals with the parallel layer perceptron (PLP) complexity control, bias and variance dilemma, using a multiobjective (MOBJ) training algorithm. To control the bias and variance the training process is rewritten as a bi-objective problem, considering the minimization of both training error and norm of the weight vector, which is a measure of the network complexity. This method is applied to regression and classification problems and compared with several other training procedures and topologies. The results show that the PLP MOBJ training algorithm presents good generalization results, outperforming traditional methods in the tested examples.     

Generalized RLS approach to the training of neural networks

Recursive least square (RLS) is an efficient approach to neural network training. However, in the classical RLS algorithm, there is no explicit decay in the energy function. This will lead to an unsatisfactory generalization ability for the trained networks. In this paper, we propose a generalized RLS (GRLS) model which includes a general decay term in the energy function for the training of feedforward neural networks. In particular, four different weight decay functions, namely, the quadratic weight decay, the constant weight decay and the newly proposed multimodal and quartic weight decay are discussed. By using the GRLS approach, not only the generalization ability of the trained networks is significantly improved but more unnecessary weights are pruned to obtain a compact network. Furthermore, the computational complexity of the GRLS remains the same as that of the standard RLS algorithm. The advantages and tradeoffs of using different decay functions are analyzed and then demonstrated with examples. Simulation results show that our approach is able to meet the design goals: improving the generalization ability of the trained network while getting a compact network.     

Divide-and-conquer learning and modular perceptron networks

A novel modular perceptron network (MPN) and divide-and-conquer learning (DCL) schemes for the design of modular neural networks are proposed. When a training process in a multilayer perceptron falls into a local minimum or stalls in a flat region, the proposed DCL scheme is applied to divide the current training data region into two easier to be learned regions. The learning process continues when a self-growing perceptron network and its initial weight estimation are constructed for one of the newly partitioned regions. Another partitioned region will resume the training process on the original perceptron network. Data region partitioning, weight estimating and learning are iteratively repeated until all the training data are completely learned by the MPN. We evaluated and compared the proposed MPN with several representative neural networks on the two-spirals problem and real-world dataset. The MPN achieved better weight learning performance by requiring much less data presentations during the network training phases, and better generalization performance, and less processing time during the retrieving phase.     

基于互补遗传算子的前馈神经网络三阶段学习方法

论文提出了一种新的基于互补遗传算子的前馈神经网络三阶段学习方法。该方法把神经网络的学习过程分为三个阶段。第一阶段为结构辨识阶段,采用遗传算法进行神经网络隐层节点数目的选择和初始参数的设定,并基于发现的遗传算子的互补效应设计高效互补遗传算子。第二阶段为参数辨识阶段,采用效率较高的神经网络算法如L-M算法进行神经网络参数的进一步学习。第三阶段为剪枝阶段,通过获得最小结构的神经网络以提高其泛化能力。在整个学习过程中,学习过程的可控性以及神经网络的逼近精度、复杂度和泛化能力之间得到了满意平衡。仿真试验结果证明了该方法的有效性。     

基于快速SVM的大规模网络流量分类方法

支持向量机方法具有良好的分类准确率、稳定性与泛化性,在网络流量分类领域已有初步应用,但在面对大规模网络流量分类问题时却存在计算复杂度高、分类器训练速度慢的缺陷。为此,提出一种基于比特压缩的快速SVM方法,利用比特压缩算法对初始训练样本集进行聚合与压缩,建立具有权重信息的新样本集,在损失尽量少原始样本信息的前提下缩减样本集规模,进一步利用基于权重的SVM算法训练流量分类器。通过大规模样本集流量分类实验对比,快速SVM方法能在损失较少分类准确率的情况下,较大程度地缩减流量分类器的训练时间以及未知样本的预测时间,同时,在无过度压缩前提下,其分类准确率优于同等压缩比例下的随机取样SVM方法。本方法在保留SVM方法较好分类稳定性与泛化性能的同时,有效提升了其应对大规模流量分类问题的能力。     

Generalization and selection of examples in feedforward neural networks

In this work, we study how the selection of examples affects the learning procedure in a boolean neural network and its relationship with the complexity of the function under study and its architecture. We analyze the generalization capacity for different target functions with particular architectures through an analytical calculation of the minimum number of examples needed to obtain full generalization (i.e., zero generalization error). The analysis of the training sets associated with such parameter leads us to propose a general architecture-independent criterion for selection of training examples. The criterion was checked through numerical simulations for various particular target functions with particular architectures, as well as for random target functions in a nonoverlapping receptive field perceptron. In all cases, the selection sampling criterion lead to an improvement in the generalization capacity compared with a pure random sampling. We also show that for the parity problem, one of the most used problems for testing learning algorithms, only the use of the whole set of examples ensures global learning in a depth two architecture. We show that this difficulty can be overcome by considering a tree-structured network of depth 2log2(N)-1.     

An exploration of the uncertainty relation satisfied by BP network learning ability and generalization ability

An overfit phenomenon exists in the BP network. The so-called overfit means that as long as the network is allowed to be sufficiently complicated, the BP network can minimize the error of the training sample set; however, in the case of a limited number of samples, the generalization ability of the network will decrease. This indicates that there is a relation between the learning ability and the generalization ability. Therefore, studying the relationship between the learning ability is the…     

Neural-network design for small training sets of high dimension

We introduce a statistically based methodology for the design of neural networks when the dimension d of the network input is comparable to the size n of the training set. If one proceeds straightforwardly, then one is committed to a network of complexity exceeding n. The result will be good performance on the training set but poor generalization performance when the network is presented with new data. To avoid this we need to select carefully the network architecture, including control over the input variables. Our approach to selecting a network architecture first selects a subset of input variables (features) using the nonparametric statistical process of difference-based variance estimation and then selects a simple network architecture using projection pursuit regression (PPR) ideas combined with the statistical idea of slicing inverse regression (SIR). The resulting network, which is then retrained without regard to the PPR/SIR determined parameters, is one of moderate complexity (number of parameters significantly less than n) whose performance on the training set can be expected to generalize well. The application of this methodology is illustrated in detail in the context of short-term forecasting of the demand for electric power from an electric utility.     

改进PSO-BP神经网络对储层参数的动态预测研究

为提高BP神经网络的收敛速度和泛化能力,防止其陷入局部最优值,在前人工作基础上对传统粒子群算法进行了改进,具体包括：设定最大限制速度、改变惯性权重因子和改进适应度函数,并把改进粒子群算法应用于BP神经网络权值和阈值的优化。之后利用改进粒子群算法优化的BP神经网络实现对储层参数的动态预测,具体步骤为：确定神经网络的输入、输出神经元,定量化时间参数[T],利用训练样本构建神经网络模型并进行检验。最后通过平均训练误差对仿真过程进行分析,结果表明改进PSO-BP算法的收敛性与泛化能力均优于BP算法和PSO-BP算法。     

动态数据约简的神经网络分类器训练方法研究

针对神经网络分类器训练时间长、泛化能力差的问题,提出了一种基于动态数据约简的神经网络分类器训练方法（DDR）。该训练方法在训练过程中赋给每个训练样本一个权重值作为样本的重要性度量,依据每次网络迭代训练样本的分类错误率动态更新每个训练样本的权重值,之后依据样本的权重值来约简训练样本,从而增加易错分类的边界样本比重,减少冗余核样本的作用。数值实验表明,基于权重的动态数据约简神经网络训练方法不仅大幅缩短了网络的训练时间,而且还能够显著提升网络的分类泛化能力。     

Radial Basis Function network learning using localized generalization error bound

Training a classifier with good generalization capability is a major issue for pattern classification problems. A novel training objective function for Radial Basis Function (RBF) network using a localized generalization error model (L-GEM) is proposed in this paper. The localized generalization error model provides a generalization error bound for unseen samples located within a neighborhood that contains all training samples. The assumption of the same width for all dimensions of a hidden neuron in L-GEM is relaxed in this work. The parameters of RBF network are selected via minimization of the proposed objective function to minimize its localized generalization error bound. The characteristics of the proposed objective function are compared with those for regularization methods. For weight selection, RBF networks trained by minimizing the proposed objective function consistently outperform RBF networks trained by minimizing the training error, Tikhonov Regularization, Weight Decay or Locality Regularization. The proposed objective function is also applied to select center, width and weight in RBF network simultaneously. RBF networks trained by minimizing the proposed objective function yield better testing accuracies when compared to those that minimizes training error only.     

初始权值优化技术在SOM网络中的应用

SOM网络是一种无导师学习方法,被广泛应用于各个领域.网络的性能受很多因素影响,如样本的选择、网络结构、初始权值的选定等.针对网络初始权值选取的不确定性问题,提出了覆盖权值初始化方法来优化SOM网络的初始权值:该方法从样本入手,并通过"覆盖"方法得出初始权值,仿真实验结果证明了此方法能有效的提高网络的识别率和稳定性.     

一种提高神经网络泛化性能的罚项最优脑外科模型*

最优脑外科过程是一种训练后网络剪枝算法,计算的复杂度非常高,通过把剪枝条件以惩罚项的形式纳入神经网络的训练目标函数中,把正则化方法的结构优化蕴涵于网络训练过程,构建面向最优脑外科过程的计算模型,实现网络训练过程和最优脑外科过程并行剪枝,既保持了最优脑外科过程的准确性,又具有正则化的高效性,提高了神经网络模型的泛化性能。该模型在理论上具有收敛性,其有效性和可行性通过给出的Levenberg-Marquardt方案仿真实验也得到了说明。     

一种基于随机GA的提高BP网络泛化能力的方法

LM-BP网络对其初始权值和阈值敏感,泛化能力不强,针对该缺点,采用遗传算法（GA）对其初始权阈值进行优化,在一定程度上能提高LM-BP网络的泛化能力。为进一步扩展GA初始种群的覆盖范围,进一步提高LM—BP网络的泛化能力,采用多次随机产生初始种群多次优化的方法。以伦河孝感段氟化物含量为实例,建立随机GA的LM—BP网络模型,对原始数据进行拟合及测试,结果表明该方法基本能100％拟合,测试误差不超过2．3％。经过对比实验,证明了该方法的有效性。     

基于贝叶斯方法的神经网络非线性模型辨识

研究了基于贝叶斯推理的多层前向神经网络训练算法,以提高网络的泛化性能。在网络目标函数中引入表示网络结构复杂性的惩罚项,以便能够在训练优化过程中降低网络结构的复杂性,达到避免网络过拟合的目的。训练过程中使用显式的概率分布假设对模型进行分析和推断,根据融入先验分布的假设和依据,获取网络参数和正则化参数的后验条件概率,并基于后验分布的贝叶斯推理得出最优化参数。利用上述算法训练前向网络,对一个微型锅炉对象进行了模型辨识,通过测试,证明所辨识出的对象模型能够较好地表现出对象的动态行为,且具有较好的泛化性能。     

基于免疫RBF神经网络的逆运动学求解

求解机械臂逆运动学问题可以采用神经网络来建立逆运动学模型,通过遗传算法或BP算法训练神经网络的权值从而得到问题的解,在求解精度和收敛速度上有待进一步改进。采用人工免疫原理对RBF网络训练数据集的泛化能力在线调整隐层结构,生成RBF网络隐层。当网络结构确定时,采用递推最小二乘法确定网络连接权值。由此对神经网络的网络结构和连接权进行自适应调整和学习。通过仿真可以看出,用免疫原理训练的神经网络收敛速度快,泛化能力强,可大幅提高机械臂逆运动学求解精度。     

Pruning recurrent neural networks for improved generalizationperformance

Determining the architecture of a neural network is an important issue for any learning task. For recurrent neural networks no general methods exist that permit the estimation of the number of layers of hidden neurons, the size of layers or the number of weights. We present a simple pruning heuristic that significantly improves the generalization performance of trained recurrent networks. We illustrate this heuristic by training a fully recurrent neural network on positive and negative strings of a regular grammar. We also show that rules extracted from networks trained with this pruning heuristic are more consistent with the rules to be learned. This performance improvement is obtained by pruning and retraining the networks. Simulations are shown for training and pruning a recurrent neural net on strings generated by two regular grammars, a randomly-generated 10-state grammar and an 8-state, triple-parity grammar. Further simulations indicate that this pruning method can have generalization performance superior to that obtained by training with weight decay.     

带比例因子的卷积神经网络压缩方法

针对卷积神经网络在图像分类任务中,分类准确率高但实时性差的问题。提出了一种含比例因子的“知识提取”算法。此方法在已有的“知识提取”算法上,加入了衡量样本类间相近关系的比例因子,充实了网络压缩手段,使得神经网络可以更精确地进行“知识提取”。其原理是将比例因子误差值作为代价函数的一部分参与训练调节神经网络的神经元参数,进而使得神经网络的泛化能力更加趋近于具有更好分类表现能力的压缩参考网络。结果表明,含比例因子的神经网络压缩算法可以更细致地刻画训练集的类间相近关系,拥有比原“知识提取”算法更好的训练性能,进而训练出泛化性能更强、精度更高的神经网络。实现了在网络分类准确率下降尽量小的前提下,较大程度地减少神经网络的分类耗时,以达到网络压缩的目的。     

Randomness in generalization ability: a source to improve it

Among several models of neurons and their interconnections, feedforward artificial neural networks (FFANNs) are most popular, because of their simplicity and effectiveness. Difficulties such as long learning time and local minima may not affect FFANNs as much as the question of generalization ability, because a network needs only one training, and then it may be used for a long time. This paper reports our observations about randomness in generalization ability of FFANNs. A novel method for measuring generalization ability is defined. This method can be used to identify degree of randomness in generalization ability of learning systems. If an FFANN architecture shows randomness in generalization ability for a given problem, multiple networks can be used to improve it. We have developed a model, called voting model, for predicting generalization ability of multiple networks. It has been shown that if correct classification probability of a single network is greater than half, then as the number of networks in a voting network is increased so does its generalization ability. Further analysis has shown that VC-dimension of the voting network model may increase monotonically as the number of networks in the voting networks is increased.     

Inversion of neural networks by gradient descent

Inversion answers the question of which input patterns to a trained multilayer neural network approximate a given output target. This method is a tool for visualization of the information processing capability of a network stored in its weights. This knowledge about the network enables one to make informed decisions on the improvement of the training task and the choice of training sets.

An inversion algorithm for multilayer perceptrons is derived from the backpropagation scheme. We apply inversion to networks for digit recognition. We observe that the multilayer perceptrons perform well with respect to generalization, i.e. correct classification of untrained digits. They are however bad on rejection of counterexamples, i.e. random patterns. Inversion gives an explanation for this drawback. We suggest an improved training scheme, and we show that a tradeoff exists between generalization and rejection of counterexamples.