Automatic model selection in a hybrid perceptron/radial network

We provide several enhancements to our previously introduced algorithm for a sequential construction of a hybrid network of radial and perceptron hidden units [6]. At each stage, the algorithm sub-divides the input space in order to reduce the entropy of the data conditioned on the clusters. The algorithm determines if a radial or a perceptron unit is required at a given region of input space, by using the local likelihood of the model under each unit type. Given an error target, the algorithm also determines the number of hidden units. This results in a final architecture which is often much smaller than an radial basis functions network or an multi-layer perceptron. A benchmark on six classification problems is given. The most striking performance improvement is achieved on the vowel data set [8].     

Reducing the complexity of an adaptive radial basis function network with a histogram algorithm

In this paper, a constructive training technique known as the dynamic decay adjustment (DDA) algorithm is combined with an information density estimation method to develop a new variant of the radial basis function (RBF) network. The RBF network trained with the DDA algorithm (i.e. RBFNDDA) is able to learn information incrementally by creating new hidden units whenever it is necessary. However, RBFNDDA exhibits a greedy insertion behaviour that absorbs both useful and non-useful information during its learning process, therefore increasing its network complexity unnecessarily. As such, we propose to integrate RBFNDDA with a histogram (HIST) algorithm to reduce the network complexity. The HIST algorithm is used to compute distribution of information in the trained RBFNDDA network. Then, hidden nodes with non-useful information are identified and pruned. The effectiveness of the proposed model, namely RBFNDDA-HIST, is evaluated using a number of benchmark data sets. A performance comparison study between RBFNDDA-HIST and other classification methods is conducted. The proposed RBFNDDA-HIST model is also applied to a real-world condition monitoring problem in a power generation plant. The results are analysed and discussed. The outcome indicates that RBFNDDA-HIST not only can reduce the number of hidden nodes significantly without requiring a long training time but also can produce promising accuracy rates.

     

Automated learning for reducing the configuration of a feedforwardneural network

In this paper, we present two learning mechanisms for artificial neural networks (ANN's) that can be applied to solve classification problems with binary outputs. These mechanisms are used to reduce the number of hidden units of an ANN when trained by the cascade-correlation learning algorithm (CAS). Since CAS adds hidden units incrementally as learning proceeds, it is difficult to predict the number of hidden units required when convergence is reached. Further, learning must be restarted when the number of hidden units is larger than expected. Our key idea in this paper is to provide alternatives in the learning process and to select the best alternative dynamically based on run-time information obtained. Mixed-mode learning (MM), our first algorithm, provides alternative output matrices so that learning is extended to find one of the many one-to-many mappings instead of finding a unique one-to-one mapping. Since the objective of learning is relaxed by this transformation, the number of learning epochs can be reduced. This in turn leads to a smaller number of hidden units required for convergence. Population-based learning for ANN's (PLAN), our second algorithm, maintains alternative network configurations to select at run time promising networks to train based on error information obtained and time remaining. This dynamic scheduling avoids training possibly unpromising ANNs to completion before exploring new ones. We show the performance of these two mechanisms by applying them to solve the two-spiral problem, a two-region classification problem, and the Pima Indian diabetes diagnosis problem.     

BP神经网络合理隐结点数确定的改进方法

合理选择隐含层结点个数是BP神经网络构造中的关键问题,对网络的适应能力、学习速率都有重要的影响.在此提出一种确定隐结点个数的改进方法.该方法基于隐含层神经元输出之间的线性相关关系与线性无关关系,对神经网络隐结点个数进行削减,缩减网络规模.以零件工艺过程中的加工参数作为BP神经网络的输入,加工完成的零件尺寸作为BP神经网络的输出建立模型,把该方法应用于此神经网络模型中,其训练结果证明了该方法的有效性.     

Connectivity and performance tradeoffs in the cascade correlationlearning architecture

The cascade correlation is a very flexible, efficient and fast algorithm for supervised learning. It incrementally builds the network by adding hidden units one at a time, until the desired input/output mapping is achieved. It connects all the previously installed units to the new unit being added. Consequently, each new unit in effect adds a new layer and the fan-in of the hidden and output units keeps on increasing as more units get added. The resulting structure could be hard to implement in VLSI, because the connections are irregular and the fan-in is unbounded. Moreover, the depth or the propagation delay through the resulting network is directly proportional to the number of units and can be excessive. We have modified the algorithm to generate networks with restricted fan-in and small depth (propagation delay) by controlling the connectivity. Our results reveal that there is a tradeoff between connectivity and other performance attributes like depth, total number of independent parameters, and learning time.     

Feedforward neural network construction using cross validation.

This article presents an algorithm that constructs feedforward neural networks with a single hidden layer for pattern classification. The algorithm starts with a small number of hidden units in the network and adds more hidden units as needed to improve the network's predictive accuracy. To determine when to stop adding new hidden units, the algorithm makes use of a subset of the available training samples for cross validation. New hidden units are added to the network only if they improve the classification accuracy of the network on the training samples and on the cross-validation samples. Extensive experimental results show that the algorithm is effective in obtaining networks with predictive accuracy rates that are better than those obtained by state-of-the-art decision tree methods.     

CARVE-a constructive algorithm for real-valued examples

A constructive neural-network algorithm is presented. For any consistent classification task on real-valued training vectors, the algorithm constructs a feedforward network with a single hidden layer of threshold units which implements the task. The algorithm, which we call CARVE, extends the "sequential learning" algorithm of Marchand et al. (1990) from Boolean inputs to the real-valued input case, and uses convex hull methods for the determination of the network weights. The algorithm is an efficient training scheme for producing near-minimal network solutions for arbitrary classification tasks. The algorithm is applied to a number of benchmark problems including German and Sejnowski's sonar data, the Monks problems and Fisher's iris data. A significant application of the constructive algorithm is in providing an initial network topology and initial weights for other neural-network training schemes, and this is demonstrated by application to backpropagation.     

Model selection of extreme learning machine based on multi-objective optimization

As a novel learning algorithm for single-hidden-layer feedforward neural networks, extreme learning machines (ELMs) have been a promising tool for regression and classification applications. However, it is not trivial for ELMs to find the proper number of hidden neurons due to the nonoptimal input weights and hidden biases. In this paper, a new model selection method of ELM based on multi-objective optimization is proposed to obtain compact networks with good generalization ability. First, a new leave-one-out (LOO) error bound of ELM is derived, and it can be calculated with negligible computational cost once the ELM training is finished. Furthermore, the hidden nodes are added to the network one-by-one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this LOO bound and the norm of output weight simultaneously in order to avoid over-fitting. Experiments on five UCI regression data sets are conducted, demonstrating that the proposed algorithm can generally obtain better generalization performance with more compact network than the conventional gradient-based back-propagation method, original ELM and evolutionary ELM.     

基于并行PSO算法的RBF建模

针对RBF网络的建模问题,提出了基于并行PSO算法的RBF网络建模方法。其中,隐层单元数由一系列随机产生的整数训练得到;中心向量从输入样本空间内随机选择。随后,通过误差适应度来评价全局最优粒子,进而实现网络性能。从对非线性系统的仿真效果看,该方法隐层单元数比较少,与相同隐层的RBF网络相比,显示出了一定的优越性。     

Use of a quasi-Newton method in a feedforward neural networkconstruction algorithm

This paper describes an algorithm for constructing a single hidden layer feedforward neural network. A distinguishing feature of this algorithm is that it uses the quasi-Newton method to minimize the sequence of error functions associated with the growing network. Experimental results indicate that the algorithm is very efficient and robust. The algorithm was tested on two test problems. The first was the n-bit parity problem and the second was the breast cancer diagnosis problem from the University of Wisconsin Hospitals. For the n-bit parity problem, the algorithm was able to construct neural network having less than n hidden units that solved the problem for n=4,...,7. For the cancer diagnosis problem, the neural networks constructed by the algorithm had small number of hidden units and high accuracy rates on both the training data and the testing data.     

Rival penalized competitive learning for clustering analysis, RBFnet, and curve detection

It is shown that frequency sensitive competitive learning (FSCL), one version of the recently improved competitive learning (CL) algorithms, significantly deteriorates in performance when the number of units is inappropriately selected. An algorithm called rival penalized competitive learning (RPCL) is proposed. In this algorithm, not only is the winner unit modified to adapt to the input for each input, but its rival (the 2nd winner) is delearned by a smaller learning rate. RPCL can be regarded as an unsupervised extension of Kohonen's supervised LVQ2. RPCL has the ability to automatically allocate an appropriate number of units for an input data set. The experimental results show that RPCL outperforms FSCL when used for unsupervised classification, for training a radial basis function (RBF) network, and for curve detection in digital images.     

一种估计前馈神经网络中隐层神经元数目的新方法

前馈神经网络中隐层神经元的数目一般凭经验的给出，这种方法往往造成隐单元数目的不足或过甚，从而导致网络存储容量不够或出现学习过拟现象，本研究提出了一种基于信息熵的估计三层前馈神经网络隐结点数目的方法，该方法首先利用训练集来训练具有足够隐单元数目的初始神经网络，然后计算训练集中能被训练过的神经网络正确识别的样本在隐层神经元的激活值，并对其进行排序，计算这些激活值的各种划分的信息增益，从而构造能将整个样本空间正确划分的决策树，最后遍历整棵树寻找重要的相关隐层神经元，并删除冗余无关的其它隐单元，从而估计神经网络中隐层神经元的较佳数目，文章最后以构造用于茶叶品质评定的具有较佳隐单元数目的神经网络为例，介绍本方法的使用，结果表明，本方法能有效估计前馈神经网络的隐单元数目。     

双向权值调整的前向神经网络学习算法

提出一种新的前向神经网络的学习算法,该算法在正向和反向阶段均可以对不同的层间的权值进行必要的调整,在正向阶段按最小范数二乘解原则确定连接隐层与输出层的权值,反向阶段则按误差梯度下降原则调整通连接输入层与隐层间的权值,具有很快的学习能力和收敛速度,并且能在一定的程度上保证所训练神经网络的泛化能力,实验结果初步验证了新算法的性能。     

Capabilities of a four-layered feedforward neural network: fourlayers versus three

Neural-network theorems state that only when there are infinitely many hidden units is a four-layered feedforward neural network equivalent to a three-layered feedforward neural network. In actual applications, however, the use of infinitely many hidden units is impractical. Therefore, studies should focus on the capabilities of a neural network with a finite number of hidden units, In this paper, a proof is given showing that a three-layered feedforward network with N-1 hidden units can give any N input-target relations exactly. Based on results of the proof, a four-layered network is constructed and is found to give any N input-target relations with a negligibly small error using only (N/2)+3 hidden units. This shows that a four-layered feedforward network is superior to a three-layered feedforward network in terms of the number of parameters needed for the training data.     

An Iterative Learning Control of Nonlinear Systems Using Neural Network Design

In this paper, a feedforward neural network with sigmoid hidden units is used to design a neural network based iterative learning controller for nonlinear systems with state dependent input gains. No prior offline training phase is necessary, and only a single neural network is employed. All the weights of the neurons are tuned during the iteration process in order to achieve the desired learning performance. The adaptive laws for the weights of neurons and the analysis of learning performance are determined via Lyapunov‐like analysis. A projection learning algorithm is used to prevent drifting of weights. It is shown that the tracking error vector will asymptotically converges to zero as the iteration goes to infinity, and the all adjustable parameters as well as internal signals remain bounded.     

Fine-tuning Cascade-Correlation trained feedforward network with backpropagation

To avoid oversized feedforward networks we propose that after Cascade-Correlation learning the network is fine-tuned with backpropagation algorithm. Our experiments show that if one uses merely Cascade-Correlation learning the network may require a large number of hidden units to reach the desired error level. However, if the network is in addition fine-tuned with backpropagation method then the desired error level can be reached with much smaller number of hidden units. It is also shown that the combined Cascade-Correlation backpropagation training is a faster scheme compared to mere backpropagation training.     

An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances

This paper presents a performance enhancement scheme for the recently developed extreme learning machine (ELM) for classifying power system disturbances using particle swarm optimization (PSO). Learning time is an important factor while designing any computational intelligent algorithms for classifications. ELM is a single hidden layer neural network with good generalization capabilities and extremely fast learning capacity. In ELM, the input weights are chosen randomly and the output weights are calculated analytically. However, ELM may need higher number of hidden neurons due to the random determination of the input weights and hidden biases. One of the advantages of ELM over other methods is that the parameter that the user must properly adjust is the number of hidden nodes only. But the optimal selection of its parameter can improve its performance. In this paper, a hybrid optimization mechanism is proposed which combines the discrete-valued PSO with the continuous-valued PSO to optimize the input feature subset selection and the number of hidden nodes to enhance the performance of ELM. The experimental results showed the proposed algorithm is faster and more accurate in discriminating power system disturbances.     

模糊神经网络算法在倒立摆控制中的应用

本文利用一种可以进行结构和参数学习的模糊神经网络成功地控制一级倒立摆,该网络是一种多层前馈网络,它将传统模糊控制器的基本要件综合到网络结构中。从而使该网络既具备神经网络的低级学习能力,从而还具备模糊逻辑系统类似人的高级推理能力。因而,给定训练数据后,该网络不仅可以学习网络参数,同时还可以学习网络结构。结构学习确定了表示了模糊规则和模糊分段数的连接类型以及隐节点数目。对一级倒立摆的实际控制效果可以证明该算法的性能和实用性。     

基于影响度剪枝的极端学习机分类算法研究

针对极端学习机(ELM)网络规模控制问题,从剪枝思路出发,提出了一种基于影响度剪枝的ELM分类算法。利用ELM网络单个隐节点连接输入层和输出层的权值向量、该隐节点的输出、初始隐节点个数以及训练样本个数,定义单个隐节点相对于整个网络学习的影响度,根据影响度判断隐节点的重要性并将其排序,采用与ELM网络规模相匹配的剪枝步长删除冗余节点,最后更新隐含层与输入层和输出层连接的权值向量。通过对多个UCI机器学习数据集进行分类实验,并将提出的算法与EM-ELM、PELM和ELM算法相比较,结果表明,该算法具有较高的稳定性和测试精度,训练速度较快,并能有效地控制网络规模。     

An Iterative Method for Deciding SVM and Single Layer Neural Network Structures

We present two new classifiers for two-class classification problems using a new Beta-SVM kernel transformation and an iterative algorithm to concurrently select the support vectors for a support vector machine (SVM) and the hidden units for a single hidden layer neural network to achieve a better generalization performance. To construct the classifiers, the contributing data points are chosen on the basis of a thresholding scheme of the outputs of a single perceptron trained using all training data samples. The chosen support vectors are used to construct a new SVM classifier that we call Beta-SVN. The number of chosen support vectors is used to determine the structure of the hidden layer in a single hidden layer neural network that we call Beta-NN. The Beta-SVN and Beta-NN structures produced by our method outperformed other commonly used classifiers when tested on a 2-dimensional non-linearly separable data set.