Multilayer neural networks for reduced-rank approximation

This paper is developed in two parts. First, the authors formulate the solution to the general reduced-rank linear approximation problem relaxing the invertibility assumption of the input autocorrelation matrix used by previous authors. The authors' treatment unifies linear regression, Wiener filtering, full rank approximation, auto-association networks, SVD and principal component analysis (PCA) as special cases. The authors' analysis also shows that two-layer linear neural networks with reduced number of hidden units, trained with the least-squares error criterion, produce weights that correspond to the generalized singular value decomposition of the input-teacher cross-correlation matrix and the input data matrix. As a corollary the linear two-layer backpropagation model with reduced hidden layer extracts an arbitrary linear combination of the generalized singular vector components. Second, the authors investigate artificial neural network models for the solution of the related generalized eigenvalue problem. By introducing and utilizing the extended concept of deflation (originally proposed for the standard eigenvalue problem) the authors are able to find that a sequential version of linear BP can extract the exact generalized eigenvector components. The advantage of this approach is that it's easier to update the model structure by adding one more unit or pruning one or more units when the application requires it. An alternative approach for extracting the exact components is to use a set of lateral connections among the hidden units trained in such a way as to enforce orthogonality among the upper- and lower-layer weights. The authors call this the lateral orthogonalization network (LON) and show via theoretical analysis-and verify via simulation-that the network extracts the desired components. The advantage of the LON-based model is that it can be applied in a parallel fashion so that the components are extracted concurrently. Finally, the authors show the application of their results to the solution of the identification problem of systems whose excitation has a non-invertible autocorrelation matrix. Previous identification methods usually rely on the invertibility assumption of the input autocorrelation, therefore they can not be applied to this case.     

Application of Sequential Learning Neural Networks to Civil Engineering Modeling Problems

A sequential orthogonal approach to the building and training of a single hidden layer neural network is presented in this paper. The Sequential Learning Neural Network (SLNN) model proposed by Zhang and Morris [1]is used in this paper to tackle the common problem encountered by the conventional Feed Forward Neural Network (FFNN) in determining the network structure in the number of hidden layers and the number of hidden neurons in each layer. The procedure starts with a single hidden neuron and sequentially increases in the number of hidden neurons until the model error is sufficiently small. The classical Gram–Schmidt orthogonalization method is used at each step to form a set of orthogonal bases for the space spanned by output vectors of the hidden neurons. In this approach it is possible to determine the necessary number of hidden neurons required. However, for the problems investigated in this paper, one hidden neuron itself is sufficient to achieve the desired accuracy. The neural network architecture has been trained and tested on two practical civil engineering problems – soil classification, and the prediction o strength and workability of high performance concrete.     

A multilayer incremental neural network architecture for classification

A new multilayer incremental neural network (MINN) architecture and its performance in classification of biomedical images is discussed. The MINN consists of an input layer, two hidden layers and an output layer. The first stage between the input and first hidden layer consists of perceptrons. The number of perceptrons and their weights are determined by defining a fitness function which is maximized by the genetic algorithm (GA). The second stage involves feature vectors which are the codewords obtained automaticaly after learning the first stage. The last stage consists of OR gates which combine the nodes of the second hidden layer representing the same class. The comparative performance results of the MINN and the backpropagation (BP) network indicates that the MINN results in faster learning, much simpler network and equal or better classification performance.     

通过最小二乘拟合方法删除神经网络的隐节点

提出了一种基于隐节点输出行向量的最小二乘拟合来删除神经网络隐节点的方法。它分成两步,首先分析隐节点输出行向量的正交投影间的关系,通过最小二乘拟合找出可以最准确地被其它隐节点输出行向量表达的隐节点输出行向量,然后将该隐节点的作用利用最小二乘拟合系数进行纵向传播分摊到其它隐节点。最后删除该隐节点,并进行再训练,实验结果表明再训练所需时间很短,因而它的实用性很强。     

CARVE-a constructive algorithm for real-valued examples

A constructive neural-network algorithm is presented. For any consistent classification task on real-valued training vectors, the algorithm constructs a feedforward network with a single hidden layer of threshold units which implements the task. The algorithm, which we call CARVE, extends the "sequential learning" algorithm of Marchand et al. (1990) from Boolean inputs to the real-valued input case, and uses convex hull methods for the determination of the network weights. The algorithm is an efficient training scheme for producing near-minimal network solutions for arbitrary classification tasks. The algorithm is applied to a number of benchmark problems including German and Sejnowski's sonar data, the Monks problems and Fisher's iris data. A significant application of the constructive algorithm is in providing an initial network topology and initial weights for other neural-network training schemes, and this is demonstrated by application to backpropagation.     

自组织前向神经网络与非线性动态系统模化

将自组织学习过程引入到前向网络的训练中,提出了一种新的三层前向神经网络的训练方法,训练过程首先利用自组织分族算法确定隐含层结点的数目以及权值,然后通过求解线性最小二乘问题估计输出层权值,自组织过程产生的激活权值对输入数据具有一种特征变换的功能,利用该方法训练的网络可以称之为自组织前向网络（ＳＯＦＮ）。文中通过实际非线性动态系统建模的例子,说明了ＳＯＦＮ网络具有良好性能。     

A Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks

In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance     

On the initialization and optimization of multilayer perceptrons

Multilayer perceptrons are now widely used for pattern recognition, although the training remains a time consuming procedure often converging toward a local optimum. Moreover, as the optimum network size and topology are usually unknown, the search of this optimum requires a lot of networks to be trained. In this paper the authors propose a method for properly initializing the parameters (weights) of a two-layer perceptron, and for identifying (without the need for any error-backpropagation training) the most suitable network size and topology for solving the problem under investigation. The initialized network can then be optimized by means of the standard error-backpropagation (EBP) algorithm. The authors' method is applicable to any two-layer perceptron comprising concentric as well as squashing units on its hidden layer. The output units are restricted to squashing units, but direct connections from the input to the output layer are also accommodated. To illustrate the power of the method, results obtained for different classification tasks are compared to similar results obtained using a traditional error-backpropagation training starting from a random initial state.     

Extreme learning machine for interval neural networks

Interval data offer a valuable way of representing the available information in complex problems where uncertainty, inaccuracy, or variability must be taken into account. Considered in this paper is the learning of interval neural networks, of which the input and output are vectors with interval components, and the weights are real numbers. The back-propagation (BP) learning algorithm is very slow for interval neural networks, just as for usual real-valued neural networks. Extreme learning machine (ELM) has faster learning speed than the BP algorithm. In this paper, ELM is applied for learning of interval neural networks, resulting in an interval extreme learning machine (IELM). There are two steps in the ELM for usual feedforward neural networks. The first step is to randomly generate the weights connecting the input and the hidden layers, and the second step is to use the Moore–Penrose generalized inversely to determine the weights connecting the hidden and output layers. The first step can be directly applied for interval neural networks. But the second step cannot, due to the involvement of nonlinear constraint conditions for IELM. Instead, we use the same idea as that of the BP algorithm to form a nonlinear optimization problem to determine the weights connecting the hidden and output layers of IELM. Numerical experiments show that IELM is much faster than the usual BP algorithm. And the generalization performance of IELM is much better than that of BP, while the training error of IELM is a little bit worse than that of BP, implying that there might be an over-fitting for BP.     

Radial basis function networks, regression weights, and theexpectation-maximization algorithm

We propose a modified radial basis function (RBF) network in which the regression weights are used to replace the constant weights in the output layer. It is shown that the modified RBF network can reduce the number of hidden units significantly. A computationally efficient algorithm, known as the expectation-maximization (EM) algorithm, is used to estimate the parameters of the regression weights. A salient feature of this algorithm is that it decomposes a complicated multiparameter optimization problem into L separate small-scale optimization problems, where L is the number of hidden units. The superior performance of the modified RB network over the standard RBF network is illustrated by computer simulations     

An analog feedback associative memory

A method for the storage of analog vectors, i.e., vectors whose components are real-valued, is developed for the Hopfield continuous-time network. An important requirement is that each memory vector has to be an asymptotically stable (i.e. attractive) equilibrium of the network. Some of the limitations imposed by the continuous Hopfield model on the set of vectors that can be stored are pointed out. These limitations can be relieved by choosing a network containing visible as well as hidden units. An architecture consisting of several hidden layers and a visible layer, connected in a circular fashion, is considered. It is proved that the two-layer case is guaranteed to store any number of given analog vectors provided their number does not exceed 1 + the number of neurons in the hidden layer. A learning algorithm that correctly adjusts the locations of the equilibria and guarantees their asymptotic stability is developed. Simulation results confirm the effectiveness of the approach.     

网络结构增长的极端学习机算法

针对极端学习机(extreme learning machine,ELM)结构设计问题,基于隐含层激活函数及其导函数提出一种前向神经网络结构增长算法.首先以Sigmoid函数为例给出了一类基函数的派生特性:导函数可以由其原函数表示.其次,利用这种派生特性提出了ELM结构设计方法,该方法自动生成双隐含层前向神经网络,其第1隐含层的结点随机逐一生成.第2隐含层的输出由第1隐含层新添结点的激活函数及其导函数确定,输出层权值由最小二乘法分析获得.最后给出了所提算法收敛性及稳定性的理论证明.对非线性系统辨识及双螺旋分类问题的仿真结果证明了所提算法的有效性.     

基于径向基函数神经网络的柔印专色配色系统

针对目前人工配色受配色者的生理、心理等主观因素影响,使产品质量难以保证这一问题,提出了基于径向基函数（RBF）神经网络的计算机柔印配色方法。该方法通过印刷实验获得样本数据,利用K均值聚类算法确定隐含层节点中心、采用伪逆法计算输出权值等参数,完成配色模型的建立。该配色模型可以快速完成柔印配色,并且具有较高的配色精度。     

The essential order of approximation for neural networks

There have been various studies on approximation ability of feedforward neural networks (FNNs). Most of the existing studies are, however, only concerned with density or upper bound estimation on how a multivariate function can be approximated by an FNN, and consequently, the essential approximation ability of an FNN cannot be revealed. In this paper, by establishing both upper and lower bound estimations on approximation order, the essential approximation ability (namely, the essential approximation order) of a class of FNNs is clarified in terms of the modulus of smoothness of functions to be approximated. The involved FNNs can not only approximate any continuous or integrable functions defined on a compact set arbitrarily well, but also provide an explicit lower bound on the number of hidden units required. By making use of multivariate approximation tools, it is shown that when the functions to be approximated are Lipschitzian with order up to 2, the approximation speed of the FNNs is uniquely deter     

基于最优误差自校正极限学习机的高频地波雷达RD谱图海面目标检测算法

高频地波雷达(High-frequency surface wave radar, HFSWR)在超视距舰船目标检测跟踪中有广泛应用.然而,HFSWR工作频段的电磁环境十分复杂,舰船目标信号往往被淹没在各种噪声中.本文提出一种基于最优误差自校正极限学习机(Optimized error self-adjustment extreme learning machine, OES-ELM)的HFSWR海面目标识别算法.该算法利用二级级联分类策略,可以显著提高目标的检测效率.首先利用灰度特征和线性分类器快速找出目标的潜在区域.然后利用Haar-like特征和OES-ELM分类器进一步辨识目标和海杂波.在OES-ELM中,首先利用L1/2正则算子裁剪隐层中的"微弱"神经元,以得到隐层神经元的最优个数;其次,通过网络误差回传至隐含层使网络的隐层权值和输出层权值迭代更新至最优状态.实验结果表明:和标准ELM相比,提出的OES-ELM网络具有更好的性能;此外,基于OES-ELM的HFSWR目标检测方法具有良好的实时性和目标检测性能.     

改进递归最小二乘RBF神经网络溶解氧预测

为提高溶解氧预测的准确性,将基于改进型递归最小二乘算法优化的径向基函数（ RBF）神经网络方法应用于溶解氧预测。利用K均值聚类算法进行隐层单元中心选择;利用改进型递归最小二乘算法优化RBF神经网络隐含层到输出层的权值。仿真结果表明：该方法对溶解氧的预测具有较好的非线性拟合能力,预测精度优于RBF神经网络和递归最小二乘算法优化的RBF神经网络。     

Synthesis of feedforward networks in supremum error bound

The main result of this paper is a constructive proof of a formula for the upper bound of the approximation error in L/sub /spl infin// (supremum norm) of multidimensional functions by feedforward networks with one hidden layer of sigmoidal units and a linear output. This result is applied to formulate a new method of neural-network synthesis. The result can also be used to estimate complexity of the maximum-error network and/or to initialize that network's weights. An example of the network synthesis is given.     

基于ELM-PNN算法的第24周太阳黑子预测预报

为了提高太阳黑子预测预报的精度,提出固定型极限学习过程神经网络(FELM-PNN)和增量型极限学习过程神经网络(IELM-PNN)两种学习算法.FELM-PNN的隐层节点数目固定,使用SVD求解隐层输出矩阵的Moore-Penrose广义逆,通过最小二乘法计算隐层输出权值;IELM-PNN逐次增加隐层节点,根据隐层输出矩阵和网络误差计算增加节点的输出权值.通过Henon时间序列预测验证了两种方法的有效性,并实际应用于第24周太阳黑子平滑月均值的中长期预测预报中.实验结果表明,两种方法的预测精度均有一定程度的提高,IELM-PNN的训练收敛性优于FELM-PNN.     

A new optimization algorithm for single hidden layer feedforward neural networks

Feedforward neural networks are the most commonly used function approximation techniques in neural networks. By the universal approximation theorem, it is clear that a single-hidden layer feedforward neural network (FNN) is sufficient to approximate the corresponding desired outputs arbitrarily close. Some researchers use genetic algorithms (GAs) to explore the global optimal solution of the FNN structure. However, it is rather time consuming to use GA for the training of FNN. In this paper, we propose a new optimization algorithm for a single-hidden layer FNN. The method is based on the convex combination algorithm for massaging information in the hidden layer. In fact, this technique explores a continuum idea which combines the classic mutation and crossover strategies in GA together. The proposed method has the advantage over GA which requires a lot of preprocessing works in breaking down the data into a sequence of binary codes before learning or mutation can apply. Also, we set up a new error function to measure the performance of the FNN and obtain the optimal choice of the connection weights and thus the nonlinear optimization problem can be solved directly. Several computational experiments are used to illustrate the proposed algorithm, which has good exploration and exploitation capabilities in search of the optimal weight for single hidden layer FNNs.     

Parameter by Parameter Algorithm for Multilayer Perceptrons

This paper presents a parameter by parameter (PBP) algorithm for speeding up the training of multilayer perceptrons (MLP). This new algorithm uses an approach similar to that of the layer by layer (LBL) algorithm, taking into account the input errors of the output layer and hidden layer. The proposed PBP algorithm, however, is not burdened by the need to calculate the gradient of the error function. In each iteration step, the weights or thresholds can be optimized directly one by one with other variables fixed. Four classes of solution equations for parameters of networks are deducted. The effectiveness of the PBP algorithm is demonstrated using two benchmarks. In comparisons with the BP algorithm with momentum (BPM) and the conventional LBL algorithms, PBP obtains faster convergences and better simulation performances.