基于影响度剪枝的极端学习机分类算法研究

针对极端学习机(ELM)网络规模控制问题,从剪枝思路出发,提出了一种基于影响度剪枝的ELM分类算法。利用ELM网络单个隐节点连接输入层和输出层的权值向量、该隐节点的输出、初始隐节点个数以及训练样本个数,定义单个隐节点相对于整个网络学习的影响度,根据影响度判断隐节点的重要性并将其排序,采用与ELM网络规模相匹配的剪枝步长删除冗余节点,最后更新隐含层与输入层和输出层连接的权值向量。通过对多个UCI机器学习数据集进行分类实验,并将提出的算法与EM-ELM、PELM和ELM算法相比较,结果表明,该算法具有较高的稳定性和测试精度,训练速度较快,并能有效地控制网络规模。     

Extreme learning machine and its applications

Recently, a novel learning algorithm for single-hidden-layer feedforward neural networks (SLFNs) named extreme learning machine (ELM) was proposed by Huang et al. The essence of ELM is that the learning parameters of hidden nodes, including input weights and biases, are randomly assigned and need not be tuned while the output weights can be analytically determined by the simple generalized inverse operation. The only parameter needed to be defined is the number of hidden nodes. Compared with other traditional learning algorithms for SLFNs, ELM provides extremely faster learning speed, better generalization performance and with least human intervention. This paper firstly introduces a brief review of ELM, describing the principle and algorithm of ELM. Then, we put emphasis on the improved methods or the typical variants of ELM, especially on incremental ELM, pruning ELM, error-minimized ELM, two-stage ELM, online sequential ELM, evolutionary ELM, voting-based ELM, ordinal ELM, fully complex ELM, and symmetric ELM. Next, the paper summarized the applications of ELM on classification, regression, function approximation, pattern recognition, forecasting and diagnosis, and so on. In the last, the paper discussed several open issues of ELM, which may be worthy of exploring in the future.     

逐层可加的急速学习机

已有的急速学习机（Extreme Learning Machine）的学习精度受隐节点数目的影响很大。无论是已提出的单隐层急速学习机还是多隐层神经网络,都是先确定隐藏层数,再通过增加每一层的神经元个数来提高精度。但当训练集规模很大时,往往需要引入很多的隐节点,导致违逆矩阵计算复杂度大,从而不利于学习效率的提高。提出逐层可加的急速学习机MHL-ELM（Extreme Learning Machine with Incremental Hidden Layers）,其思想是首先对当前隐藏层神经元（数目不大且不寻优,因而复杂度小）的权值进行随机赋值,用ELM思想求出逼近误差;若误差达不到要求,再增加一个隐含层。然后运用ELM的思想对当前隐含层优化。逐渐增加隐含层,直至满足误差精度为止。除此以外,MHL-ELM的算法复杂度为[l=1MO(N3l)]。实验使用10个UCI,keel真实数据集,通过与BP,OP-ELM等传统方法进行比较,表明MHL-ELM学习方法具有更好的泛化性,在学习精度和学习速度方面都有很大的提升。     

An integrated PSO for parameter determination and feature selection of ELM and its application in classification of power system disturbances

This paper presents a performance enhancement scheme for the recently developed extreme learning machine (ELM) for classifying power system disturbances using particle swarm optimization (PSO). Learning time is an important factor while designing any computational intelligent algorithms for classifications. ELM is a single hidden layer neural network with good generalization capabilities and extremely fast learning capacity. In ELM, the input weights are chosen randomly and the output weights are calculated analytically. However, ELM may need higher number of hidden neurons due to the random determination of the input weights and hidden biases. One of the advantages of ELM over other methods is that the parameter that the user must properly adjust is the number of hidden nodes only. But the optimal selection of its parameter can improve its performance. In this paper, a hybrid optimization mechanism is proposed which combines the discrete-valued PSO with the continuous-valued PSO to optimize the input feature subset selection and the number of hidden nodes to enhance the performance of ELM. The experimental results showed the proposed algorithm is faster and more accurate in discriminating power system disturbances.     

Evolutionary selection extreme learning machine optimization for regression

Neural network model of aggression can approximate unknown datasets with the less error. As an important method of global regression, extreme learning machine (ELM) represents a typical learning method in single-hidden layer feedforward network, because of the better generalization performance and the faster implementation. The “randomness” property of input weights makes the nonlinear combination reach arbitrary function approximation. In this paper, we attempt to seek the alternative mechanism to input connections. The idea is derived from the evolutionary algorithm. After predefining the number L of hidden nodes, we generate original ELM models. Each hidden node is seemed as a gene. To rank these hidden nodes, the larger weight nodes are reassigned for the updated ELM. We put L/2 trivial hidden nodes in a candidate reservoir. Then, we generate L/2 new hidden nodes to combine L hidden nodes from this candidate reservoir. Another ranking is used to choose these hidden nodes. The fitness-proportional selection may select L/2 hidden nodes and recombine evolutionary selection ELM. The entire algorithm can be applied for large-scale dataset regression. The verification shows that the regression performance is better than the traditional ELM and Bayesian ELM under less cost gain.     

基于径向基神经网络和正则化极限学习机的多标签学习模型*

相比径向基(RBF)神经网络,极限学习机(ELM)训练速度更快,泛化能力更强.同时,近邻传播聚类算法(AP)可以自动确定聚类个数.因此,文中提出融合AP聚类、多标签RBF(ML-RBF)和正则化ELM(RELM)的多标签学习模型(ML-AP-RBF-RELM).首先,在该模型中输入层使用ML-RBF进行映射,且通过AP聚类算法自动确定每一类标签的聚类个数,计算隐层节点个数.然后,利用每类标签的聚类个数通过K均值聚类确定隐层节点RBF函数的中心.最后,通过RELM快速求解隐层到输出层的连接权值.实验表明,ML-AP-RBF-RELM效果较好.     

Length-Changeable Incremental Extreme Learning Machine

Extreme learning machine (ELM) is a learning algorithm for generalized single-hidden-layer feed-forward networks (SLFNs). In order to obtain a suitable network architecture, Incremental Extreme Learning Machine (I-ELM) is a sort of ELM constructing SLFNs by adding hidden nodes one by one. Although kinds of I-ELM-class algorithms were proposed to improve the convergence rate or to obtain minimal training error, they do not change the construction way of I-ELM or face the over-fitting risk. Making the testing error converge quickly and stably therefore becomes an important issue. In this paper, we proposed a new incremental ELM which is referred to as Length-Changeable Incremental Extreme Learning Machine (LCI-ELM). It allows more than one hidden node to be added to the network and the existing network will be regarded as a whole in output weights tuning. The output weights of newly added hidden nodes are determined using a partial error-minimizing method. We prove that an SLFN constructed using LCI-ELM has approximation capability on a universal compact input set as well as on a finite training set. Experimental results demonstrate that LCI-ELM achieves higher convergence rate as well as lower over-fitting risk than some competitive I-ELM-class algorithms.     

Model selection of extreme learning machine based on multi-objective optimization

As a novel learning algorithm for single-hidden-layer feedforward neural networks, extreme learning machines (ELMs) have been a promising tool for regression and classification applications. However, it is not trivial for ELMs to find the proper number of hidden neurons due to the nonoptimal input weights and hidden biases. In this paper, a new model selection method of ELM based on multi-objective optimization is proposed to obtain compact networks with good generalization ability. First, a new leave-one-out (LOO) error bound of ELM is derived, and it can be calculated with negligible computational cost once the ELM training is finished. Furthermore, the hidden nodes are added to the network one-by-one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this LOO bound and the norm of output weight simultaneously in order to avoid over-fitting. Experiments on five UCI regression data sets are conducted, demonstrating that the proposed algorithm can generally obtain better generalization performance with more compact network than the conventional gradient-based back-propagation method, original ELM and evolutionary ELM.     

基于灵敏度分析法的ELM剪枝算法

针对极端学习机(ELM)网络结构设计问题,提出基于灵敏度分析法的ELM剪枝算法.利用隐含层节点输出和相对应的输出层权值向量,定义学习残差对于隐含层节点的灵敏度和网络规模适应度,根据灵敏度大小判断隐含层节点的重要性,利用网络规模适应度确定隐含层节点个数,删除重要性较低的节点.仿真结果表明,所提出的算法能够较为准确地确定与学习样本相匹配的网络规模,解决了ELM网络结构设计问题.     

Parallel Chaos Search Based Incremental Extreme Learning Machine

Recently, a simple and efficient learning steps referred to as extreme learning machine (ELM), was proposed by Huang et al. , which has shown that compared to some conventional methods, the training time of neural networks can be reduced even by thousands of times. However, recent study showed that some of random hidden nodes may paly a very minion role in the network output and thus eventually increase the network complexity. This paper proposes a parallel chaos search based incremental extreme learning machine (PC-ELM) with additional steps to obtain a more compact network architecture. At each learning step, optimal parameters of hidden node that are selected by parallel chaos optimization algorithm will be added to exist network in order to minimize the residual error between target function and network output. The optimization method is proposed parallel chaos optimization method. We prove the convergence of PC-ELM both in increased network architecture and fixed network architecture. Then we apply this approach to several regression and classification problems. Experiment of 19 benchmark testing data sets are used to test the performance of PC-ELM. Simulation results demonstrate that the proposed method provides better generalization performance and more compact network architecture.     

Knowledge-based extreme learning machines

By incorporating prior knowledge in the form of implications into extreme learning machine (ELM), a novel knowledge-based extreme learning machine (KBELM) formulation is proposed in this work. In this approach, the nonlinear prior knowledge implications are converted into linear inequalities and are then included as linear equality constraints in the ELM formulation. The proposed KBELM formulation has the advantage that it leads to solving a system of linear equations. Effectiveness of the proposed approach is demonstrated on three synthetic and the publicly available Wisconsin Prognostic Breast Cancer datasets by comparing their results with ELM and optimally pruned ELM using additive and radial basis function hidden nodes.     

An improved extreme learning machine with self-recurrent hidden layer

Extreme learning machine (ELM) is widely used in complex industrial problems, especially the online-sequential extreme learning machine (OS-ELM) plays a good role in industrial online modeling. However, OS-ELM requires batch samples to be pre-trained to obtain initial weights, which may reduce the timeliness of samples. This paper proposes a novel model for the online process regression prediction, which is called the Recurrent Extreme Learning Machine (Recurrent-ELM). The nodes between the hidden layers are connected in Recurrent-ELM, thus the input of the hidden layer receives both the information from the current input layer and the previously hidden layer. Moreover, the weights and biases of the proposed model are generated by analysis rather than random. Six regression applications are used to verify the designed Recurrent-ELM, compared with extreme learning machine (ELM), fast learning network (FLN), online sequential extreme learning machine (OS-ELM), and an ensemble of online sequential extreme learning machine (EOS-ELM), the experimental results show that the Recurrent-ELM has better generalization and stability in several samples. In addition, to further test the performance of Recurrent-ELM, we employ it in the combustion modeling of a 330 MW coal-fired boiler compared with FLN, SVR and OS-ELM. The results show that Recurrent-ELM has better accuracy and generalization ability, and the theoretical model has some potential application value in practical application.     

Uncertainty evaluation and model selection of extreme learning machine based on Riemannian metric

Considering the uncertainty of hidden neurons, choosing significant hidden nodes, called as model selection, has played an important role in the applications of extreme learning machines(ELMs). How to define and measure this uncertainty is a key issue of model selection for ELM. From the information geometry point of view, this paper presents a new model selection method of ELM for regression problems based on Riemannian metric. First, this paper proves theoretically that the uncertainty can be characterized by a form of Riemannian metric. As a result, a new uncertainty evaluation of ELM is proposed through averaging the Riemannian metric of all hidden neurons. Finally, the hidden nodes are added to the network one by one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this uncertainty evaluation and the norm of output weight simultaneously in order to obtain better generalization performance. Experiments on five UCI regression data sets and cylindrical shell vibration data set are conducted, demonstrating that the proposed method can generally obtain lower generalization error than the original ELM, evolutionary ELM, ELM with model selection, and multi-dimensional support vector machine. Moreover, the proposed algorithm generally needs less hidden neurons and computational time than the traditional approaches, which is very favorable in engineering applications.     

Self-adaptive extreme learning machine

In order to overcome the disadvantage of the traditional algorithm for SLFN (single-hidden layer feedforward neural network), an improved algorithm for SLFN, called extreme learning machine (ELM), is proposed by Huang et al. However, ELM is sensitive to the neuron number in hidden layer and its selection is a difficult-to-solve problem. In this paper, a self-adaptive mechanism is introduced into the ELM. Herein, a new variant of ELM, called self-adaptive extreme learning machine (SaELM), is proposed. SaELM is a self-adaptive learning algorithm that can always select the best neuron number in hidden layer to form the neural networks. There is no need to adjust any parameters in the training process. In order to prove the performance of the SaELM, it is used to solve the Italian wine and iris classification problems. Through the comparisons between SaELM and the traditional back propagation, basic ELM and general regression neural network, the results have proven that SaELM has a faster learning speed and better generalization performance when solving the classification problem.     

基于近似结构风险的ELM隐层节点数优化

隐层节点数是影响极端学习机(ELM)泛化性能的关键参数,针对传统的ELM隐层节点数确定算法中优化过程复杂、容易过学习或陷入局部最优的问题,提出结构风险最小化-极端学习机(SRM-ELM)算法。通过分析VC维与隐层节点数量之间的关联,对VC信任函数进行近似改进,使其为凹函数,并结合经验风险重构近似的SRM。在此基础上,将粒子群优化的位置值直接作为ELM的隐层节点数,利用粒子群算法最小化结构风险函数获得极端学习机的隐层节点数,作为最优节点数。使用6组UCI数据和胶囊缺陷数据进行仿真验证,结果表明,该算法能获得极端学习机的最优节点数,并具有更好的泛化能力。     

A hybrid extreme learning machine model with harris hawks optimisation algorithm: an optimised model for product demand forecasting applications

Accurate and real-time product demand forecasting is the need of the hour in the world of supply chain management. Predicting future product demand from historical sales data is a highly non-linear problem, subject to various external and environmental factors. In this work, we propose an optimised forecasting model - an extreme learning machine (ELM) model coupled with the Harris Hawks optimisation (HHO) algorithm to forecast product demand in an e-commerce company. ELM is preferred over traditional neural networks mainly due to its fast computational speed, which allows efficient demand forecasting in real-time. Our ELM-HHO model performed significantly better than ARIMA models that are commonly used in industries to forecast product demand. The performance of the proposed ELM-HHO model was also compared with traditional ELM, ELM auto-tuned using Bayesian Optimisation (ELM-BO), Gated Recurrent Unit (GRU) based recurrent neural network and Long Short Term Memory (LSTM) recurrent neural network models. Different performance metrics, i.e., Root Mean Squared Error (RMSE), Mean Absolute Percentage Error (MAPE) and Mean Percentage Error (MPE) were used for the comparison of the selected models. Horizon forecasting at 3 days and 7 days ahead was also performed using the proposed approach. The results revealed that the proposed approach is superior to traditional product demand forecasting models in terms of prediction accuracy and it can be applied in real-time to predict future product demand based on the previous week’s sales data. In particular, considering RMSE of forecasting, the proposed ELM-HHO model performed 62.73% better than the statistical ARIMA(7,1,0) model, 40.73% better than the neural network based GRU model, 34.05% better than the neural network based LSTM model, 27.16% better than the traditional non-optimised ELM model with 100 hidden nodes and 11.63% better than the ELM-BO model in forecasting product demand for future 3 months. The novelty of the proposed approach lies in the way the fast computational speed of ELMs has been combined with the accuracy gained by tuning hyperparameters using HHO. An increased number of hyperparameters has been optimised in our methodology compared to available models. The majority of approaches to improve the accuracy of ELM so far have only focused on tuning the weights and the biases of the hidden layer. In our hybrid model, we tune the number of hidden nodes, the number of input time lags and even the type of activation function used in the hidden layer in addition to tuning the weights and the biases. This has resulted in a significant increase in accuracy over previous methods. Our work presents an original way of performing product demand forecasting in real-time in industry with highly accurate results which are much better than pre-existing demand forecasting models.

     

改进式混合增量极限学习机算法

针对增量型极限学习机(I-ELM) 中存在大量降低学习效率及准确性的冗余节点的问题, 提出一种基于Delta 检验(DT) 和混沌优化算法(COA) 的改进式增量型核极限学习算法. 利用COA的全局搜索能力对I-ELM 中的隐含层节点参数进行寻优, 结合DT 算法检验模型输出误差, 确定有效的隐含层节点数量, 从而降低网络复杂程度, 提高算法的学习效率; 加入核函数可增强网络的在线预测能力. 仿真结果表明, 所提出的DCI-ELMK 算法具有较好的预测精度和泛化能力, 网络结构更为紧凑.

     

基于ELM的遥感影像城市道路提取

针对高分辨率遥感影像中复杂场景道路提取不理想问题,利用极限学习机ELM的快速学习能力,提出了一种基于ELM的城市道路提取方法。首先,利用改进的布谷鸟搜索CS算法自适应地选择ELM的隐含层节点数,以提高模型的稳定性;其次,引入数据样本蕴含的判别信息,弥补ELM学习不够充分问题,进而提高ELM分类性能;最后,结合数学形态学处理,对提取道路进行优化,获得最终的道路提取效果。遥感影像道路提取实验结果表明,所提方法不仅增强了网络的稳定性,同时还提高了道路提取的精确度,能较好地提取出道路信息。     

基于GSA与DE优化混合核ELM的网络异常检测模型

为了增强网络入侵检测模型的准确率与泛化性,提出一种基于引力搜索算法（GSA）与差分进化（DE）算法优化混合核极限学习机（ELM）的网络入侵检测模型。该模型针对采用单个核函数的ELM模型存在的泛化能力弱、学习能力差的问题,结合多项式核函数和径向基函数的优点,构建混合核ELM模型（HKELM）,将GSA和DE相结合优化HKELM模型参数,从而提高其在异常检测过程中的全局和局部优化能力,在此基础上利用核主成分分析算法进行入侵检测数据的数据降维和特征抽取,构建网络入侵检测模型KPCA-GSADE-HKELM。在KDD99数据集上的实验结果表明,与KDDwinner、CSVAC、CPSO-SVM、Dendron等模型进行对比,KPCA-GSADE-HKELM模型具有更高的检测精度和更快的检测速度。     

集成重复训练极限学习机的数据分类

极限学习机是一种随机化算法,它随机生成单隐含层神经网络输入层连接权和隐含层偏置,用分析的方法确定输出层连接权。给定网络结构,用极限学习机重复训练网络,会得到不同的学习模型。本文提出了一种集成模型对数据进行分类的方法。首先用极限学习机算法重复训练若干个单隐含层前馈神经网络,然后用多数投票法集成训练好的神经网络,最后用集成模型对数据进行分类,并在10个数据集上和极限学习机及集成极限学习机进行了实验比较。实验结果表明,本文提出的方法优于极限学习机和集成极限学习机。