A node pruning algorithm based on a Fourier amplitude sensitivity test method

In this paper, we propose a new pruning algorithm to obtain the optimal number of hidden units of a single layer of a fully connected neural network (NN). The technique relies on a global sensitivity analysis of model output. The relevance of the hidden nodes is determined by analysing the Fourier decomposition of the variance of the model output. Each hidden unit is assigned a ratio (the fraction of variance which the unit accounts for) that gives their ranking. This quantitative information therefore leads to a suggestion of the most favorable units to eliminate. Experimental results suggest that the method can be seen as an effective tool available to the user in controlling the complexity in NNs.     

A sensitivity-based approach for pruning architecture of Madalines

Architecture design is a very important issue in neural network research. One popular way to find proper size of a network is to prune an oversize trained network to a smaller one while keeping established performance. This paper presents a sensitivity-based approach to prune hidden Adalines from a Madaline with causing as little as possible performance loss and thus easy compensating for the loss. The approach is novel in setting up a relevance measure, by means of an Adalines’ sensitivity measure, to locate the least relevant Adaline in a Madaline. The sensitivity measure is the probability of an Adaline’s output inversions due to input variation with respect to overall input patterns, and the relevance measure is defined as the multiplication of the Adaline’s sensitivity value by the summation of the absolute value of the Adaline’s outgoing weights. Based on the relevance measure, a pruning algorithm can be simply programmed, which iteratively prunes an Adaline with the least relevance value from hidden layer of a given Madaline and then conducts some compensations until no more Adalines can be removed under a given performance requirement. The effectiveness of the pruning approach is verified by some experimental results.     

Two-Phase Construction of Multilayer Perceptrons Using Information Theory

This brief presents a two-phase construction approach for pruning both input and hidden units of multilayer perceptrons (MLPs) based on mutual information (MI). First, all features of input vectors are ranked according to their relevance to target outputs through a forward strategy. The salient input units of an MLP are thus determined according to the order of the ranking result and by considering their contributions to the network's performance. Then, the irrelevant features of input vectors can be identified and eliminated. Second, the redundant hidden units are removed from the trained MLP one after another according to a novel relevance measure. Compared with its related work, the proposed strategy exhibits better performance. Moreover, experimental results show that the proposed method is comparable or even superior to support vector machine (SVM) and support vector regression (SVR). Finally, the advantages of the MI-based method are investigated in comparison with the sensitivity analysis (SA)-based method.     

OLAP分析中一种有效的下钻路径裁减技术

0LAP技术为企业数据分析提供了极大的便利。然而,复杂的多维结构导致了复杂的下钻路径组合,从而使得用户的数据分析效率低下。解决OLAP分析中用户的探查路径过于复杂冗长是OLAP面临的主要问题之一,但是当前的研究成果大多由于与特定的分析任务相关而不能完全解决该问题。本文提出了一种与分析任务无关的下钻路径裁剪方法。该方法从多维数据结构本身出发将无效的下钻从分析过程中裁剪掉,从而达到了简化分析过程的目标。本文采用向量夹角法评估下钻操作的有效性,并给出了对应的高效下钻路径裁剪算法。该算法以有序的实事表为输入,只需一次扫描即可完成裁剪过程。实验结果证明了本文方法的可行性、高效性、抗稀疏性和抗偏斜性。     

Sensitivity Analysis for Decision Boundaries

A novel approach is presented to visualize and analyze decision boundaries for feedforward neural networks. First order sensitivity analysis of the neural network output function with respect to input perturbations is used to visualize the position of decision boundaries over input space. Similarly, sensitivity analysis of each hidden unit activation function reveals which boundary is implemented by which hidden unit. The paper shows how these sensitivity analysis models can be used to better understand the data being modelled, and to visually identify irrelevant input and hidden units.     

Improving backpropagation learning with feature selection

There exist redundant, irrelevant and noisy data. Using proper data to train a network can speed up training, simplify the learned structure, and improve its performance. A two-phase training algorithm is proposed. In the first phase, the number of input units of the network is determined by using an information base method. Only those attributes that meet certain criteria for inclusion will be considered as the input to the network. In the second phase, the number of hidden units of the network is selected automatically based on the performance of the network on the training data. One hidden unit is added at a time only if it is necessary. The experimental results show that this new algorithm can achieve a faster learning time, a simpler network and an improved performance.     

A novel pruning approach using expert knowledge for data-specific pruning

Classification is an important data mining task that discovers hidden knowledge from the labeled datasets. Most approaches to pruning assume that all dataset are equally uniform and equally important, so they apply equal pruning to all the datasets. However, in real-world classification problems, all the datasets are not equal and considering equal pruning rate during pruning tends to generate a decision tree with large size and high misclassification rate. We approach the problem by first investigating the properties of each dataset and then deriving data-specific pruning value using expert knowledge which is used to design pruning techniques to prune decision trees close to perfection. An efficient pruning algorithm dubbed EKBP is proposed and is very general as we are free to use any learning algorithm as the base classifier. We have implemented our proposed solution and experimentally verified its effectiveness with forty real world benchmark dataset from UCI machine learning repository. In all these experiments, the proposed approach shows it can dramatically reduce the tree size while enhancing or retaining the level of accuracy.     

基于权值拟熵的神经网络剪枝算法研究

“剪枝算法”是一种通过简化神经网络结构来避免网络过拟合的有效方法之一。将权值拟熵作为惩罚项加入目标函数中,使多层前向神经网络在学习过程中自动约束权值分布,并以权值敏感度作为简化标准,避免了单纯依赖权值大小剪枝的随机性。由于在剪枝过程中只剪去数值小并且敏感度低的连接权,所以网络简化后不需要重新训练,算法效率明显提高。仿真结果证明上述方法算法简单易行,并且对前向神经网络的泛化能力有较好的改善作用。     

I‐prune: Item selection for associative classification

Associative classification is characterized by accurate models and high model generation time. Most time is spent in extracting and postprocessing a large set of irrelevant rules, which are eventually pruned. We propose I‐prune, an item‐pruning approach that selects uninteresting items by means of an interestingness measure and prunes them as soon as they are detected. Thus, the number of extracted rules is reduced and model generation time decreases correspondingly. A wide set of experiments on real and synthetic data sets has been performed to evaluate I‐prune and select the appropriate interestingness measure. The experimental results show that I‐prune allows a significant reduction in model generation time, while increasing (or at worst preserving) model accuracy. Experimental evaluation also points to the chi‐square measure as the most effective interestingness measure for item pruning. © 2012 Wiley Periodicals, Inc.     

基于IF-THEN规则的决策树裁剪算法

决策树是机器学习和数据挖掘领域中一种基本的学习方法。文中分析了C4.5算法以及该算法不足之处,提出了一种决策树裁剪算法,其中以规则信息量作为判断标准。实验结果表明这种方法可以提高最终模型的预测精度,并能够很好克服数据中的噪音。     

A Novel Pruning Algorithm for Optimizing Feedforward Neural Network of Classification Problems

Optimizing the structure of neural networks is an essential step for the discovery of knowledge from data. This paper deals with a new approach which determines the insignificant input and hidden neurons to detect the optimum structure of a feedforward neural network. The proposed pruning algorithm, called as neural network pruning by significance (N2PS), is based on a new significant measure which is calculated by the Sigmoidal activation value of the node and all the weights of its outgoing connections. It considers all the nodes with significance value below the threshold as insignificant and eliminates them. The advantages of this approach are illustrated by implementing it on six different real datasets namely iris, breast-cancer, hepatitis, diabetes, ionosphere and wave. The results show that the proposed algorithm is quite efficient in pruning the significant number of neurons on the neural network models without sacrificing the networks performance.     

一种改进的神经网络模型结构优化方法

提出了一种改进的神经网络剪枝方法,针对基于灵敏度的剪枝算法存在因计算量大而需要工程近似存在误删节点的问题,以及基于相关度的剪枝算法无法处理输入节点的问题,提出对输入节点和隐节点采用不同的方法分别进行剪枝,对于输入节点基于灵敏度剪枝算法的基本思想进行剪枝,由于不用计算隐节点灵敏度,避免了因计算量大而需要采用工程近似导致误删节点的问题;对于隐节点,采用相关度的剪枝方法,避免了传统基于灵敏度的剪枝方法在进行跨层比较决定删除何节点时不够准确,导致误删节点的问题,最后通过仿真验证了该文所提出方法的有效性。     

数据流中基于滑动窗口的最大频繁项集挖掘算法*

挖掘数据流中最大频繁项集是从数据流中获得信息的一种有效手段,是数据流挖掘研究的热点之一。结合数据流的特点,提出了一种新的基于滑动窗口的最大频繁项集挖掘算法。该算法用位图来存储数据流中流动的数据;采用直接覆盖的方法存储和更新数据流上的数据;在深度优先搜索挖掘最大频繁项集时,除采用经典的剪枝策略外,还提出了与父等价原理相对应的子等价剪枝策略;最后将挖掘结果存储在索引链表中以提高超集检测效率,进一步减少挖掘最大频繁项集的时间。理论分析和实验结果证实了该算法在时间和空间上的有效性。     

A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks

In this paper, we present a fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks (FAOS-PFNN), where a novel structure learning algorithm incorporating a pruning strategy into new growth criteria is developed. The proposed growing procedure without pruning not only speeds up the online learning process but also facilitates a more parsimonious fuzzy neural network while achieving comparable performance and accuracy by virtue of the growing and pruning strategy. The FAOS-PFNN starts with no hidden neurons and parsimoniously generates new hidden units according to the proposed growth criteria as learning proceeds. In the parameter learning phase, all the free parameters of hidden units, regardless of whether they are newly created or originally existing, are updated by the extended Kalman filter (EKF) method. The effectiveness and superiority of the FAOS-PFNN paradigm is compared with other popular approaches like resource allocation network (RAN), RAN via the extended Kalman filter (RANEKF), minimal resource allocation network (MRAN), adaptive-network-based fuzzy inference system (ANFIS), orthogonal least squares (OLS), RBF-AFS, dynamic fuzzy neural networks (DFNN), generalized DFNN (GDFNN), generalized GAP-RBF (GGAP-RBF), online sequential extreme learning machine (OS-ELM) and self-organizing fuzzy neural network (SOFNN) on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification, chaotic time-series prediction and real-world regression problems. Simulation results demonstrate that the proposed FAOS-PFNN algorithm can achieve faster learning speed and more compact network structure with comparably high accuracy of approximation and generalization.     

一种改进的RBF神经网络参数优化方法

提出了一种改进的RBF神经网络参数优化算法。通过资源分配网络算法确定隐含层节点个数,引入剪枝策略删除对网络贡献不大的节点,用改进的粒子群算法对RBF网络的中心、宽度、权值进行优化,使RBF网络不仅可以得到合适的结构,同时也可以得到合适的控制参数。将此算法用于连续搅拌釜反应器模型的预测,结果表明,此算法优化后的RBF网络结构小,并且具有较高的泛化能力。     

Polyline approach for approximating Hausdorff distance between planar free-form curves

This paper presents a practical polyline approach for approximating the Hausdorff distance between planar free-form curves. After the input curves are approximated with polylines using the recursively splitting method, the precise Hausdorff distance between polylines is computed as the approximation of the Hausdorff distance between free-form curves, and the error of the approximation is controllable. The computation of the Hausdorff distance between polylines is based on an incremental algorithm that computes the directed Hausdorff distance from a line segment to a polyline. Furthermore, not every segment on polylines contributes to the final Hausdorff distance. Based on the bound properties of the Hausdorff distance and the continuity of polylines, two pruning strategies are applied in order to prune useless segments. The R-Tree structure is employed as well to accelerate the pruning process. We experimented on Bezier curves, B-Spline curves and NURBS curves respectively with our algorithm, and there are 95% segments pruned on approximating polylines in average. Two comparisons are also presented: One is with an algorithm computing the directed Hausdorff distance on polylines by building Voronoi diagram of segments. The other comparison is with equation solving and pruning methods for computing the Hausdorff distance between free-form curves.     

用于文本分类的改进KNN算法

采用灵敏度方法对距离公式中文本特征的权重进行修正;提出一种基于CURE算法和Tabu算法的训练样本库的裁减方法,采用CURE聚类算法获得每个聚类的代表样本组成新的训练样本集合,然后用Tabu算法对此样本集合进行进一步维护(添加或删除样本),添加样本时只考虑增加不同类交界处的样本,添加或删除样本以分类精度最高、与原始训练样本库距离最近为原则。     

基于敏感度方差重要性的RBF 神经网络结构优化算法

针对RBF 神经网络的结构设计问题, 提出一种基于输出敏感度方差重要性的结构优化算法. 首先, 检验网络隐层节点的输出敏感度在样本集上的方差是否与零有显著差异, 以此作为依据增加或删除相应的隐层节点; 然后,对调整后的网络参数进行修正, 使网络具有更好的拟合精度和收敛性; 最后, 对所提出的优化算法进行仿真实验, 结果表明, 所提出的算法可根据研究对象自适应地调整RBF 的网络结构, 具有良好的逼近能力和泛化能力.

     

基于深度神经网络二阶信息的结构化剪枝算法

现有结构化剪枝算法通常运用深度神经网络(DNN)的一阶或者零阶信息对通道进行剪枝,为利用二阶信息加快DNN网络模型收敛速度,借鉴HAWQ算法思想提出一种新的结构化剪枝算法.采用幂迭代法得到经过预训练的网络参数对应Hessian矩阵的主特征向量,据此向量衡量网络通道的重要性并进行通道剪枝,同时对剪枝后的网络参数进行微调提...     

基于联邦强化学习的分布式模型剪枝

联邦学习系统中, 在资源受限的边缘端进行本地模型训练存在一定的挑战. 计算、存储、能耗等方面的限制时刻影响着模型规模及效果. 传统的联邦剪枝方法在联邦训练过程中对模型进行剪裁, 但仍存在无法根据模型所处环境自适应修剪以及移除一些重要参数导致模型性能下降的情况. 本文提出基于联邦强化学习的分布式模型剪枝方法以解决此问题. 首先, 将模型剪枝过程抽象化, 建立马尔可夫决策过程, 使用DQN算法构建通用强化剪枝模型, 动态调整剪枝率, 提高模型的泛化性能. 其次设计针对稀疏模型的聚合方法, 辅助强化泛化剪枝方法, 更好地优化模型结构, 降低模型的复杂度. 最后, 在多个公开数据集上将本方法与不同基线方法进行比较. 实验结果表明, 本文所提出的方法在保持模型效果的同时减少模型复杂度.