基于稀疏正则化的卷积神经网络模型剪枝方法

doi:10.19678/j.issn.1000-3428.0059375

计算机工程 ›› 2021, Vol. 47 ›› Issue (10): 61-66. doi: 10.19678/j.issn.1000-3428.0059375

基于稀疏正则化的卷积神经网络模型剪枝方法

韦越^1,2, 陈世超^2,3, 朱凤华², 熊刚²

1. 中国科学院大学人工智能学院, 北京 100049;
2. 中国科学院自动化研究所复杂系统管理与控制国家重点实验室, 北京 100190;
3. 澳门科技大学资讯科技学院, 澳门 999078

收稿日期:2020-08-27 修回日期:2020-10-20 发布日期:2020-10-20
作者简介:韦越(1996-),男,硕士研究生,主研方向为模型压缩与加速;陈世超,助理研究员、博士;朱凤华,副研究员、博士。
基金资助:
国家自然科学基金委员会-浙江省人民政府两化融合联合基金（U1909204）；广东省基础与应用基础研究基金（2019B1515120030）。

Pruning Method for Convolutional Neural Network Models Based on Sparse Regularization

WEI Yue^1,2, CHEN Shichao^2,3, ZHU Fenghua², XIONG Gang²

1. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China;
2. State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China;
3. Faculty of Information Technology, Macau University of Science and Technology, Macao 999078, China

Received:2020-08-27 Revised:2020-10-20 Published:2020-10-20

摘要/Abstract

摘要： 现有卷积神经网络模型剪枝方法仅依靠自身参数信息难以准确评估参数重要性，容易造成参数误剪且影响网络模型整体性能。提出一种改进的卷积神经网络模型剪枝方法，通过对卷积神经网络模型进行稀疏正则化训练，得到参数较稀疏的深度卷积神经网络模型，并结合卷积层和BN层的稀疏性进行结构化剪枝去除冗余滤波器。在CIFAR-10、CIFAR-100和SVHN数据集上的实验结果表明，该方法能有效压缩网络模型规模并降低计算复杂度，尤其在SVHN数据集上，压缩后的VGG-16网络模型在参数量和浮点运算量分别减少97.3%和91.2%的情况下，图像分类准确率仅损失了0.57个百分点。

关键词: 深度学习, 模型剪枝, 卷积神经网络, 稀疏约束, 模型压缩

Abstract: The existing pruning algorithms for Convolutional Neural Network(CNN) models exhibit a low accuracy in evaluating the importance of parameters by relying on their own parameter information, which would easily lead to mispruning and affect the performance of model.To address the problem, an improved pruning method for CNN models is proposed.By training the model with sparse regularization, a deep convolutional neural network model with sparse parameters is obtained.Structural pruning is performed by combining the sparsity of the convolution layer and the BN layer to remove redundant filters.Experimental results on CIFAR-10, CIFAR-100 and SVHN datasets show that the proposed pruning method can effectively compress the network model scale and reduce the computational complexity.Especially on the SVHN dataset, the compressed VGG-16 network model reduces the amount of parameters and FLOPs by 97.3% and 91.2%, respectively, and the accuracy of image classification only loses 0.57 percentage points.

Key words: deep learning, model pruning, Convolutional Neural Network(CNN), sparse constraint, model compression

中图分类号:

TP399

韦越, 陈世超, 朱凤华, 熊刚. 基于稀疏正则化的卷积神经网络模型剪枝方法[J]. 计算机工程, 2021, 47(10): 61-66.

WEI Yue, CHEN Shichao, ZHU Fenghua, XIONG Gang. Pruning Method for Convolutional Neural Network Models Based on Sparse Regularization[J]. Computer Engineering, 2021, 47(10): 61-66.

http://www.ecice06.com/CN/Y2021/V47/I10/61

图/表 6

20211015190256

20211015190300

20211015190304

20211015190307

20211015190310

20211015190313

参考文献

[1] KRIZHEVSKY A, SUTSKEVER I, HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM, 2017, 60(6):84-90.
[2] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-07-14].https://arxiv.org/abs/1409.1556.
[3] SZEGEDY C, LIU W, JIA Y Q, et al.Going deeper with convolutions[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2015:1-9.
[4] HE K M, ZHANG X Y, REN S Q, et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2016:770-778.
[5] DENIL M, SHAKIBI B, DINH L, et al.Predicting parameters in deep learning[EB/OL].[2020-07-14].http://export.arxiv.org/pdf/1306.0543.
[6] LI H, KADAV A, DURDANOVIC I, et al.Pruning filters for efficient ConvNets[EB/OL].[2020-07-14].https://arxiv.org/abs/1608.08710.
[7] HE Y, LIU P, WANG Z W, et al.Filter pruning via geometric Median for deep convolutional neural networks acceleration[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEEPress, 2019:1-8.
[8] HAN S, POOL J, TRAN J, et al.Learning both weights and connections for efficient neural network[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.New York, USA:ACM Press, 2015:1135-1143.
[9] HAN S, MAO H, DALLY W J.Deep compression:compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL].[2020-07-14].https://arxiv.org/abs/1510.00149.
[10] 叶会娟, 刘向阳.基于稀疏卷积核的卷积神经网络研究及其应用[J].信息技术, 2017, 41(10):5-9. YE H J, LIU X Y.Research and application of convolutional neural network based on sparse convolution kernel[J].Information Technology, 2017, 41(10):5-9.(in Chinese)
[11] WEN W, WU C P, WANG Y D, et al.Learning structured sparsity in deep neural networks[EB/OL].[2020-07-14].https://arxiv.org/abs/1608.03665.
[12] CHANGPINYO S, SANDLER M, ZHMOGINOV A.The power of sparsity in convolutional neural networks[EB/OL].[2020-07-14].https://arxiv.org/abs/1702.06257.
[13] RANZATO M A, POULTNEY C, CHOPRA S, et al.Efficient learning of sparse representations with an energy-based model[C]//Proceedings of NIPS'07.New York, USA:ACMPress, 2007:1137-1144.
[14] LEE H, EKANADHAM C, NG A Y.Sparse deep belief net model for visual area V2[C]//Proceedings of the 20th International Conference on Neural Information Processing Systems.NewYork, USA:ACM Press, 2008:873-880.
[15] RANZATO M A, BOUREAU Y L, LECUN Y.Sparse feature learning for deep belief networks[C]//Proceedings of NIPS'07.New York, USA:ACM Press, 2007:1185-1192.
[16] LEE H, BATTLE A, RAINA R, et al.Efficient sparse coding algorithms[C]//Proceedings of NIPS'07.New York, USA:ACM Press, 2007:801-808.
[17] MAO H Z, HAN S, POOL J, et al.Exploring the regularity of sparse structure in convolutional neural networks[EB/OL].[2020-07-14].https://arxiv.org/abs/1705.08922.
[18] DAI B, ZHU C, WIPF D.Compressing neural networks using the variational information bottleneck[EB/OL].[2020-07-14].https://arxiv.org/abs/1802.10399.
[19] LOUIZOS C, WELLING M, KINGMA D P.Learning sparse neural networks through L0 regularization[EB/OL].[2020-07-14].https://arxiv.org/abs/1712.01312.
[20] THEIS L, KORSHUNOVA I, TEJANI A, et al.Faster gaze prediction with dense networks and Fisher pruning[EB/OL].[2020-07-14].https://arxiv.org/abs/1801.05787.
[21] HU H Y, PENG R, TAI Y W, et al.Network trimming:a data-driven neuron pruning approach towards efficient deep architectures[EB/OL].[2020-07-14].https://arxiv.org/abs/1607.03250v1.
[22] 卢海伟, 夏海峰, 袁晓彤.基于滤波器注意力机制与特征缩放系数的动态网络剪枝[J].小型微型计算机系统, 2019, 40(9):1832-1838. LU H W, XIA H F, YUAN X T.Dynamic network pruning via filter attention mechanism and feature scaling factor[J].Journal of Chinese Computer Systems, 2019, 40(9):1832-1838.(in Chinese)
[23] LIU Z, LI J G, SHEN Z Q, et al.Learning efficient convolutional networks through network slimming[C]//Proceedings of2017 IEEE International Conference on Computer Vision.Washington D.C., USA:IEEE Press, 2017:2755-2763.
[24] SCARDAPANE S, COMMINIELLO D, HUSSAIN A, et al.Group sparse regularization for deep neural networks[J].Neurocomputing, 2017, 241:81-89.
[25] HE K M, ZHANG X Y, REN S Q, et al.Identity mappings in deep residual networks[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2016:630-645.
[26] HUANG G, LIU Z, VAN DER MAATEN L, et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEEPress, 2017:2261-2269.
[27] LIU Z, SUN M J, ZHOU T H, et al.Rethinking the value of network pruning[EB/OL].[2020-07-14].https://arxiv.org/abs/1810.05270v2.

选择文件类型/文献管理软件名称

选择包含的内容

基于稀疏正则化的卷积神经网络模型剪枝方法

Pruning Method for Convolutional Neural Network Models Based on Sparse Regularization

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[2]	李泽水, 冀俊忠, 杨翠翠. 基于边权重信息深度网络嵌入的PPIN功能模块检测[J]. 计算机工程, 2023, 49(8): 69-76.
[3]	王可铮, 徐玉芬, 周尚波. 结合对比感知损失和融合注意力的图像去雾模型[J]. 计算机工程, 2023, 49(8): 207-214.
[4]	刘俊豪, 王美林, 谢兴, 宋烨兴, 许莉花. 基于改进YOLOv5的皮革瑕疵检测算法[J]. 计算机工程, 2023, 49(8): 240-249.
[5]	曹坪, 杨怀志, 薄一军, 尤嘉, 张淳杰, 李丹勇. 面向低质量裂缝图像的多知识蒸馏分类[J]. 计算机工程, 2023, 49(7): 204-213.
[6]	白明昌. 基于折叠路径聚合的属性网络节点嵌入方法[J]. 计算机工程, 2023, 49(7): 76-84.
[7]	闫兴亚, 匡娅茜, 白光睿, 李月. 基于深度学习的学生课堂行为识别方法[J]. 计算机工程, 2023, 49(7): 251-258.
[8]	李军侠, 王星驰, 殷梓, 石德硕. 边缘深度挖掘的弱监督显著性目标检测[J]. 计算机工程, 2023, 49(7): 169-178.
[9]	吴珊, 周凤. 基于改进SSD算法的小目标检测[J]. 计算机工程, 2023, 49(7): 179-188.
[10]	席建锐, 唐红梅, 梁春阳, 刘鑫. 基于改进隐函数的点云物体重建[J]. 计算机工程, 2023, 49(7): 214-222.
[11]	齐咏生, 杜晓旭, 朱俊峰, 高胜利, 刘利强. 基于增强型轻量深度网络的牧区牲畜高效检测[J]. 计算机工程, 2023, 49(7): 278-287.
[12]	谌雨章, 黄逸姿, 张钧涵. 基于多速率空洞卷积的多尺度水下小目标检测[J]. 计算机工程, 2023, 49(6): 257-264.
[13]	张博旭, 蒲智, 程曦. 基于提示学习的维吾尔语文本分类研究[J]. 计算机工程, 2023, 49(6): 292-299,313.
[14]	于海洋, 景鹏, 张文涛, 谢赛飞, 滑志华, 宋草原. 基于残差与注意力机制的道路裂缝检测U-Net改进模型[J]. 计算机工程, 2023, 49(6): 265-273.
[15]	代祖华, 刘园园, 狄世龙. 语义增强的图神经网络方面级文本情感分析[J]. 计算机工程, 2023, 49(6): 71-80.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于稀疏正则化的卷积神经网络模型剪枝方法

Pruning Method for Convolutional Neural Network Models Based on Sparse Regularization

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献

相关文章 15

编辑推荐

Metrics

本文评价