基于关系型蒸馏的分步神经网络压缩方法

针对关系型知识蒸馏方法中教师网络与学生网络的层数差距过大导致蒸馏效果下降的问题,提出一种基于关系型蒸馏的分步神经网络压缩方法.该方法的要点在于,在教师网络和学生网络之间增加一个中间网络分步进行关系型蒸馏,同时在每一次蒸馏过程中都增加额外的单体信息来进一步优化和增强学生模型的学习能力,实现神经网络压缩.实验结果表明,本文的方法在CIFAR-10和CIFAR-100图像分类数据集上的分类准确度相较于原始的关系型知识蒸馏方法均有0.2％左右的提升.

Compression Method for Stepwise Neural Network Based on Relational Distillation

This study aims at the problem that the distillation effect decreases when the gap between the teacher network and the student network in relational knowledge distillation is too large. A stepwise neural network compression method based on relational distillation is proposed. The key point of this method is to add an intermediate network between the teacher network and the student network for relational distillation step by step. Moreover, in each distillation process, additional monomer information is added to further optimize and enhance the learning ability of the student model. The experimental results show that the classification accuracy of the proposed method on CIFAR-10 and CIFAR-100 image classification datasets is improved by about 0.2% compared with that of the original relational knowledge distillation method.