深度神经网络测试研究综述

随着深度神经网络技术的快速发展、大数据的涌现和计算能力的显著提升,深度神经网络被越来越多地应用到各个安全攸关领域,例如自动驾驶、人脸识别、飞机碰撞检测等.传统的软件系统通常由开发人员手工编写代码实现其内部的决策逻辑,并依据相应的测试覆盖准则设计测试用例来测试系统代码.与传统的软件系统不同,深度学习定义了一种新的数据驱动的编程范式,开发人员仅编写代码来规定深度学习系统的网络结构,其内部逻辑则由训练过程获得的神经元连接权值所决定.因此,针对传统软件的测试方法及度量指标无法直接被移植到深度神经网络系统上.近年来,越来越多的研究致力于解决深度神经网络的测试问题,例如提出新的测试评估标准、测试用例生成方法等.调研了92篇相关领域的学术论文,从深度神经网络测试度量指标、测试输入生成、测试预言这3个角度对目前已有的研究成果进行了系统梳理.同时,分析了深度神经网络测试在图像处理、语音处理以及自然语言处理上的已有成果,并介绍了深度神经网络测试中应用到的数据集及工具.最后,对深度神经网络测试的未来工作进行了展望,以期为该领域的研究人员提供参考.

Survey on Testing of Deep Neural Networks

With the rapid development of deep neural networks as well as the computational power of hardware, DNN has been widely applied in safety-critical domains such as autonomous driving, automatic face recognition and aircraft collision avoidance systems. Traditional software is implemented by developers with carefully designed programming logics and tested with test cases which are designed based on specific coverage criteria. Unlike traditional software development, DNN defines a novel data-driven programming paradigm, i.e., developers only design the structure of networks and the inner logic is reflected by weights which are learned during training. Traditional software testing methods cannot be applied to DNN directly. Driven by the emerging demand, more and more research works have focused on testing of DNN, including proposing new testing evaluation criteria, generation of test cases, etc. In this paper, we provide a thorough survey on testing DNN. Our survey summarizes 92 works from related fields. We systematically review these works from three perspectives, i.e., DNN testing metrics, test input generation and test oracle. We also introduce existing achievements in image processing, speech processing and natural language processing. We survey the datasets and tools used in DNN testing and finally summarize our thoughts on potential future research directions on DNN testing, which, hopefully, provide references for researchers interested in the related directions.