采用多目标优化的深度学习测试优化方法

随着深度学习技术的快速发展,对其质量保障的研究也逐步增多.传感器等技术的迅速发展,使得收集测试数据变得不再困难,但对收集到的数据进行标记却需要花费高昂的代价.已有工作尝试从原始测试集中筛选出一个测试子集以降低标记成本,这些测试子集保证了与原始测试集具有相近的整体准确率(即待测深度学习模型在测试集全体测试输入上的准确率),但却不能保证在其他测试性质上与原始测试集相近.例如,不能充分覆盖原始测试集中各个类别的测试输入.提出了一种基于多目标优化的深度学习测试输入选择方法 DMOS(deep multi-objectiveselection),其首先基于HDBSCAN(hierarchicaldensity-basedspatialclusteringofapplicationswith noise)聚类方法初步分析原始测试集的数据分布,然后基于聚类结果的特征设计多个优化目标,接着利用多目标优化求解出合适的选择方案.在8组经典的深度学习测试集和模型上进行了大量实验,结果表明, DMOS方法选出的最佳测试子集(性能最好的Pareto最优解对应的测试子集)不仅能够覆盖原始测试集中更多的测试输入类别...

Deep Learning Test Optimization Method Using Multi-objective Optimization

With the rapid development of deep learning technology, the research on its quality assurance is raising more attention. Meanwhile, it is no longer difficult to collect test data owing to the mature sensor technology, but it costs a lot to label the collected data. In order to reduce the cost of labeling, the existing work attempts to select a test subset from the original test set. They only ensure that the overall accuracy (the accuracy of the target deep learning model on all test inputs of the test set) of the test subset is similar to that of the original test set. However, existing work only focuses on estimating overall accuracy, ignoring other properties of the original test set. For example, it can not fully cover all kinds of test input in the original test set. In this paper, we propose a method based on multi-objective optimization called DMOS (DeepMulti-ObjectiveSelection). It firstly analyzes the data distribution of the original test set based on HDBSCAN clustering method. Then, it designs the optimization objective based on the characteristics of the clustering results and then carries out multi-objective optimization to find out the appropriate selection solution. A large number of experiments are carried out on 8 pairs of classic deep learning test sets and models. The results show that the best test subset selected by DMOS method (It is corresponding to the Pareto optimal solution with the best performance) can not only cover more test input categories in the original test set, but also estimate the accuracy of each test input category extremely close to the original test set. Meanwhile, It can also ensure that the overall accuracy and test adequacy are close to the original test set: the average error of the overall accuracy estimation is only 1.081%, which is 0.845% less than the PACE, with the improvement of 43.87%; The average error of the accuracy estimation of each category of test input is only 5.547%, which is 2.926% less than PACE, with the improvement of 34.53%; The average estimation error of the five test adequacy measures is only 8.739%, which is 7.328% lower than PACE, with the increase improvement of 45.61%.