面向安卓自动化测试工具综合评估

自动化测试工具是安卓应用质量保障的主要手段.随着安卓版本多样性、底层硬件差异性(碎片化)以及逻辑复杂性增加,自动化测试迎来新的挑战.为解决这些问题,近年来,业界开发出大量自动化测试工具.但是现有工具数量多,并且测试重点多样,测试人员选择工具时存在一定的困扰.为帮助测试人员选择最佳测试工具,实现对自动化测试工具的统一评估,提出了面向安卓自动化测试工具多特征综合评估方法(comprehensiveevaluation of Android automated testing, CEAT),并将其实现为便于测试人员使用的平台. CEAT在引入测试领域广泛接受的3个评估指标,即代码覆盖率、异常检出率、融合多版本兼容度得分的基础上,进一步基于变异测试的思想引入变异杀死率,并从用户体验出发引入UI控件覆盖率.以上5个指标构成CEAT整个体系,从而实现安卓自动化测试工具的综合多维评估.为验证CEAT的效果,生成了1 089个变异应用的待测应用集,在包含6个移动设备的真机集群中部署实验,对5个自动化测试工具适配并执行5 040次测试任务.最终结果表明:i) 5个指标从不同角度对自动化测试工具进行评估,...

Towards Comprehensive Evaluation for Android Automated Testing Tools

Automated testing tools are the primary means of quality assurance for Android applications. With the increase in Android version diversity, underlying hardware variability (fragmentation), and logical complexity, automated testing faces new challenges. Numerous automated testing tools have been developed in recent years to address the above issues. However, there are vast tools with various testing focuses, making it hard for testers to choose the right one. To help testers select the best tool for testing and achieve a unified evaluation for automated testing tools, a multi-characteristic comprehensive evaluation of the Android automated testing (CEAT) method is proposed and an easy-to-use platform is implemented for testers. CEAT introduces three widely accepted evaluation metrics: code coverage, exception detection rate, fusion multi-version compatibility score, and further introduces mutation kill rate based on the mutation testing concept, and UI control widget coverage from the perspective of the user. The five metrics constitute the whole CEAT system, thus realizing a comprehensive multi-dimensional evaluation of Android automated testing tool. To verify the effectiveness of CEAT, a set of 1,089 mutated applications is generated for testing, the experiments are deployed in a real-world cluster containing six mobile devices, and 5,040 test tasks are executed for the testing tools. The results suggest that: (i) the five indicators evaluate the automated testing tools from different perspectives, reflecting the testing performance of different tools in a more multi-dimensional way and validating the effectiveness of CEAT; (ii) CEAT supports testers to assign different weights to the five metrics and obtain comprehensive evaluation results depending on the practical testing requirements, which has certain flexibility; (iii) CEAT automatically reconstructs the APP to obtain mutant APPs and set a specific platform for testing the tool, making it convenient to operate. CEAT effectively provides a reference for testers to select the best Android automated testing tool according to different testing requirements.