CMuJava:一个面向Java程序并发变异体生成系统

并发程序由多个共享存储空间并发执行的流程组成.由于流程之间执行次序的不确定性,使得并发软件系统的测试比较困难.变异测试是一种基于故障的软件测试技术,广泛用于评估测试用例集的充分性和测试技术的有效性.将变异测试应用于并发程序的一个关键问题是,如何高效地生成大量的模拟并发故障的变异体集合.给出了一种并发程序的变异测试框架,...     

基于规则与学习的变异技术对比研究

变异测试是一种有效的软件测试技术,通过生成变异体来模拟软件缺陷,帮助提升现有测试套件的缺陷检测能力.变异体的质量对于变异测试的有效性具有显著影响.传统的变异测试方法通常采用人工设计的基于语法规则的变异算子生成变异体,并已取得一定的研究成果.近年来,许多研究开始结合深度学习技术,通过学习开源项目历史代码生成变异体.目前该新方法在变异体生成方面取得了初步的成果.基于语法规则和基于学习的两种变异技术,其机理不同,但其目标均是通过生成变异体来提高测试套件的缺陷检测能力,因此全面比较这两种变异技术对于变异测试及其下游任务(如缺陷定位和修复)至关重要.针对这一问题,本文设计实现一项针对基于语法规则和基于学习的变异技术的实证研究,旨在了解不同机理的变异技术在变异测试任务上的性能,以及生成的变异体在程序语义上的差异性.具体地,本文以Defect4J v1.2.0数据集为实验对象,比较MAJOR和PIT为代表的基于语法规则的变异技术和以DeepMutation、μBERT和LEAM为代表的基于深度学习的变异技术.实验结果表明,基于规则与学习的变异技术均可有效支持变异测试实践,但MAJOR的测试效果最优,能够检测85.4%的真实缺陷.在语义表示上,MAJOR具有最强的语义代表能力,基于其构造的测试套件能够杀死其余变异技术生成的超过95%占比的变异体.在缺陷表征上,两类技术均具有独特性.例如,基于规则的技术具有更稳定的代码元素变异能力,而基于学习的技术具有更强的上下文理解能力.     

MujavaX:一个支持非均匀分布的变异生成系统

变异分析是一种广泛用来评估软件测试技术性能的方法.已有的变异分析技术通常将变异算子平均地应用于原始程序.由于现实程序中的故障分布往往具有群束的特征,采用平均分布的变异分析方法不能客观地评估软件测试技术的性能.前期研究工作中提出了非均匀分布的变异分析方法,采用实例研究验证了不同的故障分布对测试技术性能评估的影响.为了增强非均匀分布的变异分析方法的实用性,开发了支持非均匀分布的变异生成系统MujavaX,该系统是对广泛实践的Mujava工具的扩展与改进.采用一个实例系统验证了开发的MujavaX的正确性与可行性,实验结果表明该系统能够生成指定分布的非均匀变体集合.     

一种Web服务的测试数据自动生成方法

软件测试是保证Web服务质量的重要技术手段.测试数据生成是Web服务测试的重要内容.测试数据的质量将直接影响Web服务测试的效率和成本.文章基于合约式设计的 Web服务测试技术,提出一种 Web服务的测试数据自动生成方法.首先根据WSDL文档采用随机法自动生成初始测试数据,然后使用合约变异技术进行测试数据的选择,据此可以生成一组达到一定合约变异充分度的有效测试数据,从而提高 Web服务的测试质量和效率.最后实现了一个Web服务的测试数据自动生成工具原型,并通过实验验证了方法的有效性.     

一种新的变异测试数据自动生成方法

变异测试是一种行之有效的软件测试方法,通过使用变异算子产生变异体系统地模拟软件中的各种缺陷,然后构造能够杀死这些变异体的测试数据集.自动生成能够杀死变异体的测试数据将提高变异测试的效率和有效性.当前的研究工作只考虑生成杀死单个变异体的测试数据.文中根据杀死同一位置的多个变异体的条件相近的特点,提出一种对杀死这些变异体的条件进行组合,然后生成同时杀死该位置多个变异体的测试数据的方法;给出相应的支持工具,并且通过实验验证方法的有效性.     

基于变异的Web复合服务测试数据选择方法

Web服务集成测试的测试数据数量巨大,影响了测试的质量和效率。提出一种基于变异技术的Web复合服务测试数据选择方法,根据BPEL(Business Process Execution Language)文档获取复合服务的路径集,采用接口变异算子和路径变异算子生成变异体,根据初始测试数据杀死变异体的能力选择有效的测试数据。通过实验验证了方法的有效性,选择出的测试数据数量较少且有效。     

变异测试:原理、优化和应用

变异测试是一种基于缺陷的软件测试技术,在近四十年得到国内外学者的广泛关注,并取得了一些研究成果。对已有的研究工作进行总结,将其分为变异测试原理、优化和应用三个模块。其中在变异测试原理模块中,给出变异测试的基本假设,对变异测试分析流程进行介绍,并对其中的重要概念依次给出定义,从静态检测和动态检测两个角度对等价变异体检测技术进行总结。在变异测试优化模块中,从变异体选择优化和变异体执行优化两个角度对已有研究工作进行总结。在变异测试应用模块中,选择了测试用例集充分性评估、测试用例生成和回归测试三个应用领域,对研究工作进行分类总结。最后对变异测试的未来研究方向进行了展望。     

基于代数式规范的变异测试方法的设计与实现

软件测试是软件工程中保证软件产品质量的重要组成部分.变异测试是一种衡量测试用例集完备性的测试策略,也被用于生成完备的测试用例集.为了提出一种基于代数式规范的新的变异测试方法,为此设计了12类针对代数式规范的变异操作符,对5个代数式规范进行了实验,并进行了结果分析.结果表明基于代数式规范的变异测试方法相比基于代码的传统变异测试方法,生成更少的变异体,也大幅度提升了变异测试的效率.     

基于缺陷分析的软件测试有效性评估方法

为利用软件测试活动检测到的缺陷信息进行测试活动的有效性评估,改进软件测试活动,提出一种基于缺陷度量和缺陷基线的测试有效性评估方法.以正交缺陷分类方法为基础建立软件缺陷的分类框架,研究缺陷触发特征、缺陷类型统计信息与测试活动的关联,建立缺陷关联基线.通过分析实际缺陷度量与缺陷基线的偏离情况,评估测试活动的有效性,明确测试活动的改进方向.实例分析表明,该模型能够便捷地应用于测试活动有效性的定性评估.     

基于模型检测技术的变异测试用例生成方法

为了进行基于模型的软件测试变异分析,文中提出了一种基于模型检测的变异测试用例生成方法。基于模型检测工具UPPAAL的形式化分析与测试框架,首先用符合规范的时间自动机模型描述被测系统;然后基于时间自动机模型的基本结构和语法,对系统模型进行一组变异操作,并模拟实现时可能出现的一些错误;对变异后的模型分别使用UPPAAL Yggdrasil工具,生成一组能覆盖变异区域的测试用例;在系统变异模型上执行生成的测试用例,根据测试执行结果(是否能“杀死”变异体)筛选出一组有效的测试用例。通过实例验证,所提方案生成的测试用例是有效的,且测试用例集变异分数优于现有的基于状态机复制的变异测试用例自动生成方法和基于模型中变换覆盖的变异测试用例生成方法。     

Evolutionary mutation testing

Context

Mutation testing is a testing technique that has been applied successfully to several programming languages. However, it is often regarded as computationally expensive, so several refinements have been proposed to reduce its cost. Moreover, WS-BPEL compositions are being widely adopted by developers, but present new challenges for testing, since they can take much longer to run than traditional programs of the same size. Therefore, it is interesting to reduce the number of mutants required.

Objective

We present Evolutionary Mutation Testing (EMT), a novel mutant reduction technique for finding mutants that help derive new test cases that improve the quality of the initial test suite. It uses evolutionary algorithms to reduce the number of mutants that are generated and executed with respect to the exhaustive execution of all possible mutants, keeping as many difficult to kill and potentially equivalent mutants (strong mutants) as possible in the reduced set.

Method

To evaluate EMT we have developed GAmera, a mutation testing system powered by a co-evolutive genetic algorithm. We have applied this system to three WS-BPEL compositions to estimate its effectiveness, comparing it with random selection.

Results

The results obtained experimentally show that EMT can select all strong mutants generating 15% less mutants than random selection in over 20% less time for complex compositions. When generating a percentage of all mutants, EMT finds on average more strong mutants than random selection. This has been confirmed to be statistically significant within a 99.9% confidence interval.

Conclusions

EMT has reduced for the three tested compositions the number of mutants required to select those which are useful to derive new test cases that improve the quality of the test suite. The directed search performed by EMT makes it more effective than random selection, especially as compositions become more complex and the search space widens.     

Parallel mutation testing

Despite the existing techniques to reduce the costs of mutation analysis, the computational cost to apply mutation testing with large applications can be very high. One effective technique to improve the efficiency of mutation without losing effectiveness is parallel execution, where mutants and tests are executed in parallel processors, reducing the total time needed to perform mutation analysis. This paper presents a study of this technique adapted to current technologies. Five algorithms to execute mutants in parallel are analysed with three studies that use different network configurations and different number of processors with diverse characteristics. The experiments are performed with Bacterio ^P, a tool that is also presented. Unlike previous studies about parallel mutant execution, which date from the mid‐1990s, in the studies in this paper, the communication time in parallel systems no longer acts as a bottleneck. Thus, dynamic strategies, which require more communication, combined with other mutant cost reduction techniques, are the best strategies to run mutants in parallel.Copyright © 2012 John Wiley & Sons, Ltd.     

MuJava: an automated class mutation system

Several module and class testing techniques have been applied to object‐oriented (OO) programs, but researchers have only recently begun developing test criteria that evaluate the use of key OO features such as inheritance, polymorphism, and encapsulation. Mutation testing is a powerful testing technique for generating software tests and evaluating the quality of software. However, the cost of mutation testing has traditionally been so high that it cannot be applied without full automated tool support. This paper presents a method to reduce the execution cost of mutation testing for OO programs by using two key technologies, mutant schemata generation (MSG) and bytecode translation. This method adapts the existing MSG method for mutants that change the program behaviour and uses bytecode translation for mutants that change the program structure. A key advantage is in performance: only two compilations are required and both the compilation and execution time for each is greatly reduced. A mutation tool based on the MSG/bytecode translation method has been built and used to measure the speedup over the separate compilation approach. Experimental results show that the MSG/bytecode translation method is about five times faster than separate compilation. Copyright © 2004 John Wiley & Sons, Ltd.     

Automatic generation of basis test paths using variable length genetic algorithm

Path testing is the strongest coverage criterion in white box testing. Finding target paths is a key challenge in path testing. Genetic algorithms have been successfully used in many software testing activities such as generating test data, selecting test cases and test cases prioritization. In this paper, we introduce a new genetic algorithm for generating test paths. In this algorithm the length of the chromosome varies from iteration to another according to the change in the length of the path. Based on the proposed algorithm, we present a new technique for automatically generating a set of basis test paths which can be used as testing paths in any path testing method. The proposed technique uses a method to verify the independency of the generated paths to be included in the basis set of paths. In addition, this technique employs a method for checking the feasibility of the generated paths. We introduce new definitions for the key concepts of genetic algorithm such as chromosome representation, crossover, mutation, and fitness function to be compatible with path generation. In addition, we present a case study to show the efficiency of our technique. We conducted a set of experiments to evaluate the effectiveness of the proposed path generation technique. The results showed that the proposed technique causes substantial reduction in path generation effort, and that the proposed GA algorithm is effective in test path generation.     

GUI testing assisted by human knowledge: Random vs. functional

Software testing is a labor-intensive task in software development life-cycle. Human knowledge is useful in the practices of software testing, especially GUI testing. There are many strategies for GUI testing assisted by human knowledge, in which manual random testing and manual functional testing are two of widely used ones. In this paper, an empirical study is conducted to compare random testing and functional testing in order to provide guidelines for GUI testing. 234 participants were recruited to create thousands of random and functional test cases for open source GUI applications. Some of these test cases were selected with certain coverage criteria and then run on GUI applications to evaluate random testing and functional testing. We study three aspects on the two testing strategies: effectiveness, complementarity and impact of test case length. Some useful observations in the empirical study are: (1) Random testing is more effective in the early stage of testing on small applications and functional testing has more extensive applicability for testing large sized applications. (2) Random testing and functional testing exhibit some complementarity in our experiment. (3) Short test cases can reveal some faults more quickly and long test cases can reveal more faults lastingly.     

Should software testers use mutation analysis to augment a test set?

Mutation testing has historically been used to assess the fault-finding effectiveness of a test suite or other verification technique. Mutation analysis, rather, entails augmenting a test suite to detect all killable mutants. Concerns about the time efficiency of mutation analysis may prohibit its widespread, practical use. The goal of our research is to assess the effectiveness of the mutation analysis process when used by software testers to augment a test suite to obtain higher statement coverage scores. We conducted two empirical studies and have shown that mutation analysis can be used by software testers to effectively produce new test cases and to improve statement coverage scores in a feasible amount of time. Additionally, we find that our user study participants view mutation analysis as an effective but relatively expensive technique for writing new test cases. Finally, we have shown that the choice of mutation tool and operator set can play an important role in determining how efficient mutation analysis is for producing new test cases.     

关于蜕变测试和特殊用例测试的实例研究

提出了基于蜕变测试方法的集成测试环境MTest,进而为检验蜕变测试方法的能力和效率,以稀疏矩阵乘法程序为例设计了一组实验.该实验基于变异分析技术,分别以mutation score和错误发现率为度量指标,定量地分析和对比了特殊用例测试,以特殊测试用例和随机测试用例为源测试用例的蜕变测试这3种方法的测试能力和效率.该实验可在MTest测试环境下自动完成.实验结果表明,蜕变测试与特殊用例测试之间是互补的,而且就蜕变测试的源测试用例而言,随机测试用例在测试能力和效率上优于特殊测试用例.     

Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random testing normally uses fewer test cases than random testing for detecting the first software failure. However, under many practical situations, testing should not be withheld after the detection of a failure. Thus, it is important to investigate the effectiveness with respect to the detection of multiple failures. In this paper, we compare adaptive random testing and random testing under various scenarios and examine whether adaptive random testing is still able to use fewer test cases than random testing to detect multiple software failures. Our study delivers some interesting results and highlights a number of promising research projects. Copyright © 2011 John Wiley & Sons, Ltd.     

Evolutionary mutation testing for IoT with recorded and generated events

Mutation testing is a testing technique that has been applied successfully to several programming languages. Despite its benefits for software testing, the high computational cost of mutation testing has kept it from being widely used. Several refinements have been proposed to reduce its cost by reducing the number of generated mutants; one of those is evolutionary mutation testing (EMT). Evolutionary mutation testing aims at generating a reduced set of mutants with an evolutionary algorithm, which searches for potentially equivalent and difficult to kill mutants that help improve the test suite. Evolutionary mutation testing has been evaluated in two contexts so far, ie, web service compositions and object-oriented C++ programmes. This study explores its performance when applied to event processing language queries of various domains. This study also considers the impact of the test data, since a lack of events or the need to have specific values in them can hinder testing. The effectiveness of evolutionary mutation testing with the original test data generators and the new internet of things test event generator tool is compared in multiple case studies.