Comparing logic coverage criteria on test case prioritization

Logic coverage criteria have been widely used in the testing of safety-critical software.In the past few years,fault-based logic coverage criteria have been studied intensively both in theory and in practice.However,there is a lack of authentic evidence of the comparison of fault-based logic coverage criteria with other logic coverage criteria,such as branch coverage and modified condition/decision coverage(MC/DC).In this paper,we present a comprehensive empirical study that compares these logic coverage criteria on test case prioritization,which is currently a hot topic in software testing.Several useful conclusions are drawn from our research:(1) Fine-grained coverage criteria are always more effective and efficient.(2) The effectiveness of fault-based logic coverage criteria is not significantly different from that of MC/DC in terms of statistics,but the former is more stable.(3) A random strategy is more effective than branch coverage if a certain number of test cases are redundant.     

基于OTT策略的组合测试用例优先级排序方法

在组合测试用例优先级排序问题中,通常采用组合覆盖率为排序标准,该方法能够尽快满足覆盖率的要求,但其排序影响因子较为单一,缺陷检测能力不稳定。针对此问题,该文结合One-Test-at-a-Time（OTT）策略提出了一种在线调整的组合测试用例优先级排序方法,引入多重待覆盖率、测试用例失效率和测试用例重要程度3个影响因子用于衡量组合测试用例优先级。根据测试用例的在线测试反馈信息,该方法能够实时调整组合测试用例的优先级,增加了优先级排序的准确性。实验结果表明,相较于original、基于单一覆盖和多重覆盖的组合测试优先级排序方法,该方法在缺陷检测能力和稳定性上具有更强的竞争力。     

XML-manipulating test case prioritization for XML-manipulating services

A web service may evolve autonomously, making peer web services in the same service composition uncertain as to whether the evolved behaviors are compatible with its original collaborative agreement. Although peer services may wish to conduct regression testing to verify the agreed collaboration, the source code of the former service may be inaccessible to them. Owing to the black-box nature of peer services, traditional code-based approaches to regression testing are inapplicable. In addition, traditional techniques assume that a regression test suite for verifying a web service is available. The location to store a regression test suite is also a problem. On the other hand, we note that the rich interface specifications of a web service provide peer services with a means to formulate black-box testing strategies. In this paper, we provide a strategy for black-box service-oriented testing. We also formulate new test case prioritization strategies using tags embedded in XML messages to reorder regression test cases, and reveal how the test cases use the interface specifications of web services. We experimentally evaluate the effectiveness of these black-box strategies in revealing regression faults in modified WS-BPEL programs. The results show that the new techniques can have a high chance of outperforming random ordering. Moreover, our experiment shows that prioritizing test cases based on WSDL tag coverage can achieve a smaller variance than that based on the number of tags in XML messages in regression test cases, even though their overall fault detection rates are similar.     

System testing for object‐oriented systems with test case prioritization

This paper presents an approach to generate test cases from UML 2.0 sequence diagrams and subsequently prioritize those test cases using model information encapsulated in the sequence diagrams. The test cases generated according to the proposed approach satisfy the scenario coverage criterion and are suitable for system‐level testing. For prioritizing test cases, three different prioritization metrics are proposed. The values of these prioritization metrics can be analytically computed from the model information only. This paper also presents an approach to generate test data using a concept called rule‐based matrix. The prioritization metrics are used to control the number of test data without compromising the test adequacy. The effectiveness of the proposed approach has been verified using two industrial designs. Copyright © 2009 John Wiley & Sons, Ltd.     

利用佳点集遗传算法的白盒测试用例优先排序

在软件演化过程中,测试用例优先排序作为一种高效实用的回归测试技术,对于提高缺陷的早期检测速率和降低测试成本有重要意义。针对传统遗传算法在白盒测试用例优先排序中收敛速度慢和稳定性差的问题,采用佳点集遗传算法求解白盒测试用例优先排序问题。算法根据程序实体覆盖矩阵对个体进行编码,以程序实体覆盖平均百分比作为适应度函数,采用随机抽样选择算子和佳点集交叉算子产生新一代种群。实验选择6个典型的基准开源项目,以语句、分支和方法作为程序实体,实验结果表明佳点集遗传算法收敛速度快、稳定性好,为回归测试提供了一个有效的测试用例优先排序方法,有助于尽早发现软件缺陷,降低测试成本。     

Risk-based test case prioritization using a fuzzy expert system

Context: The use of system requirements and their risks enables software testers to identify more important test cases that can reveal the faults associated with system components.Objective: The goal of this research is to make the requirements risk estimation process more systematic and precise by reducing subjectivity using a fuzzy expert system. Further, we provide empirical results that show that our proposed approach can improve the effectiveness of test case prioritization.Method: In this research, we used requirements modification status, complexity, security, and size of the software requirements as risk indicators and employed a fuzzy expert system to estimate the requirements risks. Further, we employed a semi-automated process to gather the required data for our approach and to make the risk estimation process less subjective.Results: The results of our study indicated that the prioritized tests based on our new approach can detect faults early, and also the approach can be effective at finding more faults earlier in the high-risk system components compared to the control techniques.Conclusion: We proposed an enhanced risk-based test case prioritization approach that estimates requirements risks systematically with a fuzzy expert system. With the proposed approach, testers can detect more faults earlier than with other control techniques. Further, the proposed semi-automated, systematic approach can easily be applied to industrial applications and can help improve regression testing effectiveness.     

基于蚁群算法的测试用例优先排序

测试用例优先排序技术通过优化测试用例的执行次序来提高软件测试的效率,是增强型软件测试和回归测试的重要研究课题。针对基于需求的测试用例优先排序问题,提出了一种基于蚁群算法的求解方法,采用不同的测试用例间距离及用例序列评价策略,给出了该方法的2种不同实现方式。首先,针对黑盒测试特点,设计了基于需求的一般性测试用例序列评价指标;其次,提出测试用例吸引度概念,基于测试用例吸引度定义了测试用例间的距离;然后,给出了信息素更新策略、最优解集更新策略、局部最优解突变策略等主要设计策略,分别实现了该方法基于距离和基于指标的2种实现方式。实验结果表明,该方法具有很好的全局寻优能力,整体效果上优于粒子群算法、遗传算法和随机测试。     

Research on the logic coverage method of white-box testing

Software testing is an effective and direct way to improve the reliability of software. This paper presents a class of white-boxes testing methods-the logic coverage testing methods, and discuses how to design testing cases of the various programmes by using condition coverage method with examples. And makes condition coverage method, statement coverage method and decision coverage method in-depth and detailed comparison. At last, analyzes the current software testing methods.     

UCov: a user‐defined coverage criterion for test case intent verification

The goal of regression testing is to ensure that the behaviour of existing code, believed correct by previous testing, is not altered by new program changes. This paper argues that the primary focus of regression testing should be on code associated with (1) earlier bug fixes and (2) particular application scenarios considered to be important by the developer or tester. Existing coverage criteria do not enable such focus, for example, 100% branch coverage does not guarantee that a given bug fix is exercised or a given application scenario is tested. Therefore, there is a need for a new and complementary coverage criterion in which the user can definea test requirement characterizing a given behaviour to be covered as opposed to choosing from a pool of pre‐defined and generic program elements. This paper proposes this new methodology and calls it UCov, a user‐defined coverage criterion wherein a test requirement is an execution pattern of program elements, and possibly predicates, that a test case must satisfy. The proposed criterion is not meant to replace existing criteria, but to complement them as it focuses the testing on important code patterns that could go untested otherwise. UCov supports test case intent verification. For example, following a bug fix, the testing team may augment the regression suite with the test case that revealed the bug. However, this test case might become obsolete due to code modifications not related to the bug. But if a test requirement characterizing the bug was defined by the user, UCov would determine that test case intent verification failed. The UCov methodology was implemented for the Java platform, was successfully applied onto 10 real‐life case studies and was shown to have advantages over JUnit. The implementation comprises the following tools: (1) TRSpec: allows the user to easily specify complex test requirements; (2) TRCheck: checks whether user‐defined test requirements were satisfied, that is, supports test case intent verification; and (3) TRMigrate: migrates user‐defined test requirements to subsequent versions of a given program. Copyright © 2016 John Wiley & Sons, Ltd.     

基于离散粒子群算法的测试用例优先排序

测试用例优先排序技术能够有效提高回归测试效率,是软件测试的热点研究课题之一。针对基于需求的测试用例优先排序方法可操作性差的问题,提出了一种改进的基于测试点覆盖和离散粒子群优化算法的求解方法（TCP-DPSO）。首先,把影响排序的各种因素分为测试收益型因素和测试成本型因素两大类,通过加权平均的方式进行归一化,得到基于需求的通用测试平均收益率评价指标;然后,利用交换子和基本交换序列定义粒子的位置和速度,借鉴遗传算法（GA）变异策略引入变异算子,采用时变惯性权重调整粒子的探索能力和开发能力,促进可持续进化和逼近优化目标。实验结果表明,TCP-DPSO在最优解质量上与遗传算法相当,大幅优于随机测试,在最优解成功率和平均求解时间上优于遗传算法,具有更好的算法稳定性。     

基于MDA的软件测试用例生成

将MDA中模型驱动的软件代码自动化生成思想应用于模型驱动的软件测试用例自动化生成。从UML/OCL模型出发,采用缺陷测试理论、变异分析技术,结合约束处理规则,开发一个可以自动生成单元测试用例的框架,提高软件测试的自动化程度,从整体上提高软件的开发效率。     

Declarative paradigm of test coverage

Two facts about declarative programming prevent the application of conventional testing methods. First, the classical test coverage measures such as statement, branch or path coverage, cannot be used, since in declarative programs no control flow notion exists. Second, there is no widely accepted language available for formal specification, since predicate logic, which is the most common formalism for declarative programming, is already a very high-level abstract language. This paper presents a new approach exending previous work by the authors on test input generation for declarative programs. For this purpose, the existing program instrumentation notion is extended and a new logic coverage measure is introduced. The approach is mathematically formalized and the goal of achieving 100% program logic coverage controls the automatic test input generation. The method is depicted by means of logic programming; the results are, however, generally applicable. Finally, the concepts introduced have been used practically within a test environment. © 1998 John Wiley & Sons, Ltd.     

基于历史数据和多目标优化的测试用例排序方法

针对如何提高测试用例序列的揭错效率和回归测试效益问题,提出一种基于历史数据和多目标优化的测试用例排序方法。首先,根据测试用例的文本主题相似性和代码覆盖相似性对测试用例集进行聚类,并根据历史执行信息对测试用例间的执行失败关系进行关联规则挖掘,从而为后续过程做准备;然后,利用多目标优化算法对每个类簇内的测试用例进行排序,在此之后生成最终排序序列,使相似的测试用例分隔开;最后,利用测试用例间的关联规则,动态调整测试用例执行次序,从而使可能失败的测试用例优先执行,以进一步提高缺陷检测效率。与随机排序方法、基于聚类的排序方法、基于主题模型的排序方法、基于关联规则和多目标优化的排序方法相比,所提方法的平均故障检测率（APFD）平均值分别提高了12.59%、5.98%、3.01%和2.95%,基于成本的平均故障检测率（APFDc）平均值分别提高了17.17%、5.04%、5.08%和8.21%。实验结果表明,所提方法能有效提高回归测试效益。     

基于Tent混沌的测试用例优先级排序

针对标准粒子群算法（Particle Swarm Optimization,PSO）后期出现的早熟收敛,提出了一种基于Tent混沌的粒子群优化算法（Tent-Chaos Particle Swarm Optimization,TCPSO）用于测试用例优先级排序。首先,利用改进的Tent映射的三大特性初始化种群,使得粒子均匀分布,提高初始解的质量;并通过非线性递减的惯性权重函数对学习因子进行改进,以更新粒子速度与位置信息;其次,对陷入局部最优的粒子p_id进行混沌搜索,跳出局部最优,同时对当前种群中部分最差粒子p_iw进行混沌搜索,改善种群多样性;最后,采用测试用例的分支覆盖率和缺陷检测率作为评价标准,评判测试用例优劣程度。实验表明,提出的改进方法在分支覆盖率和缺陷检测率指标上均有优势。     

用于回归测试数据验算的筛选方法

通过利用覆盖数据技术和回归测试集选择技术,提出一种用于回归测试数据验算的筛选方法,该方法通过深度优先遍历程序的相关记录来筛选测试用例,能有效地提高回归测试的准确性,减少回归测试的测试时间和省略无需测试的测试用例,以达到降低回归测试的成本的目的.对顺序、循环和分支3类程序设计了相关的实验,比较了该算法在这3类程序上的使用效果.     

Regression testing minimization,selection and prioritization: a survey

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.     

Experiments with test case prioritization using relevant slices

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible and to gain confidence that changes to software do not introduce defects. Once developed, test suites are reused and updated frequently, and their sizes grow as software evolves. Due to time and resource constraints, an important goal during regression testing of software is to prioritize the execution of test cases in a suite so as to improve the chances of increasing the rate of fault detection. Prior techniques for test case prioritization are based on the total number of coverage requirements exercised by the test cases. In this paper, we present a new approach to prioritize test cases that takes into account the coverage requirements present in the relevant slices of the outputs of test cases. We have implemented three different heuristics based on our relevant slicing based approach to prioritize test cases and conducted experiments to compare the effectiveness of our techniques with those of the traditional techniques that only account for the total requirement coverage. Our detailed experimental study and results provide interesting insights into the effectiveness of using relevant slices for test case prioritization in terms of ability to achieve high rate of fault detection.     

Fault-based test suite prioritization for specification-based testing

Context

Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in nature and not theoretically driven; hence, they do not necessarily provide sound estimates. Also, these techniques are not applicable when the source code is not available or when the software is tested for the first time.

Objective

We propose and develop the novel notion of fault-based prioritization of test cases which directly utilizes the theoretical knowledge of their fault-detecting ability and the relationships among the test cases and the faults in the prescribed fault model, based on which the test cases are generated.

Method

We demonstrate our approach of fault-based prioritization by applying it to the testing of the implementation of logical expressions against their specifications. We then validate our proposal by an empirical study that evaluates the effectiveness of prioritization techniques using two different metrics.

Results

A theoretically guided fault-based prioritization technique generally outperforms other techniques under study, as assessed by two different metrics. Our empirical results also show that the technique helps to reveal all target faults by executing only about 72% of the prioritized test suite, thereby reducing the effort required in testing.

Conclusions

The fault-based prioritization approach is not only applicable to the instance empirically validated in this paper, but should also be adaptable to other fault-based testing strategies. We also envisage new research directions to be opened up by our work.     

Generating test data for both path coverage and fault detection using genetic algorithms

The aim of software testing is to find faults in a program under test, so generating test data that can expose the faults of a program is very important. To date, current stud- ies on generating test data for path coverage do not perform well in detecting low probability faults on the covered path. The automatic generation of test data for both path coverage and fault detection using genetic algorithms is the focus of this study. To this end, the problem is first formulated as a bi-objective optimization problem with one constraint whose objectives are the number of faults detected in the traversed path and the risk level of these faults, and whose constraint is that the traversed path must be the target path. An evolution- ary algorithm is employed to solve the formulated model, and several types of fault detection methods are given. Finally, the proposed method is applied to several real-world programs, and compared with a random method and evolutionary opti- mization method in the following three aspects： the number of generations and the time consumption needed to generate desired test data, and the success rate of detecting faults. The experimental results confirm that the proposed method can effectively generate test data that not only traverse the target path but also detect faults lying in it.