An improved method for test case prioritization by incorporating historical test case data

Test case prioritization reorders test cases from a previous version of a software system for the current release to optimize regression testing. We have previously introduced a technique for test case prioritization using historical test case performance data. The technique was based on a test case prioritization equation, which directly computes the priority of each test case using the historical information of the test case using an equation with constant coefficients. This technique was compared just with random ordering approach. In this paper, we present an enhancement of the aforementioned technique in two ways. First, we propose a new prioritization equation with variable coefficients gained according to the available historical performance data, which acts as a feedback from the previous test sessions. Second, a family of comprehensive empirical studies has been conducted to evaluate the performance of the technique. We have compared the proposed technique with our previous technique and the technique proposed by Kim and Porter. The experimental results demonstrate the effectiveness of the proposed technique in accelerating the rate of fault detection in history-based test case prioritization.     

Towards the prioritization of system test cases

During software development, companies are frequently faced with lack of time and resources, which limits their ability to effectively complete testing efforts. This paper presents a system‐level, value‐driven approach to test case prioritization called the Prioritization of Requirements for Test (PORT). PORT involves analysing and assigning value to each requirement using the following four factors: requirements volatility, customer priority, implementation complexity, and fault proneness. System test cases are prioritized such that the test cases for requirements with higher priority are executed earlier during system test. PORT was applied to four student team projects as well as an industrial case study. The results show that PORT improves the rate of detection of severe failures over random prioritization. Additionally, the results indicate that customer priority was the most important contributor towards improved rate of failure detection. Copyright © 2013 John Wiley & Sons, Ltd.     

Prioritizing test cases for regression testing

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites     

Improving test efficiency through system test prioritization

Software testing is an expensive process consuming at least 50% of the total development cost. Among the types of testing, system testing is the most expensive and complex. Companies are frequently faced with budgetary constraints, which may limit their ability to effectively complete testing efforts before delivering a software product. We build upon prior test case prioritization research and present a system-level approach to test case prioritization called Prioritization of Requirements for Test (PORT). PORT prioritizes system test cases based on four factors for each requirement: customer priority, implementation complexity, fault proneness, and requirements volatility. Test cases for requirements with higher priority based upon a weighted average of these factors are executed earlier in system test. An academic feasibility study and three post hoc industrial studies were conducted. Results indicate that PORT can be used to improve the rate of failure detection when compared with a random and operational profile-driven random approach. Furthermore, we investigated the contribution of the prioritization factors towards the improved rate of failure detection and found customer priority was the most significant contributor. Tool support is provided for the PORT scheme which allows for automatic collection of the four factor values and the resultant test case prioritization.     

基于聚类算法的测试用例排序

在软件开发过程中,回归测试是一项重要而复杂耗时的工作。测试用例排序技术作为提高测试效率的一种有效手段,是近年的研究热点之一。测试用例排序技术根据测试目标对测试用例进行排序,使得最优的测试用例能够优先执行。同时,各种软件信息的底层结构及关联信息有助于提高测试效率。不同的代码单元在测试过程中拥有不同的测试需求。因此,利用聚类算法能够有效地分析测试用例间的结构信息,从而指导测试用例的排序过程。实验结果表明,新的测试用例排序策略有效地提高了测试用例排序的效果,能够更早地检测到软件错误。     

Similarity-based test case prioritization using ordered sequences of program entities

Test suites often grow very large over many releases, such that it is impractical to re-execute all test cases within limited resources. Test case prioritization rearranges test cases to improve the effectiveness of testing. Code coverage has been widely used as criteria in test case prioritization. However, the simple way may not reveal some bugs, such that the fault detection rate decreases. In this paper, we use the ordered sequences of program entities to improve the effectiveness of test case prioritization. The execution frequency profiles of test cases are collected and transformed into the ordered sequences. We propose several novel similarity-based test case prioritization techniques based on the edit distances of ordered sequences. An empirical study of five open source programs was conducted. The experimental results show that our techniques can significantly increase the fault detection rate and be effective in detecting faults in loops. Moreover, our techniques are more cost-effective than the existing techniques.     

Prioritizing MCDC test cases by spectral analysis of Boolean functions

Test case prioritization aims at scheduling test cases in an order that improves some performance goal. One performance goal is a measure of how quickly faults are detected. Such prioritization can be performed by exploiting the fault exposing potential (FEP) parameters associated to the test cases. The FEP is usually approximated by mutation analysis under certain fault assumptions. Although this technique is effective, it could be relatively expensive compared to the other prioritization techniques. This study proposes a cost‐effective FEP approximation for prioritizing modified condition decision coverage (MCDC) test cases. A strict negative correlation between the FEP of an MCDC test case and the influence value of the associated input condition allows to order the test cases easily without the need of an extensive mutation analysis. The method is entirely based on mathematics and it provides useful insight into how spectral analysis of Boolean functions can benefit software testing.     

基于定值-引用链的测试用例优先级排序算法

测试用例优先级排序作为一种高效实用的回归测试技术,通常以测试用例的覆盖度作为优先级排序的量化指标,忽略了测试用例的其他测试性能。针对该问题,提出一种基于DU链的测试用例优先级排序算法。该算法 综合考虑 测试用例的DU链覆盖度和回归测试的错误检测能力,对测试用例优先级进行量化。与已有算法相比,该算法基于数据流覆盖,充分利用了测试执行的历史信息和程序模块的耦合信息,在排序过程中动态计算测试用例的优先级量化值。实验结果表明,采用优先级排序算法的测试用例集能在测试过程中以较短的时间发现更多的错误,有效地提高了回归测试的检错效率。     

基于需求的系统级黑盒测试用例优先化方法

测试用例优先化技术有利于提高测试的质量和效率。文章提出了一种在系统测试阶段基于需求的测试用例优先化方法TRP,并通过实验结果加以验证。与随机方法对测试用例排序的测试相比,TRP方法能尽早发现软件严重缺陷并提高缺陷检测率。     

Experiments with test case prioritization using relevant slices

Software testing and retesting occurs continuously during the software development lifecycle to detect errors as early as possible and to gain confidence that changes to software do not introduce defects. Once developed, test suites are reused and updated frequently, and their sizes grow as software evolves. Due to time and resource constraints, an important goal during regression testing of software is to prioritize the execution of test cases in a suite so as to improve the chances of increasing the rate of fault detection. Prior techniques for test case prioritization are based on the total number of coverage requirements exercised by the test cases. In this paper, we present a new approach to prioritize test cases that takes into account the coverage requirements present in the relevant slices of the outputs of test cases. We have implemented three different heuristics based on our relevant slicing based approach to prioritize test cases and conducted experiments to compare the effectiveness of our techniques with those of the traditional techniques that only account for the total requirement coverage. Our detailed experimental study and results provide interesting insights into the effectiveness of using relevant slices for test case prioritization in terms of ability to achieve high rate of fault detection.     

Regression testing minimization,selection and prioritization: a survey

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.     

利用佳点集遗传算法的白盒测试用例优先排序

在软件演化过程中,测试用例优先排序作为一种高效实用的回归测试技术,对于提高缺陷的早期检测速率和降低测试成本有重要意义。针对传统遗传算法在白盒测试用例优先排序中收敛速度慢和稳定性差的问题,采用佳点集遗传算法求解白盒测试用例优先排序问题。算法根据程序实体覆盖矩阵对个体进行编码,以程序实体覆盖平均百分比作为适应度函数,采用随机抽样选择算子和佳点集交叉算子产生新一代种群。实验选择6个典型的基准开源项目,以语句、分支和方法作为程序实体,实验结果表明佳点集遗传算法收敛速度快、稳定性好,为回归测试提供了一个有效的测试用例优先排序方法,有助于尽早发现软件缺陷,降低测试成本。     

Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites. In particular, Java and the JUnit testing framework are being used extensively to build software systems in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm. To investigate the practical implications of these results, we present a set of cost-benefits models for test case prioritization, and show how the effectiveness differences observed can result in savings in practice, but vary substantially with the cost factors associated with particular testing processes.     

基于动态相似度的错误定位优先排序方法

在软件测试中,错误定位优先排序通过优化测试用例的执行次序来提高错误定位的效果,并将检测错误和定位错误相结合,以降低测试成本。 提出了一种基于动态相似度的错误定位优先排序方法,在相似度计算中,引入了语句怀疑度,提高了相似度计算的有效性以及错误定位的准确度;同时分析并验证了不同测试用例优先排序算法对后续定位错误的影响。在6个C基准程序上,针对3种广泛采用的测试用例优先排序算法和2种错误定位技术进行了实验,结果表明提出的方法能提高错误定位的准确度和效率。     

Putting your best tests forward

Test case prioritization orders tests so that they help you meet your testing goals earlier during regression testing. Prioritization techniques can, for example, order tests to achieve coverage at the fastest rate possible, exercise features in order of expected frequency of use, or reveal faults as early as possible. We focus on the last goal, which we describe as "increasing a test suite's rate of fault detection" or the speed with which the test suite reveals faults. A faster fault detection rate during regression testing provides earlier feedback on a system under test, supporting earlier strategic decisions about release schedules and letting engineers begin debugging sooner. Also, if testing time is limited or unexpectedly reduced, prioritization increases the chance that testing resources will have been spent as cost effectively as possible in the available time.     

On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general     

Test case prioritization: a systematic mapping study

Test case prioritization techniques, which are used to improve the cost-effectiveness of regression testing, order test cases in such a way that those cases that are expected to outperform others in detecting software faults are run earlier in the testing phase. The objective of this study is to examine what kind of techniques have been widely used in papers on this subject, determine which aspects of test case prioritization have been studied, provide a basis for the improvement of test case prioritization research, and evaluate the current trends of this research area. We searched for papers in the following five electronic databases: IEEE Explorer, ACM Digital Library, Science Direct, Springer, and Wiley. Initially, the search string retrieved 202 studies, but upon further examination of titles and abstracts, 120 papers were identified as related to test case prioritization. There exists a large variety of prioritization techniques in the literature, with coverage-based prioritization techniques (i.e., prioritization in terms of the number of statements, basic blocks, or methods test cases cover) dominating the field. The proportion of papers on model-based techniques is on the rise, yet the growth rate is still slow. The proportion of papers that use datasets from industrial projects is found to be 64 %, while those that utilize public datasets for validation are only 38 %. On the basis of this study, the following recommendations are provided for researchers: (1) Give preference to public datasets rather than proprietary datasets; (2) develop more model-based prioritization methods; (3) conduct more studies on the comparison of prioritization methods; (4) always evaluate the effectiveness of the proposed technique with well-known evaluation metrics and compare the performance with the existing methods; (5) publish surveys and systematic review papers on test case prioritization; and (6) use datasets from industrial projects that represent real industrial problems.     

基于离散粒子群算法的测试用例优先排序

测试用例优先排序技术能够有效提高回归测试效率,是软件测试的热点研究课题之一。针对基于需求的测试用例优先排序方法可操作性差的问题,提出了一种改进的基于测试点覆盖和离散粒子群优化算法的求解方法（TCP-DPSO）。首先,把影响排序的各种因素分为测试收益型因素和测试成本型因素两大类,通过加权平均的方式进行归一化,得到基于需求的通用测试平均收益率评价指标;然后,利用交换子和基本交换序列定义粒子的位置和速度,借鉴遗传算法（GA）变异策略引入变异算子,采用时变惯性权重调整粒子的探索能力和开发能力,促进可持续进化和逼近优化目标。实验结果表明,TCP-DPSO在最优解质量上与遗传算法相当,大幅优于随机测试,在最优解成功率和平均求解时间上优于遗传算法,具有更好的算法稳定性。     

Prioritizing test cases for early detection of refactoring faults

Refactoring edits are error‐prone, requiring cost‐effective testing. Regression test suites are often used as a safety net for decreasing the chances of behavioural changes. Because of the high costs related to handling massive test suites, prioritization techniques can be applied to reorder test case execution, fostering early fault detection. However, traditional prioritization techniques are not specifically designed for detecting refactoring‐related faults. This article proposes refactoring‐based approach (RBA), a refactoring‐aware strategy for prioritizing regression test cases. RBA reorders an existing test sequence, using a set of proposed refactoring fault models that define the refactoring's impact on program methods. Refactoring‐based approach's evaluation shows that it promotes early detection of refactoring faults and outperforms well‐known prioritization techniques in 71% of the cases. Moreover, it prioritizes fault‐revealing test cases close to one another in 73% of the cases, which can be useful for fault localization. Those findings show that RBA can considerably improve prioritization of test cases during perfective evolution, both by increasing fault‐detection rates as well as by helping to pinpoint defects introduced by an incorrect refactoring. Copyright © 2016 John Wiley & Sons, Ltd.     

融合聚类算法和缺陷预测的测试用例优先排序方法

持续集成环境下,软件快速更新加快了回归测试执行的频率,但缺陷快速反馈的需求对回归测试又提出了更高要求。测试用例优先排序技术研究测试用例的重要性,通常将缺陷探测能力强的测试用例优先执行,使其提早发现软件缺陷,其可解决持续集成环境下的快速反馈需求。缺陷预测技术可通过被测系统代码特征和历史缺陷来预估信息预测软件在新版本中发现缺陷的可能性,传统基于聚类的测试用例优先排序方法大多未考虑不同类簇数和特征子集对聚类结果的影响。文中将缺陷预测应用到聚类优先排序方法,构建测试用例和代码关联矩阵,对测试用例进行聚类分析,结合缺陷预测结果和最大最小距离策略指导簇间和簇内排序。通过实验验证发现,类簇数和聚类特征子集选择对排序效果有一定影响,当未能获取最佳类簇数和特征子集时,相比单一的聚类优先排序方法,所提方法可更有效地提高回归测试效率。