基于路径引导的回归测试用例集扩增方法

为了全面测试演化软件,回归测试通常需要生成新的测试用例。concolic测试是一种沿着具体执行路径进行符号执行的软件验证技术,通过生成测试数据来执行程序的所有可行路径。回归测试中,由于concolic测试关注于程序本身,没有利用已有测试用例和软件演化信息,导致生成大量无效测试数据,浪费资源和时间。为解决此问题,提出一种基于路径引导的回归测试用例集扩增方法。该方法将目标路径作为引导,根据软件演化信息选择有利于覆盖目标路径的测试用例,利用已有测试用例跳过重叠初始子路径,对后续目标子路径进行concolic测试并生成覆盖目标路径的测试数据。案例分析表明,本文方法相比传统concolic测试,本方法在覆盖程序可行路径的同时,可有效减少concolic测试路径,提高测试数据生成效率。     

On guiding the augmentation of an automated test suite via mutation analysis

Mutation testing has traditionally been used as a defect injection technique to assess the effectiveness of a test suite as represented by a “mutation score.” Recently, mutation testing tools have become more efficient, and industrial usage of mutation analysis is experiencing growth. Mutation analysis entails adding or modifying test cases until the test suite is sufficient to detect as many mutants as possible and the mutation score is satisfactory. The augmented test suite resulting from mutation analysis may reveal latent faults and provides a stronger test suite to detect future errors which might be injected. Software engineers often look for guidance on how to augment their test suite using information provided by line and/or branch coverage tools. As the use of mutation analysis grows, software engineers will want to know how the emerging technique compares with and/or complements coverage analysis for guiding the augmentation of an automated test suite. Additionally, software engineers can benefit from an enhanced understanding of efficient mutation analysis techniques. To address these needs for additional information about mutation analysis, we conducted an empirical study of the use of mutation analysis on two open source projects. Our results indicate that a focused effort on increasing mutation score leads to a corresponding increase in line and branch coverage to the point that line coverage, branch coverage and mutation score reach a maximum but leave some types of code structures uncovered. Mutation analysis guides the creation of additional “common programmer error” tests beyond those written to increase line and branch coverage. We also found that 74% of our chosen set of mutation operators is useful, on average, for producing new tests. The remaining 26% of mutation operators did not produce new test cases because their mutants were immediately detected by the initial test suite, indirectly detected by test suites we added to detect other mutants, or were not able to be detected by any test.

Regression test suite minimization using integer linear programming model

Software testers always face the dilemma of whether to retest the software with all the test cases or select a few of them on the basis of their fault detection ability. This paper introduces a novel approach to minimizing the test suite as an integer linear programming problem with optimal results. The minimization method uses the cohesion values of the program parts affected by the changes made to the program. The hypothesis is that the program parts with low cohesion values are more prone to errors. This assumption is validated on the mutation fault detection ability of the test cases. The experimental study carried out on 30 programs evaluates the effectiveness and usefulness of the proposed framework. The experimental results show that the minimized test suite can efficiently reveal the errors and ensure acceptable software quality. Copyright © 2017 John Wiley & Sons, Ltd.     

回归测试中测试用例集优化方法的研究

回归测试是软件演化过程中频繁进行的且开销巨大的一项任务,测试用例集的优化程度直接影响着测试的成本和效率。针对回归测试过程的特点,提出一种对测试用例集优化的新方法,即通过对测试用例集进行必要的消除冗余和调整排序,完成了对初始测试用例集的精简以及执行顺序的确定过程,使得有限的测试资源得到科学合理的分配。实验结果表明,相对于以往的测试用例集优化方法,新方法的效率和资源分配的合理性均有了显著的提高。     

An empirical study of regression test application frequency

Regression testing is an expensive process used to revalidate modified software. Regression test selection (RTS) techniques reduce the cost of regression testing by selecting a subset of a test suite. Many RTS techniques have been proposed, and studies have shown that they produce savings; other studies have shown that their cost‐effectiveness varies with characteristics of the workloads to which they are applied. It seems plausible, however, that another factor that impacts RTS techniques involves the process by which they are applied. In particular, issues such as the frequency with which regression testing is performed affect the techniques. Thus, in earlier work an experiment was conducted to assess the effects of test application frequency on the cost‐effectiveness of RTS techniques. The results exposed tradeoffs to consider when using these techniques over a series of releases. This work, however, was limited in external validity; in particular, the programs studied were relatively small. Thus, the previous experiment has been replicated on a large, multi‐version program. This second experiment confirms the findings of the first. In particular, the cost of using safe RTS techniques was strongly and negatively affected by testing interval. Conversely, the effectiveness of minimization RTS techniques was strongly and positively affected. Copyright © 2005 John Wiley & Sons, Ltd.     

成对组合测试数据的整体优化和生成方法

测试数据生成是组合测试的一个关键问题,但是组合测试用例集的构造问题的复杂度是NP完全的。提出了一种成对组合测试用例集整体优化和生成的方法。该方法通过编码机制将测试用例数据的生成问题转换为一个基于二进制编码的最优化问题,同时使用遗传算法对此编码空间进行搜索,并对所发现的最优个体进行解码,构造产生最佳测试用例集。实验结果表明,该方法简单高效,且具有解的质量高、时间消耗小的特点。     

软件测试用例生成综合策略研究

在软件测试中,测试用例生成是其关键环节,也对软件测试的自动化有着重要影响。为了提高测试用例生成的效率,提高其自动化的程度,从不同角度对软件测试用例选择策略进行介绍,并且着重介绍了采用遗传算法作为核心搜索算法来自动生成用例的方法,对其中的关键问题进行了探讨,最后,结合实际测试过程总结了测试用例生成的综合策略。     

测试用例演化研究进展

由于功能增加、性能调优、错误修复等原因,软件常常动态演化.现有测试技术难以满足软件演化过程中变化的测试需求,因此需要系统的测试用例演化技术,以有效保障演化软件的质量.回顾测试用例演化技术的研究现状,分别对测试用例选择、测试用例修复和测试用例集扩增这3部分内容进行了详细的比较和分析.最后提出测试用例演化技术领域存在的挑战和未来的研究方向.     

测试用例集约简问题研究及其进展

测试用例集约简问题是软件测试中的关键问题之一,其目的是使用尽可能少的测试用例充分满足给定的测试目标,从而提高测试效率、降低测试成本。在简要介绍了测试用例集约简问题基本概念的基础上,总结了求解该问题的几种主要方法,分析比较了这些方法的效率和特性。随后探讨了与测试用例集约简问题强相关的测试用例集错误检测效率的问题,并研究了测试用例优先级技术。最后指出了测试用例集约简问题的下一步研究方向。     

UCov: a user‐defined coverage criterion for test case intent verification

The goal of regression testing is to ensure that the behaviour of existing code, believed correct by previous testing, is not altered by new program changes. This paper argues that the primary focus of regression testing should be on code associated with (1) earlier bug fixes and (2) particular application scenarios considered to be important by the developer or tester. Existing coverage criteria do not enable such focus, for example, 100% branch coverage does not guarantee that a given bug fix is exercised or a given application scenario is tested. Therefore, there is a need for a new and complementary coverage criterion in which the user can definea test requirement characterizing a given behaviour to be covered as opposed to choosing from a pool of pre‐defined and generic program elements. This paper proposes this new methodology and calls it UCov, a user‐defined coverage criterion wherein a test requirement is an execution pattern of program elements, and possibly predicates, that a test case must satisfy. The proposed criterion is not meant to replace existing criteria, but to complement them as it focuses the testing on important code patterns that could go untested otherwise. UCov supports test case intent verification. For example, following a bug fix, the testing team may augment the regression suite with the test case that revealed the bug. However, this test case might become obsolete due to code modifications not related to the bug. But if a test requirement characterizing the bug was defined by the user, UCov would determine that test case intent verification failed. The UCov methodology was implemented for the Java platform, was successfully applied onto 10 real‐life case studies and was shown to have advantages over JUnit. The implementation comprises the following tools: (1) TRSpec: allows the user to easily specify complex test requirements; (2) TRCheck: checks whether user‐defined test requirements were satisfied, that is, supports test case intent verification; and (3) TRMigrate: migrates user‐defined test requirements to subsequent versions of a given program. Copyright © 2016 John Wiley & Sons, Ltd.     

Evolutionary test data generation: a comparison of fitness functions

Previous research using genetic algorithms to automate the generation of data for path testing has utilized several different fitness functions, assessing their usefulness by comparing them to random generation. This paper describes two sets of experiments that assess the performance of several fitness functions, relative to one another and to random generation. The results demonstrate that some fitness functions provide better results than others, generating fewer test cases to exercise a given program path. In these studies, the branch predicate and inverse path probability approaches were the best performers, suggesting that a two‐step process combining these two methods may be the most efficient and effective approach to path testing. Copyright © 2005 John Wiley & Sons, Ltd.     

一种基于程序关联性分析的软件测试方法

软件测试是保证产品整体质量的重要手段。基于关联性的软件测试方法,在加强软件内在关联关系分析的基础上,建立模块之间及模块与测试用例之间的关联关系,并给出关联的表达和检测方法,有利于快速定位错误、缩小测试范围,减少重复测试的工作量,提高回归测试和维护的效率。     

基于子域的构件软件充分性测试样本生成策略

基于构件的软件开发给软件测试带来了新的挑战。现有基于构件的软件测试充分性的研究表明,集成测试时的软件测试充分性不能保证构件的测试充分性,同时,构件的测试充分性也不能保证软件的测试充分性。提出了将构件测试充分性对集成测试的限制转化为子域的思想,将测试标准产生的软件子域和构件子域映射的软件子域进行统一的处理,从而对基于构件的软件测试样本集生成与优化提出了一个新的思路。     

Online knapsack with resource augmentation

It is known that online knapsack is not competitive. This negative result remains true even if items are removable. In this paper we consider online removable knapsack with resource augmentation, in which we hold a knapsack of capacity R?1 and aim at maintaining a feasible packing to maximize the total profit of items packed into the knapsack. Both scenarios that removal of accepted items is allowed and is not allowed are investigated. We evaluate an online algorithm by comparing the resulting packing to an optimal packing that uses a knapsack of capacity one. Optimal online algorithms are derived for both the weighted case (items have arbitrary profits) and the unweighted case (the profit of an item is equal to its size).     

Evaluating test suite characteristics,cost, and effectiveness of FSM-based testing methods

ContextTesting from finite state machines has been investigated due to its well-founded and sound theory as well as its practical application. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods.ObjectiveWe compare the methods that generate complete test suites from finite states machines. The test suites produced by the W, HSI, H, SPY, and P methods are analyzed in different configurations.MethodComplete and partial machines were randomly generated varying numbers of states, inputs, outputs, and transitions. These different configurations were used to compare test suite characteristics (number of resets, test case length) and the test suite length (i.e., the sum of the length of its test cases). The fault detection ratio was evaluated using mutation testing to produce faulty implementations with an extra state.ResultsOn average, the recent methods (H, SPY, and P) produced longer test cases but smaller test suites than the traditional methods (W, HSI). The recent methods generated test suites of similar length, though P produced slightly smaller test suites. The SPY and P methods had the highest fault detection ratios and HSI had the lowest. For all methods, there was a positive correlation between the number of resets and the test suite length and between the test case length and the fault detection ratio.ConclusionThe recent methods rely on fewer and longer test cases to reduce the overall test suite length, while the traditional methods produce more and shorter test cases. Longer test cases are correlated to fault detection ratio which favored SPY, though all methods have a ratio of over 92%.     

基于自适应粒子群优化算法的测试数据扩增方法

针对在回归测试中原有测试数据集往往难以满足新版本软件测试需求的问题,提出一种基于自适应粒子群算法（APSO）的测试数据扩增方法。首先,根据原有测试数据在新版本程序上的穿越路径与目标路径的相似度,在原有的测试数据集中选择合适的测试数据,作为初始种群的进化个体;然后,利用初始测试数据的穿越路径与目标路径的不同子路径,确定造成两者路径偏离的输入分量;最后,根据路径相似度构建适应度函数,利用APSO操作输入分量,生成新的测试数据。该方法针对四个基准程序与基于遗传算法（GA）和随机法的测试数据扩增方法相比,测试数据扩增效率分别平均提高了约56%和81%。实验结果表明,所提方法在回归测试方面有效地提高了测试数据扩增的效率,增强了其稳定性。     

求解组合测试用例集的差分进化蚁群算法

针对组合测试中测试用例集生成这一关键问题,通过构建组合空间的搜索模型,提出了一种用于求解最小测试用例集的差分进化蚁群算法（DEACA）。该算法将差分进化融入到蚁群算法中,并在搜索过程中动态更新启发信息,有效克服了标准蚁群算法在求解组合用例时出现的早熟现象。实验表明该方法优于标准蚁群算法,与其他方法相比也具有一定优势和特点。     

基于黑盒测试的航天嵌入式软件用例设计方法

为了提高测试用例设计的效率,简化测试用例数量,提出了基于黑盒测试的航天嵌入式软件用例设计方法.阐述了确认测试过程中实用的测试用例设计流程,提出了在设计测试输入数据的基础上加强测试逻辑设计的用例设计方法,并结合实际工程经验,给出了测试用例的总体设计思路.最后,对测试遗漏问题及其原因进行了分析,为今后进行更有效的星载软件测试提供了参考.     

Regression testing minimization,selection and prioritization: a survey

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.