参数配对及n-way组合覆盖算法研究

组合测试是软件测试数据生成研究的一个重要领域,其中参数配对组合测试的应用最为广泛.对常用的参数配对组合覆盖方法进行了综述分析.目前主流的配对覆盖算法AETG和IPO所存在的主要问题是在确定水平取值时,具有盲目性和随机性,从而难以控制测试用例的规模.为此提出了改进的AETG算法和IPO算法,改进算法通过对参数进行预处理以及综合考虑各因素的水平组合等手段,对测试用例的规模进行控制,采用更加完备的方法尽早确定水平取值.为验证新算法的有效性,进行了仿真实验和实际测试,结果表明,改进算法所生成的测试用例数量要少于原算法,测试用例约减效果更为明显;测试用例数与配对数之间、测试用例数与因素水平数之间存在着某些规律性的联系,分析得出了一系列相关的结论.在配对覆盖的基础上,提出了遗传算法与AETG算法相结合的n-way组合覆盖算法,证明了其时间复杂性较已有算法得到了改善.     

A backtracking search tool for constructing combinatorial test suites

Combinatorial testing is an important testing method. It requires the test cases to cover various combinations of parameters of the system under test. The test generation problem for combinatorial testing can be modeled as constructing a matrix which has certain properties. This paper first discusses two combinatorial testing criteria: covering array and orthogonal array, and then proposes a backtracking search algorithm to construct matrices satisfying them. Several search heuristics and symmetry breaking techniques are used to reduce the search time. This paper also introduces some techniques to generate large covering array instances from smaller ones. All the techniques have been implemented in a tool called EXACT (EXhaustive seArch of Combinatorial Test suites). A new optimal covering array is found by this tool.     

基于接口参数的黑箱测试用例自动生成算法

测试用例的选择与生成技术是软件测试尤其是黑箱测试的一个重要研究领域，测试用例的质量将直接决定软件测试的科学性和有效性．该文在一般的测试用例选择方法的基础上，提出了一种基于对接口参数进行组合覆盖的黑箱测试用例自动生成算法模型，据此可以生成一个对所有接口参数进行两两组合覆盖的测试用例表．并证明了该方法产生的测试用例具有数量少、能实现对接口参数最大限度组合覆盖的特点，从而可以在提高软件测试质量的同时，降低成本，提高效率．最后介绍了该算法在作者研究开发的测试数据生成工具中的实际效果．     

组合测试技术在防汛辅助决策支持系统中的应用

组合测试是一种重要的软件测试方法,能够有效地检测软件系统中各种因素间的组合可能促发的故障。防汛辅助决策支持系统需要输入多个查询条件实现相关信息的查询,影响系统正常工作的因素众多,可使用组合测试技术检测防汛辅助决策支持系统中各种参数及其相互作用对系统各种功能的影响。选择水雨情信息模块,重点探讨对水雨情信息模块建立组合测试建模,以及使用高级组合测试系统(ACTS)对该模块进行组合测试用例集的生成,再用测试用例集对系统进行测试。测试结果表明,两两组合测试仅需要执行35个测试用例,而全覆盖测试需要执行140个测试用例。     

组合测试用例生成技术

组合测试是一种科学有效的软件测试方法,该方法旨在使用较少的测试用例有效地检测软件系统中各个因素以及它们之间的相互作用对系统产生的影响,实践证明其具有较高的错误检测能力。当前组合测试研究的热点之一是组合测试用例生成问题,即如何针对具体待测软件,在满足给定组合覆盖要求的前提下,生成规模尽可能小的测试用例集,以便在保证错误检测能力的前提下尽可能降低测试成本。从N维组合覆盖和变力度组合覆盖等两类不同的组合覆盖标准出发,简要介绍了迄今为止人们在组合测试用例生成领域所取得的研究成果,对现有的组合用例生成方法进行了分类和总结。此外,还对优先级、组合约束、错误定位等条件和应用场景下的组合测试用例生成技术进行了介绍。最后,分析了现有成果中存在的问题,并对该领域未来的研究方向进行了分析和讨论。     

Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm

ContextSoftware has become an innovative solution nowadays for many applications and methods in science and engineering. Ensuring the quality and correctness of software is challenging because each program has different configurations and input domains. To ensure the quality of software, all possible configurations and input combinations need to be evaluated against their expected outputs. However, this exhaustive test is impractical because of time and resource constraints due to the large domain of input and configurations. Thus, different sampling techniques have been used to sample these input domains and configurations.ObjectiveCombinatorial testing can be used to effectively detect faults in software-under-test. This technique uses combinatorial optimization concepts to systematically minimize the number of test cases by considering the combinations of inputs. This paper proposes a new strategy to generate combinatorial test suite by using Cuckoo Search concepts.MethodCuckoo Search is used in the design and implementation of a strategy to construct optimized combinatorial sets. The strategy consists of different algorithms for construction. These algorithms are combined to serve the Cuckoo Search.ResultsThe efficiency and performance of the new technique were proven through different experiment sets. The effectiveness of the strategy is assessed by applying the generated test suites on a real-world case study for the purpose of functional testing.ConclusionResults show that the generated test suites can detect faults effectively. In addition, the strategy also opens a new direction for the application of Cuckoo Search in the context of software engineering.     

The density algorithm for pairwise interaction testing

There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one‐test‐at‐a‐time methods (such as AETG and TCG) have proven to be a reasonable compromise between the needs for small test suites, fast test‐suite generation, and flexibility to accommodate a variety of testing scenarios. However, such methods suffer from the lack of a worst‐case logarithmic guarantee on test suite size, while methods that provide such a guarantee at present are less efficient or flexible, or do not produce test suites that are competitive in size for practical testing scenarios. In this paper, a new algorithm establishes that efficient, greedy, one‐test‐at‐a‐time methods can indeed produce a logarithmic worst‐case guarantee on the test suite size. In addition, this can be done while still producing test suites that are of competitive size, and in a time that is comparable to the published methods. It is deterministic, guaranteeing reproducibility. It generates only one candidate test at a time, permits users to ‘seed’ the test suite with specified tests, and allows users to specify constraints of combinations that should be avoided. Further, statistical analysis examines the impact of five variables used to tune this density algorithm for execution time and test suite size: weighting of density for factors, scaling of density, tie‐breaking, use of multiple candidates, and multiple repetitions using randomization. Copyright © 2007 John Wiley & Sons, Ltd.     

基于符号零压缩二叉决策图的组合测试用例生成方法

组合测试是系统测试中一种非常有效的方法,能够在保证错误检出率的前提下采用较少的测试用例来测试系统。但是,组合测试用例集构造问题的复杂度是NP完全的。给出了一种基于符号零压缩二叉决策图(Zero-suppressed Binary Decision Diagram,ZBDD)的组合测试用例生成方法。该方法首先利用ZBDD的结构特性,对测试系统进行紧凑的符号表示。然后利用ZBDD的隐式操作,结合贪心算法的思想,不断地覆盖更多的组合并缩小未覆盖组合集合,生成2～4维覆盖强度的较小测试用例集。实验证明,所提方法不仅可行而且节点开销小。     

A Formal Logic Approach to Constrained Combinatorial Testing

Combinatorial testing is as an effective testing technique to reveal failures in a given system, based on input combinations coverage and combinatorial optimization. Combinatorial testing of strength t (t ≥ 2) requires that each t-wise tuple of values of the different system input parameters is covered by at least one test case. Combinatorial test suite generation algorithms aim at producing a test suite covering all the required tuples in a small (possibly minimal) number of test cases, in order to reduce the cost of testing. The most used combinatorial technique is the pairwise testing (t = 2) which requires coverage of all pairs of input values. Constrained combinatorial testing takes also into account constraints over the system parameters, for instance forbidden tuples of inputs, modeling invalid or not realizable input values combinations. In this paper a new approach to combinatorial testing, tightly integrated with formal logic, is presented. In this approach, test predicates are used to formalize combinatorial testing as a logical problem, and an external formal logic tool is applied to solve it. Constraints over the input domain are expressed as logical predicates too, and effectively handled by the same tool. Moreover, inclusion or exclusion of select tuples is supported, allowing the user to customize the test suite layout. The proposed approach is supported by a prototype tool implementation and results of experimental assessment are also presented.     

Generating combinatorial test suite using combinatorial optimization

Combinatorial testing (CT) is an effective technique to test software with multiple configurable parameters. It is used to detect interaction faults caused by the combination effect of parameters. CT test generation aims at generating covering arrays that cover all t-way parameter combinations, where t is a given covering strength. In practical CT usage scenarios, there are usually constraints between parameters, and the performance of existing constraint-handling methods degrades fast when the number of constraints increases.The contributions of this paper are (1) we propose a new one-test-at-a-time algorithm for CT test generation, which uses pseudo-Boolean optimization to generate each new test case; (2) we have found that pursuing the maximum coverage for each test case is uneconomic, and a possible balance point is to keep the approximation ratio in [0.8,0.9]; (3) we propose a new self-adaptive mechanism to stop the optimization process at a proper time when generating each test case; (4) extensive experimental results show that our algorithm works fine on existing benchmarks, and the constraint-handling ability is better than existing approaches when the number of constraints is large; and (5) we propose a method to translate shielding parameters (a common type of constraints) into normal constraints.     

基于反射特征的Android测试用例自动生成

随着Android应用软件数量的急速增加, Android应用质量的重要性越来越受到人们的重视. 测试是高质量软件的重要保证, 而测试用例生成技术是自动化测试的关键. 数据显示, 在Google Play中有将近88%的Android应用程序使用了反射. 然而, 现有的Android测试用例自动生成方法通常没有考虑反射技术的使用, 无法检测出反射隐藏的恶意行为. 为了进一步提高软件质量, 本文提出一种新的Android测试用例生成方法, 结合反射特征构造Android应用程序多粒度模型, 同时对反射关系进行分析, 生成能到达反射的调用路径, 并利用自适应遗传算法生成覆盖反射路径的测试用例, 对含反射特征的Android应用进行测试. 为验证本文方法, 分别从Android应用多粒度模型有效性及测试方法效率两方面对本文方法有效性进行评估. 实验结果表明, 本文提出的基于反射特征的Android测试用例自动生成方法对于反射的检测效果更好并且效率更高.     

基于动态调整简化粒子群优化的组合测试用例生成方法

优化的组合测试中的一个关键是生成的测试用例能够覆盖更多的组合,而粒子群算法在生成强组合覆盖用例方面有其独特的优势和能力。文中提出了一种基于动态调整简化粒子群优化的组合测试用例生成方法。该方法基于粒子群算法生成测试用例,结合混合的优先级one-test-at-a-time策略和基于动态调整的简化粒子群算法生成组合测试用例集,排除了速度因素对粒子优化过程的影响。定义了一个粒子收敛指标,以粒子群早熟收敛程度为依据来动态调整惯性权值,以防止粒子陷入局部最优和后期出现收敛速度慢的情况,从而提高粒子群算法所生成的覆盖表的覆盖组合能力。通过对比实验表明,基于动态调整的简化粒子群优化算法在用例规模和时间成本上具有一定的优势。     

基于解空间树的组合测试数据生成

在组合覆盖测试模型的基础上提出：将所有的可用测试数据表示为一棵解空间树,利用回溯法对解空间树进行路径搜索来生成测试数据,然后使用贪心算法补充生成测试数据,以满足两两组合覆盖标准.并且实现了基于该方法的测试数据生成工具,所生成的测试数据集与同类工具相比具有一定的特点和优势.     

Variable strength combinatorial testing of concurrent programs

Reachability testing is an important approach to testing concurrent programs. It generates and exercises synchronization sequences automatically and on-the-fly without saving any test history. Existing reachability testing can be classified into exhaustive and t-way testing. Exhaustive testing is impractical in many cases while t-way testing may decrease the capability of fault detection in some cases. In this paper, we present a variable strength reachability testing strategy, which adopts the dynamic framework of reachability testing and uses a variable strength combinatorial strategy. Different parameter groups are provided with different covering strength. Variable strength testing covers no t-way combinations but the necessary combinations of parameters having mutual interactions in a concurrent program. It is more reasonable than t-way testing because uniform interactions between parameters do not often exist in concurrent systems. We propose a merging algorithmthat implements the variable strength combinatorial testing strategy and conduct our experiment on several concurrent programs. The experimental results indicate that our variable strength reachability testing reaches a good tradeoff between the effectiveness and efficiency. It can keep the same capability of fault detection as exhaustive reachability testing while substantially reducing the number of synchronization sequences and decreasing the execution time in most cases.     

Using combinatorial testing to build navigation graphs for dynamic web applications

Modelling a software system is often a challenging prerequisite to automatic test case generation. Modelling the navigation structure of a dynamic web application is particularly challenging because of the presence of a large number of pages that are created dynamically and the difficulty of reaching a dynamic page unless a set of appropriate input values are provided for the parameters. To address the first challenge, some form of abstraction is required to enable scalable modelling. For the second challenge, techniques are required to select appropriate input values for parameters and systematically combine them to reach new pages. This paper presents a combinatorial approach in building a navigation graph for dynamic web applications. The navigation graph can then be used to automatically generate test sequences for testing web applications. The novelty of our approach is twofold. First, we use an abstraction scheme to control the page explosion problem, where pages that are likely to have the same navigation behaviour are grouped together and are represented as a single node in the navigation graph. Second, assuming that values of individual parameters are supplied manually or generated from other techniques, we combine parameter values such that well‐defined combinatorial coverage of input parameter values is achieved. Using combinatorial coverage can significantly reduce the number of requests that have to be submitted while still achieving effective coverage of the navigation structure. We implement our combinatorial approach in a tool, Tansuo, and apply the tool on seven open‐source web applications. We evaluate the effectiveness of Tansuo's exploration process guided by t‐way coverage, for t = 1,2,3, with respect to code coverage, and find that the navigation structure exploration by Tansuo, in general, results in high code coverage (more than 80% statement coverage for most of our subject applications when dead code is removed). We compare Tansuo's effectiveness with two other navigation graph tools and find that Tansuo is more effective. Our empirical results indicate that using pairwise coverage in Tansuo results in the efficient generation of navigation graphs and effective exploration of dynamic web applications. Copyright © 2016 John Wiley & Sons, Ltd.     

Constraint-based test generation for automotive operating systems

This work suggests a method for systematically constructing a software-level environment model for safety checking automotive operating systems by introducing a constraint specification language, OSEK_CSL. OSEK_CSL is designed to specify the usage constraints of automotive operating systems using a pre-defined set of constraint types identified from the international standard OSEK/VDX. Each constraint specified in OSEK_CSL is interpreted as either a regular language or a context-free language that can be checked by a finite automaton or a pushdown automaton. The set of usage constraints is used to systematically classify the universal usage model of OSEK-/VDX-based operating systems and to generate test sequences with varying degrees of constraint satisfaction using LTL model checking. With pre-defined constraint patterns and the full support of automation, test engineers can choose the degree of constraint satisfaction and generate test cases using combinatorial intersections of selected constraints that cover all corner cases classified by constraints. A series of experiments on an open-source automotive operating system show that our approach finds safety issues more effectively than conventional specification-based testing, scenario-based testing, and conformance testing.     

成对测试中的测试生成策略研究

测试用例生成是软件测试的关键，成对测试是根据特定的测试原则研究测试用例的产生方法。基于这一原则分别构造了IPO-H算法和IPO-V算法的测试以产生整个测试用例，具有产生的测试用例少，时间消耗小等优点。并与另一个成对测试的测试生成工具AETG(高效自动测试生成器)进行了比较、分析，证明了改进的IPO策略便于构造自动测试工具。     

基于分支覆盖的回归测试路径选择

回归测试是迭代式软件开发的重要环节,测试数据生成是回归测试的前提.传统的回归测试方法,从已有的测试数据中选择部分测试数据,并生成一些新的测试数据,以验证程序的正确性.但是,该方法容易生成冗余的测试数据,从而降低了回归测试的效率.研究了回归测试的分支覆盖问题,通过利用已有测试数据的路径覆盖信息,并选择一定个数的路径,以覆盖所有的目标分支.首先,以若干路径形成的集合作为决策变量,以路径最少、覆盖的分支最多以及包含的未覆盖路径最少为目标,建立路径选择问题的3目标优化模型;然后,采用遗传算法求解上述模型时,设计了基于目标重要性的个体评价策略;最后,基于已有的测试数据与选择的路径之间的覆盖关系,确定需要生成的测试数据.将所提方法应用于6个基准工业程序测试中,并与其他方法比较.实验结果表明,采用该方法选择的路径,能够覆盖更多的分支,需要生成的测试数据更少,回归测试消耗的时间更短.     

基于优先级的组合测试方法探究

传统组合测试方法致力于生成最小组合测试用例集合.然而在实际测试工作中,如果执行测试用例的时间有限,用传统组合测试方法生成的测试用例集难以在有限资源下完全执行.因此,在组合测试方法的基础上引入测试用例优先级概念,可以设计出在资源有限的情况下充分测试关键参数及组合的测试方法,缩短测试时间并确保缺陷检测率.     

基于试验设计的RFID应用组合测试优化研究

作为物联网的核心技术之一, 射频识别(Radio frequency identification, RFID)已广泛应用到多个行业中. 而复杂的应用环境和多样化的产品设备也给RFID应用部署带来了新的挑战. 为了解决RFID应用测试中的组合爆炸问题, 本文提出一种RFID应用组合测试优化方法, 可以在多因子多水平的条件下通过正交设计在所有试验方案中均匀地挑选出最具有代表性的少数试验方案来简化测试, 并利用方差分析法获得各因子对测试结果影响的主次因素和变化规律, 协助使用者迅速建立对输入输出关系的认识. 以RFID标签应用为例进行的组合测试数据分析表明, 利用试验设计方法对RFID应用组合测试的设计方案进行优化, 可以为RFID系统部署提供有益的参考依据.