Test frame updating in CPM testing of Prolog programs

Category Partition Method (CPM) is a general approach to specification-based program testing, where test frame reduction and refinement are two important issues. Test frame reduction is necessary since too many test frames may be produced, and test frame refinement is important since during CPM testing new information about test frame generation may be achieved and considered incrementally. Besides the information provided by testers or users, implementation related knowledge offers alternative information for reducing and refining CPM test frames. This paper explores the idea by proposing a call patterns semantics based test frame updating method for Prolog programs, in which a call patterns analysis is used to collect information about the way in which procedures are used in a program. The updated test frames will be represented as constraints. The effect of our test frame updating is two-fold. On one hand, it removes “uncared” data from the original set of test frames; on the other hand, it refines the test frames to which we should pay more attention. The first effect makes the input domain on which a procedure must be tested a subset of the procedure’s input domain, and the latter makes testers stand more chance to find out the faults that are more likely to show their presence in the use of the program under consideration. Our test frame updating method preserves the effectiveness of CPM testing with respect to the detection of faults we care. The test case generation from the updated set of test frames is also discussed. In order to show the applicability of our method an approximation call patterns semantics is proposed, and the test frame updating on the semantics is illustrated by an example.

一种针对高维输入域的适应性随机测试改进性算法

适应性随机测试ART能够保证测试用例在输入域中更加均匀地分布,从而在失效检测能力上明显强于基本的随机测试,其中,固定候选集规模的ART算法 FSCS ART因具备较好的揭错能力而被广泛采用。然而随着输入域维度的升高,FSCS ART的失效检测效果显著降低。针对该问题,在从候选集中选择正式用例时综合考虑两种距离：候选点与已测用例之间的距离和它与中心点之间的距离,这样,输入域边缘的候选点的优先级得以降低,有效地克服了FSCS ART趋向于边缘的弊端。实验结果表明,改进后的算法针对高维输入域表现出更强的失效检测能力。     

一种基于规范和程序域分析的软件测试方法

虽然软件规范规定了输入变量的定义域(功能域)，但程序实现本身也定义了其定义域(运行域)．如果二者不能完全重合，那么某些软件故障就可检测出来．为此，提出了一种基于规范和程序域分析的软件测试方法，介绍了一个运行域自动确定系统ADSOD，并讨论了边界测试数据的选取原则．对特殊表达式，采用程序切片技术，构造基于表达式的可执行模块，通过执行该模块，识别其输入变量的奇异值，进而确定其输入变量的取值范围．通过分析功能域和运行域的边界测试数据集，检测二者的重合性，并根据其分析结果产生测试用例．实验表明：所提方法产生的测试用例，可以有效地提高测试效率．     

一种基于双约束传播的面向路径自动随机测试方法

随机测试是一种常用的软件测试自动化方法。该方法随机地从程序输入域中选取测试输入,其自动测试实现容易。但随机地从输入域中选取满足某种测试准则的输入数据,测试生成效率较低。为此提出了一种基于双约束传播的面向路径自动随机测试方法,通过对输入变量的取值范围进行切分,执行两次约束传播来求取指定路径的输入域,在此基础上开发了一个自动随机测试生成系统,并与PRT方法进行了实验对比分析。结果表明,该方法计算出的路径输入域更加精确,可有效地减少对无效域的测试生成,提高了随机测试的效率。     

Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm

ContextSoftware has become an innovative solution nowadays for many applications and methods in science and engineering. Ensuring the quality and correctness of software is challenging because each program has different configurations and input domains. To ensure the quality of software, all possible configurations and input combinations need to be evaluated against their expected outputs. However, this exhaustive test is impractical because of time and resource constraints due to the large domain of input and configurations. Thus, different sampling techniques have been used to sample these input domains and configurations.ObjectiveCombinatorial testing can be used to effectively detect faults in software-under-test. This technique uses combinatorial optimization concepts to systematically minimize the number of test cases by considering the combinations of inputs. This paper proposes a new strategy to generate combinatorial test suite by using Cuckoo Search concepts.MethodCuckoo Search is used in the design and implementation of a strategy to construct optimized combinatorial sets. The strategy consists of different algorithms for construction. These algorithms are combined to serve the Cuckoo Search.ResultsThe efficiency and performance of the new technique were proven through different experiment sets. The effectiveness of the strategy is assessed by applying the generated test suites on a real-world case study for the purpose of functional testing.ConclusionResults show that the generated test suites can detect faults effectively. In addition, the strategy also opens a new direction for the application of Cuckoo Search in the context of software engineering.     

基于约束的软件失效域识别与特征分析

随机测试是实践中广泛采用的一种黑盒测试方法.近年来提出的适应性随机测试方法改进了随机测试的不足,仿真实验结果表明,改进效果取决于软件失效域的特征.提出以测试约束刻画软件失效域在输入域上的分布,探讨了基于现有的程序分析技术构造测试约束的过程,讨论了基于测试约束的软件失效域的特征分析方法.以一个实例软件验证所提出的测试约束构造过程及其软件失效域特征分析方法.测试约束揭示了软件故障的触发与传播的内在机制,基于测试约束的软件失效域的特征分析方法有助于改进测试用例的设计质量以及评价适应性随机测试方法的适用性.     

GUI testing assisted by human knowledge: Random vs. functional

Software testing is a labor-intensive task in software development life-cycle. Human knowledge is useful in the practices of software testing, especially GUI testing. There are many strategies for GUI testing assisted by human knowledge, in which manual random testing and manual functional testing are two of widely used ones. In this paper, an empirical study is conducted to compare random testing and functional testing in order to provide guidelines for GUI testing. 234 participants were recruited to create thousands of random and functional test cases for open source GUI applications. Some of these test cases were selected with certain coverage criteria and then run on GUI applications to evaluate random testing and functional testing. We study three aspects on the two testing strategies: effectiveness, complementarity and impact of test case length. Some useful observations in the empirical study are: (1) Random testing is more effective in the early stage of testing on small applications and functional testing has more extensive applicability for testing large sized applications. (2) Random testing and functional testing exhibit some complementarity in our experiment. (3) Short test cases can reveal some faults more quickly and long test cases can reveal more faults lastingly.     

An investigation of statistical software testing

Software validation embodies two notions: fault removal and fault forecasting. Statistical testing involves exercising a piece of software by supplying it with input values that are randomly selected according to a defined probability distribution on its input domain. It can be used as a practical tool for revealing faults in a fault removal phase, and for assessing software dependability in a fault forecasting phase. In both of these, its efficiency is linked to the adequacy of the input probability distribution with respect to the test experiment goal. In this paper a mixed validation strategy combining deterministic and random test data is defined, and the theoretical and experimental work performed to support the strategy is reported. The quoted results relate to the unit testing of four real programs from the nuclear field. They confirm the high fault revealing power of statistical structural testing. Two main directions for further investigation of statistical testing are indicated by the reported work. These are described and solutions to the associated problems are outlined.     

A property-based testing framework for encryption programs

In recent years, a variety of encryption algorithms were proposed to enhance the security of software and systems. Validating whether encryption algorithms are correctly implemented is a challenging issue. Software testing delivers an effective and practical solution, but it also faces the oracle problem (that is, under many practical situations, it is impossible or too computationally expensive to know whether the output for any given input is correct). In this paper, we propose a property-based approach to testing encryption programs in the absence of oracles. Our approach makes use of the so-called metamorphic properties of encryption algorithms to generate test cases and verify test results. Two case studies were conducted to illustrate the proposed approach and validate its effectiveness. Experimental results show that even without oracles, the proposed approach can detect nearly 50% inserted faults with at most three metamorphic relations (MRs) and fifty test cases.     

成对组合测试数据的整体优化和生成方法

测试数据生成是组合测试的一个关键问题,但是组合测试用例集的构造问题的复杂度是NP完全的。提出了一种成对组合测试用例集整体优化和生成的方法。该方法通过编码机制将测试用例数据的生成问题转换为一个基于二进制编码的最优化问题,同时使用遗传算法对此编码空间进行搜索,并对所发现的最优个体进行解码,构造产生最佳测试用例集。实验结果表明,该方法简单高效,且具有解的质量高、时间消耗小的特点。     

Unit Test Data Generation for C Using Rule-Directed Symbolic Execution

Unit testing is widely used in software development. One important activity in unit testing is automatic test data generation. Constraint-based test data generation is a technique for automatic generation of test data, which uses symbolic execution to generate constraints. Unit testing only tests functions instead of the whole program, where individual functions typically have preconditions imposed on their inputs. Conventional symbolic execution cannot detect these preconditions, let alone converting these preconditions into constraints. To overcome these limitations, we propose a novel unit test data generation approach using rule-directed symbolic execution for dealing with functions with missing input preconditions. Rule-directed symbolic execution uses predefined rules to detect preconditions in the individual function, and generates constraints for inputs based on preconditions. We introduce implicit constraints to represent preconditions, and unify implicit constraints and program constraints into integrated constraints. Test data generated based on integrated constraints can explore previously unreachable code and help developers find more functional faults and logical faults. We have implemented our approach in a tool called CTS-IC, and applied it to real-world projects. The experimental results show that rule-directed symbolic execution can find preconditions (implicit constraints) automatically from an individual function. Moreover, the unit test data generated by our approach achieves higher coverage than similar tools and efficiently mitigates missing input preconditions problems in unit testing for individual functions.

     

基于相似性的CITCP强化学习奖励策略

在面向持续集成测试用例优先排序(continuous integration test case prioritization,CITCP)的强化学习方法中,智能体通过对测试用例实施奖励从而调整测试用例优先排序策略以适应后续集成测试,可以满足持续集成测试频繁迭代和快速反馈的需求.智能体通常只奖励执行失效测试用例,但实际...     

Killing strategies for model‐based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Back's action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright © 2014 John Wiley & Sons, Ltd.     

An initial industrial evaluation of interactive search-based testing for embedded software

Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so.Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies.For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system.This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search.In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ‘domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions.The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently.     

Automated metamorphic testing on the analyses of feature models

Context

A feature model (FM) represents the valid combinations of features in a domain. The automated extraction of information from FMs is a complex task that involves numerous analysis operations, techniques and tools. Current testing methods in this context are manual and rely on the ability of the tester to decide whether the output of an analysis is correct. However, this is acknowledged to be time-consuming, error-prone and in most cases infeasible due to the combinatorial complexity of the analyses, this is known as the oracle problem.

Objective

In this paper, we propose using metamorphic testing to automate the generation of test data for feature model analysis tools overcoming the oracle problem. An automated test data generator is presented and evaluated to show the feasibility of our approach.

Method

We present a set of relations (so-called metamorphic relations) between input FMs and the set of products they represent. Based on these relations and given a FM and its known set of products, a set of neighbouring FMs together with their corresponding set of products are automatically generated and used for testing multiple analyses. Complex FMs representing millions of products can be efficiently created by applying this process iteratively.

Results

Our evaluation results using mutation testing and real faults reveal that most faults can be automatically detected within a few seconds. Two defects were found in FaMa and another two in SPLOT, two real tools for the automated analysis of feature models. Also, we show how our generator outperforms a related manual suite for the automated analysis of feature models and how this suite can be used to guide the automated generation of test cases obtaining important gains in efficiency.

Conclusion

Our results show that the application of metamorphic testing in the domain of automated analysis of feature models is efficient and effective in detecting most faults in a few seconds without the need for a human oracle.     

A combinatorial testing strategy for concurrent programs

One approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on‐the‐fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended to exercise all possible synchronization sequences of a concurrent program with a given input. In this paper, we present a new testing strategy, called t‐way reachability testing, that adopts the dynamic framework of reachability testing but selectively exercises a subset of synchronization sequences. The selection of the synchronization sequences is based on a combinatorial testing strategy called t‐way testing. We present an algorithm that implements t‐way reachability testing, and report the results of several case studies that were conducted to evaluate its effectiveness. The results indicate that t‐way reachability testing can substantially reduce the number of synchronization sequences exercised during reachability testing while still effectively detecting faults. Copyright © 2007 John Wiley & Sons, Ltd.     

降低最大距离测试码输入功耗研究

伪随机测试在数字系统的故障测试中已经得到了多年的应用,但传统的伪随机测试存在着效率比较低的缺陷。针对该缺陷提出了在伪随机测试方法中引入测试码之间距离的概念。根据测试码之间距离越大,能检测到不同故障的数目概率也越大的假设,基于测试码之间距离的随机测试法（简称基于距离测试法）可以生成一组测试码序列。但是由于基于距离测试法所生成的测试码相邻间距离的变大,将造成相邻输入码之间的跳变次数增多,使得输入测试码时所需要的功耗急剧增大。针对该情况,提出对伪随机测试法生成的测试码输入顺序进行重新排序和调整的概念,从而达到降低测试功耗的最终目标。