A unified framework for evaluating test criteria in model-checking-assisted test case generation

Testing is often cited as one of the most costly operations in testing dependable systems (Heimdahl et al. 2001). A particular challenging task in testing is test-case generation. To improve the efficiency of test-case generation and reduce its cost, recently automated formal verification techniques such as model checking are extended to automate test-case generation processes. In model-checking-assisted test-case generation, a test criterion is formulated as temporal logical formulae, which are used by a model checker to generate test cases satisfying the test criterion. Traditional test criteria such as branch coverage criterion and newer temporal-logic-inspired criteria such as property coverage criteria (Tan et al. 2004) are used with model-checking-assisted test generation. Two key questions in model-checking-assisted test generation are how efficiently a model checker may generate test suites for these criteria and how effective these test suites are. To answer these questions, we developed a unified framework for evaluating (1) the effectiveness of the test criteria used with model-checking-assisted test-case generation and (2) the efficiency of test-case generation for these criteria. The benefits of this work are three-fold: first, the computational study carried out in this work provides some measurements of the effectiveness and efficiency of various test criteria used with model-checking-assisted test case generation. These performance measurements are important factors to consider when a practitioner selects appropriate test criteria for an application of model-checking-assisted test generation. Second, we propose a unified test generation framework based on generalized Büchi automata. The framework uses the same model checker, in this case, SPIN model checker (Holzmann 1997), to generate test cases for different criteria and compare them on a consistent basis. Last but not least, we describe in great details the methodology and automated test generation environment that we developed on the basis of our unified framework. Such details would be of interest to researchers and practitioners who want to use and extend this unified framework and its accompanying tools.     

A framework for intelligent test data generation

Test data generation using traditional software testing methods generally requires considerable manual effort and generates only a limited number of test cases before the amount of time expanded becomes unacceptably large. A rule-based framework that will automatically generate test data to achieve maximal branch coverage is presented. The design and discovery of rules used to generate meaningful test cases are also described. The rule-based approach allows this framework to be extended to include additional testing requirements and test case generation knowledge.This work was supported in part by George C. Marshall Space Flight Center, NASA/MSFC, AL 35812 (NASA-NCC8-14).     

Cloud-based automatic test data generation framework

Designing test cases is one of the most crucial activities in software testing process. Manual test case design might result in inadequate testing outputs due to lack of expertise and/or skill requirements. This article delivers automatic test data generation framework by effectively utilizing soft computing technique with Apache Hadoop MapReduce as the parallelization framework. We have evaluated and analyzed statistically our proposed framework using real world open source libraries. The experimental results conducted on Hadoop cluster with ten nodes are effective and our framework significantly outperforms other existing cloud-based testing models.     

A rigorous approach towards test case generation

This paper presents a method to generate test cases for sequential programs and concurrent programs written in a flow based program language. Test cases of sequential programs are generated based on condition calculation, and can be combined together to form SYN-sequences for concurrent program testing. Semantics of the language provides an infrastructure for the test case generation, and thus our method may be considered as a rigorous and systematic approach to the program testing. Compared with some formal testing methods, our method can avoid hitting state explosion problem in the test formation. Besides, the complexity analysis reveals that our method is time saving. Our method has been applied to generate test cases for PPP over ATM, a subsystem of IAD that runs data and voice over DSL.     

面向路径的测试数据生成框架及应用

测试数据生成是软件测试的核心与关键,本文介绍了迭代松弛法以及对迭代松弛法进行改进,改进后的方法比原方法生成测试数据的能力更强,不仅能够用于白盒测试数据的自动生成,还能够用于黑盒测试数据的自动生成。在此基础上提出一个面向路径的测试数据生成框架。并讨论该框架在单元测试、组装测试中的应用。     

Formalizing interoperability for test case generation purpose

This study deals with interoperability testing of protocol implementations. The objective is to define a method for automatic test derivation based on formal definitions. First, the notion of interoperability criteria is introduced. They formally describes the conditions that two implementations must verify in order to be considered interoperable. Then, based on the equivalence of two of the defined interoperability criteria, we propose a method to derive automatically interoperability test cases that avoids the state-space explosion problem.     

A test case generation approach for conformance testing of SDL systems

This paper presents an approach for automatic executable test case and test sequence generation for a protocol modeled by an SDL system. Our methodology uses a unified method which tests an Extended Finite State Machine (EFSM) based system by using control and data flow techniques. To test an SDL system, it extracts an EFSM from each process then the system is tested by incrementally computing a partial product for each EFSM C, taking into account only transitions which influence (or are influenced by) C, and generating test cases for it. This process ends when the coverage achieved by the generated test cases is satisfactory or when the partial products for all EFSMs are tested. Experimental results show that this method can be applied to systems of practical size.     

基于MDA的软件测试用例生成

将MDA中模型驱动的软件代码自动化生成思想应用于模型驱动的软件测试用例自动化生成。从UML/OCL模型出发,采用缺陷测试理论、变异分析技术,结合约束处理规则,开发一个可以自动生成单元测试用例的框架,提高软件测试的自动化程度,从整体上提高软件的开发效率。     

A virtualized test automation framework: A DellEMC case study of test automation practice

As software development life-cycles continue to shorten, the need for reliable, maintainable and scalable test automation solutions becomes more and more important. This importance arises as software, test and automation activities occur earlier in the development life cycle with the aim of providing accelerated feedback on the code and software produced. In this environment, infrastructure and development environments are made available earlier to test teams, the demand of which can create environment bottlenecks as software and automation teams also require the same resources at the same time. To help maximize the use of the underlying resources virtualization and cloud based solutions have been proposed. However, at present there is no guidance on how to design and implement a test automation solution that leverages an underlying virtualized infrastructure. In this paper a virtualized test automation framework designed and implemented by DellEMC Software Quality team in Cork, Ireland is presented along with a macro benchmark analysis of its performance over different virtualization environments.     

A framework for counterexample generation and exploration

Model-checking is becoming an accepted technique for debugging hardware and software systems. Debugging is based on the “Check/Analyze/Fix” loop: check the system against a desired property, producing a counterexample when the property fails to hold; analyze the generated counterexample to locate the source of the error; fix the flawed artifact—the property or the model. The success of model-checking non-trivial systems critically depends on making this Check/Analyze/Fix loop as tight as possible. In this paper, we concentrate on the Analyze part of the debugging loop. To this end, we present a framework for generating, structuring and exploring counterexamples, implemented in a tool called KEGVis. The framework is based on the idea that the most general type of evidence to why a property holds or fails to hold is a proof. Such proofs can be presented to the user in the form of proof-like counterexamples, without sacrificing any of the intuitiveness and close relation to the model that users have learned to expect from model-checkers. Moreover, proof generation is flexible, and can be controlled by strategies, whether built into the tool or specified by the user, thus enabling generation of the most “interesting” counterexample and its interactive exploration. Moreover, proofs can be used to generate and display all relevant evidence together, a technique referred to as abstract counterexamples. Overall, our framework can be used for explaining the reason why the property failed or succeeded, determining whether the property was correct (“specification debugging”), and for general model exploration.     

A relation-based method combining functional and structural testing for test case generation

Specification-based (or functional) testing enables us to detect errors in the implementation of functions defined in specifications, but since specifications are often incomplete in practice for some reasons (e.g., lack of ideas, no time to write), it is unlikely to be sufficient for testing all parts of corresponding programs. On the other hand, implementation-based (or structural) testing focuses on the examination of program structures, which allows us to test all parts of the programs, but may not be effective to show whether the programs properly implement the corresponding specifications. To perform a comprehensive testing of a program in practice, it is important to adopt both specification-based and implementation-based testing. In this paper we describe a relation-based test method that combines the specification-based and the implementation-based testing approaches. We establish a set of relations for test case generation, illustrate how the method is used with an example, and investigate the effectiveness and weakness of the method through an experiment on testing a software tool system.     

A general framework for relation graph clustering

Relation graphs, in which multi-type (or single type) nodes are related to each other, frequently arise in many important applications, such as Web mining, information retrieval, bioinformatics, and epidemiology. In this study, We propose a general framework for clustering on relation graphs. Under this framework, we derive a family of clustering algorithms including both hard and soft versions, which are capable of learning cluster patterns from relation graphs with various structures and statistical properties. A number of classic approaches on special cases of relation graphs, such as traditional graphs with singly-type nodes and bi-type relation graphs with two types of nodes, can be viewed as special cases of the proposed framework. The theoretic analysis and experiments demonstrate the great potential and effectiveness of the proposed framework and algorithm.     

BPEL的测试用例生成研究

BPEL是能表达复杂的并发行为的一个语言,被用于描述Web服务的组合.提出一个算法来生成BPEL过程的单元测试用例.首先通过给定的转换规则把用BPEL描述的过程转换成BPEL流图(FGBPEL),然后提出CTP算法来生成测试用例.对提出的方法做适当的扩展,也可以运用到其它的业务过程处理语言中.     

Petri net based test case generation for evolved specification

Model-based testing can use a model to test a concrete program’s implementation. When the model is changed due to the evolution of the specification, it is important to maintain the test suites up to date, such that it can be used for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in practice. To address this problem effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis, such that we can generate new test cases to cover only the change-related parts of the new model. The Market Information System (MIS) is employed to demonstrate the feasibility and effectiveness of the proposed method. Our experimental results show that the use of our method saves about 31.5% test case generation cost.     

Systematic scenario test case generation for nuclear safety systems

ContextThe current validation tests for nuclear software are routinely performed by random testing, which leads to uncertain test coverage. Moreover, validation tests should directly verify the system’s compliance with the original user’s needs. Unlike current model-based testing methods, which are generally based on requirements or design models, the proposed model is derived from the original user’s needs in text through domain-specific ontology, and then used to generate validation tests systematically.ObjectiveOur first goal is to develop an objective, repeatable, and efficient systematic validation test scheme that is effective for large systems, with analyzable test coverage. Our second goal is to provide a new model-based validation testing method that reflects the user’s original safety needs.MethodA model-based scenario test case generation for nuclear digital safety systems was designed. This was achieved by converting the scenarios described in natural language in a Safety Analysis Report (SAR) prepared by the power company for licensing review, to Unified Modeling Language (UML) sequence diagrams based on a proposed ontology of a related regulatory standard. Next, we extracted the initial environmental parameters and the described operational sequences. We then performed variations on these data to systematically generate a sufficient number of scenario test cases.ResultsTest coverage criteria, which are the equivalence partition coverage of initial environment, the condition coverage, the action coverage and the scenario coverage, were met using our method.ConclusionThe proposed model-based scenario testing can provide improved testing coverage than random testing. A test suite based on user needs can be provided.     

面向云测试的并行测试用例自动生成方法

为实现云计算环境下的高效软件测试,提出一种Web 应用系统功能测试的并行测试用例自动生成方法。该方法首先根据场景流图采用深度优先遍历算法生成并行测试路径,录制组合产生并行测试脚本,并进行参数化处理;然后,使用基于搜索的软件测试(SBST)方法自动生成可经过目标路径的有效测试数据集,脚本与数据耦合形成大量可并行部署的自动化测试用例。为验证方法的有效性设计了自动化云测试原型系统。实验结果表明,该并行测试用例自动生成方法可以高效地为云平台提供测试用例输入,提高测试效率。     

Issues in using model checkers for test case generation

The use of model checkers for automated software testing has received some attention in the literature: It is convenient because it allows fully automated generation of test suites for many different test objectives. On the other hand, model checkers were not originally meant to be used this way but for formal verification, so using model checkers for testing is sometimes perceived as a “hack”. Indeed, several drawbacks result from the use of model checkers for test case generation. If model checkers were designed or adapted to take into account the needs that result from the application to software testing, this could lead to significant improvements with regard to test suite quality and performance. In this paper we identify the drawbacks of current model checkers when used for testing. We illustrate techniques to overcome these problems, and show how they could be integrated into the model checking process. In essence, the described techniques can be seen as a general road map to turn model checkers into general purpose testing tools.     

Automatic test case generation for WS-Agreements using combinatorial testing

In the scope of the applications developed under the service-based paradigm, Service Level Agreements (SLAs) are a standard mechanism used to flexibly specify the Quality of Service (QoS) that must be delivered. These agreements contain the conditions negotiated between the service provider and consumers as well as the potential penalties derived from the violation of such conditions. In this context, it is important to assure that the service based application (SBA) behaves as expected in order to avoid potential consequences like penalties or dissatisfaction between the stakeholders that have negotiated and signed the SLA. In this article we address the testing of SLAs specified using the WS-Agreement standard by means of applying testing techniques such as the Classification Tree Method and Combinatorial Testing to generate test cases. From the content of the individual terms of the SLA, we identify situations that need to be tested. We also obtain a set of constraints based on the SLA specification and the behavior of the SBA in order to guarantee the testability of the test cases. Furthermore, we define three different coverage strategies with the aim at grading the intensity of the tests. Finally, we have developed a tool named SLACT (SLA Combinatorial Testing) in order to automate the process and we have applied the whole approach to an eHealth case study.     

SVM应用于测试用例生成的方法

针对在小样本情况下BP神经网络在生成软件测试用例的过程中可能产生的过学习问题及识别正确率较差的缺点,运用支持向量机具有更好的泛化性能的原理,提出了应用支持向量机生成测试用例的方法。对五个软件测试实例针对多组不同数量的训练样本所做的实验表明,在小样本情况下与BP神经网络相比,应用支持向量机得到的测试用例预期结果的正确率提高了10个百分点以上,说明了该方法的有效性。