Specification-driven model transformation testing

Testing model transformations poses several challenges, among them the automatic generation of appropriate input test models and the specification of oracle functions. Most approaches for the generation of input models ensure a certain coverage of the source meta-model or the transformation implementation code, whereas oracle functions are frequently defined using query or graph languages. However, these two tasks are usually performed independently regardless of their common purpose, and sometimes, there is a gap between the properties exhibited by the generated input models and those considered by the transformations. Recently, we proposed a formal specification language for the declarative formulation of transformation properties (by means of invariants, pre-, and postconditions) from which we generated partial oracle functions used for transformation testing. Here, we extend the usage of our specification language for the automated generation of input test models by SAT solving. The testing process becomes more intentional because the generated models ensure a certain coverage of the transformation requirements. Moreover, we use the same specification to consistently derive both the input test models and the oracle functions. A set of experiments is presented, aimed at measuring the efficacy of our technique.     

基于分支覆盖的回归测试路径选择

回归测试是迭代式软件开发的重要环节,测试数据生成是回归测试的前提.传统的回归测试方法,从已有的测试数据中选择部分测试数据,并生成一些新的测试数据,以验证程序的正确性.但是,该方法容易生成冗余的测试数据,从而降低了回归测试的效率.研究了回归测试的分支覆盖问题,通过利用已有测试数据的路径覆盖信息,并选择一定个数的路径,以覆盖所有的目标分支.首先,以若干路径形成的集合作为决策变量,以路径最少、覆盖的分支最多以及包含的未覆盖路径最少为目标,建立路径选择问题的3目标优化模型;然后,采用遗传算法求解上述模型时,设计了基于目标重要性的个体评价策略;最后,基于已有的测试数据与选择的路径之间的覆盖关系,确定需要生成的测试数据.将所提方法应用于6个基准工业程序测试中,并与其他方法比较.实验结果表明,采用该方法选择的路径,能够覆盖更多的分支,需要生成的测试数据更少,回归测试消耗的时间更短.     

A Test Design Methodology for Protocol Testing

Communication protocol testing can be done with a test architecture consisting of remote Lower Tester and local Upper Tester processes. For real protocols, tests can be designed based on the formal specification of the protocol which uses an extended finite state machine model. The specification is transformed into a simpler form consisting of normal form transitions. It can then be modeled by a control and a data flow graph. The graphs are decomposed into subtours and data flow functions, respectively. Tests are designed by considering parameter variations of the input primitives of each data flow function and determining the expected outputs. The methodology gives complete test coverage of all data flow functions and control paths in the specification. Functional fault models are proposed for functions that are not formally specified.     

Search based algorithms for test sequence generation in functional testing

ContextThe generation of dynamic test sequences from a formal specification, complementing traditional testing methods in order to find errors in the source code.ObjectiveIn this paper we extend one specific combinatorial test approach, the Classification Tree Method (CTM), with transition information to generate test sequences. Although we use CTM, this extension is also possible for any combinatorial testing method.MethodThe generation of minimal test sequences that fulfill the demanded coverage criteria is an NP-hard problem. Therefore, search-based approaches are required to find such (near) optimal test sequences.ResultsThe experimental analysis compares the search-based technique with a greedy algorithm on a set of 12 hierarchical concurrent models of programs extracted from the literature. Our proposed search-based approaches (GTSG and ACOts) are able to generate test sequences by finding the shortest valid path to achieve full class (state) and transition coverage.ConclusionThe extended classification tree is useful for generating of test sequences. Moreover, the experimental analysis reveals that our search-based approaches are better than the greedy deterministic approach, especially in the most complex instances. All presented algorithms are actually integrated into a professional tool for functional testing.     

Automatic generation of test specifications for coverage of system state transitions

Adequate system testing of present day application programs requires satisfactory coverage of system states and transitions. This can be achieved by using a system state model. However, the system state models are rarely constructed by system developers, as these are large and complex. The only state models that are constructed by the developers are those of individual objects. However test case generation for state-based system testing based on traversal of statecharts of individual objects appears to be infeasible, since system test cases would have to be specified in the form of scenario sequences rather than transitions on individual object statecharts. In this paper, we propose a novel approach to coverage of elementary transition paths of an automatically synthesized system state model. Our technique for coverage of elementary transition paths would also ensure coverage of all states and transitions of the system model.     

A model-driven framework to enhance the consistency of logical integrity constraints: Introducing integrity regression testing

Although the importance of models continuously grows in software development, common development approaches are less able to integrate the automatic management of model integrity into the development process. These critically important constraints may ensure the coherence of models in the evolution process to prevent manipulations that could violate defined constraints on a model. This paper proposes an integrity framework in the context of model-driven architecture to achieve sufficient structural code coverage at a higher program representation level than machine code. Our framework offers to propagate the modifications from a platform-independent specification to the corresponding test template model while keeping the consistency and integrity constraints after system evolution. To examine the efficiency of the proposed framework, a quantitative analysis plan is evaluated based on two experimental case studies. In addition, we propose coverage criteria for integrity regression testing (IRT), derived from logic coverage criteria that apply different conceptual levels of testing for the formulation of integrity requirements. The defined criteria for IRT reduce the inherent complexity and cost of verifying complex design changes in regression testing while keeping the fault detection capability with respect to the changes. The framework aims to keep pace with IRT in a formal way. The framework can solve a number of restricted outlooks in model integrity and some limiting factors of incremental maintenance and retesting. The framework satisfies several valuable quality attributes in software testing, such as safety percentage, precision, abstract fault detection performance measurable coverage level, and generality.     

Mutation based test case generation via a path selection strategy

ContextGenerally, mutation analysis has been identified as a powerful testing method. Researchers have shown that its use as a testing criterion exercises quite thoroughly the system under test while it achieves to reveal more faults than standard structural testing criteria. Despite its potential, mutation fails to be adopted in a widespread practical use and its popularity falls significantly short when compared with other structural methods. This can be attributed to the lack of thorough studies dealing with the practical problems introduced by mutation and the assessment of the effort needed when applying it. Such an incident, masks the real cost involved preventing the development of easy and effective to use strategies to circumvent this problem.ObjectiveIn this paper, a path selection strategy for selecting test cases able to effectively kill mutants when performing weak mutation testing is presented and analysed.MethodThe testing effort is highly correlated with the number of attempts the tester makes in order to generate adequate test cases. Therefore, a significant influence on the efficiency associated with a test case generation strategy greatly depends on the number of candidate paths selected in order to achieve a predefined coverage goal. The effort can thus be related to the number of infeasible paths encountered during the test case generation process.ResultsAn experiment, investigating well over 55 million of program paths is conducted based on a strategy that alleviates the effects of infeasible paths. Strategy details, along with a prototype implementation are reported and analysed through the experimental results obtained by its employment to a set of program units.ConclusionThe results obtained suggest that the strategy used can play an important role in making the mutation testing method more appealing and practical.     

Formal firewall conformance testing: an application of test and proof techniques

Firewalls are an important means to secure critical ICT infrastructures. As configurable off‐the‐shelf products, the effectiveness of a firewall crucially depends on both the correctness of the implementation itself as well as the correct configuration. While testing the implementation can be done once by the manufacturer, the configuration needs to be tested for each application individually. This is particularly challenging as the configuration, implementing a firewall policy, is inherently complex, hard to understand, administrated by different stakeholders and thus difficult to validate. This paper presents a formal model of both stateless and stateful firewalls (packet filters), including NAT , to which a specification‐based conformance test case generation approach is applied. Furthermore, a verified optimisation technique for this approach is presented: starting from a formal model for stateless firewalls, a collection of semantics‐preserving policy transformation rules and an algorithm that optimizes the specification with respect of the number of test cases required for path coverage of the model are derived. We extend an existing approach that integrates verification and testing, that is, tests and proofs to support conformance testing of network policies. The presented approach is supported by a test framework that allows to test actual firewalls using the test cases generated on the basis of the formal model. Finally, a report on several larger case studies is presented. Copyright © 2014 John Wiley & Sons, Ltd.     

Coverage-biased random exploration of large models and application to testing

This paper presents several randomised algorithms for generating paths in large models according to a given coverage criterion. Using methods for counting combinatorial structures, these algorithms can efficiently explore very large models, based on a graphical representation by an automaton or by a product of several automata. This new approach can be applied to random exploration in order to optimise path coverage and can be generalised to take into account other coverage criteria, via the definition of a notion of randomised coverage satisfaction. Our main contributions are a method for drawing paths uniformly at random in composed models, i.e. models that are given as products of automata, first without and then with synchronisation; a new efficient approach to draw paths at random taking into account some other coverage criterion. Experimental results show promising agreement with theoretical predictions and significant improvement over previous randomised approaches. This work opens new perspectives for future studies of statistical testing and model checking, mainly to fight the combinatorial explosion problem.     

Use of sequencing constraints for specification-based testing ofconcurrent programs

This paper presents and evaluates a specification-based methodology for testing concurrent programs. This methodology requires sequencing constraints, which specify restrictions on the allowed sequences of synchronization events. Sequencing constraints for a concurrent program can be derived from the program's formal or informal specification. Details of the proposed testing methodology based on the use of Constraints on Succeeding and Preceding Events (CSPE) are given. How to achieve coverage and detect violations of CSPE constraints for a concurrent program, according to deterministic and nondeterministic testing of this program, are described. A coverage criterion for CSPE-based testing is defined and analyzed. The results of empirical studies of CSPE-based testing for four concurrent problems are reported. These results indicate that the use of sequencing constraints for specification-based testing of concurrent programs is a promising approach     

Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion?, an embedded system providing helicopter landing assistance.     

Automated test generation using model checking: an industrial evaluation

In software development, testers often focus on functional testing to validate implemented programs against their specifications. In safety-critical software development, testers are also required to show that tests exercise, or cover, the structure and logic of the implementation. To achieve different types of logic coverage, various program artifacts such as decisions and conditions are required to be exercised during testing. Use of model checking for structural test generation has been proposed by several researchers. The limited application to models used in practice and the state space explosion can, however, impact model checking and hence the process of deriving tests for logic coverage. Thus, there is a need to validate these approaches against relevant industrial systems such that more knowledge is built on how to efficiently use them in practice. In this paper, we present a tool-supported approach to handle software written in the Function Block Diagram language such that logic coverage criteria can be formalized and used by a model checker to automatically generate tests. To this end, we conducted a study based on industrial use-case scenarios from Bombardier Transportation AB, showing how our toolbox CompleteTest can be applied to generate tests in software systems used in the safety-critical domain. To evaluate the approach, we applied the toolbox to 157 programs and found that it is efficient in terms of time required to generate tests that satisfy logic coverage and scales well for most of the programs.     

基于BPEL的Web Service组合的数据流分析测试方法

随着Web Service组合变得越来越复杂,通过测试来保证服务质量和可靠性也变得越来越重要.将传统数据流分析方法扩展用于Web Service组合测试,提出了一种基于BPEL的Web Service组合的数据流分析测试方法.该方法基于一个测试模型:Web Service组合测试模型WSCTM,该测试模型可以捕获Web Service组合的数据流接口.采用基于服务的模型WSCTM,数据流可以从3个视点来分析:服务间、服务内和服务实现构件间.从而,Web Service组合的数据流测试可以在三层上得到实现.基于以上方法,可得到Web Service组合的定义-使用链,最终可产生满足既定测试标准以获得需求Web服务组合质量要求的测试路径.     

Generating semantically valid test inputs using constrained input grammars

ContextGenerating test cases based on software input interface is a black-box testing technique that can be made more effective by using structured input models such as input grammars. Automatically generating grammar-based test inputs may lead to structurally valid but semantically invalid inputs that may be rejected in early semantic error checking phases of a system under test.ObjectiveThis paper aims to introduce a method for specifying a grammar-based input model with the model’s semantic constraints to be used in the generation of positive test inputs. It is also important that the method can generate effective test suites based on appropriate grammar-based coverage criteria.MethodFormal specification of both input structure and input semantics provides the opportunity to use model instantiation techniques to create model instances that satisfy all specified constraints. The input interface of a subject system can be specified using a high-level specification scheme such as attribute grammars, and a transformation function from this scheme to an instantiable formal modeling language can generate the desired model instances.ResultsWe propose a declarative grammar-based input specification method that is based on a variation of attribute grammars and allows the user to specify input constraints in addition to input structure. The model can be instantiated automatically to generate structurally and semantically valid test inputs. The proposed method has the capability to specify test requirements and coverage criteria and use them to generate valid test suites that satisfy test coverage criteria requirements.ConclusionThe work presented in this paper provides a black-box test generation method for grammar-based software inputs that can automatically generate criteria-covering test suites.     

Developing comprehensive acceptance tests from use cases and robustness diagrams

In agile development processes, the rewards from acceptance testing are maximized by using the practice to drive the development process. Traditionally, User Stories are used in agile projects to describe a system’s usage scenarios and are utilized as a basis for developing acceptance tests. This paper introduces a technique that aims to achieve the benefits of acceptance testing within large-scale development projects that deploy a V-model development process, specifically those that utilize use case models. The approach is based on utilizing a number of artifacts: use case models supported by robustness diagrams and domain models. The feasibility of the proposed approach is demonstrated by applying it to a real-world system—the RestoMapper system. The results show that a comprehensive set of acceptance tests can be developed based upon use case models.     

Stopping criteria for statistical testing

The decision to stop testing can be based on a number of criteria, such as (1) the confidence in a reliability estimate; (2) the degree to which testing experience has converged to the expected use of the software; and (3) model coverage criteria based on a degree of state, arc, or path coverage during crafted and random testing. In practice it is best to use multiple stopping criteria. For example, further evaluation of the testing performed is needed if the measure of correspondence between testing experience and expected use of the software indicates that the testing experience closely matches the expected use of the software, yet the variance of the reliability estimate is unacceptably large. One test of equality of testing experience and expected use is the Kullback discriminant from the usage chain to the testing chain. A new measure of “approximate equality” is introduced here for use in conjunction with the Kullback discriminant.     

Assessing testing tools in research and education

An evaluation of three software engineering tools based on their use in research and educational environments is presented. The three testing tools are Mothra, a mutation-testing tool, Asset, a dataflow testing tool, and ATAC, a dataflow testing tool. Asset, ATAC, and Mothra were used in research projects that examined relative and general fault-detection effectiveness of testing methods, how good a test set is after functional testing based on program specification, how reliability estimates from existing models vary with the testing method used, and how improved coverage affects reliability. Students used ATAC and Mothra by treating the tools as artifacts and studying them from the point of view of documentation, coding style, and possible enhancements, solving simple problems given during testing lectures, and conducting experiments that supported ongoing research in software testing and reliability. The strengths, weaknesses, and performances of Asset, Mothra, and ATAC are discussed     

Enumerating message paths for interaction testing of object-oriented systems

MM path is integration artifact of object-oriented systems and is characterized by interleaved sequence of messages and method execution paths. In the context of UML sequence diagrams, an MM path is usually found common in many message paths and concatenation of one MM path with different MM paths results in distinct message paths. Adequate testing of all message paths in UML sequence diagrams implies the coverage of all possible concatenation of constituent MM paths in different object-states, which may not be fully achieved in practical situations. With limited test effort, practitioners are compelled to consider a subset of all message paths in ad-hoc manner or priority basis. It can be possible that the selected paths may not include at least one instance of highly critical MM path, whereas less critical MM paths may have been included with multiple instances. It is, therefore, necessary that the selected message paths would contain one instance of all MM paths and their additional instances as per their priorities. To address this problem, we propose two coverage criteria: all MM paths and prioritized MM paths and an MM path-based coverage model. To obtain such coverage model, we build a graph model called as sequence integration graph from the sequence diagram and thereafter, synthesize MM paths to merge them into a set of connected trees. Further, we capture priority information of the MM paths by means of edge weights and order of concatenation among the MM paths by means of connector edges between the trees. These information captured in the MM coverage model helps to determine effective coverage of the MM paths. Our experimental results substantiate usefulness of our MM path-based coverage model for selecting message paths to satisfy all MM paths and prioritized MM paths coverage criteria in the context of integration testing of object-oriented systems with the limited testing effort.