Model checking, testing and verification working together

We present a symbolic model checking approach that allows verifying a unit of code, e.g., a single procedure or a collection of procedures that interact with each other. We allow temporal specifications that assert over both the program counters and the program variables. We decompose the verification into two parts: (1) a search that is based on the temporal behavior of the program counters, and (2) the formulation and refutation of a path condition, which inherits conditions constraining the program variables from the temporal specification. This verification approach is modular, as we do not require that all the involved procedures are provided. Furthermore, we do not request that the code is based on a finite domain. The presented approach can also be used for automating the generation of test cases for unit testing.A preliminary version of the paper, with the title Unit Checking: Symbolic Model Checking for a Unit of Code appears in the Lecture Notes in Computer Science volume 2772, Verification– Theory and Practice, celebrating Zohar Manna’s 64th birthdayThis research was partially supported by US Army Research Office Grant number DAAAD19-01-1-0473This research was partially supported by Subcontract UTA03-031 to The University of Warwick under University of Texas at Austin’s prime National Science Foundation Grant #CCR-0205483Received February 2004Revised September 2004 and April 2005Accepted April 2005 by M. Leuschel and D. J. Cooke     

基于模型检测技术的变异测试用例生成方法

为了进行基于模型的软件测试变异分析,文中提出了一种基于模型检测的变异测试用例生成方法。基于模型检测工具UPPAAL的形式化分析与测试框架,首先用符合规范的时间自动机模型描述被测系统;然后基于时间自动机模型的基本结构和语法,对系统模型进行一组变异操作,并模拟实现时可能出现的一些错误;对变异后的模型分别使用UPPAAL Yggdrasil工具,生成一组能覆盖变异区域的测试用例;在系统变异模型上执行生成的测试用例,根据测试执行结果(是否能“杀死”变异体)筛选出一组有效的测试用例。通过实例验证,所提方案生成的测试用例是有效的,且测试用例集变异分数优于现有的基于状态机复制的变异测试用例自动生成方法和基于模型中变换覆盖的变异测试用例生成方法。     

Test sequence generation and model checking using dynamic transition relations

The task of finding a set of test sequences that provides good coverage of industrial circuits is infeasible because of the size of the circuits. For small critical subcircuits of the design, however, designers can create a set of test sequences that achieve good coverage. These sequences cannot be used on the full design because the inputs to the subcircuit may not be accessible. In this work we present an efficient test generation algorithm that receives a test sequence created for the subcircuit and finds a test sequence for the full design that reproduces the given sequence on the subcircuit. The algorithm uses a new technique called dynamic transition relations to increase its efficiency .The most common and most expensive step in our algorithm is the computation of the set of predecessors of a set of states. To make this computation more efficient we exploit a partitioning of the transition relation into a set of simpler relations. At every step we use only those that are necessary, resulting in a smaller relation than the original one. A different relation is used for each step, hence the name dynamic transition relations. The same idea can be used to improve symbolic model checking for the temporal logic CTL.We have implemented the new method in SMV and run it on several large circuits. Our experiments indicate that the new method can provide gains of up to two orders of magnitude in time and space during verification. These results show that dynamic transition relations can make it possible to verify circuits that were previously unmanageable due to their size and complexity .     

基于Ajax技术的Web应用的建模与测试用例生成

Ajax技术使Web应用能够通过异步请求从服务端获取数据,并在网页上局部刷新显示。这使得一张网页可以包含多个不同状态,状态数的激增使其关系变得更加复杂,给Web应用的建模与测试带来了更大的难度。研究基于Ajax技术的Web应用的建模与测试用例生成方法,给出一种可行的产生测试用例的技术。结合课题组自身开发的项目进行建模与测试用例的生成分析,结果表明,该技术能有效地得到所需的测试用例。     

Evaluating test suite characteristics,cost, and effectiveness of FSM-based testing methods

ContextTesting from finite state machines has been investigated due to its well-founded and sound theory as well as its practical application. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods.ObjectiveWe compare the methods that generate complete test suites from finite states machines. The test suites produced by the W, HSI, H, SPY, and P methods are analyzed in different configurations.MethodComplete and partial machines were randomly generated varying numbers of states, inputs, outputs, and transitions. These different configurations were used to compare test suite characteristics (number of resets, test case length) and the test suite length (i.e., the sum of the length of its test cases). The fault detection ratio was evaluated using mutation testing to produce faulty implementations with an extra state.ResultsOn average, the recent methods (H, SPY, and P) produced longer test cases but smaller test suites than the traditional methods (W, HSI). The recent methods generated test suites of similar length, though P produced slightly smaller test suites. The SPY and P methods had the highest fault detection ratios and HSI had the lowest. For all methods, there was a positive correlation between the number of resets and the test suite length and between the test case length and the fault detection ratio.ConclusionThe recent methods rely on fewer and longer test cases to reduce the overall test suite length, while the traditional methods produce more and shorter test cases. Longer test cases are correlated to fault detection ratio which favored SPY, though all methods have a ratio of over 92%.     

基于模型的验证及其方法

基于模型的验证（Model-based Verfication)方法在软件开发中,特别是安全性系统、复杂系统的开发中占有的地位越来越重要,MBV侧重于在开发的早期找出错误,从而避免时间、金钱的耗费及重复性的劳动,在研究各种MBV技术的基础上,着重于软件系统的模型验证方法,对各种方法作了基本的介绍,并总结了发展方向。     

Automatic test case generation for structural testing of function block diagrams

ContextFunction Block Diagram (FBD) is increasingly used in safety-critical applications. Test coverage issues for FBDs are frequently raised by regulators and users. However, there is little work at this aspect on testing FBD at model level. Our previous study has designed a new data-flow test coverage criterion, FB-Path Complete Condition Test Coverage (FPCC), that can directly test FBD structures and effectively detect function mutation errors. Nevertheless, because FPCC scheme involves several data-flow concepts and thus it is somewhat complicated to comprehend and to generate FPCC-complied test cases. An automatic test suite generator for FPCC is highly desirable.ObjectiveThis study designs an automatic test case generator, FPCCTestGen, for FPCC so as to enhance the practicability and acceptance of the FPCC approach.MethodFirst, a supporting infrastructure for performing automatic FBD-to-UPPAAL-for-FPCC transformation is designed. The supporting infrastructure includes templates, declarations, and functions as building blocks for transformation. Then, for each input FBD, represented in PLCopen XML format, FPCCTestGen performs parsing and converts FBD components into corresponding UPPAAL model components using aforementioned building blocks. After that, queries related to FPCC characteristics are submitted to UPPAAL model checker for verification. Finally, the verification traces are analyzed to obtain a FPCC-complied test suite.ResultsA safety injection system is used as a case study. Preliminary results show that the generated test suite achieves the highest FPCC percentage with a near optimal number of test cases.ConclusionThis automatic test case generation tool is effective and thus, can promote the use of the new test coverage criterion. Methodology used in FPCCTestGen is generic and can be applied to test suite generation for other test criteria on data-flow programs.     

Model-based testing using UML activity diagrams: A systematic mapping study

Context:The Unified Modeling Language (UML) has become the de facto standard for software modeling. UML models are often used to visualize, understand, and communicate the structure and behavior of a system. UML activity diagrams (ADs) are often used to elaborate and visualize individual use cases. Due to their higher level of abstraction and process-oriented perspective, UML ADs are also highly suitable for model-based test generation. In the last two decades, different researchers have used UML ADs for test generation. Despite the growing use of UML ADs for model-based testing, there are currently no comprehensive and unbiased studies on the topic.Objective:To present a comprehensive and unbiased overview of the state-of-the-art on model-based testing using UML ADs.Method:We review and structure the current body of knowledge on model-based testing using UML ADs by performing a systematic mapping study using well-known guidelines. We pose nine research questions, outline our selection criteria, and develop a classification scheme.Results:The results comprise 41 primary studies analyzed against nine research questions. We also highlight the current trends and research gaps in model-based testing using UML ADs and discuss some shortcomings for researchers and practitioners working in this area. The results show that the existing approaches on model-based testing using UML ADs tend to rely on intermediate formats and formalisms for model verification and test generation, employ a multitude of graph-based coverage criteria, and use graph search algorithms.Conclusion:We present a comprehensive overview of the existing approaches on model-based testing using UML ADs. We conclude that (1) UML ADs are not being used for non-functional testing, (2) only a few approaches have been validated against realistic, industrial case studies, (3) most approaches target very restricted application domains, and (4) there is currently a clear lack of holistic approaches for model-based testing using UML ADs.     

基于程序切片的测试用例生成系统研究与实现

介绍了一种基于程序行为切片的测试用例生成系统的实现方案,系统在不扫描全部程序路径的情况下,生成可以覆盖全部程序行为的测试用例集。系统分为静态分析、动态符号执行以及测试用例生成3个模块。在静态分析模块中根据输入的程序代码分析程序的控制流和信息流,提取程序的控制依赖和数据依赖,并计算程序的潜在依赖;动态符号执行模块求解约束条件、生成测试用例和分析代码执行过程;测试用例生成模块根据执行路径和依赖关系计算被路径覆盖的程序行为切片和未被覆盖的程序行为切片,然后根据未被覆盖的程序行为切片,引导符号执行生成能覆盖新的程序行为切片的测试用例。实验证明,本系统生成的测试用例集可以保证覆盖所有的程序行为,同时能显著减少生成的测试用例数量。     

Bottom-up reuse for multi-level testing

Lifecycle models divide the test process into consecutive test levels that are considered independently. This strict separation obstructs the view on the test process as a whole and fails to reflect the commonalities across test levels. Multi-level testing is an emerging approach that addresses the challenge of integrating test levels, putting particular emphasis on embedded systems. In this paper, we introduce a test level integration strategy based on reuse that is called bottom-up reuse. In addition, we present a test level instrument that seamlessly supports this strategy: multi-level test cases. We also provide a case study that reflects the positive results we have obtained in practice so far and demonstrates the feasibility of our test level integration approach. Bottom-up reuse and multi-level test cases lead to testing earlier on in the development process while reducing the effort required by test specification, test design, and test implementation.     

Mutation testing on an object-oriented framework: An experience report

Context

The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners.

Objective

This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective.

Method

In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT.

Results

Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field.

Conclusion

We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.     

Event-driven web application testing based on model-based mutation testing

ContextEvent-Driven Software (EDS) is a class of software whose behavior is driven by incoming events. Web and desktop applications that respond to user-initiated events on their Graphical User Interface (GUI), or embedded software responding to events and signals received from the equipment in its operating environment are examples of EDS. Testing EDS poses great challenges to software testers. One of these challenges is the need to generate a huge number of possible event sequences that could sufficiently cover the EDS’s state space.ObjectiveIn this paper, we introduce a new six-stage testing procedure for event-driven web applications to overcome EDS testing challenges.MethodThe stages of the testing procedure include dividing the application based on its structure, creating functional graphs for each section, creating mutants from functional graphs, choosing coverage criteria to produce test paths, merging event sequences to make longer ones, and deriving and running test cases. We have analyzed our proposed testing procedure with the help of four metrics consisting of Fault Detection Density (FDD), Fault Detection Effectiveness (FDE), Mutation Score, and Unique Fault.ResultsUsing this procedure, we have prepared prioritized test cases and also discovered a list of unique faults by running the suggested test cases on a sample real-world web application called Academic E-mail System.ConclusionWe propose that our suggested testing procedure has some advantages such as creating functional graphs with requirements document, resolving the problem of removing infeasible test cases with these graphs and conditions on the “add edge” operator before creating mutants. But the suggested testing procedure, like any other method, had some drawbacks. Because most of the stages in the approach were performed manually, the testing time was increased.     

Control flow and data flow oriented integration testing methods

The testing of modular software systems can be divided into a module testing phase and an integration testing phase. While module testing checks the modules separately, integration testing examines the use of interfaces in a modular system. Integration testing allows errors to be found which cannot be found by module testing. The aim of this paper is to propose a new approach to integration testing. The main principle is to transfer and adapt module testing methods to the level of integration testing. The approach is described for control flow and data flow oriented testing methods. To decrease the testing effort and increase the probability of finding errors, integration testing can be limited to statically detectable anomalous applications of interfaces. This is accomplished by the combination of static analysis with dynamic execution and by the possibility of using information already provided by the module tests. To find further test data to execute interfaces, symbolic execution is applied. One great advantage here is to prove whether statically determined interface anomalies can be dynamically executed and can therefore occur at all.     

An evaluation of the effectiveness of symbolic testing

The effectiveness in discovering errors of symbolic evaluation and of testing sad static program analysis are studied. The three techniques are applied to a diverse collection of programs and the results compared. Symbolic evaluation is used to carry out symbolic testing and to generate symbolic systems of path predicates. The use of the predicates for automated test data selection is analysed. Several conventional types of program testing strategies are evaluated. The strategies include branch testing, structured testing and testing on input values having special properties. The static source analysis techniques that are studied include anomaly analysis and interface analysis. Examples are included which describe typical situations in which one technique is reliable but another unreliable. The effectiveness of symbolic testing is compared with testing on actual data and with the use of an integrated methodology that includes both testing and static source analysis. Situations in which symbolic testing is difficult to apply or not effective are discussed. Different ways in which symbolic evaluation can be used for generating test data are described. Those ways for which it is most effective are isolated. The paper concludes with a discussion of the most effective uses to which symbolic evaluation can he put in an integrated system which contains all three of the validation techniques that are studied.     

基于模型的若干逻辑边界覆盖测试准则

鉴于 现有的基于规约的逻辑覆盖测试准则很少考虑到边界情况,对边界值分析法进行形式化,提出了一系列基于模型的逻辑边界覆盖测试准则。结果表明,相对于传统的逻辑覆盖测试准则,满足这些测试准则生成的测试用例能检测出系统更多的错误,既满足相应的逻辑覆盖测试准则,又能检测系统的边界情况。     

Interprocedural test path evaluation

In automated test data generation a program is analyzed to derive a set of test paths, that is a set of predicted executions. Each path is characterized by some path constraints: data satisfying these constraints will perform the predicted execution.

This paper discusses some problems which arise during path constraints evaluation when block structured languages are involved and when test paths traverse several procedures (possibly with recursive activations). In these cases partial test paths, independently selected for each procedure, must be pre-processed (enriched and or modified) to explicitly consider the semantics associated with dynamic memory allocation, parameter transmission (by value, value-result, name and reference), and co-existing instances of a recursive procedure.     

Using symbolic execution and data flow criteria to aid test data selection

The utilization is described of a data flow path selection criterion in a symbolic execution system. The system automatically generates a subset of program paths according to a certain control flow criterion. This subset is called the ZOT-subset, since it requires paths that traverse loops zero, one and two times. Experience indicates that traversing this subset of program paths is enough to cover most control flow and data flow components of the program. The problem with the ZOT-subset is that it might contain, for large programs, a large number of paths. The number of paths in this subset can be reduced by concentrating on executable paths that cover vital components of programs such as data flow components. This object is achieved by employing a data flow path selection criterion in the system. The system symbolically executes the paths of the ZOT-subset, and creates a system of branch conditions for each one. The user determines infeasible paths by checking the consistency of each system of conditions. The system selects feasible paths from the ZOT-subset that cover the data flow criterion. Solving the systems of conditions of the selected paths provides the user with test data to fulfil the given data flow criterion.     

An empirical investigation into branch coverage for C programs using CUTE and AUSTIN

Automated test data generation has remained a topic of considerable interest for several decades because it lies at the heart of attempts to automate the process of Software Testing. This paper reports the results of an empirical study using the dynamic symbolic-execution tool, CUTE, and a search based tool, AUSTIN on five non-trivial open source applications. The aim is to provide practitioners with an assessment of what can be achieved by existing techniques with little or no specialist knowledge and to provide researchers with baseline data against which to measure subsequent work. To achieve this, each tool is applied ‘as is’, with neither additional tuning nor supporting harnesses and with no adjustments applied to the subject programs under test. The mere fact that these tools can be applied ‘out of the box’ in this manner reflects the growing maturity of Automated test data generation. However, as might be expected, the study reveals opportunities for improvement and suggests ways to hybridize these two approaches that have hitherto been developed entirely independently.     

基于OWL-S文档的Web服务测试用例生成

若从使用者的角度测试Web服务,只能从服务的接口文档中提取信息进行测试工作.现有工作提出通过分析文档中的输入或输出参数获取测试数据的方法.利用OWL-S文档中的输入输出参数信息生成初始的测试数据,同时充分利用文档中的服务过程信息,提取出服务的控制流程图,据此约简测试数据,生成最终的测试用例,从而提高测试用例的生成效率,降低测试成本.