Generating minimal fault detecting test suites for general Boolean specifications

Context

Boolean expressions are a central aspect of specifications and programs, but they also offer dangerously many ways to introduce faults. To counter this effect, various criteria to generate and evaluate tests have been proposed. These are traditionally based on the structure of the expressions, but are not directly related to the possible faults. Often, they also require expressions to be in particular formats such as disjunctive normal form (DNF), where a strict hierarchy of faults is available to prove fault detection capability.

Objective

This paper describes a method that generates test cases directly from an expression’s possible faults, guaranteeing that faults of any chosen class will be detected. In contrast to many previous criteria, this approach does not require the Boolean expressions to be in DNF, but allows expressions in any format, using any deliberate fault classes.

Method

The presented approach is based on creating test objectives for individual faults, such that efficient, modern satisfiability solvers can be used to derive test cases that directly address the faults. Although the number of such test objectives can be high depending on the considered fault classes, a number of optimizations can be applied to reduce the test generation effort.

Results

Evaluation on a set of commonly used benchmarks shows that despite guaranteeing fault coverage, the number of test cases can be reduced even further than that produced by other state of the art strategies. At the same time, the fault detection capability is not affected negatively, and clearly improves over state of the art criteria for general form Boolean expressions.

Conclusion

The approach presented in this paper is shown to improve over the state of the art with respect to the types of expressions that can be handled, the fault classes that are guaranteed to be covered, and the sizes of test suites generated automatically. This has implications for several fields of software testing: A main application is specification based testing, but Boolean expressions also exist in normal source code and need to be tested there as well.     

A semantic approach for automated test oracle generation

This paper presents the design, implementation, and applications of a software testing tool, TAO, which allows users to specify and generate test cases and oracles in a declarative way. Extended from its previous grammar-based test generation tool, TAO provides a declarative notation for defining denotational semantics on each productive grammar rule, such that when a test case is generated, its expected semantics will be evaluated automatically as well, serving as its test oracle. TAO further provides a simple tagging mechanism to embed oracles into test cases for bridging the automation between test case generation and software testing. Two practical case studies are used to illustrate how automated oracle generation can be effectively integrated with grammar-based test generation in different testing scenarios: locating fault-inducing input patterns on Java applications; and Selenium-based automated web testing.     

基于UML活动图生成功能测试线索

测试自动化是提高软件测试效率的重要途径。基于UML模型的面向对象软件测试是当前研究的热点。这些研究大都是类或集成测试,对于如何自动生成较为完整、合理的系统测试用例较少。在总结研究现状的基础上,以应用举例的方式提出一种从描述系统用例问顺序依赖关系的活动图和用例实现的活动图中获取系统功能测试线索的方法。     

Distributing test cases more evenly in adaptive random testing

Adaptive random testing (ART) has recently been proposed to enhance the failure-detection capability of random testing. In ART, test cases are not only randomly generated, but also evenly spread over the input domain. Various ART algorithms have been developed to evenly spread test cases in different ways. Previous studies have shown that some ART algorithms prefer to select test cases from the edge part of the input domain rather than from the centre part, that is, inputs do not have equal chance to be selected as test cases. Since we do not know where the failure-causing inputs are prior to testing, it is not desirable for inputs to have different chances of being selected as test cases. Therefore, in this paper, we investigate how to enhance some ART algorithms by offsetting the edge preference, and propose a new family of ART algorithms. A series of simulations have been conducted and it is shown that these new algorithms not only select test cases more evenly, but also have better failure detection capabilities.     

基于UML规格说明测试用例生成工具

利用UML状态图,采用基于状态的测试数据生成标准生成测试用例。其中UML状态图是测试用例生成的关键部分,在某种意义上,UML状态图能够图容易生成测试用例。     

基于测试需求的互操作性测试用例生成方法

网络化应用将成为未来软件技术发展的主导模式。为了保证网络化应用之间能够有效地协同工作,必须对其进行互操作性测试。互操作性测试具有测试需求复杂易变、测试用例设计困难的特点,因而需要消耗大量成本。为此,提出一种基于测试需求的互操作性测试用例生成方法。该方法采用模型驱动的测试思想,以测试需求模型描述互操作性测试需求,以状态图描述各被测应用的规格说明,通过两者的结合生成满足测试需求的测试用例。     

Calibrating embedded protocols on asynchronous systems

Embedding is a method of projecting one topology into another. In one-to-one node embedding, paths in the target topology correspond to links in the original topology. A protocol running on the original topology can be modified to be executed on a target topology by means of embedding. However, if the protocol is tolerant to a number of faults - faults that affect the data but not the code of a distributed protocol executed by the nodes in a distributed systems - then the adapted protocol will not have the fault tolerance property preserved, due to the fact that links in the original topology can be embedded into paths of length greater than one: faults at the intermediate nodes on such paths are not accounted for in the protocol. We propose a communication protocol in the target topology that preserves the fault tolerance characteristics of any protocol designed for the original topology, namely by our mechanism the modification preserves fault tolerance.     

基于UML测试用例生成法的研究

在当今社会随着经济和科学技术的发展,UML也逐渐应用于诸多领域,其中最常用的是建立软件系统的模型。同时伴随着UML的推广,对于在其基础之上的测试用例生成发法的研究也逐渐得到了人们的关注。本文通过介绍UML的发展及具体相关内容,来探讨基于UML测试用例生成法的研究。     

The automated generation of test cases for compilers

The testing of a compiler is a difficult process, since often a very high quality product is expected by the user, particularly with respect to the compiler's correctness. This paper seeks to review the main methods that have been proposed for the automation of part of this process, using the automatic generation of test cases and the automatic checking of the results. It then summarizes the advantages and disadvantages of the general approach and highlights several areas for further research.     

XML-manipulating test case prioritization for XML-manipulating services

A web service may evolve autonomously, making peer web services in the same service composition uncertain as to whether the evolved behaviors are compatible with its original collaborative agreement. Although peer services may wish to conduct regression testing to verify the agreed collaboration, the source code of the former service may be inaccessible to them. Owing to the black-box nature of peer services, traditional code-based approaches to regression testing are inapplicable. In addition, traditional techniques assume that a regression test suite for verifying a web service is available. The location to store a regression test suite is also a problem. On the other hand, we note that the rich interface specifications of a web service provide peer services with a means to formulate black-box testing strategies. In this paper, we provide a strategy for black-box service-oriented testing. We also formulate new test case prioritization strategies using tags embedded in XML messages to reorder regression test cases, and reveal how the test cases use the interface specifications of web services. We experimentally evaluate the effectiveness of these black-box strategies in revealing regression faults in modified WS-BPEL programs. The results show that the new techniques can have a high chance of outperforming random ordering. Moreover, our experiment shows that prioritizing test cases based on WSDL tag coverage can achieve a smaller variance than that based on the number of tags in XML messages in regression test cases, even though their overall fault detection rates are similar.     

基于方法序列规范的测试用例生成

1 引言面向对象软件测试技术的研究是最近几年才引起重视的一个新领域。OO程序由于引入了封装、继承、多态等概念和机制,在程序的组织结构和运行机制上与传统的程序有很大差别,因而也对软件测试技术提出了新要求。无论是传统程序还是OO程序,测试活动通常分4个步骤进行:(1)确定测试准则;(2)基于测试准则构造     

随机测试用例的优化技术研究

提出了一种基于粗糙集的不完备测试数据填补方法。该方法首先利用粗糙集中下近似集的性质对随机生成的测试数据进行填补,然后根据属性数据的取值概率函数求出的结果进行二次填补,从而完成对不完备测试数据的完备化处理,生成最优测试用例。采用本方法可以较好地反映待测系统所蕴含的规则,且可以避免测试数据的冲突。     

基于概率统计的测试用例生成技术研究

软件可靠性测试是软件工程中的一个新的研究领域。为了缩短软件测试周期,测试用例的生成是关键,提出将测试用例的生成问题转化为一个数学问题。通过实例证明,这种测试技术具有一定实用性和有效性。     

面向对象语言编译器的自动测试平台

提出了一种新的面向对象语言编译器自动测试的方法和过程,并在此基础上设计开发了面向对象编译器自动测试工具(O_OCTT)。测试用例的自动生成是编译器自动测试工具的核心,介绍了C 编译器测试用例自动生成的手段和方法,阐述了。O_OCTT设计与实现过程中的关键技术。     

Synchronous,asynchronous, and causally ordered communication

Summary.  This article studies characteristic properties of synchronous and asynchronous message communications in distributed systems. Based on the causality relation between events in computations with asynchronous communications, we characterize computations which are realizable with synchronous communications, which respect causal order, or where messages between two processes are always received in the order sent. It is shown that the corresponding computation classes form a strict hierarchy. Furthermore, an axiomatic definition of distributed computations with synchronous communications is given, and it is shown that several informal characterizations of such computations are equivalent when they are formalized appropriately. As an application, we use our results to show that the distributed termination detection algorithm by Dijkstra et al. is correct under a weaker synchrony assumption than originally stated. Received: November 1992/Accepted: July 1995     

基于规则引擎的测试用例自动生成方法研究与实现

测试用例自动生成是软件自动化测试的基础与关键。本文实现了基于规则引擎的自动测试用例生成技术,探讨了相应的测试覆盖准则,分析了测试数据的生成。     

基于黑盒测试的航天嵌入式软件用例设计方法

为了提高测试用例设计的效率,简化测试用例数量,提出了基于黑盒测试的航天嵌入式软件用例设计方法.阐述了确认测试过程中实用的测试用例设计流程,提出了在设计测试输入数据的基础上加强测试逻辑设计的用例设计方法,并结合实际工程经验,给出了测试用例的总体设计思路.最后,对测试遗漏问题及其原因进行了分析,为今后进行更有效的星载软件测试提供了参考.     

Detecting buffer overflow via automatic test input data generation

Buffer overflows cause serious problems in various categories of software systems. In critical systems, such as health-care, nuclear or aerospace software applications, a buffer overflow may cause severe threats to humans or severe economic losses. If they occur in network or security applications, they can be exploited to gain administrator privileges, perform system attacks, access unauthorized data, or misuse the system. This paper proposes a combination of genetic algorithms, linear programming, evolutionary testing, and static and dynamic information to detect buffer overflows. The newly proposed test input generation process avoids the need for human intervention to define and tune genetic algorithm weights and therefore it becomes completely automated. The process that guides the genetic search towards the detection of buffer overflow relies on a fitness function that takes into account static and dynamic information. Reported results of our case studies, consisting of two sets of open-source programs show that the new process and fitness function outperform previously published approaches.