Development of formal models and conformance testing for systems with asynchronous interfaces and telecommunications protocols

There is a gap between the formal modeling and testing methods for modern protocols and asynchronous software systems: due to high complexity of such systems, attempts to include formal models in testing procedures fail. In this paper, we propose an approach to filling this gap based on a formalization of the behavior of systems with asynchronous interfaces using contract specifications followed by the use of these specifications to design adaptive test suites. This approach was used for testing various software systems including implementations of the IPv6 Internet protocols stack and implementations of the POSIX and Linux Standard Base software interfaces.     

Interoperability test suite derivation for communication protocols

This paper addresses the problem of interoperability testing for communication protocols. We develop a coherent framework of interoperability testing in which the notions of interoperability, interoperability testing, interoperability test case and interoperability test architecture are interrelated and a systematic interoperability test suite derivation method based on the framework. The approach to interoperability testing is illustrated with the example of the ATM Signaling Protocol. To demonstrate practicality of the approach, we implemented executable test suites derived by the method on a commercial ATM test platform, applied them for interoperability testing of various ATM equipment and analyzed the testing results.     

FSM-based conformance testing methods: A survey annotated with experimental evaluation

The development of test cases is an important issue for testing software, communication protocols and other reactive systems. A number of methods are known for the development of a test suite based on a formal specification given in the form of a finite state machine. In this paper, we overview and experiment with these methods to assess their complexity, applicability, completeness, fault detection capability, length and derivation time of their test suites. The experiments are conducted on randomly generated specifications and on two realistic protocols called the Simple Connection Protocol and the ITU-T V.76 Recommendation.     

基于一致性测试理论的Statechart描述的测试用例自动生成

本文研究Statechart描述的测试语义和测试用例的自动生成.基于Tretmans的从标记转换系统描述自动生成测试用例的方法,我们研究如何从Statechart描述自动生成测试用例.本文的主要贡献在于建立了基于Statechart描述的一致性测试和测试用例生成的形式化基础.为Statechart描述建立了形式化测试语...     

Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software

Software is increasingly being developed/maintained by multiple, often geographically distributed developers working concurrently. Consequently, rapid-feedback-based quality assurance mechanisms such as daily builds and smoke regression tests, which help to detect and eliminate defects early during software development and maintenance, have become important. This paper addresses a major weakness of current smoke regression testing techniques, i.e., their inability to automatically (re)test graphical user interfaces (GUIs). Several contributions are made to the area of GUI smoke testing. First, the requirements for GUI smoke testing are identified and a GUI smoke test is formally defined as a specialized sequence of events. Second, a GUI smoke regression testing process called daily automated regression tester (DART) that automates GUI smoke testing is presented. Third, the interplay between several characteristics of GUI smoke test suites including their size, fault detection ability, and test oracles is empirically studied. The results show that: 1) the entire smoke testing process is feasible in terms of execution time, storage space, and manual effort, 2) smoke tests cannot cover certain parts of the application code, 3) having comprehensive test oracles may make up for not having long smoke test cases, and 4) using certain oracles can make up for not having large smoke test suites.     

Evolution of the UniTESK test development technology

The basic principles of UniTESK technology designed for constructing tests on the basis of formal models are presented. Experience in using UniTESK in large test development projects for software and hardware systems, including telecommunication protocols, basic and standard interfaces of operating systems, and microprocessor units, is summarized. The paper provides an overview of possible directions for the development of the technology with a view to providing higher scalability of test suites.     

基于漏洞库的fuzzing测试技术

鉴于主流fuzzing(模糊)测试技术生成的测试用例随机性和针对性无法兼顾的问题,提出一种基于漏洞库的fuzzing测试技术.根据漏洞产生的原因和重现的方法对漏洞库进行整理分类,构造出测试用例集用于fuzzing测试.这样生成的测试用例集直接与漏洞相关,具有更强的针对性;同时扩展的测试用例集可以用于检测未知的同类漏洞,覆盖面更广,因此保证了一定的随机性.以ftp服务器测试为例,选取了W indow s平台下的4款ftp服务器进行测试,共发现了3款软件的6个新漏洞,提交Se-curityFocus并通过,其中5个漏洞得到了国际权威漏洞数据库CVE的认可.     

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction     

通信协议的一致性测试

一致性测试在通信协议工程中是非常重要的。本文给出有关一致性测试的一般概览。一致性测试的基本原理,抽象测试方法的比较和选择,测试集和TTCN,一致性评估过程以及结果的分析将在文章中讨论。     

Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication failure, and timing. Furthermore, they do not provide the engineer with a means to evaluate the relative effectiveness of different criteria, nor the relative effectiveness of adequate test suites satisfying a given criterion. This paper makes three contributions to the development and use of test adequacy criteria for distributed systems: (1) a testing method based on discrete-event simulations; (2) a fault-based analysis technique for evaluating test suites and adequacy criteria; and (3) a series of case studies that validate the method and technique. The testing method uses a discrete-event simulation as an operational specification of a system, in which the behavioral effects of distribution are explicitly represented. Adequacy criteria and test cases are then defined in terms of this simulation-based specification. The fault-based analysis involves mutation of the simulation-based specification to provide a foil against which test suites and the criteria that formed them can be evaluated. Three distributed systems were used to validate the method and technique, including DNS, the Domain Name System.     

基于程序路径分析的有效蜕变测试

蜕变测试对于预期输出难以构造的程序是实用和高效的.作者在系统研究已有蜕变测试方法和路径分析技术的基础上,首先针对使用二元蜕变关系的测试提出了一组蜕变测试准则,以在多个不同的层次上定义蜕变测试用例集的充分性;然后给出了3种能够生成相应测试用例集的算法;最后通过变异分析的方法证实这些算法的有效性.实验结果表明,蜕变关系和测试准则的选取直接影响到测试的效果,另外,使用蜕变关系全路径覆盖可满足性算法(APCEMsT)可以快速准确地发现待测程序中的错误,而生成的测试用例的数量却比传统技术要少.     

On the use of a similarity function for test case selection in the context of model‐based testing

Test case selection in model‐based testing is discussed focusing on the use of a similarity function. Automatically generated test suites usually have redundant test cases. The reason is that test generation algorithms are usually based on structural coverage criteria that are applied exhaustively. These criteria may not be helpful to detect redundant test cases as well as the suites are usually impractical due to the huge number of test cases that can be generated. Both problems are addressed by applying a similarity function. The idea is to keep in the suite the less similar test cases according to a goal that is defined in terms of the intended size of the test suite. The strategy presented is compared with random selection by considering transition‐based and fault‐based coverage. The results show that, in most of the cases, similarity‐based selection can be more effective than random selection when applied to automatically generated test suites. Copyright © 2009 John Wiley & Sons, Ltd.     

Prioritizing test cases for regression testing

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites     

The density algorithm for pairwise interaction testing

There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one‐test‐at‐a‐time methods (such as AETG and TCG) have proven to be a reasonable compromise between the needs for small test suites, fast test‐suite generation, and flexibility to accommodate a variety of testing scenarios. However, such methods suffer from the lack of a worst‐case logarithmic guarantee on test suite size, while methods that provide such a guarantee at present are less efficient or flexible, or do not produce test suites that are competitive in size for practical testing scenarios. In this paper, a new algorithm establishes that efficient, greedy, one‐test‐at‐a‐time methods can indeed produce a logarithmic worst‐case guarantee on the test suite size. In addition, this can be done while still producing test suites that are of competitive size, and in a time that is comparable to the published methods. It is deterministic, guaranteeing reproducibility. It generates only one candidate test at a time, permits users to ‘seed’ the test suite with specified tests, and allows users to specify constraints of combinations that should be avoided. Further, statistical analysis examines the impact of five variables used to tune this density algorithm for execution time and test suite size: weighting of density for factors, scaling of density, tie‐breaking, use of multiple candidates, and multiple repetitions using randomization. Copyright © 2007 John Wiley & Sons, Ltd.     

Two case studies in grammar-based test generation

Grammar-based test generation (GBTG) has seen extensive study and practical use since the 1970s. GBTG was introduced to generate source code for testing compilers from context-free grammars specifying language syntax. More recently, GBTG has been applied to many other testing problems, including the generation of eXtensible Markup Language (XML) documents and the generation of packets for testing communications protocols. Recent research has shown how to integrate covering-array techniques such as pairwise testing into GBTG tools. While the integration offers considerable power to the tester, there are few practical demonstrations in the literature. We present two case studies showing how to use grammars and covering arrays for automated software testing. The first case study exposes HTML injection vulnerabilities in an RSS feed parser. The second case study determines the effectiveness of network firewalls when faced with TCP flag attacks. The case studies illustrate the use of covering arrays in a GBTG context, the use of visualization to understand large test logs, and the issues and tradeoffs in the design of fully automated GBTG test suites.     

The automatic generation of load test suites and the assessment ofthe resulting software

Three automatic test case generation algorithms intended to test the resource allocation mechanisms of telecommunications software systems are introduced. Although these techniques were specifically designed for testing telecommunications software, they can be used to generate test cases for any software system that is modelable by a Markov chain provided operational profile data can either be collected or estimated. These algorithms have been used successfully to perform load testing for several real industrial software systems. Experience generating test suites for five such systems is presented. Early experience with the algorithms indicate that they are highly effective at detecting subtle faults that would have been likely to be missed if load testing had been done in the more traditional way, using hand-crafted test cases. A domain-based reliability measure is applied to systems after the load testing algorithms have been used to generate test data. Data are presented for the same five industrial telecommunications systems in order to track the reliability as a function of the degree of system degradation experienced     

A two-folded impact analysis of schema changes on database applications

Database applications are becoming increasingly popular, mainly due to the advanced data management facilities that the underlying database management system offers compared against traditional legacy software applications. The interaction, however, of such applications with the database system introduces a number of issues, among which, this paper addresses the impact analysis of the changes performed at the database schema level. Our motivation is to provide the software engineers of database applications with automated methods that facilitate major maintenance tasks, such as source code corrections and regression testing, which should be triggered by the occurrence of such changes. The presented impact analysis is thus two-folded： the impact is analysed in terms of both the affected source code statements and the affected test suites concerning the testing of these applications. To achieve the former objective, a program slicing technique is employed, which is based on an extended version of the program dependency graph. The latter objective requires the analysis of test suites generated for database applications, which is accomplished by employing testing techniques tailored for this type of applications. Utilising both the slicing and the testing techniques enhances program comprehension of database applications, while also supporting the development of a number of practical metrics regarding their maintainability against schema changes. To evaluate the feasibility and effectiveness of the presented techniques and metrics, a software tool, called DATA, has been implemented. The experimental results from its usage on the TPC-C case study are reported and analysed.     

On the order of test goals in specification-based testing

Model-based testing techniques often select test cases according to test goals such as coverage criteria or mutation adequacy. Complex criteria and large models lead to large test suites, and a test case created for one coverage item usually covers several other items as well. This can be problematic if testing is expensive and resources are limited. Therefore, test case generation can be optimized in order to avoid unnecessary test cases and minimize the test generation and execution costs. Because of this optimization the order in which test goals are selected is expected to have an impact on both the performance of the test case generation and the size of resulting test suites, although finding the optimal order is not feasible in general. In this paper we report on experiments to determine the effects of the order in which test goals are selected on performance and the size of resulting test suites, and evaluate different heuristics to select test goals such that the time required to generate test suites as well as their size are minimized. The test case generation approach used for experimentation uses model checkers, and experimentation shows that good results can be achieved with any random ordering, but some improvement is still possible with simple heuristics.     

基于Java的邮件服务器的设计与实现

文章在比较了二层与三层结构模型和多层结构模型之间的异同和优缺点的基础上,提出了一种基于Java的邮件服务器设计方案,并给出了该方案的设计思想和具体实现,剖析了EJB在该邮件服务器(MailServer)中的应用,EJB提供了Java服务器端服务框架的规范,使得设计者可以专注于应用业务逻辑的设计,而不必考虑底层框架的实现,增强了系统的可移植性和可扩展性,并且更易于维护和升级,同时简单分析了邮件服务器的SMTP与POP3协议的设计实现。