一种新的软件测试方法的研究

提出基于蒙特卡罗的软件可靠性测试方法。它是软件工程中的一个新的研究领域,主要探讨蒙特卡罗方法和软件工程的交叉。一方面借鉴一些传统的软件测试方法,另一方面借助于蒙特卡罗方法,提出的一套全新的测试技术。     

An experimental study of adaptive testing for software reliability assessment

Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previous work has shown that adaptive testing can be formulated and guided in theory to minimize the variance of an unbiased software reliability estimator and to achieve optimal software reliability assessment. In this paper, we present an experimental study of adaptive testing for software reliability assessment, where the adaptive testing strategy, the random testing strategy and the operational profile based testing strategy were applied to the Space program in four experiments. The experimental results demonstrate that the adaptive testing strategy can really work in practice and may noticeably outperform the other two. Therefore, the adaptive testing strategy can serve as a preferable alternative to the random testing strategy and the operational profile based testing strategy if high confidence in the reliability estimates is required or the real-world operational profile of the software under test cannot be accurately identified.     

Enlargement of vulnerable web applications for testing

There are two main kinds of vulnerable web applications, usual applications developed with a specific aim and applications which are vulnerable by design. On one hand, the usual applications are those that are used everywhere and on a daily basis, and where vulnerabilities are detected, and often mended, such as online banking systems, newspaper sites, or any other Web site. On the other hand, vulnerable by design web applications are developed for proper evaluation of web vulnerability scanners and for training in detecting web vulnerabilities. The main drawback of vulnerable by design web applications is that they used to include just a short set of well-known types of vulnerabilities, usually from famous classifications like the OWASP Top Ten. They do not include most of the types of web vulnerabilities. In this paper, an analysis and assessment of vulnerable web applications is conducted in order to select the applications that include the larger set of types of vulnerabilities. Then those applications are enlarged with more types of web vulnerabilities that vulnerable web applications do not include. Lastly, the new vulnerable web applications have been analyzed to check whether web vulnerability scanners are able to detect the new added vulnerabilities, those vulnerabilities that vulnerable by design web applications do not include. The results show that the tools are not very successful in detecting those vulnerabilities, less than well-known vulnerabilities.     

Model-based testing of global properties on large-scale distributed systems

ContextLarge-scale distributed systems are becoming commonplace with the large popularity of peer-to-peer and cloud computing. The increasing importance of these systems contrasts with the lack of integrated solutions to build trustworthy software. A key concern of any large-scale distributed system is the validation of global properties, which cannot be evaluated on a single node. Thus, it is necessary to gather data from distributed nodes and to aggregate these data into a global view. This turns out to be very challenging because of the system’s dynamism that imposes very frequent changes in local values that affect global properties. This implies that the global view has to be frequently updated to ensure an accurate validation of global properties.ObjectiveIn this paper, we present a model-based approach to define a dynamic oracle for checking global properties. Our objective is to abstract relevant aspects of such systems into models. These models are updated at runtime, by monitoring the corresponding distributed system.MethodWe conduce real-scale experimental validation to evaluate the ability of our approach to check global properties. In this validation, we apply our approach to test two open-source implementations of distributed hash tables. The experiments are deployed on two clusters of 32 nodes.ResultsThe experiments reveal an important defect on one implementation and show clear performance differences between the two implementations. The defect would not be detected without a global view of the system.ConclusionTesting global properties on distributed software consists of gathering data from different nodes and building a global view of the system, where properties are validated. This process requires a distributed test architecture and tools for representing and validating global properties. Model-based techniques are an expressive mean for building oracles that validate global properties on distributed systems.     

Software testing: A selected annotated bibliography

A collection of 69 references drawn from books, papers and conference proceedings on the subject of software testing is presented. Each reference is accompanied by a paragraph describing its contents. The aim when selecting the references was to minimize the number and maximize the coverage of the subject. For this reason, the bibliography is primarily intended for the researcher or practitioner who is relatively new to the subject. The bibliography is organized according to the following sections: general introductions; background; testing methods; experimental studies; and tools.     

A systematic mapping study of software product lines testing

Context

In software development, Testing is an important mechanism both to identify defects and assure that completed products work as specified. This is a common practice in single-system development, and continues to hold in Software Product Lines (SPL). Even though extensive research has been done in the SPL Testing field, it is necessary to assess the current state of research and practice, in order to provide practitioners with evidence that enable fostering its further development.

Objective

This paper focuses on Testing in SPL and has the following goals: investigate state-of-the-art testing practices, synthesize available evidence, and identify gaps between required techniques and existing approaches, available in the literature.

Method

A systematic mapping study was conducted with a set of nine research questions, in which 120 studies, dated from 1993 to 2009, were evaluated.

Results

Although several aspects regarding testing have been covered by single-system development approaches, many cannot be directly applied in the SPL context due to specific issues. In addition, particular aspects regarding SPL are not covered by the existing SPL approaches, and when the aspects are covered, the literature just gives brief overviews. This scenario indicates that additional investigation, empirical and practical, should be performed.

Conclusion

The results can help to understand the needs in SPL Testing, by identifying points that still require additional investigation, since important aspects regarding particular points of software product lines have not been addressed yet.     

Distributing test cases more evenly in adaptive random testing

Adaptive random testing (ART) has recently been proposed to enhance the failure-detection capability of random testing. In ART, test cases are not only randomly generated, but also evenly spread over the input domain. Various ART algorithms have been developed to evenly spread test cases in different ways. Previous studies have shown that some ART algorithms prefer to select test cases from the edge part of the input domain rather than from the centre part, that is, inputs do not have equal chance to be selected as test cases. Since we do not know where the failure-causing inputs are prior to testing, it is not desirable for inputs to have different chances of being selected as test cases. Therefore, in this paper, we investigate how to enhance some ART algorithms by offsetting the edge preference, and propose a new family of ART algorithms. A series of simulations have been conducted and it is shown that these new algorithms not only select test cases more evenly, but also have better failure detection capabilities.     

Software testing processes as a linear dynamic system

Software testing is essential for software reliability improvement and assurance, and the processes of software testing are intrinsically dynamic. However they are seldom investigated in a mathematically rigorous manner. In this paper a theoretical study is presented to examine the dynamic behavior of software testing. More specifically, a set of simplifying assumptions is adopted to formulate and quantify the software testing processes. The mathematical formulae for the expected number of observed software failures are rigorously derived, the bounds and trends of the expected number of observed software failures are analyzed, and the variance of the number of observed software failures is examined. On the other hand, it is demonstrated that under the simplifying assumptions, the software testing processes can be treated as a linear dynamic system. This suggests that the software testing processes could be classified as linear or non-linear, and there be intrinsic link between software testing and system dynamics.     

A relation-based method combining functional and structural testing for test case generation

Specification-based (or functional) testing enables us to detect errors in the implementation of functions defined in specifications, but since specifications are often incomplete in practice for some reasons (e.g., lack of ideas, no time to write), it is unlikely to be sufficient for testing all parts of corresponding programs. On the other hand, implementation-based (or structural) testing focuses on the examination of program structures, which allows us to test all parts of the programs, but may not be effective to show whether the programs properly implement the corresponding specifications. To perform a comprehensive testing of a program in practice, it is important to adopt both specification-based and implementation-based testing. In this paper we describe a relation-based test method that combines the specification-based and the implementation-based testing approaches. We establish a set of relations for test case generation, illustrate how the method is used with an example, and investigate the effectiveness and weakness of the method through an experiment on testing a software tool system.     

Evolutionary functional testing

The development and testing of software-based systems is an essential activity for the automotive industry. The 50–70 software-based systems with different complexities and developed by various suppliers are installed in today's premium vehicles, communicating with each other via different bus systems. The integration and testing of systems of this complexity is a very challenging task. The aim of testing is to detect faults in the systems under test and to convey confidence in the correct functioning of the systems if no faults are found during comprehensive testing. Faults not found in the different testing phases could have significant consequences that range from customer dissatisfaction to damage of physical property or, in safety-relevant areas, even to the endangering of human lives. Therefore, the thorough testing of developed systems is essential. Evolutionary testing tries to improve the effectiveness and efficiency of the testing process by transforming testing objectives into search problems, and applying evolutionary computation in order to solve them. The most important class of testing methods is functional testing. However, functional testing is difficult to automate by evolutionary testing. This work will describe how evolutionary testing could be applied to automate functional testing in general and the testing of complex automotive systems in particular. It presents two case studies and shows how evolutionary testing is effective at finding faults in the functional behaviour of these systems. In addition, a quantitative comparison with manual and random test case selection is done for one application.     

Modeling presentation layers of web applications for testing

Web software applications have become complex, sophisticated programs that are based on novel computing technologies. Their most essential characteristic is that they represent a different kind of software deployment—most of the software is never delivered to customers’ computers, but remains on servers, allowing customers to run the software across the web. Although powerful, this deployment model brings new challenges to developers and testers. Checking static HTML links is no longer sufficient; web applications must be evaluated as complex software products. This paper focuses on three aspects of web applications that are unique to this type of deployment: (1) an extremely loose form of coupling that features distributed integration, (2) the ability that users have to directly change the potential flow of execution, and (3) the dynamic creation of HTML forms. Taken together, these aspects allow the potential control flow to vary with each execution, thus the possible control flows cannot be determined statically, prohibiting several standard analysis techniques that are fundamental to many software engineering activities. This paper presents a new way to model web applications, based on software couplings that are new to web applications, dynamic flow of control, distributed integration, and partial dynamic web application development. This model is based on the notion of atomic sections, which allow analysis tools to build the analog of a control flow graph for web applications. The atomic section model has numerous applications in web applications; this paper applies the model to the problem of testing web applications.     

An adaptive and trustworthy software testing framework on the grid

Grid computing, which is characterized by large-scale sharing and collaboration of dynamic distributed resources has quickly become a mainstream technology in distributed computing and is changing the traditional way of software development. In this article, we present a grid-based software testing framework for unit and integration test, which takes advantage of the large-scale and cost-efficient computational grid resources to establish a testbed for supporting automated software test in complex software applications. Within this software testing framework, a dynamic bag-of-tasks model using swarm intelligence is developed to adaptively schedule unit test cases. Various high-confidence computing mechanisms, such as redundancy, intermediate value checks, verification code injection, and consistency checks are employed to verify the correctness of each test case execution on the grid. Grid workflow is used to coordinate various test units for integration test. Overall, we expect that the grid-based software testing framework can provide efficient and trustworthy services to significantly accelerate the testing process with large-scale software testing.

基于Java的Web应用全球化自动测试设计

随着Internet的广泛应用,基于Web页面的应用已成为当前计算机应用软件中最常用的形式。每一个Web软件产品都会有很多Web页面,尤其是支持多种语言的应用,由于如今对Web应用的测试绝大部分还是手工完成,进而带来了测试难度和工作量问题。描述了如何用Java和J2SE1.4来部分实现Web页面全球化的自动测试架构。     

Mirror adaptive random testing

Recently, adaptive random testing (ART) has been introduced to improve the fault-detection effectiveness of random testing for non-point types of failure patterns. However, ART requires additional computations to ensure an even spread of test cases, which may render ART less cost-effective than random testing. This paper presents a new technique, namely mirror ART, to reduce these computations. It is an integration of the technique of mirroring and ART. Our simulation results clearly show that mirror ART does improve the cost-effectiveness of ART.