GA-based multiple paths test data generator

Developers have learned over time that software testing costs a considerable amount of a software project budget. Hence, software quality managers have been looking for solutions to reduce testing costs and time. Considering path coverage as the test adequacy criterion, we propose using genetic algorithms (GA) for automating the generation of test data for white-box testing. There are evidences that GA has been already successful in generating test data. However, existing GA-based test data generators suffer from some problems. This paper presents our approach to overcome one of these problems; that is the inefficiency in covering multiple target paths. We have designed a GA-based test data generator that is, in one run, able to synthesize multiple test data to cover multiple target paths. Moreover, we have implemented a set of variations of the generator. Experimental results show that our test data generator is more efficient and more effective than others.     

A study of user acceptance tests

The user acceptance test (UAT) is the final stage of testing in application software development. When testing results meet the acceptance criteria, the software system can be released for operational use.This paper first compares the different testing phases of software development (i.e. unit test, integration test, system test and UAT) in terms of important testing elements so as to highlight the uniqueness of UAT relative to the other test phases. Then, we describe several approaches for acceptance test, including the behaviour-based approach, the black-box strategy and a new UAT strategy called operation-based testing. The new strategy uses the operational profile for testing purposes, includes a well-defined acceptance criteria, and satisfies the test requirements of ISO9001 standards.     

Test criteria and coverage measures for software integration testing

The aim of this paper is to introduce a systematic approach to integration testing of software systems. Various test data selection criteria for integration testing are presented, coverage measures are introduced, and interconnection between them are discussed. The main principle is to transfer and adapt test criteria and coverage measures which are useful for unit testing to the level of integration testing. Test criteria help the tester to organise the test process. They should be chosen in accordance with the available test effort. Test coverage measures are defined as a ratio between the test cases required for satisfying the criteria and those of these which have been executed. The measures are used to obtain information about the completeness of integration tests. The approach is described for data flow and control flow oriented criteria and measures. The intention is to enable the tester to specify integration tests in advance in terms of effort, and to evaluate the results in terms of test completeness.     

Deriving a Fault Architecture to Guide Testing

Defect analysis of software components can be used to guide testing, with the goal of focusing on parts of the software that were fault-prone in earlier releases or earlier life cycle phases, such as development. We replicate a study that adapted a reverse architecting technique using defect reports to derive fault architectures. A fault architecture determines and visualizes components that are fault-prone in their relationships with other components, as well as those that are locally fault-prone. Our case study uses defect data from three releases of a large medical record system to identify relationships among system components, based on whether they are involved in the same defect report.We investigate measures that assess the fault-proneness of components and component relationships. Component relationships are used to derive a fault architecture. The resulting fault architecture indicates what the most fault-prone relationships are in a release. We also apply the technique in a new way. Not only do we derive fault architectures for each release, we derive fault architectures for the development, system test and post release phases within each release. Comparing across releases, makes it possible to see whether some components are repeatedly in fault-prone relationships. Comparing across phases, makes it possible to see whether development fault architectures can be used to identify those parts of the software that need to be tested more. We validate our predictions using system test data from the same release. We also use the development and system test fault architectures to identify fault-prone components after release, and validate our predictions using post release data.     

嵌入式汇编语言软件单元测试方法

单元测试是在整个软件开发过程中的一个重要环节,单元是软件的构成基础,因此单元的质量是整个软件质量的保证。嵌入式软件由于与CPu密切相关,多采用汇编语言编程,使得其测试一直是个难点,特别是单元测试。针对嵌入式软件的特点,本文根据软件单元测试的理论,介绍一种使用旋极AsmTester对嵌入式汇编语言编程的软件进行单元测试的方法。     

Robustness testing for software components

Component-based development allows one to build software from existing components and promises to improve software reuse and reduce costs. For critical applications, the user of a component must ensure that it fits the requirements of the application. To achieve this, testing is a well-suited means when the source code of the components is not available. Robustness testing is a testing methodology to detect the vulnerabilities of a component under unexpected inputs or in a stressful environment. As components may fail differently in different states, we use a state machine based approach to robustness testing. First, a set of paths is generated to cover transitions of the state machine, and it is used by the test cases to bring the component into a specific control state. Second, method calls with invalid inputs are fed to the component in different states to test the robustness. By traversing the paths, the test cases cover more states and transitions compared to stateless API testing. We apply our approach to several components, including open source software, and compare our results with existing approaches.     

Unit Test Data Generation for C Using Rule-Directed Symbolic Execution

Unit testing is widely used in software development. One important activity in unit testing is automatic test data generation. Constraint-based test data generation is a technique for automatic generation of test data, which uses symbolic execution to generate constraints. Unit testing only tests functions instead of the whole program, where individual functions typically have preconditions imposed on their inputs. Conventional symbolic execution cannot detect these preconditions, let alone converting these preconditions into constraints. To overcome these limitations, we propose a novel unit test data generation approach using rule-directed symbolic execution for dealing with functions with missing input preconditions. Rule-directed symbolic execution uses predefined rules to detect preconditions in the individual function, and generates constraints for inputs based on preconditions. We introduce implicit constraints to represent preconditions, and unify implicit constraints and program constraints into integrated constraints. Test data generated based on integrated constraints can explore previously unreachable code and help developers find more functional faults and logical faults. We have implemented our approach in a tool called CTS-IC, and applied it to real-world projects. The experimental results show that rule-directed symbolic execution can find preconditions (implicit constraints) automatically from an individual function. Moreover, the unit test data generated by our approach achieves higher coverage than similar tools and efficiently mitigates missing input preconditions problems in unit testing for individual functions.

     

A practical model‐based statistical approach for generating functional test cases: application in the automotive industry

With the growing complexity of industrial software applications, industrials are looking for efficient and practical methods to validate the software. This paper develops a model‐based statistical testing approach that automatically generates online and offline test cases for embedded software. It discusses an integrated framework that combines solutions for three major software testing research questions: (i) how to select test inputs; (ii) how to predict the expected results of a test; and (iii) when to stop testing software. The automatic selection of test inputs is based on a stochastic test model that accounts for the main particularity of embedded software: time sensitivity. Software test practitioners may design one or more test models when they generate random, user‐oriented, or fault‐oriented test inputs. A formal framework integrating existing and appropriate specification techniques was developed for the design of automated test oracles (executable software specifications) and the formal measurement of functional coverage. The decision to stop testing software is based on both test coverage objectives and cost constraints. This approach was tested on two representative case studies from the automotive industry. The experiment was performed at unit testing level in a simulated environment on a host personal computer (automatic test execution). The two software functionalities tested had previously been unit tested and validated using the test design approach conventionally used in the industry. Applying the proposed model‐based statistical testing approach to these two case studies, we obtained significant improvements in performing functional unit testing in a real and complex industrial context: more bugs were detected earlier and in a shorter time. Copyright © 2012 John Wiley & Sons, Ltd.     

面向对象软件的测试技术的研究

面向对象软件测试是面向对象软件开发不可缺少的环节,对保证软件质量、提高软件可靠性至关重要。传统的测试技术已经很成熟,但无法对其进行有效测试。针对面向对象软件所具有的特性,提出面向对象软件的测试模型,详细讨论和分析了其测试方法和解决方案。     

Test Case Generation as an AI Planning Problem

While Artificial Intelligence techniques have been applied to a variety of software engineering applications, the area of automated software testing remains largely unexplored. Yet, test cases for certain types of systems (e.g., those with command language interfaces and transaction based systems) are similar to plans. We have exploited this similarity by constructing an automated test case generator with an AI planning system at its core. We compared the functionality and output of two systems, one based on Software Engineering techniques and the other on planning, for a real application: the StorageTek robot tape library command language. From this, we showed that AI planning is a viable technique for test case generation and that the two approaches are complementary in their capabilities.     

一种面向增量式开发的虚拟单元测试框架

单元测试的目标是检验程序模块的正确性,从而为集成测试、系统测试提供符合预期效果的部件。随着对软件质量保证的要求提高,许多单元测试的技术已相继提出。但由于软件运行环境复杂度不断增加以及软件测试技术与软件工程的结合日益紧密,对单元测试框架技术的研究有了新的需求。论文结合国际软件测试标准,讨论了面向增量式开发的虚拟单元测试框架组成,给出了一个能够提高软件弹性,保证软件质量,测试环境独立、简单的单元测试解决方案。     

浮力调节软件的单元测试技术研究

为了提高嵌入式软件的单元测试效率,同时能达到很好的测试效果,针对嵌入式软件制定单元测试充分性准则和单元测试策略,并使用测试工具对嵌入式软件进行单元测试具有实际意义;以浮力调节软件为例,通过研究基于控制流的单元测试充分性准则,结合浮力调节软件的单元测试需求,提出了针对浮力调节软件的单元测试充分性准则;同时,考虑到圈复杂度和函数节点数对函数正确实现的影响,制定了基于优先级的单元测试策略;利用自动化单元测试工具Testbed搭建了浮力调节软件动态测试环境,通过代码覆盖率分析,帮助创建测试用例以达到单元测试充分性要求,从而实现了浮力调节软件单元测试自动化;在自动化测试工具的帮助下,结合单元测试充分性准则和单元测试策略,最终实现严格而高效的单元测试。     

Comparing model-based and dynamic event-extraction based GUI testing techniques: An empirical study

Graphical user interfaces are pervasive in modern software systems, and to ensure their quality it is important to test them. Two primary classes of automated GUI testing approaches, those based on static models and those based on dynamic event-extraction, present tradeoffs in cost and effectiveness. For example, static model-based GUI testing techniques can create test cases that contain nonexecutable events, whereas dynamic event-extraction based GUI testing techniques can create larger numbers of duplicate test cases. To better understand the effects of these tradeoffs, we created a GUI testing framework that facilitates fair comparison of different GUI testing techniques, and we conducted a controlled experiment comparing representative versions of static-model based and dynamic event-extraction based testing techniques on several GUI-based Java applications. Our study reveals several cost and effectiveness tradeoffs between the techniques, with implications for research and practice.     

Iterative design and testing within the software development life cycle

The activity of testing begins during system development and spans all subsequent phases. Some system development lifecycles describe testing which is performed after the coding phase, but this may cause the software to be delivered without sufficient testing. In this paper, we present a software system development lifecycle model, called the Test design Stages Processed model (TSP model), in which we emphasize that iterative test design stages should be incorporated at each phase of the software development lifecycle. When a phase is completed, testing of the phase should also be completed at that time. Within this paper we have added unit, integration and system testing processes into BoochÕs micro–design process to generate a new designs and test model. This shows the process of iterative and incremental software development. Comparing this with our model, we explain how the TSP model can be used for developing and testing an object-oriented software system.     

Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact

Where the creation, understanding, and assessment of software testing and regression testing techniques are concerned, controlled experimentation is an indispensable research methodology. Obtaining the infrastructure necessary to support such experimentation, however, is difficult and expensive. As a result, progress in experimentation with testing techniques has been slow, and empirical data on the costs and effectiveness of techniques remains relatively scarce. To help address this problem, we have been designing and constructing infrastructure to support controlled experimentation with testing and regression testing techniques. This paper reports on the challenges faced by researchers experimenting with testing techniques, including those that inform the design of our infrastructure. The paper then describes the infrastructure that we are creating in response to these challenges, and that we are now making available to other researchers, and discusses the impact that this infrastructure has had and can be expected to have. Editor: Forrest Shull and Natalia Juristo     

Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random testing normally uses fewer test cases than random testing for detecting the first software failure. However, under many practical situations, testing should not be withheld after the detection of a failure. Thus, it is important to investigate the effectiveness with respect to the detection of multiple failures. In this paper, we compare adaptive random testing and random testing under various scenarios and examine whether adaptive random testing is still able to use fewer test cases than random testing to detect multiple software failures. Our study delivers some interesting results and highlights a number of promising research projects. Copyright © 2011 John Wiley & Sons, Ltd.     

Issues in using model checkers for test case generation

The use of model checkers for automated software testing has received some attention in the literature: It is convenient because it allows fully automated generation of test suites for many different test objectives. On the other hand, model checkers were not originally meant to be used this way but for formal verification, so using model checkers for testing is sometimes perceived as a “hack”. Indeed, several drawbacks result from the use of model checkers for test case generation. If model checkers were designed or adapted to take into account the needs that result from the application to software testing, this could lead to significant improvements with regard to test suite quality and performance. In this paper we identify the drawbacks of current model checkers when used for testing. We illustrate techniques to overcome these problems, and show how they could be integrated into the model checking process. In essence, the described techniques can be seen as a general road map to turn model checkers into general purpose testing tools.     

测试用例复用研究与实践

测试用例设计在军用软件功能测试中需要占用大量的时间和人力。有效复用已有的测试用例能够缩短测试用例设计的时间,减少测试用例设计的工作量,从而提高测试的效率。针对大型军用软件功能测试用例的复用进行了探讨,提出了一种军用软件的测试用例复用方法,并在工程中进行了实践应用。     

Selecting a Cost-Effective Test Case Prioritization Technique

Regression testing is an expensive testing process used to validate modified software and detect whether new faults have been introduced into previously tested code. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One goal of prioritization is to increase a test suite's rate of fault detection. Previous empirical studies have shown that several prioritization techniques can significantly improve rate of fault detection, but these studies have also shown that the effectiveness of these techniques varies considerably across various attributes of the program, test suites, and modifications being considered. This variation makes it difficult for a practitioner to choose an appropriate prioritization technique for a given testing scenario. To address this problem, we analyze the fault detection rates that result from applying several different prioritization techniques to several programs and modified versions. The results of our analyses provide insights into which types of prioritization techniques are and are not appropriate under specific testing scenarios, and the conditions under which they are or are not appropriate. Our analysis approach can also be used by other researchers or practitioners to determine the prioritization techniques appropriate to other workloads.     

基于UML活动图生成功能测试线索

测试自动化是提高软件测试效率的重要途径。基于UML模型的面向对象软件测试是当前研究的热点。这些研究大都是类或集成测试,对于如何自动生成较为完整、合理的系统测试用例较少。在总结研究现状的基础上,以应用举例的方式提出一种从描述系统用例问顺序依赖关系的活动图和用例实现的活动图中获取系统功能测试线索的方法。