Coverage-driven Automated Compiler Test Suite Generation

The paper presents a novel approach to automated compiler test suite generation based on the source level specification. Several coverage criteria are introduced. The application of the proposed methodology to testing the realistic programming language is discussed.     

基于规格说明的若干逻辑覆盖测试准则

基于规格说明的测试可以在不需要了解软件程序代码的情况下对软件进行功能测试.判定是形式规格说明中用于描述前、后置条件的主要形式.分析了基于规格说明的逻辑覆盖测试准则,针对已有的决定性逻辑覆盖测试准则的不足,提出了掩盖性逻辑覆盖测试准则,并对其进行了详细分析.提出了掩盖性逻辑覆盖测试准则的一个可行的测试生成算法.根据该准则生成的测试用例能够发现条件的掩盖性带来的错误.然后,从判定的结构入手,分析了条件之间的约束关系、复杂判定的分解与合成、判定之间的关系.这些分别能够阐明逻辑覆盖中条件间的耦合性问题、同一个条件在判定中的多次出现问题以及判定在程序中的位置问题.继而提出了全真判定覆盖、全假判定覆盖、完全子判定覆盖、唯一条件真覆盖以及唯一条件假覆盖等测试准则.满足这些测试准则的测试用例集能检测出不同类型的错误.最后,给出了这些测试准则之间的包含关系图,并建议了不同测试准则适用的应用场景.     

基于规格说明的若干逻辑覆盖测试准则

基于规格说明的测试可以在不需要了解软件程序代码的情况下对软件进行功能测试.判定是形式规格说明中用于描述前、后置条件的主要形式.分析了基于规格说明的逻辑覆盖测试准则,针对已有的决定性逻辑覆盖测试准则的不足,提出了掩盖性逻辑覆盖测试准则,并对其进行了详细分析.提出了掩盖性逻辑覆盖测试准则的一个可行的测试生成算法.根据该准则生成的测试用例能够发现条件的掩盖性带来的错误.然后,从判定的结构入手,分析了条件之间的约束关系、复杂判定的分解与合成、判定之间的关系.这些分别能够阐明逻辑覆盖中条件间的耦合性问题、同一个条件在判定中的多次出现问题以及判定在程序中的位置问题.继而提出了全真判定覆盖、全假判定覆盖、完全子判定覆盖、唯一条件真覆盖以及唯一条件假覆盖等测试准则.满足这些测试准则的测试用例集能检测出不同类型的错误.最后,给出了这些测试准则之间的包含关系图,并建议了不同测试准则适用的应用场景.     

Test generation from P systems using model checking

This paper presents some testing approaches based on model checking and using different testing criteria. First, test sets are built from different Kripke structure representations. Second, various rule coverage criteria for transitional, non-deterministic, cell-like P systems, are considered in order to generate adequate test sets. Rule based coverage criteria (simple rule coverage, context-dependent rule coverage and variants) are defined and, for each criterion, a set of LTL (Linear Temporal Logic) formulas is provided. A codification of a P system as a Kripke structure and the sets of LTL properties are used in test generation: for each criterion, test cases are obtained from the counterexamples of the associated LTL formulas, which are automatically generated from the Kripke structure codification of the P system. The method is illustrated with an implementation using a specific model checker, NuSMV.     

Interface Mutation: an approach for integration testing

The need for test adequacy criteria is widely recognized. Several criteria have been proposed for the assessment of adequacy of tests at the unit level. However, there remains a lack of criteria for the assessment of the adequacy of tests generated during integration testing. We present a mutation based interprocedural criterion, named Interface Mutation (IM), suitable for use during integration testing. A case study to evaluate the proposed criterion is reported. In the study, the UNIX sort utility was seeded with errors and Interface Mutation evaluated by measuring the cost of its application and its error revealing effectiveness. Alternative IM criteria using different sets of Interface Mutation operators were also evaluated. While comparing the error revealing effectiveness of these Interface Mutation-based test sets with same size randomly generated test sets, we observed that in most cases Interface Mutation based test sets are superior. The results suggest that Interface Mutation offers a viable test adequacy criteria for use at the integration level     

A unified framework for evaluating test criteria in model-checking-assisted test case generation

Testing is often cited as one of the most costly operations in testing dependable systems (Heimdahl et al. 2001). A particular challenging task in testing is test-case generation. To improve the efficiency of test-case generation and reduce its cost, recently automated formal verification techniques such as model checking are extended to automate test-case generation processes. In model-checking-assisted test-case generation, a test criterion is formulated as temporal logical formulae, which are used by a model checker to generate test cases satisfying the test criterion. Traditional test criteria such as branch coverage criterion and newer temporal-logic-inspired criteria such as property coverage criteria (Tan et al. 2004) are used with model-checking-assisted test generation. Two key questions in model-checking-assisted test generation are how efficiently a model checker may generate test suites for these criteria and how effective these test suites are. To answer these questions, we developed a unified framework for evaluating (1) the effectiveness of the test criteria used with model-checking-assisted test-case generation and (2) the efficiency of test-case generation for these criteria. The benefits of this work are three-fold: first, the computational study carried out in this work provides some measurements of the effectiveness and efficiency of various test criteria used with model-checking-assisted test case generation. These performance measurements are important factors to consider when a practitioner selects appropriate test criteria for an application of model-checking-assisted test generation. Second, we propose a unified test generation framework based on generalized Büchi automata. The framework uses the same model checker, in this case, SPIN model checker (Holzmann 1997), to generate test cases for different criteria and compare them on a consistent basis. Last but not least, we describe in great details the methodology and automated test generation environment that we developed on the basis of our unified framework. Such details would be of interest to researchers and practitioners who want to use and extend this unified framework and its accompanying tools.     

Structural testing criteria for message‐passing parallel programs

Parallel programs present some features such as concurrency, communication and synchronization that make the test a challenging activity. Because of these characteristics, the direct application of traditional testing is not always possible and adequate testing criteria and tools are necessary. In this paper we investigate the challenges of validating message‐passing parallel programs and present a set of specific testing criteria. We introduce a family of structural testing criteria based on a test model. The model captures control and data flow of the message‐passing programs, by considering their sequential and parallel aspects. The criteria provide a coverage measure that can be used for evaluating the progress of the testing activity and also provide guidelines for the generation of test data. We also describe a tool, called ValiPar, which supports the application of the proposed testing criteria. Currently, ValiPar is configured for parallel virtual machine (PVM) and message‐passing interface (MPI). Results of the application of the proposed criteria to MPI programs are also presented and analyzed. Copyright © 2008 John Wiley & Sons, Ltd.     

面向对象语言编译器的自动测试平台

提出了一种新的面向对象语言编译器自动测试的方法和过程,并在此基础上设计开发了面向对象编译器自动测试工具(O_OCTT)。测试用例的自动生成是编译器自动测试工具的核心,介绍了C 编译器测试用例自动生成的手段和方法,阐述了。O_OCTT设计与实现过程中的关键技术。     

Efficient Techniques for Automatic Verification-Oriented Test Set Optimization

Most Systems-on-a-Chips include a custom microprocessor core, and time and resource constraints make the design of such devices a challenging task. This paper presents a simulation-based methodology for the automatic completion and refinement of verification test sets. The approach extends the μGP, an evolutionary test program generator, with the possibility to enhance existing test sets. Already devised test programs are not merely included in the new set, but assimilated and used as a starting point for a new test-program cultivation task. Reusing existing material cuts down the time required to generate a verification test set during the microprocessor design. Experimental results are reported on a small pipelined microprocessor, and show the effectiveness of the approach. Additionally, the use of the proposed methodology enabled to experimentally analyze the relationship of the different code coverage metrics used in the test program generation     

Multiple detection test generation with diversified fault partitioning paths

The dependability of current and future nanoscale technologies highly depends on the ability of the testing process to detect emerging defects that cannot be modeled traditionally. Generating test sets that detect each fault more than one times has been shown to increase the effectiveness of a test set to detect non-modeled faults, either static or dynamic. Traditional n-detect test sets guarantee to detect a modeled fault with minimum n different tests. Recent techniques examine how to quantify and maximize the difference between the various tests for a fault. The proposed methodology introduces a new systematic test generation algorithm for multiple-detect (including n-detect) test sets that increases the diversity of the fault propagation paths excited by the various tests per fault. A novel algorithm tries to identify different propagating paths (if such a path exists) for each one of the multiple (n) detections of the same fault. The proposed method can be applied to any linear, to the circuit size, static or dynamic fault model for multiple fault detections, such as the stuck-at or transition delay fault models, and avoids any path or path segment enumeration. Experimental results show the effectiveness of the approach in increasing the number of fault propagating paths when compared to traditional n-detect test sets.     

活动图模型驱动的Web应用程序测试方法

提出了一种活动图(AD)模型驱动的Web应用程序测试方法,从活动图中生成满足往返路径覆盖准则的测试序列,基于测试输入语法生成驱动测试序列执行的测试数据,将测试数据整合到测试序列中生成测试用例,运行测试用例进行测试并生成测试结果的报告。实现了一个原型工具以支持活动图模型驱动的Web应用程序测试方法,设计并完成了一个实验,验证了该方法的可行性与有效性。     

组合测试用例生成技术

组合测试是一种科学有效的软件测试方法,该方法旨在使用较少的测试用例有效地检测软件系统中各个因素以及它们之间的相互作用对系统产生的影响,实践证明其具有较高的错误检测能力。当前组合测试研究的热点之一是组合测试用例生成问题,即如何针对具体待测软件,在满足给定组合覆盖要求的前提下,生成规模尽可能小的测试用例集,以便在保证错误检测能力的前提下尽可能降低测试成本。从N维组合覆盖和变力度组合覆盖等两类不同的组合覆盖标准出发,简要介绍了迄今为止人们在组合测试用例生成领域所取得的研究成果,对现有的组合用例生成方法进行了分类和总结。此外,还对优先级、组合约束、错误定位等条件和应用场景下的组合测试用例生成技术进行了介绍。最后,分析了现有成果中存在的问题,并对该领域未来的研究方向进行了分析和讨论。     

面向安全需求的测试用例自动生成技术研究

传统的测试用例生成技术通常都是面向系统功能性需求,并不适用于安全苛求软件系统的安全性测试。面向安全需求,通过定义安全因子,建立带有安全因子的UML顺序图,采用最小安全因子路径,提出了基于最小安全因子路径完全覆盖准则的测试用例自动生成算法,并将其成功应用到高速铁路列车运行控制系统的仿真测试中。     

一种基于软件体系结构配置的测试用例生成算法研究

从软件体系结构配置生成测试用例是软件测试领域中的一个重要分支。本文首先提出了一组基于软件体系结构配置的测试标准及其计算规则。基于此，实现了基于软件体系结构配置的测试用例生成算法并分析了其时间复杂性。对人机结口（Marr-Machine Interface，MMI）实验表明，所提出的算法能够根据测试标准生成测试用例。     

Hybrid probabilistic-harmony search algorithm methodology in generation scheduling problem

This paper attempts to propose a fair solution in generation scheduling problem in the presence of inherent uncertainties in short-term power system operation. The proposed methodology incorporates probabilistic methodology in the uncertainties representation section, while harmony search algorithm is adopted as a fast and reliable soft computing algorithm to solve the proposed nonlinear, non-convex, large-scaled and combinatorial problem. As an indispensable step towards a more economical power system operation, the optimal generation scheduling strategy in the presence of mixed hydro-thermal generation mix, deemed to be the most techno-economically efficient scheme, comes to the play and is profoundly taken under concentration in this study. This paper devises a comprehensive hybrid optimisation approach by which all the crucial aspects of great influence in the generation scheduling process can be accounted for. Two-point estimation method is also adopted probabilistically approaching the involved uncertain criteria. In the light of the proposed methodology being implemented on an adopted test system, the anticipated efficiency of the proposed method is well verified.     

Application of a Methodology for the Development and Validation of Reliable Process Control Software

This paper discusses the necessity of a good methodology for the development of reliable software, especialy with respect to the final software validation and testing activities. A formal specification development and validation methodology is proposed. This methodology has been applied to the development and validation of a pilot software, incorporating typical features of critical software for nuclear power plant safety protection. The main features of the approach indude the use of a formal specification language and the independent development of two sets of specifications. Analyses on the specifications consists of three-parts: validation against the functional requirements consistency and integrity of the specifications, and dual specification comparison based on a high-level symbolic execution technique. Dual design, implementation, and testing are performed. Automated tools to facilitate the validation and testing activities are developed to support the methodology. These includes the symbolic executor and test data generator/dual program monitor system. The experiences of applying the methodology to the pilot software are discussed, and the impact on the quality of the software is assessed.     

A cumulative belief degree approach for group decision‐making problems with heterogeneous information

In some complex group decision‐making (GDM) problems, the information needing to be processed may be heterogeneous. This may involve consideration of objective and subjective criteria by experts who have their own particular set of criteria, their own preference format for assessing alternatives under these criteria, and who may themselves be assigned differing importance weights as experts. This paper presents a cumulative belief degree approach to cope with heterogeneous information in multiple attribute GDM problems. The proposed approach focuses to aggregate subjective expert assessments and objective criteria that are presented in various representation formats and scales. The methodology employs transformation formulae for several preference representation scales to belief structure, including 2‐tuple representation, classical fuzzy sets, hesitant fuzzy sets, and intuitionistic fuzzy sets. Aggregation formulae are proposed to combine expert criteria evaluations and find a collective preference. A consensus degree is calculated for measuring the agreement between the experts. An illustrative example is presented to clarify the steps of the methodology, and validity of the approach is assured through comparative analysis with the existing methods.     

A methodology of testing high-level Petri nets

Petri nets have been extensively used in the modelling and analysis of concurrent and distributed systems. The verification and validation of Petri nets are of particular importance in the development of concurrent and distributed systems. As a complement to formal analysis techniques, testing has been proven to be effective in detecting system errors and is easy to apply. An open problem is how to test Petri nets systematically, effectively and efficiently. An approach to solve this problem is to develop test criteria so that test adequacy can be measured objectively and test cases can be generated efficiently, even automatically. In this paper, we present a methodology of testing high-level Petri nets based on our general theory of testing concurrent software systems. Four types of testing strategies are investigated, which include state-oriented testing, transition-oriented testing, flow-oriented testing and specification-oriented testing. For each strategy, a set of schemes to observe and record testing results and a set of coverage criteria to measure test adequacy are defined. The subsumption relationships and extraction relationships among the proposed testing methods are systematically investigated and formally proved.     

Generating test data from state‐based specifications

Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system‐level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated. This paper presents general criteria for generating test inputs from state‐based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults. Copyright © 2003 John Wiley & Sons, Ltd.