Issues in using model checkers for test case generation

The use of model checkers for automated software testing has received some attention in the literature: It is convenient because it allows fully automated generation of test suites for many different test objectives. On the other hand, model checkers were not originally meant to be used this way but for formal verification, so using model checkers for testing is sometimes perceived as a “hack”. Indeed, several drawbacks result from the use of model checkers for test case generation. If model checkers were designed or adapted to take into account the needs that result from the application to software testing, this could lead to significant improvements with regard to test suite quality and performance. In this paper we identify the drawbacks of current model checkers when used for testing. We illustrate techniques to overcome these problems, and show how they could be integrated into the model checking process. In essence, the described techniques can be seen as a general road map to turn model checkers into general purpose testing tools.     

关于两两测试的研究

在软件开发过程中软件测试的成本是很高的。但是，在许多情况下两两测试可用于减少测试的成本，并对于许多软件系统在实践中证明是实用和有效的。它要求对于系统的每一对输入参数，这两个参数有效值的每一个组合至少要被一个测试用例覆盖。介绍了两两测试的概念以及生成测试用例的两个算法。最后，对应用两两测试时应注意的一些问题进行了讨论．两两测试的许多思想可以扩展到t路测试．     

On the effect of test-suite reduction on automatically generated model-based tests

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage—often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications. This work has been partially supported by NASA grant NAG-1-224 and NASA contract NCC-01001. We also want to thank the McKnight Foundation for their generous support over the years.

---

     

Automated test generation technique for aspectual features in AspectJ

ContextAspect-oriented programming (AOP) has been promoted as a means for handling the modularization of software systems by raising the abstraction level and reducing the scattering and tangling of crosscutting concerns. Studies from literature have shown the usefulness and application of AOP across various fields of research and domains. Despite this, research shows that AOP is currently used in a cautious way due to its natural impact on testability and maintainability.ObjectiveTo realize the benefits of AOP and to increase its adoption, aspects developed using AOP should be subjected to automated testing. Automated testing, as one of the most pressing needs of the software industry to reduce both effort and costs in assuring correctness, is a delicate issue in testing aspect-oriented programs that still requires advancement and has a way to go before maturity.MethodPrevious attempts and studies in automated test generation process for aspect-oriented programs have been very limited. This paper proposes a rigorous automated test generation technique, called RAMBUTANS, with its tool support based on guided random testing for the AspectJ programs.ResultsThe paper reports the results of a thorough empirical study of 9 AspectJ benchmark programs, including non-trivial and larger software, by means of mutation analysis to compare RAMBUTANS and the four existing automated AOP testing approaches for testing aspects in terms of fault detection effectiveness and test effort efficiency. The results of the experiment and statistical tests supplemented by effect size measures presented evidence of the effectiveness and efficiency of the proposed technique at 99% confidence level (i.e. p < 0.01).ConclusionThe study showed that the resulting randomized tests were reasonably good for AOP testing, thus the proposed technique could be worth using as an effective and efficient AOP-specific automated test generation technique.     

An overview on test generation from functional requirements

Despite the fact that the test phase is described in the literature as one of the most relevant for quality assurance in software projects, this test phase is not usually developed, among others, with enough resources, time or suitable techniques.To offer solutions which supply the test phase, with appropriate tools for the automation of tests generation, or even, for their self-execution, could become a suitable way to improve this phase and reduce the cost constraints in real projects.This paper focuses on answering a concrete research question: is it possible to generate test cases from functional requirements described in an informal way? For this aim, it presents an overview of a set of relevant approaches that works in this field and offers a set of comparative analysis to determine which the state of the art is.     

基于黑盒测试的航天嵌入式软件用例设计方法

为了提高测试用例设计的效率,简化测试用例数量,提出了基于黑盒测试的航天嵌入式软件用例设计方法.阐述了确认测试过程中实用的测试用例设计流程,提出了在设计测试输入数据的基础上加强测试逻辑设计的用例设计方法,并结合实际工程经验,给出了测试用例的总体设计思路.最后,对测试遗漏问题及其原因进行了分析,为今后进行更有效的星载软件测试提供了参考.     

关于软件测试模型的探讨

近年来,计算机进入了千万百姓家,在教育、医学、军事等领域贡献了巨大力量,而作为信息技术的一种,软件规模越来越大,数量和种类越来越多,软件结构也愈加复杂,但是软件在社会各个领域都是不可或缺的,因此如何提高软件质量、安全性成为了企业、研究机构所关注的重要问题.大量实践表明,软件测试是决定软件质量的重要因素,软件开发有过程模型,而软件测试有测试模型.使用测试模型的主要目的是描述软件测试中包含的活动和其相互之间的关系,运用软件测试模型能够有效提高测试效率,减少人力和财力消耗.     

Automatic testing environment for multi-core embedded software—ATEMES

Software testing during the development process of embedded software is not only complex, but also the heart of quality control. Multi-core embedded software testing faces even more challenges. Major issues include: (1) how demanding efforts and repetitive tedious actions can be reduced; (2) how resource restraints of embedded system platform such as temporal and memory capacity can be tackled; (3) how embedded software parallelism degree can be controlled to empower multi-core CPU computing capacity; (4) how analysis is exercised to ensure sufficient coverage test of embedded software; (5) how to do data synchronization to address issues such as race conditions in the interrupt driven multi-core embedded system; (6) high level reliability testing to ensure customer satisfaction. To address these issues, this study develops an automatic testing environment for multi-core embedded software (ATEMES). Based on the automatic mechanism, the system can parse source code, instrument source code, generate testing programs for test case and test driver, support generating primitive, structure and object types of test input data, multi-round cross-testing, and visualize testing results. To both reduce test engineer's burden and enhance his efficiency when embedded software testing is in process, this system developed automatic testing functions including unit testing, coverage testing, multi-core performance monitoring. Moreover, ATEMES can perform automatic multi-round cross-testing benchmark testing on multi-core embedded platform for parallel programs adopting Intel TBB library to recommend optimized parallel parameters such as pipeline tokens. Using ATEMES on the ARM11 multi-core platform to conduct testing experiments, the results show that our constructed testing environment is effective, and can reduce burdens of test engineer, and can enhance efficiency of testing task.     

Effects of computer-based testing on test performance and testing motivation

Computer-based testing (CBT) is a green computing strategies used to reduce paper consumption. However, some scholars have questioned the effectiveness of CBT and suggested for conducting systematic studies on CBT to carefully check its reliability and validity before opting for it. Recently, some studies have evaluated the effectiveness of CBT by comparing it with paper–pencil testing (PPT), and the findings were inconsistent. Besides this, most of these studies have been conducted using quasi or basic experimental designs without identifying testing effects on test takers. The limitation of these design is testing effects might occur when a participant in the control or treatment group is tested at least twice on a same test, and it influences the outcomes of taking a posttest, therefore, the findings might be misinterpreted. This study employed a Solomon four-group experimental design (2 × 3 factorial design) on a group of student teachers (n = 140) to compare CBT and PPT on test performance (test scores), testing time and testing motivation. Results indicated that the CBT mode is more reliable in terms of internal and external validity. The CBT significantly reduced testing time and developed stronger self-efficacy, intrinsic and social testing motivation in the participants.     

基于Windows Mobile的智能终端上自动化测试研究

开发windows Mobile的设备应用过程中,软件测试费用占总成本的50％左右,因此,有必要实现软件测试自动化以降低测试费用。分析了Windows Mobile的自动化测试框架,在产品中设计了UI自动化测试用例。该自动化测试测试效率高、可靠性好、易维护,并有助于实现Windows Mobile设备中其它方面的自动化测试。     

组合测试用例生成技术

组合测试是一种科学有效的软件测试方法,该方法旨在使用较少的测试用例有效地检测软件系统中各个因素以及它们之间的相互作用对系统产生的影响,实践证明其具有较高的错误检测能力。当前组合测试研究的热点之一是组合测试用例生成问题,即如何针对具体待测软件,在满足给定组合覆盖要求的前提下,生成规模尽可能小的测试用例集,以便在保证错误检测能力的前提下尽可能降低测试成本。从N维组合覆盖和变力度组合覆盖等两类不同的组合覆盖标准出发,简要介绍了迄今为止人们在组合测试用例生成领域所取得的研究成果,对现有的组合用例生成方法进行了分类和总结。此外,还对优先级、组合约束、错误定位等条件和应用场景下的组合测试用例生成技术进行了介绍。最后,分析了现有成果中存在的问题,并对该领域未来的研究方向进行了分析和讨论。     

基于过程与场景分析的航天应用软件测试方法

航天应用软件系统具有软件种类多、规模大、信息交换关系复杂等特点,如何开展软件测试是工程实践中的一个难题.结合某测控信息应用软件系统实例,提出了一种基于形式化模型的软件系统测试方法.基于软件系统形式化定义和线索分析技术,绘制软件系统运行流程图、阶段分析图,建立可表述软件系统级行为的场景树模型和场景树图;根据测试充分性要求...     

软件测试用例自动生成算法综述*

按照测试用例自动生成技术的不同,将测试用例自动生成算法分为随机、遗传、蚁群、粒子群四类,对上述各类算法的现状和进展进行介绍、分析和探讨。最后,对软件测试用例自动生成的研究进行了总结。     

遗传算法生成变异测试用例

实际测试用例一般不能满足变异测试充分,但遗传算法搜索空间较大,可使用其生成变异测试充分度较高的测试用例集.适应值函数的构造使用分支函数插装法.首先根据杀死弱变异体的必要性条件,构造必要性条件分支函数,插装于源程序中;然后根据可达性条件,构造可达性条件的分支函数并插装.使用基于面向路径的遗传算法来搜索杀死弱变异体的测试用例.将终止条件改为程序最终结果的不同,插装函数不变,生成满足条件的强变异测试用例.对于多重弱变异,按熙可达路径实施等价类划分,每一个等价类采用与单重弱变异相同的方法.实验结果表明,遗传算法可生成杀死各类变异体的测试用例,优于随机生成的测试用例.     

基于XML的通用测试文档生成方法研究

针对目前软件测试过程中通用文档自动生成所面临的问题,介绍了一种基于XML的通用测试文档生成方法.分析了软件测试文档的基本种类及文档生成的重要性,论述了将XML技术用于通用文档生成中的优势,给出了一个基于XML的通用测试文档生成模型,并探讨了该模型的关键方案设计.     

单元测试自动化工具的研究与实现

软件测试的自动化技术是软件测试技术研究的重点问题,对于提高软件测试的客观性和效率有着非常重要的意义.实现了一个可以自动生成、执行测试用例的工具,该工具针对软件测试过程中的单元测试,以容易发生异常例外的条件语句为中心,对其进行覆盖检查,实现了单元测试的自动化.     

InRob: An approach for testing interoperability and robustness of real-time embedded software

Advances in digital technologies have contributed for significant reduction in accidents caused by hardware failures. However, the growing complexity of functions performed by embedded software has increased the number of accidents caused by software faults in critical systems. Moreover, due to the highly competitive market, software intensive subsystems are usually developed by different suppliers. Often these subsystems are required to interact with each other in order to provide a collaborative service. Testing approaches for subsystems integration support verification of the quality of service, focusing on the subsystems interfaces. The increasing complexity and tight coupling of real-time subsystems make integration testing unmanageable. The ad-hoc approach for testing is becoming less effective and more expensive. This article presents an integration testing approach denominated InRob, designed to verify the interoperability and robustness related to timing constraints of real-time embedded software. InRob guides the construction of services, based on formal models, aiming at the specifications of interoperability and robustness of test cases related to delays and time-outs of the messages exchanged in the interfaces of interconnected subsystems. The proposed formalism supports automatic test cases generation by verifying the relevant properties in the service behavioral model. As timing constraints are critical properties of aerospace systems, the feasibility of InRob is showed in the integration testing process of a telescope onboard in a satellite. The process is instantiated with existing testing tools and the case study is the software embedded in the telescope.     

成对组合测试数据的整体优化和生成方法

测试数据生成是组合测试的一个关键问题,但是组合测试用例集的构造问题的复杂度是NP完全的。提出了一种成对组合测试用例集整体优化和生成的方法。该方法通过编码机制将测试用例数据的生成问题转换为一个基于二进制编码的最优化问题,同时使用遗传算法对此编码空间进行搜索,并对所发现的最优个体进行解码,构造产生最佳测试用例集。实验结果表明,该方法简单高效,且具有解的质量高、时间消耗小的特点。