Constraint-based automatic test data generation

A novel technique for automatically generating test data is presented. The technique is based on mutation analysis and creates test data that approximate relative adequacy. It is a fault-based technique that uses algebraic constraints to describe test cases designed to find particular types of faults. A set of tools (collectively called Godzilla) that automatically generates constraints and solves them to create test cases for unit and module testing has been implemented. Godzilla has been integrated with the Mothra testing system and has been used as an effective way to generate test data that kill program mutants. The authors present an initial list of constraints and discuss some of the problems that have been solved to develop the complete implementation of the technique     

Should software testers use mutation analysis to augment a test set?

Mutation testing has historically been used to assess the fault-finding effectiveness of a test suite or other verification technique. Mutation analysis, rather, entails augmenting a test suite to detect all killable mutants. Concerns about the time efficiency of mutation analysis may prohibit its widespread, practical use. The goal of our research is to assess the effectiveness of the mutation analysis process when used by software testers to augment a test suite to obtain higher statement coverage scores. We conducted two empirical studies and have shown that mutation analysis can be used by software testers to effectively produce new test cases and to improve statement coverage scores in a feasible amount of time. Additionally, we find that our user study participants view mutation analysis as an effective but relatively expensive technique for writing new test cases. Finally, we have shown that the choice of mutation tool and operator set can play an important role in determining how efficient mutation analysis is for producing new test cases.     

一种Web服务的测试数据自动生成方法

软件测试是保证Web服务质量的重要技术手段.测试数据生成是Web服务测试的重要内容.测试数据的质量将直接影响Web服务测试的效率和成本.文章基于合约式设计的 Web服务测试技术,提出一种 Web服务的测试数据自动生成方法.首先根据WSDL文档采用随机法自动生成初始测试数据,然后使用合约变异技术进行测试数据的选择,据此可以生成一组达到一定合约变异充分度的有效测试数据,从而提高 Web服务的测试质量和效率.最后实现了一个Web服务的测试数据自动生成工具原型,并通过实验验证了方法的有效性.     

Testing robot controllers using constraint programming and continuous integration

ContextTesting complex industrial robots (CIRs) requires testing several interacting control systems. This is challenging, especially for robots performing process-intensive tasks such as painting or gluing, since their dedicated process control systems can be loosely coupled with the robot’s motion control.ObjectiveCurrent practices for validating CIRs involve manual test case design and execution. To reduce testing costs and improve quality assurance, a trend is to automate the generation of test cases. Our work aims to define a cost-effective automated testing technique to validate CIR control systems in an industrial context.MethodThis paper reports on a methodology, developed at ABB Robotics in collaboration with SIMULA, for the fully automated testing of CIRs control systems. Our approach draws on continuous integration principles and well-established constraint-based testing techniques. It is based on a novel constraint-based model for automatically generating test sequences where test sequences are both generated and executed as part of a continuous integration process.ResultsBy performing a detailed analysis of experimental results over a simplified version of our constraint model, we determine the most appropriate parameterization of the operational version of the constraint model. This version is now being deployed at ABB Robotics’s CIR testing facilities and used on a permanent basis. This paper presents the empirical results obtained when automatically generating test sequences for CIRs at ABB Robotics. In a real industrial setting, the results show that our methodology is not only able to detect reintroduced known faults, but also to spot completely new faults.ConclusionOur empirical evaluation shows that constraint-based testing is appropriate for automatically generating test sequences for CIRs and can be faithfully deployed in an industrial context.     

An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing

Experimental work in software testing has generally focused on evaluating the effectiveness and effort requirements of various coverage criteria. The important issue of testing efficiency has not been sufficiently addressed. In this paper, we describe an approach for comparing the effectiveness and efficiency of test coverage criteria using mutation analysis. For each coverage criterion under study, we generate multiple coverage-adequate minimal test suites for a test-program from a test-pool, which are then executed on a set of systematically generated program mutants to obtain the fault data. We demonstrate the applicability of the proposed approach by describing the results of an experiment comparing the three code-based testing criteria, namely, block coverage, branch coverage, and predicate coverage. Our results suggest that there is a trade-off between effectiveness and efficiency of a coverage criterion. Specifically, the predicate coverage criterion was found to be most effective but least efficient whereas the block coverage criterion was most efficient but least effective. We observed high variability in the performance of block test suites whereas branch and predicate test suites were relatively consistent. Overall results suggest that the branch coverage criterion performs consistently with good efficiency and effectiveness, and it appears to be the most viable option for code-based control flow testing.     

基于软件层次化模型的软件测试数据生成

针对基于Markov链模型的软件测试技术在测试数据生成时不考虑软件的结构信息,生成的测试数据集对代码路径的覆盖能力以及缺陷检测能力都较低的问题,将统计测试与基于Markov链模型的测试相结合,提出了一种新的软件测试模型——软件层次化模型。该模型涵盖了软件与外部环境之间的交互,同时描述了软件内部结构信息。还给出了该模型测试数据集的生成算法：首先生成符合使用情况的测试序列,然后为测试序列生成覆盖软件内部结构的输入数据。通过针对示例软件的实验结果表明,与基于Markov链模型的测试方法对比,基于软件层次化模型的测试在满足软件测试充分性要求的同时,提高了测试数据集的代码路径覆盖能力和缺陷检测能力。     

Ct coverage — initial results

Software testing techniques based upon automated analysis of control flow are useful for improving software reliability. The fundamental types of control flow analysis, in ascending order of effectiveness, are statement, branch, and path coverage. Automated tools that perform branch coverage analysis are now accepted practice and researchers are exploring the area that exists between branch and path analysis. Path testing is complicated by the huge number of paths in ordinary programs and by anomalies, such as infeasible paths. The Ct testing strategy is a method for obtaining a manageable set of path classes by specifying a minimum iteration count k. The Ct test coverage metric is a measurement of the proportion of Ct path classes that are exercised within a program. This paper describes an initial experiment in which a special, automated tool was used to evaluate the Ct coverage of tests for a C program of significant size. The results of the study showed that high values of branch coverage may not necessarily imply high path coverage, that approximately 15% of the functions analysed were too complex to analyse effectively, and that the majority of the remaining functions achieved on the order of 11% Ct k=1 coverage. Experimental work in the attainment of high Ct coverage suggests the development of a new, more efficient software testing strategy, which combines Ct path and data flow analysis.     

专家系统规则库覆盖度量的研究及实践

针对专家系统规则库的软件测试和评价是保证专家系统质量与可靠性的重要环节。基于数据流的结构化测试技术,提出了一种规则库执行流图RBEF,用于描述规则库中不同规则之间相互连结、制约关系,进而给出了一组规则库覆盖度量准则和测试用例的设计方法,最后讨论了一个已实现并付诸于应用的基于RBEF的软件测试工具ESRTS。     

基于统计占优分析的变异测试

为数众多的变异体产生的高昂测试代价严重影响了变异测试技术在实际程序中的应用.为了大幅度减少弱变异测试中变异体的数量,提出基于统计占优分析的变异体约简方法.该方法首先利用变异前后的语句构造变异分支,并将所有变异分支集成到原程序中,形成新的被测程序;然后,通过统计测试用例对各个变异分支的覆盖信息,确定变异分支之间的占优关系;最后得到非被占优分支集,其对应的变异体就是约简后的变异体.将该方法用于8个程序的测试,结果表明:该方法能够约简平均90%的变异体,从而显著提高了变异测试的效率.