Fully automated interoperability test suite derivation for communication protocols

We propose an automatic test derivation method for testing interoperability of communication protocols. It first derives a skeleton test suite for control part through a sort of reachability analysis. Next, it parameterizes each test case. Finally, it completes the test suite by actualizing test cases by assigning values to their parameters. By constructing test case space based on stable states and by basing test case actualization step on orthogonal arrays used in experimental design techniques, state space explosion is avoided. In order to demonstrate the feasibility and effectiveness, out method is applied to the ATM/B-ISDN signaling protocol and TCP. The application results show that our method surpasses the conventional methods in terms of both test size and coverage.     

Automatic test case generation for structural testing of function block diagrams

ContextFunction Block Diagram (FBD) is increasingly used in safety-critical applications. Test coverage issues for FBDs are frequently raised by regulators and users. However, there is little work at this aspect on testing FBD at model level. Our previous study has designed a new data-flow test coverage criterion, FB-Path Complete Condition Test Coverage (FPCC), that can directly test FBD structures and effectively detect function mutation errors. Nevertheless, because FPCC scheme involves several data-flow concepts and thus it is somewhat complicated to comprehend and to generate FPCC-complied test cases. An automatic test suite generator for FPCC is highly desirable.ObjectiveThis study designs an automatic test case generator, FPCCTestGen, for FPCC so as to enhance the practicability and acceptance of the FPCC approach.MethodFirst, a supporting infrastructure for performing automatic FBD-to-UPPAAL-for-FPCC transformation is designed. The supporting infrastructure includes templates, declarations, and functions as building blocks for transformation. Then, for each input FBD, represented in PLCopen XML format, FPCCTestGen performs parsing and converts FBD components into corresponding UPPAAL model components using aforementioned building blocks. After that, queries related to FPCC characteristics are submitted to UPPAAL model checker for verification. Finally, the verification traces are analyzed to obtain a FPCC-complied test suite.ResultsA safety injection system is used as a case study. Preliminary results show that the generated test suite achieves the highest FPCC percentage with a near optimal number of test cases.ConclusionThis automatic test case generation tool is effective and thus, can promote the use of the new test coverage criterion. Methodology used in FPCCTestGen is generic and can be applied to test suite generation for other test criteria on data-flow programs.     

A semantic approach for automated test oracle generation

This paper presents the design, implementation, and applications of a software testing tool, TAO, which allows users to specify and generate test cases and oracles in a declarative way. Extended from its previous grammar-based test generation tool, TAO provides a declarative notation for defining denotational semantics on each productive grammar rule, such that when a test case is generated, its expected semantics will be evaluated automatically as well, serving as its test oracle. TAO further provides a simple tagging mechanism to embed oracles into test cases for bridging the automation between test case generation and software testing. Two practical case studies are used to illustrate how automated oracle generation can be effectively integrated with grammar-based test generation in different testing scenarios: locating fault-inducing input patterns on Java applications; and Selenium-based automated web testing.     

A method of test case set generation in the commutativity test of reduce functions

MapReduce framework has become one of the more popular big data processing frameworks. In the MapReduce framework, the test of the commutativity problem of the reduce function may take a lot of time and space. The aim of this study is to reduce the size of the test case set required to detect the commutativity problem of the reduce function when the initial test cases are available. With the initial test case as input, this method will determine the pattern of the function according to the information when the function is running, and select the corresponding test case according to the characteristics of each pattern to generate the subsequent test case set. Experiments on 170 reduce functions can verify that the test case set generated by our method is effective when testing the commutativity of reduce functions. And compared with the existing test case generation strategy, our method can have a smaller test case set to achieve the same accuracy. When detecting the commutativity of reduce functions, the test case set generation strategy we proposed can detect the commutativity of functions with a smaller test scale, and has a high recall.     

Code-coverage guided prioritized test generation

Most automatic test generation research focuses on generation of test data from pre-selected program paths or input domains or program specifications. This paper presents a methodology for a full solution to code-coverage-based test case generation, which includes code coverage-based path selection, test data generation and actual test case representation in program’s original languages. We implemented this method in an automatic testing framework, eXVantage. Experimental results and industrial trials show that the framework is able to generate tests to achieve program line coverage from 20% to 98% with reduced overall testing effort. Our major contributions include an innovative coverage-based program prioritization algorithm, a novel path selection algorithm that takes into consideration program priority and functional calling relationship, and a constraint solver for test data generation that derives constraints from bytecode and solves complex constraints involving strings and dynamic objects.     

DLJ: A dynamic line-justification algorithm for test generation

Lne justification is a basic factor in affecting the efficiency of algorithms for test generation.The existence of reconvergent fanouts in the circuit under test results in backtracks in the process of line justification.In order to reduce the number of backtracks and shorten the processing time between backtracks,we present a new algorithm called DLJ(dynamic line justification)in whic two techniques are employed.1.A cost function called “FOCOST” is proposed as heuristic information to represent the cost of justifying a certain line.Whn the relations among the lines being justified are“and”,the line having the highest FOCOST should be chosen.When the relations are“or”,the line having the lowest FOCOST should be chosen.The computing of the FOCOST of lines is very simple.2.Disjoint justification cubes dynamically generated to perform backtracks make the backtrack number of the algorithm minimal.When the backtrace with cube C1 does not yield a solution,the next cube to be chosen is C‘2=C2-{C1,C2}.Experimental results demonstrate that the combination of the two techniques effectively reduces the backtracks and accelerates the test generation.     

Dynamic method for software test data generation

Test data generation in program testing is the process of identifying a set of test data which satisfies a given testing criterion. Existing pathwise test data generators proceed by selecting program paths that satisfy the selected criterion and then generating program inputs for these paths. One of the problems with this approach is that unfeasible paths are often selected; as a result, significant computational effort can be wasted in analysing those paths. In this paper, an approach to test data generation, referred to as a dynamic approach for test data generation, is presented. In this approach, the path selection stage is eliminated. Test data are derived based on the actual execution of the program under test and function minimization methods. The approach starts by executing a program for an arbitrary program input. During program execution for each executed branch, a search procedure decides whether the execution should continue through the current branch or an alternative branch should be taken. If an undesirable execution flow is observed at the current branch, then a real-valued function is associated with this branch, and function minimization search algorithms are used to locate values of input variables automatically, which will change the flow of execution at this branch.     

Manufacturing interoperability

As manufacturing and commerce become ever more global, companies are dependent increasingly upon the efficient and effective sharing of information with their partners, wherever they may be. Leading manufacturers perform this sharing with computers, which must therefore have the required software to encode and decode the associated electronic transmissions. Because no single company can dictate that all its partners use the same software, standards for how the information is represented become critical for error-free transmission and translation. The terms interoperability and integration are frequently used to refer to this error-free transmission and translation. This paper summarizes two projects underway at the National Institute of Standards and Technology in the areas of interoperability testing and integration automation. These projects lay the foundation for at tomorrow’s standards, which we believe will rely heavily upon the use of formal logic representations, commonly called ontologies. Received February 2005: /Accepted: January 2006     

基于MDA的软件测试用例生成

将MDA中模型驱动的软件代码自动化生成思想应用于模型驱动的软件测试用例自动化生成。从UML/OCL模型出发,采用缺陷测试理论、变异分析技术,结合约束处理规则,开发一个可以自动生成单元测试用例的框架,提高软件测试的自动化程度,从整体上提高软件的开发效率。     

Towards mass customized interoperability

This paper describes an approach to interoperability in design projects that is based on computational agents customizing the representation of product data to individual design tools. The agents can augment their translator capabilities autonomously at runtime. This has the potential to lessen the need to manually pre-define an appropriate set of tool translators. The process of generating tailored product models uses an approach to organizing agent knowledge that limits computational complexity when large numbers of tools are involved. We report the implementation and evaluation of an agent-based system that demonstrates the fundamental features of mass customized interoperability. Results include an average reduction of manual work by 88% using this approach.     

Separating sequence overlap for automated test sequence generation

Finite state machines have been used to model a number of classes of system and there has thus been much interest in the automatic generation of test sequences from finite state machines. Many finite state machine based test techniques utilize sequences that check the final states of transitions, the most general such sequence being a separating sequence: an input sequence that distinguishes between two states of an FSM. When using such techniques the test sequence length can be reduced by utilizing overlap. This paper investigates overlap for separating sequences and shows how this can be incorporated into test sequence generation.     

Multilingual language resources and interoperability

This article introduces the topic of “Multilingual language resources and interoperability”. We start with a taxonomy and parameters for classifying language resources. Later we provide examples and issues of interoperatability, and resource architectures to solve such issues. Finally we discuss aspects of linguistic formalisms and interoperability.

Detecting buffer overflow via automatic test input data generation

Buffer overflows cause serious problems in various categories of software systems. In critical systems, such as health-care, nuclear or aerospace software applications, a buffer overflow may cause severe threats to humans or severe economic losses. If they occur in network or security applications, they can be exploited to gain administrator privileges, perform system attacks, access unauthorized data, or misuse the system. This paper proposes a combination of genetic algorithms, linear programming, evolutionary testing, and static and dynamic information to detect buffer overflows. The newly proposed test input generation process avoids the need for human intervention to define and tune genetic algorithm weights and therefore it becomes completely automated. The process that guides the genetic search towards the detection of buffer overflow relies on a fitness function that takes into account static and dynamic information. Reported results of our case studies, consisting of two sets of open-source programs show that the new process and fitness function outperform previously published approaches.     

基于测试需求的互操作性测试用例生成方法

网络化应用将成为未来软件技术发展的主导模式。为了保证网络化应用之间能够有效地协同工作,必须对其进行互操作性测试。互操作性测试具有测试需求复杂易变、测试用例设计困难的特点,因而需要消耗大量成本。为此,提出一种基于测试需求的互操作性测试用例生成方法。该方法采用模型驱动的测试思想,以测试需求模型描述互操作性测试需求,以状态图描述各被测应用的规格说明,通过两者的结合生成满足测试需求的测试用例。     

SVM应用于测试用例生成的方法

针对在小样本情况下BP神经网络在生成软件测试用例的过程中可能产生的过学习问题及识别正确率较差的缺点,运用支持向量机具有更好的泛化性能的原理,提出了应用支持向量机生成测试用例的方法。对五个软件测试实例针对多组不同数量的训练样本所做的实验表明,在小样本情况下与BP神经网络相比,应用支持向量机得到的测试用例预期结果的正确率提高了10个百分点以上,说明了该方法的有效性。     

应用UML顺序图的联锁测试用例生成方法

计算机联锁软件是安全完善度等级最高的软件, 因此, 在投入使用前对联锁软件的测试是必不可少的。为生成联锁软件测试用例, 结合包含有充分测试信息的UML模型, 提出一种以UML顺序图为主要模型的软件测试用例生成方法。通过提出有向图SG生成算法, 将顺序图转换为SG; 然后基于特定的覆盖准则和采用深度优先搜索算法遍历SG, 得到场景的输入、预期输出、约束条件以及场景环境, 生成测试用例; 最后, 以进路建立过程为例验证了该方法的可行性和正确性。     

InRob: An approach for testing interoperability and robustness of real-time embedded software

Advances in digital technologies have contributed for significant reduction in accidents caused by hardware failures. However, the growing complexity of functions performed by embedded software has increased the number of accidents caused by software faults in critical systems. Moreover, due to the highly competitive market, software intensive subsystems are usually developed by different suppliers. Often these subsystems are required to interact with each other in order to provide a collaborative service. Testing approaches for subsystems integration support verification of the quality of service, focusing on the subsystems interfaces. The increasing complexity and tight coupling of real-time subsystems make integration testing unmanageable. The ad-hoc approach for testing is becoming less effective and more expensive. This article presents an integration testing approach denominated InRob, designed to verify the interoperability and robustness related to timing constraints of real-time embedded software. InRob guides the construction of services, based on formal models, aiming at the specifications of interoperability and robustness of test cases related to delays and time-outs of the messages exchanged in the interfaces of interconnected subsystems. The proposed formalism supports automatic test cases generation by verifying the relevant properties in the service behavioral model. As timing constraints are critical properties of aerospace systems, the feasibility of InRob is showed in the integration testing process of a telescope onboard in a satellite. The process is instantiated with existing testing tools and the case study is the software embedded in the telescope.     

Genetic algorithm based multi-agent system applied to test generation

Automatic test generating system in distributed computing context is one of the most important links in on-line evaluation system. Although the issue has been argued long since, there is not a perfect solution to it so far. This paper proposed an innovative approach to successfully addressing such issue by the seamless integration of genetic algorithm (GA) and multi-agent system. In the design phase, a test ontology was firstly defined for smoothing the communication among agents. For the implementation of GA, The fitness function and the structure of chromosome were identified on the basis of the analysis of constraint conditions associated with a test. To demonstrate the task execution flow and messages passing among agents, the activity diagram and sequence diagram were also shown on the AUML basis. In the phase of implementation, the JADE based agent behavior model was described in detail and the implementation platform was also demonstrated. The final simulation results validated the feasibility of the proposed approach.     

软件测试用例自动生成算法综述*

按照测试用例自动生成技术的不同,将测试用例自动生成算法分为随机、遗传、蚁群、粒子群四类,对上述各类算法的现状和进展进行介绍、分析和探讨。最后,对软件测试用例自动生成的研究进行了总结。     

Issues in using model checkers for test case generation

The use of model checkers for automated software testing has received some attention in the literature: It is convenient because it allows fully automated generation of test suites for many different test objectives. On the other hand, model checkers were not originally meant to be used this way but for formal verification, so using model checkers for testing is sometimes perceived as a “hack”. Indeed, several drawbacks result from the use of model checkers for test case generation. If model checkers were designed or adapted to take into account the needs that result from the application to software testing, this could lead to significant improvements with regard to test suite quality and performance. In this paper we identify the drawbacks of current model checkers when used for testing. We illustrate techniques to overcome these problems, and show how they could be integrated into the model checking process. In essence, the described techniques can be seen as a general road map to turn model checkers into general purpose testing tools.