On the effect of test-suite reduction on automatically generated model-based tests

Model checking techniques can be successfully employed as a test-case generation technique to generate tests from formal models. The number of tests-cases produced, however, is typically large for complex coverage criteria such as MC/DC. Test-suite reduction can provide us with a smaller set of test-cases that preserve the original coverage—often a dramatically smaller set. Nevertheless, one potential drawback with test-suite reduction is that this might affect the quality of the test-suite in terms of fault finding. Previous empirical studies provide conflicting evidence on this issue. To further investigate the problem and determine its effect when testing implementations derived from formal models of software we performed an experiment using a large case example of a Flight Guidance System, generated reduced test-suites for a variety of structural coverage criteria while preserving coverage, and recorded their fault finding effectiveness. Our results indicate that the size of the specification based test-suites can be dramatically reduced and that the fault detection of the reduced test-suites is adversely affected. In this report we describe our experiment, analyze the results, and discuss the implications for testing based on formal specifications. This work has been partially supported by NASA grant NAG-1-224 and NASA contract NCC-01001. We also want to thank the McKnight Foundation for their generous support over the years.

---

     

Establishing flight software reliability: testing,model checking,constraint-solving,monitoring and learning

In this paper we discuss the application of a range of techniques to the verification of mission-critical flight software at NASA’s Jet Propulsion Laboratory. For this type of application we want to achieve a higher level of confidence than can be achieved through standard software testing. Unfortunately, given the current state of the art, especially when efforts are constrained by the tight deadlines and resource limitations of a flight project, it is not feasible to produce a rigorous formal proof of correctness of even a well-specified stand-alone module such as a file system (much less more tightly coupled or difficult-to-specify modules). This means that we must look for a practical alternative in the area between traditional testing and proof, as we attempt to optimize rigor and coverage. The approaches we describe here are based on testing, model checking, constraint-solving, monitoring, and finite-state machine learning, in addition to static code analysis. The results we have obtained in the domain of file systems are encouraging, and suggest that for more complex properties of programs with complex data structures, it is possibly more beneficial to use constraint solvers to guide and analyze execution (i.e., as in testing, even if performed by a model checking tool) than to translate the program and property into a set of constraints, as in abstraction-based and bounded model checkers. Our experience with non-file-system flight software modules shows that methods even further removed from traditional static formal methods can be assisted by formal approaches, yet readily adopted by test engineers and software developers, even as the key problem shifts from test generation and selection to test evaluation.     

基于通信多端口有限状态机的协议互操作性测试生成研究

协议测试是一种保证网络通信协议实现质量的重要技术，互操作性测试是一类常用的协议测试技术．文章提出了一种基于通信多端口有限状态机模型的协议互操作忡测试生成方法．首先采用已有的基于可达性分析的方法生成集中式测试序列；然后采用单一错误模型对其进行系统的错误覆盖分析，为达到更高的错误覆盖度，进一步提出一种增强的测试生成算法；最后讨论了互操作性测试巾的控制观察问题，选择适当的分布式测试架构，并进而生成分布式同步测试序列．实验结果表明：与原有方法相比，该方法可以有效地提高测试集的错误覆盖，并具备一定的可行性和有效性．     

Test development for communication protocols: towards automation

In this paper we give an introduction to methods and tools for testing communication protocols and distributed systems. In this context, we try to answer the following questions: Why are we testing? What are we testing? Against what are we testing?... We present the different approaches of test automation and explain the industrial point of view (automatic test execution) and the research point of view (automatic test generation). The complete automation of the testing process requires the use of formal methods for providing a model of the required system behavior. We show the importance of modelling the aspects to be tested (the right model for the right problem!) and point out the different aspects of interest (control, data, time and communication). We present the problem of testing based on models, in the form of finite state machines (FSMs), extended FSMs, timed FSMs and communicating FSMs, and give an overview of the proposed solutions and their limitations. Finally, we present our own experience in automatic test generation based on SDL specifications, and discuss some related work and existing tools.     

DeepRanger:覆盖制导的深度森林测试方法

深度学习软件的结构特征与传统软件存在明显差异,因此即使展开了大量测试,依然无法有效衡量测试数据对深度学习软件的覆盖情况和测试充分性,并造成后续使用过程中依然可能存在大量未知错误.深度森林是一种新型深度学习模型,其克服了深度神经网络存在的一些缺点,例如:需要大量训练数据、需要高算力平台、需要大量超参数.但目前还没有相关工作对深度森林的测试方法进行研究.针对深度森林的结构特点,制定了一组由随机森林结点覆盖率RFNC、随机森林叶子覆盖率RFLC、级联森林类型覆盖率CFCC和级联森林输出覆盖率CFOC组成的测试覆盖率评价指标.在此基础上,基于遗传算法设计了覆盖制导的测试数据自动生成方法DeepRanger,可自动生成能有效提高模型覆盖率的测试数据集.为对所提出覆盖指标的有效性进行验证,在深度森林开源项目gcForest和MNIST数据集上设计并进行了一组实验.实验结果表明,所提出的4种覆盖指标均能有效评价测试数据集对深度森林模型的测试充分性.此外,与基于随机选择的遗传算法相比,使用覆盖信息制导的测试数据生成方法 DeepRanger能达到更高的模型覆盖率.     

A methodology of testing high-level Petri nets

Petri nets have been extensively used in the modelling and analysis of concurrent and distributed systems. The verification and validation of Petri nets are of particular importance in the development of concurrent and distributed systems. As a complement to formal analysis techniques, testing has been proven to be effective in detecting system errors and is easy to apply. An open problem is how to test Petri nets systematically, effectively and efficiently. An approach to solve this problem is to develop test criteria so that test adequacy can be measured objectively and test cases can be generated efficiently, even automatically. In this paper, we present a methodology of testing high-level Petri nets based on our general theory of testing concurrent software systems. Four types of testing strategies are investigated, which include state-oriented testing, transition-oriented testing, flow-oriented testing and specification-oriented testing. For each strategy, a set of schemes to observe and record testing results and a set of coverage criteria to measure test adequacy are defined. The subsumption relationships and extraction relationships among the proposed testing methods are systematically investigated and formally proved.     

Assessing and generating test sets in terms of behavioural adequacy

Identifying a finite test set that adequately captures the essential behaviour of a program such that all faults are identified is a well‐established problem. This is traditionally addressed with syntactic adequacy metrics (e.g. branch coverage), but these can be impractical and may be misleading even if they are satisfied. One intuitive notion of adequacy, which has been discussed in theoretical terms over the past three decades, is the idea of behavioural coverage: If it is possible to infer an accurate model of a system from its test executions, then the test set can be deemed to be adequate. Despite its intuitive basis, it has remained almost entirely in the theoretical domain because inferred models have been expected to be exact (generally an infeasible task) and have not allowed for any pragmatic interim measures of adequacy to guide test set generation. This paper presents a practical approach to incorporate behavioural coverage. Our BESTEST approach (1) enables the use of machine learning algorithms to augment standard syntactic testing approaches and (2) shows how search‐based testing techniques can be applied to generate test sets with respect to this criterion. An empirical study on a selection of Java units demonstrates that test sets with higher behavioural coverage significantly outperform current baseline test criteria in terms of detected faults. © 2015 The Authors. Software Testing, Verification and Reliability published by John Wiley & Sons, Ltd.     

Using coverage to automate and improve test purpose based testing

Test purposes have been presented as a solution to avoid the state space explosion when selecting test cases from formal models. Although such techniques work very well with regard to the speed of the test derivation, they leave the tester with one important task that influences the quality of the overall testing process: test purposes have to be formulated manually. In this paper, we present an approach that assists a test engineer with test purpose design in two ways: it allows automatic generation of coverage based test suites and can be used to automatically exercise those aspects of the system that are missed by hand-crafted test purposes. We consider coverage of Lotos specifications, and show how labeled transition systems derived from such specifications have to be extended in order to allow the application of logical coverage criteria to Lotos specifications. We then show how existing tools can be used to efficiently derive test cases and suggest how to use the coverage information to minimize test suites while generating them.     

Measuring and specifying combinatorial coverage of test input configurations

A key issue in testing is how many tests are needed for a required level of coverage or fault detection. Estimates are often based on error rates in initial testing, or on code coverage. For example, tests may be run until a desired level of statement or branch coverage is achieved. Combinatorial methods present an opportunity for a different approach to estimating required test set size, using characteristics of the test set. This paper describes methods for estimating the coverage of, and ability to detect, t-way interaction faults of a test set based on a covering array. We also develop a connection between (static) combinatorial coverage and (dynamic) code coverage, such that if a specific condition is satisfied, 100 % branch coverage is assured. Using these results, we propose practical recommendations for using combinatorial coverage in specifying test requirements, and for improving estimates of the fault detection capacity of a test set.     

基于规格说明的若干逻辑覆盖测试准则

基于规格说明的测试可以在不需要了解软件程序代码的情况下对软件进行功能测试.判定是形式规格说明中用于描述前、后置条件的主要形式.分析了基于规格说明的逻辑覆盖测试准则,针对已有的决定性逻辑覆盖测试准则的不足,提出了掩盖性逻辑覆盖测试准则,并对其进行了详细分析.提出了掩盖性逻辑覆盖测试准则的一个可行的测试生成算法.根据该准则生成的测试用例能够发现条件的掩盖性带来的错误.然后,从判定的结构入手,分析了条件之间的约束关系、复杂判定的分解与合成、判定之间的关系.这些分别能够阐明逻辑覆盖中条件间的耦合性问题、同一个条件在判定中的多次出现问题以及判定在程序中的位置问题.继而提出了全真判定覆盖、全假判定覆盖、完全子判定覆盖、唯一条件真覆盖以及唯一条件假覆盖等测试准则.满足这些测试准则的测试用例集能检测出不同类型的错误.最后,给出了这些测试准则之间的包含关系图,并建议了不同测试准则适用的应用场景.     

基于规格说明的若干逻辑覆盖测试准则

基于规格说明的测试可以在不需要了解软件程序代码的情况下对软件进行功能测试.判定是形式规格说明中用于描述前、后置条件的主要形式.分析了基于规格说明的逻辑覆盖测试准则,针对已有的决定性逻辑覆盖测试准则的不足,提出了掩盖性逻辑覆盖测试准则,并对其进行了详细分析.提出了掩盖性逻辑覆盖测试准则的一个可行的测试生成算法.根据该准则生成的测试用例能够发现条件的掩盖性带来的错误.然后,从判定的结构入手,分析了条件之间的约束关系、复杂判定的分解与合成、判定之间的关系.这些分别能够阐明逻辑覆盖中条件间的耦合性问题、同一个条件在判定中的多次出现问题以及判定在程序中的位置问题.继而提出了全真判定覆盖、全假判定覆盖、完全子判定覆盖、唯一条件真覆盖以及唯一条件假覆盖等测试准则.满足这些测试准则的测试用例集能检测出不同类型的错误.最后,给出了这些测试准则之间的包含关系图,并建议了不同测试准则适用的应用场景.     

基于Robot Framework的MVC GUI模块自动化测试框架

基于Robot Framework提出了一种面向MVC GUI模块的自动化测试框架。该框架将GUI自动化测试分为四大方面：用户界面功能测试、信息设置功能测试、信息查询功能测试和特殊项测试。整个框架将上层 GUI和底层Database进行结合测试,使GUI自动化测试范围得以全面覆盖;并根据不同系统环境,编写Python脚本,灵活调用底层数据;而选用黑盒与白盒有机结合的测试方式,使测试者可以迅速锁定代码缺陷,保证了开发程序的正确性和完整性。同时,就OpenWRT LuCI模块作为具体实例,验证了本框架的适用性。     

Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion?, an embedded system providing helicopter landing assistance.     

Verification and validation of safety applications based on PLCopen safety function blocks

Functional Safety is a major concern in the design of automation systems today. Many of those systems are realized using Programmable Logic Controllers (PLCs) programmed according to IEC 61131-3. PLCopen - as IEC 61131 user organization - semi-formally specified a set of software function blocks to be used in safety applications according to IEC 61508. In the presented work, formal models in the form of timed automata for the safety function blocks (SFBs) are constructed from the semi-formal specifications. The accordance of the formalized blocks to the specification is verified using model checking. Furthermore, their behaviour is validated against specified test cases by simulation. The resulting verified and validated library of formal models is used to build a formal model of a given safety application - built from SFBs - and to verify and validate its properties.     

基于强化学习算法的神经网络模糊测试技术优化研究

现有神经网络模糊测试技术在测试样本生成阶段通常对初始样本进行随机变异,导致生成样本质量不高,从而测试覆盖率不高;针对以上问题,提出一种基于强化学习算法的神经网络模糊测试技术,将模糊测试过程建模为马尔可夫决策过程,在该模型中,测试样本被看作环境状态,不同的变异方法被看作可供选择的动作空间,神经元覆盖率被看作奖励反馈,使用强化学习算法来学习最优的变异策略,指导生成最优测试样本,使其能够获得最高的神经元覆盖率;通过与现有的主流神经网络模糊测试方法的对比实验表明,基于强化学习算法的神经网络模糊测试技术,可以提升在不同粒度下的神经元覆盖。     

Java自动化基本路径测试研究

针对Java单元测试自动化程度和测试效率较低的问题,对基于Java程序的基本路径测试方法进行研究,提出了基于Java代码的基本路径生成方法和程序插桩方法,给出了插桩节点和控制流图节点的定义。首先,通过对Java源代码进行分析,构建程序的控制流图,进而对控制流图进行遍历生成基本路径集合;然后,对被测程序进行插桩,以获取程序的执行路径,插桩过程中保持节点和基本路径中的节点一致,使得插桩后的被测程序执行时得到的路径能够和基本路径集合进行自动化比对;最后,通过以测试数据为输入执行被测程序,对执行路径和基本路径进行比较,判断测试数据集对基本路径的覆盖度。通过实验,验证了所提出方法的有效性。     

Testing a probabilistic FSM using interval estimation

In this paper, we propose a method to test a probabilistic FSM. The testing process consists of two parts. First, we check if there are any output faults or transfer faults in transitions. In order to identify a state of a PFSM, the characterization set is extended such that states are identified not only by observing output sequences but also by comparing probabilities. Second, we test whether the transition probabilities are correctly implemented. Interval estimation is used to assert the correctness of transition probabilities where a test verdict is assigned with a given confidence level. From a given confidence level and confidence interval length, a method is presented to determine the test sequence repetition numbers for testing probabilities. Fault coverage evaluation is carried out based on extended fault types where probabilities are changed. As an application, we apply the proposed method to a probabilistic non-repudiation protocol.     

基于控制流模型的Web系统测试方法研究

Web应用系统的使用与日俱增,技术层出不穷,但是Web应用系统的测试却是一个难点.论文提出一种基于控制流的测试方法,对Web应用系统建立控制流模型,并给出形式化的定义,采用模型进行测试用例的产生.为了使测试自动化,论文还谈到脚本技术和测试执行器的设计.     

基于扩展有限状态机的SCA符合性测试方法研究

软件通信体系结构（Software Communication Architecture,SCA）标准是针对软件无线电设备缺乏互操作性而制定的标准,通过定义设备组件对底层的硬件进行屏蔽,实现上层应用与底层硬件相互分离,SCA符合性测试通过验证实际的应用是否符合SCA标准中的定义,从而保证应用的跨平台性与通用性。当前国内外对SCA符合性测试的研究较少,测试过程中存在诸多问题,如各需求的测试之间存在依赖关系,某些需求测试的进行可能造成后续需求测试无法执行等,导致目前缺乏自动化的测试方法。针对这些问题,提出了一种测试序列自动生成方法,并基于扩展有限状态机的基本原理构造了SCA符合性测试模型。该模型能够直观地反映测试过程,分析测试中软件无线电系统存在的各种状态,判断测试用例执行序列的可行性,指导测试序列的构建,再结合基于集合的贪心算法对测试序列集进行优化。实验结果表明该方法能有效地保证测试覆盖度,并且提升测试效率。     

A Test Design Methodology for Protocol Testing

Communication protocol testing can be done with a test architecture consisting of remote Lower Tester and local Upper Tester processes. For real protocols, tests can be designed based on the formal specification of the protocol which uses an extended finite state machine model. The specification is transformed into a simpler form consisting of normal form transitions. It can then be modeled by a control and a data flow graph. The graphs are decomposed into subtours and data flow functions, respectively. Tests are designed by considering parameter variations of the input primitives of each data flow function and determining the expected outputs. The methodology gives complete test coverage of all data flow functions and control paths in the specification. Functional fault models are proposed for functions that are not formally specified.