基于相似性的CITCP强化学习奖励策略

在面向持续集成测试用例优先排序(continuous integration test case prioritization,CITCP)的强化学习方法中,智能体通过对测试用例实施奖励从而调整测试用例优先排序策略以适应后续集成测试,可以满足持续集成测试频繁迭代和快速反馈的需求.智能体通常只奖励执行失效测试用例,但实际...     

Test case selection for black-box regression testing of database applications

ContextThis paper presents an approach for selecting regression test cases in the context of large-scale database applications. We focus on a black-box (specification-based) approach, relying on classification tree models to model the input domain of the system under test (SUT), in order to obtain a more practical and scalable solution. We perform an experiment in an industrial setting where the SUT is a large database application in Norway’s tax department.ObjectiveWe investigate the use of similarity-based test case selection for supporting black box regression testing of database applications. We have developed a practical approach and tool (DART) for functional black-box regression testing of database applications. In order to make the regression test approach scalable for large database applications, we needed a test case selection strategy that reduces the test execution costs and analysis effort. We used classification tree models to partition the input domain of the SUT in order to then select test cases. Rather than selecting test cases at random from each partition, we incorporated a similarity-based test case selection, hypothesizing that it would yield a higher fault detection rate.MethodAn experiment was conducted to determine which similarity-based selection algorithm was the most suitable in selecting test cases in large regression test suites, and whether similarity-based selection was a worthwhile and practical alternative to simpler solutions.ResultsThe results show that combining similarity measurement with partition-based test case selection, by using similarity-based test case selection within each partition, can provide improved fault detection rates over simpler solutions when specific conditions are met regarding the partitions.ConclusionsUnder the conditions present in the experiment the improvements were marginal. However, a detailed analysis concludes that the similarity-based selection strategy should be applied when a large number of test cases are contained in each partition and there is significant variability within partitions. If these conditions are not present, incorporating similarity measures is not worthwhile, since the gain is negligible over a random selection within each partition.     

支持用例集并行测试的接口测试平台

针对手工测试成本高、时效性弱和传统的接口自动化测试工具扩展能力不足问题,提出了一个支持用例集并行测试的接口测试平台(OLa). OLa采用分层架构模式将系统分为用户展示层、应用逻辑层、数据服务层和用例执行层.其中,用户展示层基于Vue框架开发,结合Vue Router、Vuex等工具实现单页应用;应用逻辑层基于Spring Boot框架开发;数据服务层基于MyBatis-Plus框架和Spring Data框架开发;用例执行层使用okhttp3、fastjson、jackson等工具实现接口测试.此外,基于系统技术架构、Java网络编程和面向抽象编程的思想,创新性地提出了基于C/S模式的用例执行流程和基于参数识别的自动匹配校验方法,解决了传统的自动化测试工具无法支持并行测试的问题.实验结果表明本文设计与实现的OLa接口测试平台支持单用例测试、用例集的串行测试和并行测试,能够自动识别用例参数并对接口响应内容进行校验,提高了接口测试的灵活性和有效性,降低了测试难度,并能够在用例之间无相互依赖的情况下提高接口测试的效率.     

Model-based testing for real

Model-based testing relies on abstract behavior models for test case generation. These models are abstractions, i.e., simplifications. For deterministic reactive systems, test cases are sequences of input and expected output. To bridge the different levels of abstraction, input must be concretized before being applied to the system under test. The systems output must then be abstracted before being compared to the output of the model.The concepts are discussed along the lines of a feasibility study, an inhouse smart card case study. We describe the modeling concepts of the CASE tool AutoFocus and an approach to model-based test case generation that is based on symbolic execution with Constraint Logic Programming.Different search strategies and algorithms for test case generation are discussed. Besides validating the model itself, generated test cases were used to verify the actual hardware with respect to these traces.     

Test Case Generation for Mutation-based Testing of Timeliness

Temporal correctness is crucial for real-time systems. Few methods exist to test temporal correctness and most methods used in practice are ad-hoc. A problem with testing real-time applications is the response-time dependency on the execution order of concurrent tasks. Execution order in turn depends on execution environment properties such as scheduling protocols, use of mutual exclusive resources as well as the point in time when stimuli is injected. Model based mutation testing has previously been proposed to determine the execution orders that need to be verified to increase confidence in timeliness. An effective way to automatically generate such test cases for dynamic real-time systems is still needed. This paper presents a method using heuristic-driven simulation to generate test cases.     

Test generation from state based use case models

We present a strategy for the automatic generation of test cases from parametrised use case templates that capture control flow, state, input and output. Our approach allows test scenario selection based on particular traces or states of the model. The templates are internally represented as CSP processes with explicit input and output alphabets, and test generation is expressed as counter-examples of refinement checking, mechanised using the FDR tool. Soundness is addressed through an input–output conformance relation formally defined in the CSP traces model. This purely process algebraic characterisation of testing has some potential advantages, mainly an easy automation of conformance verification and test case generation via model checking, without the need to develop any explicit algorithm.     

Systematic testing: A means for validating reactive systems

The validity of the first (formal) model of a system to be developed is crucial for the whole development process. Systematically checking this validity helps avoid costs that could arise if it were discovered too late that the system does not satisfy the customer's needs and expectations. This paper addresses how to validate synchronous reactive programs using the technique of systematic testing. Testing reactive systems differs from testing sequential systems: instead of checking simple pairs of inputs and outputs, sequences of inputs and outputs have to be checked. Thus, testing cannot be based on a simple function model, mapping input values onto output values nor on a control flow graph model (where a path from the start node to the final node represents one execution through the represented program). The model widely used instead is that of a finite-state machine. A systematic testing approach is presented that is both effective and efficient for validating reactive systems. It uses an additional specification based on a finite-state machine model. The approach is demonstrated for the well-known lift example. It is shown how to use the specification for carefully choosing a set of test criteria that address different types of fault; a procedure for selecting test cases and test data that satisfy the chosen criteria is presented.     

Extending symbolic execution for automated testing of stored procedures

Stored procedures in database management systems are often used to implement complex business logic. Correctness of these procedures is critical for flawless working of the system. However, testing them remains difficult due to many possible database states and constraints on data. This leads to mostly manual testing. Newer tools offer automated execution for unit testing of stored procedures but the test cases are still written manually. We propose an approach of using dynamic symbolic execution for generating automated test cases and corresponding database states for stored procedures. We model the constraints on data imposed by the schema and the SQL statements, treating values in database tables as symbolic. We use SMT solver to find values that will drive the stored procedure on a particular execution path. We instrument the internal execution plans generated by PostgreSQL to extract constraints. We use Z3 to generate test cases consisting of table data and procedure inputs. Our evaluation using stored procedures from a large business application and various GitHub repositories quantifies the evidence of effectiveness of our technique by generating test cases that lead to schema constraint violations and user-defined exceptions.     

基于蚁群算法的测试用例优先排序

测试用例优先排序技术通过优化测试用例的执行次序来提高软件测试的效率,是增强型软件测试和回归测试的重要研究课题。针对基于需求的测试用例优先排序问题,提出了一种基于蚁群算法的求解方法,采用不同的测试用例间距离及用例序列评价策略,给出了该方法的2种不同实现方式。首先,针对黑盒测试特点,设计了基于需求的一般性测试用例序列评价指标;其次,提出测试用例吸引度概念,基于测试用例吸引度定义了测试用例间的距离;然后,给出了信息素更新策略、最优解集更新策略、局部最优解突变策略等主要设计策略,分别实现了该方法基于距离和基于指标的2种实现方式。实验结果表明,该方法具有很好的全局寻优能力,整体效果上优于粒子群算法、遗传算法和随机测试。     

Complete model-based equivalence class testing

In this article, we present a model-based black-box equivalence partition testing strategy, together with a formal proof of its completeness properties. The results apply to reactive systems with large, possibly infinite input data types and finite internal and output data ranges that may be enumerated with acceptable effort. The investigation is performed on a semantic level and applies to all concrete test models whose behavioural semantics can be encoded as a variant of state transition systems. Test suite construction is performed in relation to a given fault model \(\mathcal{F}\) for which a finite black-box test suite can be constructed which is complete with respect to \(\mathcal{F}\). It is shown how the test suite generation can be effectively implemented by model-based testing tools, using propositional representations of behavioural model semantics and constraint solvers. A SysML model of the ceiling speed monitoring function of the European Train Control System is presented as a case study, to explain theory application to a concrete modelling formalism.     

Automated product line test case selection: industrial case study and controlled experiment

Automated test case selection for a new product in a product line is challenging due to several reasons. First, the variability within the product line needs to be captured in a systematic way; second, the reusable test cases from the repository are required to be identified for testing a new product. The objective of such automated process is to reduce the overall effort for selection (e.g., selection time), while achieving an acceptable level of the coverage of testing functionalities. In this paper, we propose a systematic and automated methodology using a feature model for testing (FM_T) to capture commonalities and variabilities of a product line and a component family model for testing (CFM_T) to capture the overall structure of test cases in the repository. With our methodology, a test engineer does not need to manually go through the repository to select a relevant set of test cases for a new product. Instead, a test engineer only needs to select a set of relevant features using FM_T at a higher level of abstraction for a product and a set of relevant test cases will be selected automatically. We evaluated our methodology via three different ways: (1) We applied our methodology to a product line of video conferencing systems called Saturn developed by Cisco, and the results show that our methodology can reduce the selection effort significantly; (2) we conducted a questionnaire-based study to solicit the views of test engineers who were involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology and models (FM_T and CFM_T) in their current practice; (3) we conducted a controlled experiment with 20 graduate students to assess the performance (i.e., cost, effectiveness and efficiency) of our automated methodology as compared to the manual approach. The results showed that our methodology is cost-effective as compared to the manual approach, and at the same time, its efficiency is not affected by the increased complexity of products.     

Web系统测试用例自动执行系统

由于Web应用系统的迅速发展,如何进行有效的Web测试对于保证Web系统的可靠性越来越重要。设计的Web功能测试用例自动执行系统,改进了传统的录制/回放方式,可以模拟用户操作,对Web系统进行功能性测试。提出一种基于XML的测试用例的结构描述,称之为X-Case,对于经常更新的Web系统,修改X-Case结构内的相关数据就可以进行回归测试,不需要重新生成大量的测试用例,节约Web测试的成本。该系统支持部分测试数据自动生成,对动态和静态网页均能执行自动化测试。     

Understanding the effects of changes on the cost‐effectiveness of regression testing techniques

Regression testing is an expensive testing process used to validate modified software. Regression test selection and test‐case prioritization can reduce the costs of regression testing by selecting a subset of test cases for execution, or scheduling test cases to meet testing objectives better. The cost‐effectiveness of these techniques can vary widely, however, and one cause of this variance is the type and magnitude of changes made in producing a new software version. Engineers unaware of the causes and effects of this variance can make poor choices in designing change integration processes, selecting inappropriate regression testing techniques, designing excessively expensive regression test suites and making unnecessarily costly changes. Engineers aware of causal factors can perform regression testing more cost‐effectively. This article reports the results of an embedded multiple case study investigating the modifications made in the evolution of four software systems and their impact on regression testing techniques. The results of this study expose tradeoffs and constraints that affect the success of techniques and provide guidelines for designing and managing regression testing processes. Copyright © 2003 John Wiley & Sons, Ltd.     

一个基于下推自动机的Web测试自动执行器

随着Web应用系统的广泛应用，对其质量要求也越来越高。如何进行有效的测试以保证Web应用系统的质量是值得关注的研究问题。支持测试用例自动执行的测试执行器（Test runner）是其中的一个热点和难点。文章研究并提出了基于下推自动机的Web测试自动执行器，并实现了其原型系统。谊执行器支持测试用例的自动执行并给出测试结果报告，并通过Web应用测试实例验证了其有效性。     

基于模型驱动的Web应用自动化测试平台设计与应用

为了提高Web应用的测试效率和测试覆盖率,保证Web应用软件的质量,设计了基于模型驱动的Web应用自动化测试平台。该平台结合了基于UML模型的测试用例生成、基于关键字驱动思想的框架设计和复杂多层的自动化测试框架,实现了测试用例自动设计生成及测试自动化执行,增加了测试脚本的复用性,显著提高了测试效率和测试覆盖率。最后,给出应用实例,并与现有的测试方法和平台进行对比,突出本平台的可行性和应用价值。     

Formal specification and testing: A case study

Z is a formal notation for writing system specifications that has been growing in popularity over recent years. This paper examines some of the issues involved in applying a ‘partition based’ testing method to a system specified in Z. Details of an extensive case study are given, from specification and implementation of the system to the development and execution of test cases. The strategy is found to have benefits compared to those based on less formal specifications, but there are limitations to the approach, and difficulties that need addressing.     

一种面向形式化表格需求模型的测试用例生成方法

现代安全关键性系统的软件规模和复杂性的快速增长给这类安全关键性软件系统的开发带来了很多挑战。传统文本文档的需求描述方法无法保证此类系统的开发进度和系统可靠性要求。为此文中提出了一种兼具可读性和可自动分析的形式化表格需求建模方法。文中介绍了一种针对这种表格模型测试用例的自动生成方法,工作包括对该形式化需求表格模型展开语义分析,建立需求模型的控制树结构,得到其测试等价类;为了减少不必要的测试,定义了不同安全级别的软件需求模型的测试覆盖标准,并针对不同覆盖率准则分别给出基于控制树结构的测试路径约束选择方法;对于每条路径约束测试等价类,提出了基于域错误的测试用例选择方法,能够自动生成所需的检测域错误的测试用例集。最后,通过一个需求模型实例展示了所提方法的有效性。     

Verifying Timing Constraints of Real-Time Systems by Means of Evolutionary Testing

Many industrial products are based on the use of embedded computer systems. Usually, these systems have to fulfil real-time requirements, and correct system functionality depends on their logical correctness as well as on their temporal correctness. In order to verify the temporal behavior of real-time systems, previous scientific work has, to a large extent, concentrated on static analysis techniques. Although these techniques offer the possibilty of providing safe estimates of temporal behavior for certain cases, there are a number of cases in practice for which static analysis can not be easily applied. Furthermore, no commercial tools for timing analysis of real-world programs are available. Therefore, the developed systems have to be thoroughly tested in order to detect existing deficiencies in temporal behavior, as well as to strengthen the confidence in temporal correctness. An investigation of existing test methods shows that they mostly concentrate on testing for logical correctness. They are not specialised in the examination of temporal correctness which is also essential to real-time systems. For this reason, existing test procedures must be supplemented by new methods which concentrate on determining whether the system violates its specified timing constraints. Normally, a violation means that outputs are produced too early, or their computation takes too long. The task of the tester therefore is to find the input situations with the longest or shortest execution times, in order to check whether they produce a temporal error. If the search for such inputs is interpreted as a problem of optimization, evolutionary computation can be used to automatically find the inputs with the longest or shortest execution times. This automatic search for accurate test data by means of evolutionary computation is called evolutionary testing. Experiments using evolutionary testing on a number of programs with up to 1511 LOC and 5000 input parameters have successfully identified new longer and shorter execution times than had been found using other testing techniques. Evolutionary testing, therefore, seems to be a promising approach for the verification of timing constraints. A combination of evolutionary testing and systematic testing offers further opportunities to improve the test quality, and could lead to an effective test strategy for real-time systems.