Prioritizing test cases for regression testing

Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. Various goals are possible; one involves rate of fault detection, a measure of how quickly faults are detected within the testing process. An improved rate of fault detection during testing can provide faster feedback on the system under test and let software engineers begin correcting faults earlier than might otherwise be possible. One application of prioritization techniques involves regression testing, the retesting of software following modifications; in this context, prioritization techniques can take advantage of information gathered about the previous execution of test cases to obtain test case orderings. We describe several techniques for using test execution information to prioritize test cases for regression testing, including: 1) techniques that order test cases based on their total coverage of code components; 2) techniques that order test cases based on their coverage of code components not previously covered; and 3) techniques that order test cases based on their estimated ability to reveal faults in the code components that they cover. We report the results of several experiments in which we applied these techniques to various test suites for various programs and measured the rates of fault detection achieved by the prioritized test suites, comparing those rates to the rates achieved by untreated, randomly ordered, and optimally ordered suites     

Prioritizing test cases for early detection of refactoring faults

Refactoring edits are error‐prone, requiring cost‐effective testing. Regression test suites are often used as a safety net for decreasing the chances of behavioural changes. Because of the high costs related to handling massive test suites, prioritization techniques can be applied to reorder test case execution, fostering early fault detection. However, traditional prioritization techniques are not specifically designed for detecting refactoring‐related faults. This article proposes refactoring‐based approach (RBA), a refactoring‐aware strategy for prioritizing regression test cases. RBA reorders an existing test sequence, using a set of proposed refactoring fault models that define the refactoring's impact on program methods. Refactoring‐based approach's evaluation shows that it promotes early detection of refactoring faults and outperforms well‐known prioritization techniques in 71% of the cases. Moreover, it prioritizes fault‐revealing test cases close to one another in 73% of the cases, which can be useful for fault localization. Those findings show that RBA can considerably improve prioritization of test cases during perfective evolution, both by increasing fault‐detection rates as well as by helping to pinpoint defects introduced by an incorrect refactoring. Copyright © 2016 John Wiley & Sons, Ltd.     

Prioritizing MCDC test cases by spectral analysis of Boolean functions

Test case prioritization aims at scheduling test cases in an order that improves some performance goal. One performance goal is a measure of how quickly faults are detected. Such prioritization can be performed by exploiting the fault exposing potential (FEP) parameters associated to the test cases. The FEP is usually approximated by mutation analysis under certain fault assumptions. Although this technique is effective, it could be relatively expensive compared to the other prioritization techniques. This study proposes a cost‐effective FEP approximation for prioritizing modified condition decision coverage (MCDC) test cases. A strict negative correlation between the FEP of an MCDC test case and the influence value of the associated input condition allows to order the test cases easily without the need of an extensive mutation analysis. The method is entirely based on mathematics and it provides useful insight into how spectral analysis of Boolean functions can benefit software testing.     

基于依赖结构的功能测试集排序方法

为解决依赖结构优先级在测试用例权值相等时存在的问题,针对流行的深度优先搜索算法进行改进。通过结合测试用例之间的功能依赖和测试用例的代码覆盖率,推导出有向无环图和算法流程图;利用推导出的有向无环图和流程图,使用权值和代码覆盖率算出最长路径作为测试集,以达到缩减测试集,同时保证代码覆盖率的目的。结合实例,将依赖结构优先级和现存的技术进行对比,验证了依赖结构优先级技术在提高错误检测率方面的可行性和实用性。     

Prioritizing test scenarios from UML communication and activity diagrams

Due to the large size and complexity of software, exhaustive testing becomes impossible. Hence, testing must be done in an optimized way keeping in mind factors, such as requirements of the customer, cost and time. For this, there is a need to generate test cases and exercise them to gain maximum throughput by uncovering defects. Test case/scenario prioritization is a well known and efficient technique to ensure the software quality. Prioritization of test scenarios helps in early detection of bugs. In this paper, we present an integrated approach and a prioritization technique to generate cluster-level test scenarios from UML communication and activity diagrams. In our approach, we first construct a tree representation of communication diagrams, and then a tree representation of activity diagrams. We convert them into an intermediate tree named as COMMACT tree. We, then carry out a post-order traversal of the constructed tree for selecting conditional predicates from the intermediate tree. Then, we propose an algorithm to generate test scenarios from the constructed tree. Next, the necessary information, such as method-activity sequence, associated objects, and constraint conditions is extracted from test scenario. The test sequences are a set of theoretical paths starting from initialization to end, while taking conditions (pre- and post-condition) into consideration. Each generated test sequence corresponds to a particular scenario of the considered use case. The third phase is to generate test scenarios from the tree satisfying the message–activity path test adequacy criteria. Preliminary results obtained on a case-study indicate that the technique is effective in extracting the critical scenarios from the communication and activity diagrams. Our approach generates redundant test scenarios and still achieves adequate test coverage.     

一种融合多种编辑距离的字符串相似度计算方法*

针对中西文混合字符串,采用了将汉字作为西文字符的等价单位计算编辑距离的方法,并从输入法的角度提出了采用拼音编码和五笔编码计算编辑距离的方法,最后给出了融合三种编辑距离计算字符串相似度的算法。仿真结果表明,该方法在提高相似重复记录检测的查全率的同时,也能获得较高的查准率。     

EFSM模型的字符串类型测试数据自动生成

基于软件描述模型的测试数据自动生成研究中,字符串类型测试数据生成是一个研究热点和难点。EFSM模型是一种重要的软件描述模型。分析了EFSM模型的特点,针对面向EFSM模型目标路径的字符串测试数据生成,建立了字符串输入变量模型和操作模型,结合静态测试的特点,给出了通过字符串变量模型在目标路径上的符号执行结果生成字符串类型测试数据的方法。实验结果表明,该方法能够达到预期效果,提高测试生成效率。     

A test for spatial randomness based on k-NN distances

This paper examines a new statistic based on k-NN distances for assessing the uniformity of a set of points in a d-dimensional space and proposes the growing frame method for dealing with unknown sampling windows. The proposed statistic is evaluated in three Monte Carlo experiments and applied to the Iris Data.     

Automatic string test data generation for detecting domain errors

Domain testing is designed to detect domain errors that result from a small boundary shift in a path domain. Although many researchers have studied domain testing, automatic domain test data generation for string predicates has seldom been explored. This paper presents a novel approach for the automatic generation of ON–OFF test points for string predicate borders, and describes a corresponding test data generator. Our empirical work is conducted on a set of programs with string predicates, where extensive trials have been done for each string predicate, and the results are analysed using the SPSS tool. Conclusions are drawn that: (i) the approach is promising and effective; (ii) there is a strong linear relationship between the performance of the test generator and the length of target string in the predicate tested; and (iii) initial inputs, no shorter than the target string and with characters generated randomly, may enhance the performance in the test data generation for string predicates. Copyright © 2009 John Wiley & Sons, Ltd.     

振弦传感器极限拉伸室内试验研究

在大型水利水电工程建设的原位监测工作中,时有锚杆应力计应力超量程问题发生,超过量程后的测试数据的可靠性备受争议。分别对量程为210,310,400 MPa三种振弦式应力计进行了极限拉伸试验,对其超量程极限应力状态下的工作状态进行了研究。结果表明:量程范围内荷载—位移曲线呈线性,一直到锚杆应力计屈服前其测量读数是可靠的,一般荷载超过1.6~2.0倍量程范围的读数存在较大误差。     

Field modeling with sampled distances

Traditional mesh-based approaches to the modeling and analysis of physical fields within geometric models require some form of topological reconstruction and conversion in the mesh generation process. Such manipulations tend to be tedious and error-prone manual processes that are not easily automated. We show that most field problems may be solved directly by using approximate distance fields computed from designed or sampled geometric data, thus avoiding many of the difficult reconstruction and meshing problems. With distances we can model fields that satisfy boundary conditions while approximating the governing differential equations to arbitrary precision. Because the method is based on sampling, it provides natural control for multi-resolution both in geometric detail of the domain and in accuracy of the computed physical field. We demonstrate the field modeling capability with several heat transfer applications, including a typical transient problem and a ‘scan and solve’ approach to the simulation of a physical field in a real-world artifact.     

Shape recognition with spectral distances

Recent works have shown the use of diffusion geometry for various pattern recognition applications, including nonrigid shape analysis. In this paper, we introduce spectral shape distance as a general framework for distribution-based shape similarity and show that two recent methods for shape similarity due to Rustamov and Mahmoudi and Sapiro are particular cases thereof.     

基于差分进化算法的组合测试用例集生成

为了减小所生成的组合测试用例集规模,提出了一种基于one-test-at-a-time策略的差分进化算法来求解该问题的方案。通过实验,比较了不同变异方式的优化性能,考察了各个算法参数对优化性能的影响,并与其他常用方法进行了对比。实验结果表明,与其他常用的组合测试用例生成方法相比,基于差分进化算法的生成方法能够生成规模更小的组合测试用例集。并且组合数越多,该算法的优势就越明显。     

Modeling test cases for security protocols with SecureMDD

Designing and executing test cases for security-critical protocols is a technically complicated and tedious process. SecureMDD is a model-driven approach that enables development of security-critical applications based on cryptographic protocols. In this paper we introduce a method which combines the model-driven approach used in SecureMDD with the design of functional and security tests. We construct and evaluate new modeling guidelines that allow the modeler to easily define such test cases during the modeling stage. We also implement model transformation routines to generate runnable tests for actual implementation of applications developed with SecureMDD.     

Prioritizing computer literacy topics

This article presents a process of prioritizing selected computer literacy topics as an initial step in implementing computer literacy programs for secondary schools. Ninety-five Nebraska educators formed the sample. A 45-item questionnaire was developed as the instrument. Educators were asked to choose more preferable topics among other topics for teaching secondary school students about computers. Thurstone's Law of Comparative Judgment was used to scale or locate educator's opinions about computer literacy topics on an interval level continuum. A general knowledge about computers was emphasized by educators surveyed. Hands-on experiences were viewed as necessary for students who want to be competent in using computers. Programming skills were suggested but not essential for computer literate students. Findings of this study provide implications of integrating computer literacy with existing curriculum. Results also provide guidelines for curriculum planners and persons responsible for the acquisition of hardware, software and staff development.     

Prioritizing static analysis results

We begin by describing the system Fortify uses for ranking vulnerabilities and our method for assigning a program an over-all score. We then consider another popular vulnerability ranking system (CVSS) and explain why it is less useful for ranking static analysis results. We use the second half of the paper to explain the motivation and method behind our ranking system.     

Prioritizing Point-Based POMDP Solvers

Recent scaling up of partially observable Markov decision process (POMDP) solvers toward realistic applications is largely due to point-based methods that quickly converge to an approximate solution for medium-sized domains. These algorithms compute a value function for a finite reachable set of belief points, using backup operations. Point-based algorithms differ on the selection of the set of belief points and on the order by which backup operations are executed on the selected belief points. We first show how current algorithms execute a large number of backups that can be removed without reducing the quality of the value function. We demonstrate that the ordering of backup operations on a predefined set of belief points is important. In the simpler domain of MDP solvers, prioritizing the order of equivalent backup operations on states is known to speed up convergence. We generalize the notion of prioritized backups to the POMDP framework, showing how existing algorithms can be improved by prioritizing backups. We also present a new algorithm, which is the prioritized value iteration, and show empirically that it outperforms current point-based algorithms. Finally, a new empirical evaluation measure (in addition to the standard runtime comparison), which is based on the number of atomic operations and the number of belief points, is proposed in order to provide more accurate benchmark comparisons.      

Generating asynchronous test cases from test purposes

Context

Input/output transition system (IOTS) models are commonly used when next input can arrive even before outputs are produced. The interaction between the tester and an implementation under test (IUT) is usually assumed to be synchronous. However, as the IUT can produce outputs at any moment, the tester should be prepared to accept all outputs from the IUT, or else be able to block (refuse) outputs of the implementation. Testing distributed, remote applications under the assumptions that communication is synchronous and actions can be blocked is unrealistic, since synchronous communication for such applications can only be achieved if special protocols are used. In this context, asynchronous tests can be more appropriate, reflecting the underlying test architecture which includes queues.

Objective

In this paper, we investigate the problem of constructing test cases for given test purposes and specification input/output transition systems, when the communication between the tester and the implementation under test is assumed to be asynchronous, performed via multiple queues.

Method

When issuing verdicts, asynchronous tests should take into account a distortion caused by the queues in the observed interactions. First, we investigate how the test purpose can be transformed to account for this distortion when there are a single input queue and a single output queue. Then, we consider a more general problem, when there may be multiple queues.

Results

We propose an algorithm which constructs a sound test case, by transforming the test purpose prior to composing it with the specification without queues.

Conclusion

The proposed algorithm mitigates the state explosion problem which usually occurs when queues are directly involved in the composition. Experimental results confirm the resulting state space reduction.     

Shift-or string matching with super-alphabets

Given a text T[1…n] and a pattern P[1…m] over some alphabet Σ of size σ, we want to find all the (exact) occurrences of P in T. The well-known shift-or algorithm solves this problem in time O(n⌈m/w⌉), where w is the number of bits in machine word, using bit-parallelism. We show how to extend the bit-parallelism in another direction, using super-alphabets. This gives a speed-up by a factor s, where s is the number of characters processed simultaneously. The algorithm is implemented, and we show that it works well in practice too. The result is the fastest known algorithm for exact string matching for short patterns and small alphabets.     

Generalizing predicates with string arguments

The least general generalization (LGG) of strings may cause an over-generalization in the generalization process of the clauses of predicates with string arguments. We propose a specific generalization (SG) for strings to reduce over-generalization. SGs of strings are used in the generalization of a set of strings representing the arguments of a set of positive examples of a predicate with string arguments. In order to create a SG of two strings, first, a unique match sequence between these strings is found. A unique match sequence of two strings consists of similarities and differences to represent similar parts and differing parts between those strings. The differences in the unique match sequence are replaced to create a SG of those strings. In the generalization process, a coverage algorithm based on SGs of strings or learning heuristics based on match sequences are used. Ilyas Cicekli received a Ph.D. in computer science from Syracuse University in 1991. He is currently a professor of the Department of Computer Engineering at Bilkent University. From 2001 till 2003, he was a visiting faculty at University of Central Florida. His current research interests include example-based machine translation, machine learning, natural language processing, and inductive logic programming. Nihan Kesim Cicekli is an Associate Professor of the Department of Computer Engineering at the Middle East Technical University (METU). She graduated in computer engineering at the Middle East Technical University in 1986. She received the MS degree in computer engineering at Bilkent University in 1988; and the PhD degree in computer science at Imperial College in 1993. She was a visiting faculty at University of Central Florida from 2001 till 2003. Her current research interests include multimedia databases, semantic web, web services, data mining, and machine learning.