一种随机TBFL方法

许多学者研究了运用测试集对程序错误语句定位的问题,并提出了许多行之有效的方法,这些方法统称为TBFL(testing based fault localization)方法。后来人们发现,测试集里如果出现冗余,则这些冗余测试用例会伤害这些定位方法的功效。为了解决这个问题,Hao等人提出了SAFL(similarity aware fault localization)方法。实际上完全避免冗余是不可能的,因此从另一个角度构造了一个新的TBFL方法,称为随机TBFL方法。该方法的基本思想是:测试前对程序的语句错误概率进行先验分布,并把测试集看成随机变量,用测试用例反映的程序语句有关信息对程序语句的概率作一些调整,调整后的概率称为后验校正概率,最后根据这个后验概率对错误语句进行定位。将传统的TB-FL方法如Dicing方法、TARANTULA方法、SAFL方法纳入随机信息分析并通过几个实例进行分析和比较,结果表明,随机TBFL方法不仅能够正确定位错误语句,而且冗余对该方法的功效伤害不大。     

Interactive Fault Localization Using Test Information

Debugging is a time-consuming task in software development. Although various automated approaches have been proposed, they are not effective enough. On the other hand, in manual debugging, developers have difficulty in choosing breakpoints. To address these problems and help developers locate faults effectively, we propose an interactive fault-localization framework, combining the benefits of automated approaches and manual debugging. Before the fault is found, this framework continuously recommends checking points based on statements' suspicions, which are calculated according to the execution information of test cases and the feedback information from the developer at earlier checking points. Then we propose a naive approach, which is an initial implementation of this framework. However, with this naive approach or manual debugging, developers' wrong estimation of whether the faulty statement is executed before the checking point (breakpoint) may make the debugging process fail. So we propose another robust approach based on this framework, handling cases where developers make mistakes during the fault-localization process. We performed two experimental studies and the results show that the two interactive approaches are quite effective compared with existing fault-localization approaches. Moreover, the robust approach can help developers find faults when they make wrong estimation at some checking points.     

Test input reduction for result inspection to facilitate fault localization

Testing-based fault-localization (TBFL) approaches often require the availability of high-statement-coverage test suites that sufficiently exercise the areas around the faults. However, in practice, fault localization often starts with a test suite whose quality may not be sufficient to apply TBFL approaches. Recent capture/replay or traditional test-generation tools can be used to acquire a high-statement-coverage test collection (i.e., test inputs only) without expected outputs. But it is expensive or even infeasible for developers to manually inspect the results of so many test inputs. To enable practical application of TBFL approaches, we propose three strategies to reduce the test inputs in an existing test collection for result inspection. These three strategies are based on the execution traces of test runs using the test inputs. With the three strategies, developers can select only a representative subset of the test inputs for result inspection and fault localization. We implemented and applied the three test-input-reduction strategies to a series of benchmarks: the Siemens programs, DC, and TCC. The experimental results show that our approach can help developers inspect the results of a smaller subset (less than 10%) of test inputs, whose fault-localization effectiveness is close to that of the whole test collection.     

State dependency probabilistic model for fault localization

ContextFault localization is an important and expensive activity in software debugging. Previous studies indicated that statistically-based fault-localization techniques are effective in prioritizing the possible faulty statements with relatively low computational complexity, but prior works on statistical analysis have not fully investigated the behavior state information of each program element.ObjectiveThe objective of this paper is to propose an effective fault-localization approach based on the analysis of state dependence information between program elements.MethodIn this paper, state dependency is proposed to describe the control flow dependence between statements with particular states. A state dependency probabilistic model uses path profiles to analyze the state dependency information. Then, a fault-localization approach is proposed to locate faults by differentiating the state dependencies in passed and failed test cases.ResultsWe evaluated the fault-localization effectiveness of our approach based on the experiments on Siemens programs and four UNIX programs. Furthermore, we compared our approach with current state-of-art fault-localization methods such as SOBER, Tarantula, and CP. The experimental results show that, our approach can locate more faults than the other methods in every range on Siemens programs, and the overall efficiency of our approach in the range of 10–30% of analyzed source code is higher than the other methods on UNIX programs.ConclusionOur studies show that our approach consistently outperforms the other evaluated techniques in terms of effectiveness in fault localization on Siemens programs. Moreover, our approach is highly effective in fault localization even when very few test cases are available.     

Improving Fault Detection Capability by Selectively Retaining Test Cases during Test Suite Reduction

Software testing is a critical part of software development. As new test cases are generated over time due to software modifications, test suite sizes may grow significantly. Because of time and resource constraints for testing, test suite minimization techniques are needed to remove those test cases from a suite that, due to code modifications over time, have become redundant with respect to the coverage of testing requirements for which they were generated. Prior work has shown that test suite minimization with respect to a given testing criterion can significantly diminish the fault detection effectiveness (FDE) of suites. We present a new approach for test suite reduction that attempts to use additional coverage information of test cases to selectively keep some additional test cases in the reduced suites that are redundant with respect to the testing criteria used for suite minimization, with the goal of improving the FDE retention of the reduced suites. We implemented our approach by modifying an existing heuristic for test suite minimization. Our experiments show that our approach can significantly improve the FDE of reduced test suites without severely affecting the extent of suite size reduction     

Improving the effectiveness of test suite reduction for user-session-based testing of web applications

ContextTest suite reduction is the problem of creating and executing a set of test cases that are smaller in size but equivalent in effectiveness to an original test suite. However, reduced suites can still be large and executing all the tests in a reduced test suite can be time consuming.ObjectiveWe propose ordering the tests in a reduced suite to increase its rate of fault detection. The ordered reduced test suite can be executed in time constrained situations, where, even if test execution is stopped early, the best test cases from the reduced suite will already be executed.MethodIn this paper, we present several approaches to order reduced test suites using experimentally verified prioritization criteria for the domain of web applications. We conduct an empirical study with three subject applications and user-session-based test cases to demonstrate how ordered reduced test suites often make a practical contribution. To enable comparison between test suites of different sizes, we develop Mod_APFD_C, a modification of the traditional prioritization effectiveness measure.ResultsWe find that by ordering the reduced suites, we create test suites that are more effective than unordered reduced suites. In each of our subject applications, there is at least one ordered reduced suite that outperforms the best unordered reduced suite and the best prioritized original suite.ConclusionsOur results show that when a tester does not have enough time to execute the entire reduced suite, executing an ordered reduced suite often improves the rate of fault detection. By coupling the underlying system’s characteristics with observations from our study on the criteria that produce the best ordered reduced suites, a tester can order their reduced test suites to obtain increased testing effectiveness.     

Directed test suite augmentation: an empirical investigation

Test suite augmentation techniques are used in regression testing to identify code elements in a modified program that are not adequately tested and to generate test cases to cover those elements. A defining feature of test suite augmentation techniques is the potential for reusing existing regression test suites. Our preliminary work suggests that several factors influence the efficiency and effectiveness of augmentation techniques that perform such reuse. These include the order in which target code elements are considered while generating test cases, the manner in which existing regression test cases and newly generated test cases are used, and the algorithm used to generate test cases. In this work, we present the results of two empirical studies examining these factors, considering two test case generation algorithms (concolic and genetic). The results of our studies show that the primary factor affecting augmentation using these approaches is the test case generation algorithm utilized; this affects both cost and effectiveness. The manner in which existing and newly generated test cases are utilized also has a substantial effect on efficiency and in some cases a substantial effect on effectiveness. The order in which target code elements are considered turns out to have relatively few effects when using concolic test case generation but in some cases influences the efficiency of genetic test case generation. The results of our first study, on four relatively small programs using a large number of test suites, are supported by our second study of a much larger program available in multiple versions. Together, the studies reveal a potential opportunity for creating a more cost‐effective hybrid augmentation approach leveraging both concolic and genetic test case generation techniques, while appropriately utilizing our understanding of the factors that affect them. Copyright © 2014 John Wiley & Sons, Ltd.     

A learning algorithm for optimizing continuous integration development and testing practice

Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large-scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault-detection effectiveness of continuous integration tests using historical test records and combining this information with coverage-based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest-all approach. The results further demonstrate no significant reduction in fault-detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.     

Regression testing minimization,selection and prioritization: a survey

Regression testing is a testing activity that is performed to provide confidence that changes do not harm the existing behaviour of the software. Test suites tend to grow in size as software evolves, often making it too costly to execute entire test suites. A number of different approaches have been studied to maximize the value of the accrued test suite: minimization, selection and prioritization. Test suite minimization seeks to eliminate redundant test cases in order to reduce the number of tests to run. Test case selection seeks to identify the test cases that are relevant to some set of recent changes. Test case prioritization seeks to order test cases in such a way that early fault detection is maximized. This paper surveys each area of minimization, selection and prioritization technique and discusses open problems and potential directions for future research. Copyright © 2010 John Wiley & Sons, Ltd.     

Petri net based test case generation for evolved specification

Model-based testing can use a model to test a concrete program’s implementation. When the model is changed due to the evolution of the specification, it is important to maintain the test suites up to date, such that it can be used for regression testing. A complete regeneration of the whole test suite from the new model, although inefficient, is still frequently used in practice. To address this problem effectively, we propose a test case reusability analysis technique to identify reusable test cases of the original test suite based on graph analysis, such that we can generate new test cases to cover only the change-related parts of the new model. The Market Information System (MIS) is employed to demonstrate the feasibility and effectiveness of the proposed method. Our experimental results show that the use of our method saves about 31.5% test case generation cost.