Using coverage information to predict the cost-effectiveness ofregression testing strategies

Selective regression testing strategies attempt to choose an appropriate subset of test cases from among a previously run test suite for a software system, based on information about the changes made to the system to create new versions. Although there has been a significant amount of research in recent years on the design of such strategies, there has been very little investigation of their cost-effectiveness. The paper presents some computationally efficient predictors of the cost-effectiveness of the two main classes of selective regression testing approaches. These predictors are computed from data about the coverage relationship between the system under test and its test suite. The paper then describes case studies in which these predictors were used to predict the cost-effectiveness of applying two different regression testing strategies to two software systems. In one case study, the TESTTUBE method selected an average of 88.1 percent of the available test cases in each version, while the predictor predicted that 87.3 percent of the test cases would be selected on average     

Comparing the effectiveness of several modeling methods for fault prediction

We compare the effectiveness of four modeling methods—negative binomial regression, recursive partitioning, random forests and Bayesian additive regression trees—for predicting the files likely to contain the most faults for 28 to 35 releases of three large industrial software systems. Predictor variables included lines of code, file age, faults in the previous release, changes in the previous two releases, and programming language. To compare the effectiveness of the different models, we use two metrics—the percent of faults contained in the top 20% of files identified by the model, and a new, more general metric, the fault-percentile-average. The negative binomial regression and random forests models performed significantly better than recursive partitioning and Bayesian additive regression trees, as assessed by either of the metrics. For each of the three systems, the negative binomial and random forests models identified 20% of the files in each release that contained an average of 76% to 94% of the faults.     

The automatic generation of load test suites and the assessment ofthe resulting software

Three automatic test case generation algorithms intended to test the resource allocation mechanisms of telecommunications software systems are introduced. Although these techniques were specifically designed for testing telecommunications software, they can be used to generate test cases for any software system that is modelable by a Markov chain provided operational profile data can either be collected or estimated. These algorithms have been used successfully to perform load testing for several real industrial software systems. Experience generating test suites for five such systems is presented. Early experience with the algorithms indicate that they are highly effective at detecting subtle faults that would have been likely to be missed if load testing had been done in the more traditional way, using hand-crafted test cases. A domain-based reliability measure is applied to systems after the load testing algorithms have been used to generate test data. Data are presented for the same five industrial telecommunications systems in order to track the reliability as a function of the degree of system degradation experienced     

Reliability testing of rule-based systems

Rule-based software systems are becoming more common in industrial settings, particularly to monitor and control large, real-time systems. The authors describe an algorithm for reliability testing of rule-based systems and their experience using it to test an industrial network surveillance system     

A formal analysis of the fault-detecting ability of testing methods

Several relationships between software testing criteria, each induced by a relation between the corresponding multisets of subdomains, are examined. The authors discuss whether for each relation R and each pair of criteria, C₁ and C₂ , R(C₁, C₂) guarantees that C₁ is better at detecting faults than C₂ according to various probabilistic measures of fault-detecting ability. It is shown that the fact that C ₁ subsumes C₂ does not guarantee that C₁ is better at detecting faults. Relations that strengthen the subsumption relation and that have more bearing on fault-detecting ability are introduced     

Provable improvements on branch testing

This paper compares the fault-detecting ability of several software test data adequacy criteria. It has previously been shown that if C₁ properly covers C₂, then C₁ is guaranteed to be better at detecting faults than C₂, in the following sense: a test suite selected by independent random selection of one test case from each subdomain induced by C₁ is at least as likely to detect a fault as a test suite similarly selected using C₂. In contrast, if C₁ subsumes but does not properly cover C₂, this is not necessarily the case. These results are used to compare a number of criteria, including several that have been proposed as stronger alternatives to branch testing. We compare the relative fault-detecting ability of data flow testing, mutation testing, and the condition-coverage techniques, to branch testing, showing that most of the criteria examined are guaranteed to be better than branch testing according to two probabilistic measures. We also show that there are criteria that can sometimes be poorer at detecting faults than substantially less expensive criteria     

The cost of data flow testing: an empirical study

A family of test data adequacy criteria employing data-flow information was previously proposed, and a theoretical complexity analysis was performed. The author describes an empirical study to determine the actual cost of using these criteria. The aim is to establish the practical usefulness of these criteria in testing software and provide a basis for predicting the amount of testing needed for a given program. The first goal of the study is to confirm the belief that the family of software testing criteria considered is practical to use. An attempt is made to show that even as the program size increases, the amount of testing, expressed in terms of the number of test cases sufficient to satisfy a given criterion, remains modest. Several ways of evaluating this hypothesis are explored. The second goal is to provide the prospective user of these criteria with a way of predicting the number of test cases that will be needed to satisfy a given criterion for a given program. This provides testers with a basis for selecting the most comprehensive criterion that they can expect to satisfy. Several plausible bases for such a prediction are considered