Selecting a Cost-Effective Test Case Prioritization Technique

Regression testing is an expensive testing process used to validate modified software and detect whether new faults have been introduced into previously tested code. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One goal of prioritization is to increase a test suite's rate of fault detection. Previous empirical studies have shown that several prioritization techniques can significantly improve rate of fault detection, but these studies have also shown that the effectiveness of these techniques varies considerably across various attributes of the program, test suites, and modifications being considered. This variation makes it difficult for a practitioner to choose an appropriate prioritization technique for a given testing scenario. To address this problem, we analyze the fault detection rates that result from applying several different prioritization techniques to several programs and modified versions. The results of our analyses provide insights into which types of prioritization techniques are and are not appropriate under specific testing scenarios, and the conditions under which they are or are not appropriate. Our analysis approach can also be used by other researchers or practitioners to determine the prioritization techniques appropriate to other workloads.     

The (in)accuracy of novice rover operators' perception of obstacle height from monoscopic images

Researchers have previously described a mobile robot, or rover, operator's difficulty in accurately perceiving the rover's tilt and roll, which can lead to rollover accidents. Safe mobile robot navigation and effective mission planning also require an operator to accurately interpret and understand the geometry and scale of features in the rover's environment. This work presents an experiment that measures an observer's ability to estimate height of distant (5-15 m) obstacles given an accurate local model (e.g., within 0-5 m of the rover), a panoramic image, and a physical mock-up of the local terrain. The experimental conditions were intended to represent a best-case scenario for a stopped rover equipped with short base-line stereoscopic cameras. The participants' task was to extrapolate the well-modeled local geometry to monoscopic images of the more distant terrain. The experiment compared two estimation techniques. With the first technique, each observer physically indicated his or her direct estimates of the obstacle distance and height. With the second estimation technique, which we call horizon analysis, the observer indicated the position of the top and bottom of each rock on an image and the height was calculated by measuring the visual angle between the theoretical horizon and the points indicated by the observer. The direct estimation technique overestimated the height of the rocks by an average of 190%; the horizon analysis technique overestimated by 80%. The results suggest that even when provided with a rich set of supplementary and context information, rover operators have significant difficulty in vertically perceiving the scale of distant terrain. The results also suggest that horizon analysis is a more accurate method for determining the height of distant rover navigation obstacles, when the local terrain is nearly level.     

Trace anomalies as precursors of field failures: an empirical study

Reproducing and learning from failures in deployed software is costly and difficult. Those activities can be facilitated, however, if the circumstances leading to a failure can be recognized and properly captured. To anticipate failures we propose to monitor system field behavior for simple trace instances that deviate from a baseline behavior experienced in-house. In this work, we empirically investigate the effectiveness of various simple anomaly detection schemes to identify the conditions that precede failures in deployed software. The results of our experiment provide a preliminary assessment of these schemes, and expose the tradeoffs between different anomaly detection algorithms applied to several types of observable attributes under varying levels of in-house testing.

Understanding the effects of changes on the cost‐effectiveness of regression testing techniques

Regression testing is an expensive testing process used to validate modified software. Regression test selection and test‐case prioritization can reduce the costs of regression testing by selecting a subset of test cases for execution, or scheduling test cases to meet testing objectives better. The cost‐effectiveness of these techniques can vary widely, however, and one cause of this variance is the type and magnitude of changes made in producing a new software version. Engineers unaware of the causes and effects of this variance can make poor choices in designing change integration processes, selecting inappropriate regression testing techniques, designing excessively expensive regression test suites and making unnecessarily costly changes. Engineers aware of causal factors can perform regression testing more cost‐effectively. This article reports the results of an embedded multiple case study investigating the modifications made in the evolution of four software systems and their impact on regression testing techniques. The results of this study expose tradeoffs and constraints that affect the success of techniques and provide guidelines for designing and managing regression testing processes. Copyright © 2003 John Wiley & Sons, Ltd.