Program testing by specification mutation

Both theoretical and empirical arguments suggest that specifications and implementations are equally important sources of information for generating test cases. Nevertheless, the majority of test generation procedures described in the literature deal only with the program source, ignoring specifications. In this paper we outline a procedure for measuring test case effectiveness using specifications given in predicate calculus form. This method is similar to the mutation analysis method of testing programs.     

A conceptual model of software testing

As code is executed correctly under test, confidence in the correctness of the code increases. In this context, an intuitive conceptual model of the process of software testing which draws upon experience gained with mutation analysis is presented. The model is used to explain how the testing of one path can influence confidence in other (possibly unexecuted) paths. It is also discussed in relation to software reliability and systematic structural testing, and is shown to be consistent with observations made during these forms of testing.     

Concerning ordered mutation testing of relational operators

The aim of this paper is to correct a misapprehension concerning mutation testing of relational operators in imperative programming languages. An apparently plausible, but fallacious argument is stated concerning the order in which mutants are best considered. Subsequently, a counter-example is presented which disproves the argument.     

现代生物技术在环境检测中的应用

21世纪以来,随着我国电子通信等技术的迅猛成长,使得我国经济在相关方面发展上得到了科技保障.现代生物技术基于生命科学的发展逐渐有了雏形.它主要是研究生物相关组织或者有机体,把研究结果有效地运用现实生活中去,从而让企业完成新品的开发.如今,我国相关专家对现代生物技术的不断探索与发现,该技术可以运用在多个领域与行业.     

Evolutionary mutation testing

Context

Mutation testing is a testing technique that has been applied successfully to several programming languages. However, it is often regarded as computationally expensive, so several refinements have been proposed to reduce its cost. Moreover, WS-BPEL compositions are being widely adopted by developers, but present new challenges for testing, since they can take much longer to run than traditional programs of the same size. Therefore, it is interesting to reduce the number of mutants required.

Objective

We present Evolutionary Mutation Testing (EMT), a novel mutant reduction technique for finding mutants that help derive new test cases that improve the quality of the initial test suite. It uses evolutionary algorithms to reduce the number of mutants that are generated and executed with respect to the exhaustive execution of all possible mutants, keeping as many difficult to kill and potentially equivalent mutants (strong mutants) as possible in the reduced set.

Method

To evaluate EMT we have developed GAmera, a mutation testing system powered by a co-evolutive genetic algorithm. We have applied this system to three WS-BPEL compositions to estimate its effectiveness, comparing it with random selection.

Results

The results obtained experimentally show that EMT can select all strong mutants generating 15% less mutants than random selection in over 20% less time for complex compositions. When generating a percentage of all mutants, EMT finds on average more strong mutants than random selection. This has been confirmed to be statistically significant within a 99.9% confidence interval.

Conclusions

EMT has reduced for the three tested compositions the number of mutants required to select those which are useful to derive new test cases that improve the quality of the test suite. The directed search performed by EMT makes it more effective than random selection, especially as compositions become more complex and the search space widens.     

Methods of detection of program bugs in computer systems

A problem of information protection in computer systems, namely, protection from program bugs, is considered. The existing methods of detection of viruses, program bugs, and other destructive program means are considered. The stages of bug life cycle are shown. A safety filter is proposed, i.e., a protection system based on an on-line multiparameter check of program execution. Translated from Kibernetika i Sistemnyi Analiz, No. 4, pp. 184–188, July–August, 1999.     

Effects of computer-based testing on test performance and testing motivation

Computer-based testing (CBT) is a green computing strategies used to reduce paper consumption. However, some scholars have questioned the effectiveness of CBT and suggested for conducting systematic studies on CBT to carefully check its reliability and validity before opting for it. Recently, some studies have evaluated the effectiveness of CBT by comparing it with paper–pencil testing (PPT), and the findings were inconsistent. Besides this, most of these studies have been conducted using quasi or basic experimental designs without identifying testing effects on test takers. The limitation of these design is testing effects might occur when a participant in the control or treatment group is tested at least twice on a same test, and it influences the outcomes of taking a posttest, therefore, the findings might be misinterpreted. This study employed a Solomon four-group experimental design (2 × 3 factorial design) on a group of student teachers (n = 140) to compare CBT and PPT on test performance (test scores), testing time and testing motivation. Results indicated that the CBT mode is more reliable in terms of internal and external validity. The CBT significantly reduced testing time and developed stronger self-efficacy, intrinsic and social testing motivation in the participants.     

A mutation carol: Past, present and future

Context

The field of mutation analysis has been growing, both in the number of published papers and the number of active researchers. This special issue provides a sampling of recent advances and ideas. But do all the new researchers know where we started?

Objective

To imagine where we are going, we must first know where we are. To understand where we are, we must know where we have been. This paper reviews past mutation analysis research, considers the present, then imagines possible future directions.

Method

A retrospective study of past trends lets us the ability to see the current state of mutation research in a clear context, allowing us to imagine and then create future vectors.

Results

The value of mutation has greatly expanded since the early view of mutation as an expensive way to unit test subroutines. Our understanding of what mutation is and how it can help has become much deeper and broader.

Conclusion

Mutation analysis has been around for 35 years, but we are just now beginning to see its full potential. The papers in this issue and future mutation workshops will eventually allow us to realize this potential.     

Patterns: from system design to software testing

Design patterns are used extensively in the design of software systems. Patterns codify effective solutions for recurring design problems and allow software engineers to reuse these solutions, tailoring them appropriately to their particular applications, rather than reinventing them from scratch. In this paper, we consider the following question: How can system designers and implementers test whether their systems, as implemented, are faithful to the requirements of the patterns used in their design? A key consideration underlying our work is that the testing approach should enable us, in testing whether a particular pattern P has been correctly implemented in different systems designed using P, to reuse the common parts of this effort rather than having to do it from scratch for each system. Thus in the approach we present, corresponding to each pattern P, there is a set of pattern test case templates (PTCTs). A PTCT codifies a reusable test case structure designed to identify defects associated with applications of P in all systems designed using P. Next we present a process using which, given a system designed using P, the system tester can generate a test suite from the PTCTs for P that can be used to test the particular system for bugs in the implementation of P in that system. This allows the tester to tailor the PTCTs for P to the needs of the particular system by specifying a set of specialization rules that are designed to reflect the scenarios in which the defects codified in this set of PTCTs are likely to manifest themselves in the particular system. We illustrate the approach using the Observer pattern.     

Analyzing the relationships between inspections and testing to provide a software testing focus

ContextQuality assurance effort, especially testing effort, is frequently a major cost factor during software development. Consequently, one major goal is often to reduce testing effort. One promising way to improve the effectiveness and efficiency of software quality assurance is the use of data from early defect detection activities to provide a software testing focus. Studies indicate that using a combination of early defect data and other product data to focus testing activities outperforms the use of other product data only. One of the key challenges is that the use of data from early defect detection activities (such as inspections) to focus testing requires a thorough understanding of the relationships between these early defect detection activities and testing. An aggravating factor is that these relationships are highly context-specific and need to be evaluated for concrete environments.ObjectiveThe underlying goal of this paper is to help companies get a better understanding of these relationships for their own environment, and to provide them with a methodology for finding relationships in their own environments.MethodThis article compares three different strategies for evaluating assumed relationships between inspections and testing. We compare a confidence counter, different quality classes, and the F-measure including precision and recall.ResultsOne result of this case-study-based comparison is that evaluations based on the aggregated F-measures are more suitable for industry environments than evaluations based on a confidence counter. Moreover, they provide more detailed insights about the validity of the relationships.ConclusionWe have confirmed that inspection results are suitable data for controlling testing activities. Evaluated knowledge about relationships between inspections and testing can be used in the integrated inspection and testing approach In²Test to focus testing activities. Product data can be used in addition. However, the assumptions have to be evaluated in each new context.     

A mutation analysis tool for Java programs

Program mutation is a fault-based technique for measuring the effectiveness of test cases that, although powerful, is computationally expensive. The principal expense of mutation is that many faulty versions of the program under test, called mutants, must be created and repeatedly executed. This paper describes a tool, called JavaMut, that implements 26 traditional and object-oriented mutation operators for supporting mutation analysis of Java programs. The current version of that tool is based on syntactic analysis and reflection for implementing mutation operators. JavaMut is interactive; it provides a graphical user interface to make mutation analysis faster and less painful. Thanks to such automated tools, mutation analysis should be achieved within reasonable costs.     

Optimal testing resource allocation during module testing considering cost, testing effort and reliability

Software reliability is one of the most important quality attributes of commercial software. During software testing, software reliability growth models (SRGMs) are commonly used to describe the phenomenon of failure occurrence and/or fault removal which consequently enhancements software reliability. Large software systems are developed by integrating a number of relatively small and independent modules, which are tested independently during module testing phase. The amount of testing resource available is limited which is desired to be consumed judiciously so as to optimize the testing process. In this paper we formulate a resource allocation problem of minimizing the cost of software testing under available amount of testing resource, given a reliability constraint. We use a flexible SRGM considering testing effort which, depending upon the values of parameters, can describe either exponential or S-shaped failure pattern of software modules. A systematic and sequential Algorithm is proposed to solve the optimization problem formulated. Numerical examples are given to illustrate the formulation and solution procedures. Sensitivity analysis is performed to examine the behavior of some parameters of SRGM with most significant influence.     

Test Case Generation for Mutation-based Testing of Timeliness

Temporal correctness is crucial for real-time systems. Few methods exist to test temporal correctness and most methods used in practice are ad-hoc. A problem with testing real-time applications is the response-time dependency on the execution order of concurrent tasks. Execution order in turn depends on execution environment properties such as scheduling protocols, use of mutual exclusive resources as well as the point in time when stimuli is injected. Model based mutation testing has previously been proposed to determine the execution orders that need to be verified to increase confidence in timeliness. An effective way to automatically generate such test cases for dynamic real-time systems is still needed. This paper presents a method using heuristic-driven simulation to generate test cases.     

An investigation of statistical software testing

Software validation embodies two notions: fault removal and fault forecasting. Statistical testing involves exercising a piece of software by supplying it with input values that are randomly selected according to a defined probability distribution on its input domain. It can be used as a practical tool for revealing faults in a fault removal phase, and for assessing software dependability in a fault forecasting phase. In both of these, its efficiency is linked to the adequacy of the input probability distribution with respect to the test experiment goal. In this paper a mixed validation strategy combining deterministic and random test data is defined, and the theoretical and experimental work performed to support the strategy is reported. The quoted results relate to the unit testing of four real programs from the nuclear field. They confirm the high fault revealing power of statistical structural testing. Two main directions for further investigation of statistical testing are indicated by the reported work. These are described and solutions to the associated problems are outlined.     

Efficient multi-objective higher order mutation testing with genetic programming

It is said 90% of faults that survive manufacturer’s testing procedures are complex. That is, the corresponding bug fix contains multiple changes. Higher order mutation testing is used to study defect interactions and their impact on software testing for fault finding. We adopt a multi-objective Pareto optimal approach using Monte Carlo sampling, genetic algorithms and genetic programming to search for higher order mutants which are both hard-to-kill and realistic. The space of complex faults (higher order mutants) is much larger than that of traditional first order mutations which correspond to simple faults, nevertheless search based approaches make this scalable. The problems of non-determinism and efficiency are overcome. Easy to detect faults may become harder to detect when they interact and impossible to detect single faults may be brought to light when code contains two such faults. We use strong typing and BNF grammars in search based mutation testing to find examples of both in ancient heavily optimised every day C code.     

Higher Order Mutation Testing

This paper introduces a new paradigm for Mutation Testing, which we call Higher Order Mutation Testing (HOM Testing). Traditional Mutation Testing considers only first order mutants, created by the injection of a single fault. Often these first order mutants denote trivial faults that are easily killed. Higher order mutants are created by the insertion of two or more faults. The paper introduces the concept of a subsuming HOM; one that is harder to kill than the first order mutants from which it is constructed. By definition, subsuming HOMs denote subtle fault combinations. The paper reports the results of an empirical study of HOM Testing using 10 programs, including several non-trivial real-world subjects for which test suites are available.     

Interactive visualization for testing Markov Decision Processes: MDPVIS

Markov Decision Processes (MDPs) are a formulation for optimization problems in sequential decision making. Solving MDPs often requires implementing a simulator for optimization algorithms to invoke when updating decision making rules known as policies. The combination of simulator and optimizer are subject to failures of specification, implementation, integration, and optimization that may produce invalid policies. We present these failures as queries for a visual analytic system (MDPVIS). MDPVIS addresses three visualization research gaps. First, the data acquisition gap is addressed through a general simulator-visualization interface. Second, the data analysis gap is addressed through a generalized MDP information visualization. Finally, the cognition gap is addressed by exposing model components to the user. MDPVIS generalizes a visualization for wildfire management. We use that problem to illustrate MDPVIS and show the visualization's generality by connecting it to two reinforcement learning frameworks that implement many different MDPs of interest in the research community.