A fuzzy logic based approach for phase-wise software defects prediction using software metrics

ContextThe software defect prediction during software development has recently attracted the attention of many researchers. The software defect density indicator prediction in each phase of software development life cycle (SDLC) is desirable for developing a reliable software product. Software defect prediction at the end of testing phase may not be more beneficial because the changes need to be performed in the previous phases of SDLC may require huge amount of money and effort to be spent in order to achieve target software quality. Therefore, phase-wise software defect density indicator prediction model is of great importance.ObjectiveIn this paper, a fuzzy logic based phase-wise software defect prediction model is proposed using the top most reliability relevant metrics of the each phase of the SDLC.MethodIn the proposed model, defect density indicator in requirement analysis, design, coding and testing phase is predicted using nine software metrics of these four phases. The defect density indicator metric predicted at the end of the each phase is also taken as an input to the next phase. Software metrics are assessed in linguistic terms and fuzzy inference system has been employed to develop the model.ResultsThe predictive accuracy of the proposed model is validated using twenty real software project data. Validation results are satisfactory. Measures based on the mean magnitude of relative error and balanced mean magnitude of relative error decrease significantly as the software project size increases.ConclusionIn this paper, a fuzzy logic based model is proposed for predicting software defect density indicator at each phase of the SDLC. The predicted defects of twenty different software projects are found very near to the actual defects detected during testing. The predicted defect density indicators are very helpful to analyze the defect severity in different artifacts of SDLC of a software project.     

Event-driven web application testing based on model-based mutation testing

ContextEvent-Driven Software (EDS) is a class of software whose behavior is driven by incoming events. Web and desktop applications that respond to user-initiated events on their Graphical User Interface (GUI), or embedded software responding to events and signals received from the equipment in its operating environment are examples of EDS. Testing EDS poses great challenges to software testers. One of these challenges is the need to generate a huge number of possible event sequences that could sufficiently cover the EDS’s state space.ObjectiveIn this paper, we introduce a new six-stage testing procedure for event-driven web applications to overcome EDS testing challenges.MethodThe stages of the testing procedure include dividing the application based on its structure, creating functional graphs for each section, creating mutants from functional graphs, choosing coverage criteria to produce test paths, merging event sequences to make longer ones, and deriving and running test cases. We have analyzed our proposed testing procedure with the help of four metrics consisting of Fault Detection Density (FDD), Fault Detection Effectiveness (FDE), Mutation Score, and Unique Fault.ResultsUsing this procedure, we have prepared prioritized test cases and also discovered a list of unique faults by running the suggested test cases on a sample real-world web application called Academic E-mail System.ConclusionWe propose that our suggested testing procedure has some advantages such as creating functional graphs with requirements document, resolving the problem of removing infeasible test cases with these graphs and conditions on the “add edge” operator before creating mutants. But the suggested testing procedure, like any other method, had some drawbacks. Because most of the stages in the approach were performed manually, the testing time was increased.     

A correlation study between automated program repair and test-suite metrics

Automated program repair is increasingly gaining traction, due to its potential to reduce debugging cost greatly. The feasibility of automated program repair has been shown in a number of works, and the research focus is gradually shifting toward the quality of generated patches. One promising direction is to control the quality of generated patches by controlling the quality of test-suites used for automated program repair. In this paper, we ask the following research question: “Can traditional test-suite metrics proposed for the purpose of software testing also be used for the purpose of automated program repair?” We empirically investigate whether traditional test-suite metrics such as statement/branch coverage and mutation score are effective in controlling the reliability of generated repairs (the likelihood that repairs cause regression errors). We conduct the largest-scale experiments of this kind to date with real-world software, and for the first time perform a correlation study between various test-suite metrics and the reliability of generated repairs. Our results show that in general, with the increase of traditional test suite metrics, the reliability of repairs tend to increase. In particular, such a trend is most strongly observed in statement coverage. Our results imply that the traditional test suite metrics proposed for software testing can also be used for automated program repair to improve the reliability of repairs.     

An experimental study of adaptive testing for software reliability assessment

Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previous work has shown that adaptive testing can be formulated and guided in theory to minimize the variance of an unbiased software reliability estimator and to achieve optimal software reliability assessment. In this paper, we present an experimental study of adaptive testing for software reliability assessment, where the adaptive testing strategy, the random testing strategy and the operational profile based testing strategy were applied to the Space program in four experiments. The experimental results demonstrate that the adaptive testing strategy can really work in practice and may noticeably outperform the other two. Therefore, the adaptive testing strategy can serve as a preferable alternative to the random testing strategy and the operational profile based testing strategy if high confidence in the reliability estimates is required or the real-world operational profile of the software under test cannot be accurately identified.     

Test case generation for the task tree type of architecture

ContextEmerging multicores and clusters of multicores that may operate in parallel have set a new challenge – development of massively parallel software composed of thousands of loosely coupled or even completely independent threads/processes, such as MapReduce and Java 3.0 workers, or Erlang processes, respectively. Testing and verification is a critical phase in the development of such software products.ObjectiveGenerating test cases based on operational profiles and certifying declared operational reliability figure of the given software product is a well-established process for the sequential type of software. This paper proposes an adaptation of that process for a class of massively parallel software – large-scale task trees.MethodThe proposed method uses statistical usage testing and operational reliability estimation based on operational profiles and novel test suite quality indicators, namely the percentage of different task trees and the percentage of different paths.ResultsAs an example, the proposed method is applied to operational reliability certification of a parallel software infrastructure named the TaskTreeExecutor. The paper proposes an algorithm for generating random task trees to enable that application. Test runs in the experiments involved hundreds and thousands of Win32/Linux threads thus demonstrating scalability of the proposed approach. For practitioners, the most useful result presented is the method for determining the number of task trees and the number of paths, which are needed to certify the given operational reliability of a software product. The practitioners may also use the proposed coverage metrics to measure the quality of automatically generated test suite.ConclusionThis paper provides a useful solution for the test case generation that enables the operational reliability certification process for a class of massively parallel software called the large-scale task trees. The usefulness of this solution was demonstrated by a case study – operational reliability certification of the real parallel software product.     

Automated test generation technique for aspectual features in AspectJ

ContextAspect-oriented programming (AOP) has been promoted as a means for handling the modularization of software systems by raising the abstraction level and reducing the scattering and tangling of crosscutting concerns. Studies from literature have shown the usefulness and application of AOP across various fields of research and domains. Despite this, research shows that AOP is currently used in a cautious way due to its natural impact on testability and maintainability.ObjectiveTo realize the benefits of AOP and to increase its adoption, aspects developed using AOP should be subjected to automated testing. Automated testing, as one of the most pressing needs of the software industry to reduce both effort and costs in assuring correctness, is a delicate issue in testing aspect-oriented programs that still requires advancement and has a way to go before maturity.MethodPrevious attempts and studies in automated test generation process for aspect-oriented programs have been very limited. This paper proposes a rigorous automated test generation technique, called RAMBUTANS, with its tool support based on guided random testing for the AspectJ programs.ResultsThe paper reports the results of a thorough empirical study of 9 AspectJ benchmark programs, including non-trivial and larger software, by means of mutation analysis to compare RAMBUTANS and the four existing automated AOP testing approaches for testing aspects in terms of fault detection effectiveness and test effort efficiency. The results of the experiment and statistical tests supplemented by effect size measures presented evidence of the effectiveness and efficiency of the proposed technique at 99% confidence level (i.e. p < 0.01).ConclusionThe study showed that the resulting randomized tests were reasonably good for AOP testing, thus the proposed technique could be worth using as an effective and efficient AOP-specific automated test generation technique.     

Spectral‐based fault localization using hyperbolic function

Debugging is crucial for producing reliable software. One of the effective bug localization techniques is spectral‐based fault localization. It tries to locate a buggy statement by applying an evaluation metric to program spectra and ranking program components on the basis of the score it computes. Here, we propose a restricted class of “hyperbolic” metrics, with a small number of numeric parameters. This class of functions is based on past theoretical and empirical results. We show that optimization methods such as genetic programming and simulated annealing can reliably discover effective metrics over a wide range of data sets of program spectra. We evaluate the performance for both real programs and model programs with single bugs, multiple bugs, “deterministic” bugs, and nondeterministic bugs and find that the proposed class of metrics performs as well as or better than the previous best‐performing metrics over a broad range of data.     

Applying a smoothing filter to improve IR-based traceability recovery processes: An empirical investigation

ContextTraceability relations among software artifacts often tend to be missing, outdated, or lost. For this reason, various traceability recovery approaches—based on Information Retrieval (IR) techniques—have been proposed. The performances of such approaches are often influenced by “noise” contained in software artifacts (e.g., recurring words in document templates or other words that do not contribute to the retrieval itself).AimAs a complement and alternative to stop word removal approaches, this paper proposes the use of a smoothing filter to remove “noise” from the textual corpus of artifacts to be traced.MethodWe evaluate the effect of a smoothing filter in traceability recovery tasks involving different kinds of artifacts from five software projects, and applying three different IR methods, namely Vector Space Models, Latent Semantic Indexing, and Jensen–Shannon similarity model.ResultsOur study indicates that, with the exception of some specific kinds of artifacts (i.e., tracing test cases to source code) the proposed approach is able to significantly improve the performances of traceability recovery, and to remove “noise” that simple stop word filters cannot remove.ConclusionsThe obtained results not only help to develop traceability recovery approaches able to work in presence of noisy artifacts, but also suggest that smoothing filters can be used to improve performances of other software engineering approaches based on textual analysis.     

An approximation of the activity duration distribution in PERT

PERT is widely used as a tool for managing large-scale projects. The traditional PERT approach uses the beta distribution as the distribution of activity duration and estimates the mean and the variance of activity duration based on the “pessimistic”, “optimistic” and “most likely” time estimates. Several authors have modified the original three point PERT estimators to improve the accuracy of the estimates. This article proposes new approximations for the mean and the variance of activity based on “pessimistic”, “optimistic” and “most likely” time estimates. By numerical comparison with actual values, the proposal is shown as more accurate than the original PERT estimates and its modifications. Another advantage of the proposed approximation is that it requires no assumptions about the parameters of the beta distribution as in the case of existing ones.Scope and purposeThe traditional PERT model uses beta distribution as the distribution of activity duration and estimates the mean and the variance of activity duration using “pessimistic”, “optimistic” and “most likely” time estimates proposed by an expert. In the past several authors have modified the original PERT estimators to improve the accuracy. This article proposes new approximations for the mean and the variance of activity duration which are more accurate than the original PERT estimates and their modifications. Another advantage of the proposal is that it is not based on any assumptions about the parameters of the beta distribution as in the case of the existing ones.     

Defining a test coverage criterion for model-level testing of FBD programs

ContextThe Programmable Logic Controller (PLC) is being integrated into the automation and control of computer systems in safety–critical domains at an increasing rate. Thoroughly testing such software to ensure safety is crucial. Function Block Diagram (FBD) is a popular data-flow programming language for PLC. Current practice often involves translating an FBD program into an equivalent C program for testing. Little research has been conducted on coverage of direct testing a data-flow program, such as an FBD program, at the model level. There are no commonly accepted structural test coverage criteria for data-flow programs. The objective of this study is to develop effective structural test coverage criterion for testing model-level FBD programs. The proposed testing scheme can be used to detect mutation errors at the logical function level.ObjectiveThe purpose of this study is to design a new test coverage criterion that can directly test FBD programs and effectively detect logical function mutation errors.MethodA complete test set for each function and function block in an FBD program are defined. Moreover, this method augments the data-flow path concept with a sensitivity check to avoid fault masking and effectively detect logical function mutation errors.ResultsPreliminary experiments show that this test coverage criterion is comprehensive and effective for error detection.ConclusionThe proposed coverage criterion is general and can be applied to real cases to improve the quality of data-flow program design.     

An architectural model for software testing lesson learned systems

ContextSoftware testing is a key aspect of software reliability and quality assurance in a context where software development constantly has to overcome mammoth challenges in a continuously changing environment. One of the characteristics of software testing is that it has a large intellectual capital component and can thus benefit from the use of the experience gained from past projects. Software testing can, then, potentially benefit from solutions provided by the knowledge management discipline. There are in fact a number of proposals concerning effective knowledge management related to several software engineering processes.ObjectiveWe defend the use of a lesson learned system for software testing. The reason is that such a system is an effective knowledge management resource enabling testers and managers to take advantage of the experience locked away in the brains of the testers. To do this, the experience has to be gathered, disseminated and reused.MethodAfter analyzing the proposals for managing software testing experience, significant weaknesses have been detected in the current systems of this type. The architectural model proposed here for lesson learned systems is designed to try to avoid these weaknesses. This model (i) defines the structure of the software testing lessons learned; (ii) sets up procedures for lesson learned management; and (iii) supports the design of software tools to manage the lessons learned.ResultsA different approach, based on the management of the lessons learned that software testing engineers gather from everyday experience, with two basic goals: usefulness and applicability.ConclusionThe architectural model proposed here lays the groundwork to overcome the obstacles to sharing and reusing experience gained in the software testing and test management. As such, it provides guidance for developing software testing lesson learned systems.     

Dynamic stopping criteria for search-based test data generation for path testing

ContextEvolutionary algorithms have proved to be successful for generating test data for path coverage testing. However in this approach, the set of target paths to be covered may include some that are infeasible. It is impossible to find test data to cover those paths. Rather than searching indefinitely, or until a fixed limit of generations is reached, it would be desirable to stop searching as soon it seems likely that feasible paths have been covered and all remaining un-covered target paths are infeasible.ObjectiveThe objective is to develop criteria to halt the evolutionary test data generation process as soon as it seems not worth continuing, without compromising testing confidence level.MethodDrawing on software reliability growth models as an analogy, this paper proposes and evaluates a method for determining when it is no longer worthwhile to continue searching for test data to cover un-covered target paths. We outline the method, its key parameters, and how it can be used as the basis for different decision rules for early termination of a search. Twenty-one test programs from the SBSE path testing literature are used to evaluate the method.ResultsCompared to searching for a standard number of generations, an average of 30–75% of total computation was avoided in test programs with infeasible paths, and no feasible paths were missed due to early termination. The extra computation in programs with no infeasible paths was negligible.ConclusionsThe method is effective and efficient. It avoids the need to specify a limit on the number of generations for searching. It can help to overcome problems caused by infeasible paths in search-based test data generation for path testing.     

Enhancing mirror adaptive random testing through dynamic partitioning

ContextAdaptive random testing (ART), originally proposed as an enhancement of random testing, is often criticized for the high computation overhead of many ART algorithms. Mirror ART (MART) is a novel approach that can be generally applied to improve the efficiency of various ART algorithms based on the combination of “divide-and-conquer” and “heuristic” strategies.ObjectiveThe computation overhead of the existing MART methods is actually on the same order of magnitude as that of the original ART algorithms. In this paper, we aim to further decrease the order of computation overhead for MART.MethodWe conjecture that the mirroring scheme in MART should be dynamic instead of static to deliver a higher efficiency. We thus propose a new approach, namely dynamic mirror ART (DMART), which incrementally partitions the input domain and adopts new mirror functions.ResultsOur simulations demonstrate that the new DMART approach delivers comparable failure-detection effectiveness as the original MART and ART algorithms while having much lower computation overhead. The experimental studies further show that the new approach also delivers a better and more reliable performance on programs with failure-unrelated parameters.ConclusionIn general, DMART is much more cost-effective than MART. Since its mirroring scheme is independent of concrete ART algorithms, DMART can be generally applied to improve the cost-effectiveness of various ART algorithms.     

Approach for estimating similarity between procedures in differently compiled binaries

ContextDetection of an unauthorized use of a software library is a clone detection problem that in case of commercial products has additional complexity due to the fact that only binary code is available.ObjectiveThe goal of this paper is to propose an approach for estimating the level of similarity between the procedures originating from different binary codes. The assumption is that the clones in the binary codes come from the use of a common software library that may be compiled with different toolsets.MethodThe approach uses a set of software metrics adapted from the high level languages and it also extends the set with new metrics that take into account syntactical changes that are introduced by the usage of different toolsets and optimizations. Moreover, the approach compares metric values and introduces transformers and formulas that can use training data for production of measure of similarities between the two procedures in binary codes. The approach has been evaluated on programs from STAMP benchmark and BusyBox tool, compiled with different toolsets in different modes.ResultsThe experiments with programs from STAMP benchmark show that detecting the same procedures recall can be up to 1.44 times higher using new metrics. Knowledge about the used compiling toolset can bring up to 2.28 times improvement in recall. The experiment with BusyBox tool shows 43% recall for 43% precision.ConclusionThe most useful newly proposed metrics are those that consider the frequency of arithmetic instructions, the number and frequency of occurrences for instructions, and the number of occurrences for target addresses in calls. The best way to combine the results of comparing metrics is to use a geometric mean or when previous knowledge is available, to use an arithmetic mean with appropriate transformer.     

专家系统规则库覆盖度量的研究及实践

针对专家系统规则库的软件测试和评价是保证专家系统质量与可靠性的重要环节。基于数据流的结构化测试技术,提出了一种规则库执行流图RBEF,用于描述规则库中不同规则之间相互连结、制约关系,进而给出了一组规则库覆盖度量准则和测试用例的设计方法,最后讨论了一个已实现并付诸于应用的基于RBEF的软件测试工具ESRTS。     

Mutation-oriented test data augmentation for GUI software fault localization

ContextFault localization lies at the heart of program debugging and often proceeds by contrasting the statistics of program constructs executed by passing and failing test cases. A vital issue here is how to obtain these “suitable” test cases. Techniques presented in the literature mostly assume the existence of a large test suite a priori. However, developers often encounter situations where a failure occurs, but where no or no appropriate test suite is available for use to localize the fault.ObjectiveThis paper aims to alleviate this key limitation of traditional fault localization techniques for GUI software particularly, namely, it aims at enabling cost-effective fault localization process for GUI software in the described scenario.MethodTo address this scenario, we propose a mutation-oriented test data augmentation technique, which actually is directed by the “similarity” criterion in GUI software’s test case context towards the generation of test suite with excellent fault localization capabilities. More specifically, the technique mainly uses four proposed novel mutation operators to iteratively mutate some failing GUI test cases’ event sequences to derive new test cases potentially useful to localize the specific encountered fault. We then compare the fault localization performance of the test suite generated using this technique with that of an original provided large event-pair adequate test suite on some GUI applications.ResultsThe results indicate that the proposed technique is capable of generating a test suite that has comparable, if not better, fault localization effectiveness to the event-pair adequate test suite, but it is much smaller and it is generated immediately once a failure is encountered by developers.ConclusionIt is concluded that the proposed technique can truly enable quick-start cost-effective fault localization process under the investigated all-too-common scenario, greatly alleviating one key limitation of traditional fault localization techniques and prompting the test–diagnose–repair cycle.     

NONLINEAR ROBUST CONTROL FOR PARALLEL AC/DC TRANSMISSION SYSTEMS: A NEW ADAPTIVE BACK-STEPPING APPROACH

ABSTRACT

By utilizing the controllability of High Voltage Direct Current (HVDC), which means that the power delivered can be modulated, to improve the stability and operation performances of the parallel AC/DC transmission system, a new adaptive back-stepping approach for this system is developed. Compared with the existing controller based on “classical” adaptive back-stepping, the approach does not follow the classical certainty-equivalence principle. We introduce this approach, for the first time, into parallel AC/DC systems containing unknown parameters and present a novel parameter estimator and dynamics feedback controller. Besides the preserving useful nonlinearities and the real-time estimation of uncertainty parameter, the proposed approach possesses better performances with respect to the response of the system and the speed of adaptation. Simulation results demonstrate that the proposed approach is better than the design based on “classical” adaptive back-stepping in terms of properties of stability and parameter estimation and that it recovers the performance of the “full-information” controller, which is obtained by assuming that the parameters are known and apply standard back-stepping, hence it will be an alternative to practice engineering and applications.     

Optimal and adaptive testing for software reliability assessment

Optimal software testing is concerned with how to test software such that the underlying testing goal is achieved in an optimal manner. Our previous work shows that the optimal testing problem for software reliability growth can be treated as closed-loop or feedback control problem, where the software under test serves as a controlled object and the software testing strategy serves as the corresponding controller. More specifically, the software under test is modeled as controlled Markov chains (CMCs) and the control theory of Markov chains is used to synthesize the required optimal testing strategy. In this paper, we show that software reliability assessment can be treated as a feedback control problem and the CMC approach is also applicable to dealing with the optimal testing problem for software reliability assessment. In this problem, the code of the software under test is frozen and the software testing process is optimized in the sense that the variance of the software reliability estimator is minimized. An adaptive software testing strategy is proposed that uses the testing data collected on-line to estimate the required parameters and selects next test cases. Simulation results show that the proposed adaptive software testing strategy can really work in the sense that the resulting variance of the software reliability estimate is much smaller than that resulting from the random testing strategies. The work presented in this paper is a contribution to the new area of software cybernetics that explores the interplay between software and control.     

Graphical user interface (GUI) testing: Systematic mapping and repository

ContextGUI testing is system testing of a software that has a graphical-user interface (GUI) front-end. Because system testing entails that the entire software system, including the user interface, be tested as a whole, during GUI testing, test cases—modeled as sequences of user input events—are developed and executed on the software by exercising the GUI’s widgets (e.g., text boxes and clickable buttons). More than 230 articles have appeared in the area of GUI testing since 1991.ObjectiveIn this paper, we study this existing body of knowledge using a systematic mapping (SM).MethodThe SM is conducted using the guidelines proposed by Petersen et al. We pose three sets of research questions. We define selection and exclusion criteria. From the initial pool of 230 articles, published in years 1991–2011, our final pool consisted of 136 articles. We systematically develop a classification scheme and map the selected articles to this scheme.ResultsWe present two types of results. First, we report the demographics and bibliometrics trends in this domain, including: top-cited articles, active researchers, top venues, and active countries in this research area. Moreover, we derive the trends, for instance, in terms of types of articles, sources of information to derive test cases, types of evaluations used in articles, etc. Our second major result is a publicly-accessible repository that contains all our mapping data. We plan to update this repository on a regular basis, making it a “live” resource for all researchers.ConclusionOur SM provides an overview of existing GUI testing approaches and helps spot areas in the field that require more attention from the research community. For example, much work is needed to connect academic model-based techniques with commercially available tools. To this end, studies are needed to compare the state-of-the-art in GUI testing in academic techniques and industrial tools.