MSR4SM: Using topic models to effectively mining software repositories for software maintenance tasks

ContextMining software repositories has emerged as a research direction over the past decade, achieving substantial success in both research and practice to support various software maintenance tasks. Software repositories include bug repository, communication archives, source control repository, etc. When using these repositories to support software maintenance, inclusion of irrelevant information in each repository can lead to decreased effectiveness or even wrong results.ObjectiveThis article aims at selecting the relevant information from each of the repositories to improve effectiveness of software maintenance tasks.MethodFor a maintenance task at hand, maintainers need to implement the maintenance request on the current system. In this article, we propose an approach, MSR4SM, to extract the relevant information from each software repository based on the maintenance request and the current system. That is, if the information in a software repository is relevant to either the maintenance request or the current system, this information should be included to perform the current maintenance task. MSR4SM uses the topic model to extract the topics from these software repositories. Then, relevant information in each software repository is extracted based on the topics.ResultsMSR4SM is evaluated for two software maintenance tasks, feature location and change impact analysis, which are based on four subject systems, namely jEdit, ArgoUML, Rhino and KOffice. The empirical results show that the effectiveness of traditional software repositories based maintenance tasks can be greatly improved by MSR4SM.ConclusionsThere is a lot of irrelevant information in software repositories. Before we use them to implement a maintenance task at hand, we need to preprocess them. Then, the effectiveness of the software maintenance tasks can be improved.     

A systematic mapping study of web application testing

ContextThe Web has had a significant impact on all aspects of our society. As our society relies more and more on the Web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 147 papers in the area of web application testing, which have appeared between 2000 and 2011.ObjectiveAs this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends in this specialized field.MethodWe review and structure the body of knowledge related to web application testing through a systematic mapping (SM) study. As part of this study, we pose two sets of research questions, define selection and exclusion criteria, and systematically develop and refine a classification schema. In addition, we conduct a bibliometrics analysis of the papers included in our study.ResultsOur study includes a set of 79 papers (from the 147 retrieved papers) published in the area of web application testing between 2000 and 2011. We present the results of our systematic mapping study. Our mapping data is available through a publicly-accessible repository. We derive the observed trends, for instance, in terms of types of papers, sources of information to derive test cases, and types of evaluations used in papers. We also report the demographics and bibliometrics trends in this domain, including top-cited papers, active countries and researchers, and top venues in this research area.ConclusionWe discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our systematic mapping can help researchers to obtain an overview of existing web application testing approaches and indentify areas in the field that require more attention from the research community.     

Web application testing: A systematic literature review

ContextThe web has had a significant impact on all aspects of our society. As our society relies more and more on the web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 193 papers in the area of web application testing, which have appeared between 2000 and 2013.ObjectiveAs this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends and empirical evidence in this specialized field.MethodsWe systematically review the body of knowledge related to functional testing of web application through a systematic literature review (SLR) study. This SLR is a follow-up and complimentary study to a recent systematic mapping (SM) study that we conducted in this area. As part of this study, we pose three sets of research questions, define selection and exclusion criteria, and synthesize the empirical evidence in this area.ResultsOur pool of studies includes a set of 95 papers (from the 193 retrieved papers) published in the area of web application testing between 2000 and 2013. The data extracted during our SLR study is available through a publicly-accessible online repository. Among our results are the followings: (1) the list of test tools in this area and their capabilities, (2) the types of test models and fault models proposed in this domain, (3) the way the empirical studies in this area have been designed and reported, and (4) the state of empirical evidence and industrial relevance.ConclusionWe discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our SLR can help researchers to obtain an overview of existing web application testing approaches, fault models, tools, metrics and empirical evidence, and subsequently identify areas in the field that require more attention from the research community.     

Mutation-oriented test data augmentation for GUI software fault localization

ContextFault localization lies at the heart of program debugging and often proceeds by contrasting the statistics of program constructs executed by passing and failing test cases. A vital issue here is how to obtain these “suitable” test cases. Techniques presented in the literature mostly assume the existence of a large test suite a priori. However, developers often encounter situations where a failure occurs, but where no or no appropriate test suite is available for use to localize the fault.ObjectiveThis paper aims to alleviate this key limitation of traditional fault localization techniques for GUI software particularly, namely, it aims at enabling cost-effective fault localization process for GUI software in the described scenario.MethodTo address this scenario, we propose a mutation-oriented test data augmentation technique, which actually is directed by the “similarity” criterion in GUI software’s test case context towards the generation of test suite with excellent fault localization capabilities. More specifically, the technique mainly uses four proposed novel mutation operators to iteratively mutate some failing GUI test cases’ event sequences to derive new test cases potentially useful to localize the specific encountered fault. We then compare the fault localization performance of the test suite generated using this technique with that of an original provided large event-pair adequate test suite on some GUI applications.ResultsThe results indicate that the proposed technique is capable of generating a test suite that has comparable, if not better, fault localization effectiveness to the event-pair adequate test suite, but it is much smaller and it is generated immediately once a failure is encountered by developers.ConclusionIt is concluded that the proposed technique can truly enable quick-start cost-effective fault localization process under the investigated all-too-common scenario, greatly alleviating one key limitation of traditional fault localization techniques and prompting the test–diagnose–repair cycle.     

Software clone detection: A systematic review

ContextReusing software by means of copy and paste is a frequent activity in software development. The duplicated code is known as a software clone and the activity is known as code cloning. Software clones may lead to bug propagation and serious maintenance problems.ObjectiveThis study reports an extensive systematic literature review of software clones in general and software clone detection in particular.MethodWe used the standard systematic literature review method based on a comprehensive set of 213 articles from a total of 2039 articles published in 11 leading journals and 37 premier conferences and workshops.ResultsExisting literature about software clones is classified broadly into different categories. The importance of semantic clone detection and model based clone detection led to different classifications. Empirical evaluation of clone detection tools/techniques is presented. Clone management, its benefits and cross cutting nature is reported. Number of studies pertaining to nine different types of clones is reported. Thirteen intermediate representations and 24 match detection techniques are reported.ConclusionWe call for an increased awareness of the potential benefits of software clone management, and identify the need to develop semantic and model clone detection techniques. Recommendations are given for future research.     

Employing user profiles to test a new version of a GUI component in its context of use

Advancement in reusable component technology has had a significant impact on the development of complex graphical user interfaces (GUIs), which are front-ends to most of today’s software. Software developers can, with very little effort, integrate components into their software’s GUI. Problems, however, arise when new versions of GUI components replace their predecessors in an implementation. Often, the inclusion of a new version of a component breaks some part of the software, i.e., tasks that end-users were able to perform before modifications were made can no longer be performed. Software developers (who also happen to be component users) are unable to perform adequate regression testing in this context because of several factors, including lack of source code, environmental side-effects on GUI rendering, event-driven nature of GUIs, and large number of possible permutations of events. This paper presents a new technique that end-users’ quality assurance (QA) teams can employ to test the new version of a component in its application context by using the existing version as a baseline. The technique combines lightweight event-level dynamic profiling to collect user profiles in a transparent manner, GUI reverse engineering to extract the structure of the component’s GUI, test case execution to replay the collected profiles on the new version, and GUI oracles that collect properties from the existing version. Empirical studies demonstrate the practicality, usefulness, and limitations of the technique. Atif M. Memon is an Assistant Professor at the Department of Computer Science, University of Maryland. He received his BS, MS, and Ph.D. in Computer Science in 1991, 1995, and 2001 respectively. He was awarded a Gold Medal in BS. He was awarded Fellowships from the Andrew Mellon Foundation for his Ph.D. research. He received the NSF CAREER award in 2005. His research interests include program testing, software engineering, artificial intelligence, plan generation, reverse engineering, and program structures. He is a member of the ACM and the IEEE Computer Society.     

Controlling Testing Using Three-Tier Model Architecture

In this paper, based on our earlier work, we introduce test model architecture for model-based testing through a GUI. The model architecture consists of three tiers. The tiers separate important concerns in GUI testing: navigation in the GUI using keywords, high-level concepts expressed as action words, and test-control related issues defined as control words. For test control, we define a novel coverage language to express coverage objectives. Furthermore, we introduce our refined vision for the associated tool platform enabling system level testing in the Symbian environment. The architecture includes a commercial GUI testing tool that we have extended with components enabling the use of test models.     

Communication and personality profiles of global software developers

ContextPrior research has established that a small proportion of individuals dominate team communication during global software development. It is not known, however, how these members’ contributions affect their teams’ knowledge diffusion process, or whether their personality profiles are responsible for their dominant presence.ObjectiveWe set out to address this gap through the study of repository artifacts.MethodArtifacts from ten teams were mined from the IBM Rational Jazz repository. We employed social network analysis (SNA) to group practitioners into two clusters, Top Members and Others, based on the numbers of messages they communicated and their engagement in task changes. SNA metrics (density, in-degree and closeness) were then used to study practitioners’ importance in knowledge diffusion. Thereafter, we performed psycholinguistic analysis on practitioners’ messages using linguistic dimensions that had been previously correlated with the Big Five personality profiles.ResultsFor our sample of 146 practitioners we found that Top Members occupied critical roles in knowledge diffusion, and demonstrated more openness to experience than the Others. Additionally, all personality profiles were represented during teamwork, although openness to experience, agreeableness and extroversion were particularly evident. However, no specific personality predicted members’ involvement in knowledge diffusion.ConclusionTask assignment that promotes highly connected team communication networks may mitigate tacit knowledge loss in global software teams. Additionally, while members expressing openness to experience are likely to be particularly driven to perform, this is not entirely responsible for a global team’s success.     

Answering software evolution questions: An empirical evaluation

ContextDevelopers often need to find answers to questions regarding the evolution of a system when working on its code base. While their information needs require data analysis pertaining to different repository types, the source code repository has a pivotal role for program comprehension tasks. However, the coarse-grained nature of the data stored by commit-based software configuration management systems often makes it challenging for a developer to search for an answer.ObjectiveWe present Replay, an Eclipse plug-in that allows developers to explore the change history of a system by capturing the changes at a finer granularity level than commits, and by replaying the past changes chronologically inside the integrated development environment, with the source code at hand.MethodWe conducted a controlled experiment to empirically assess whether Replay outperforms a baseline (SVN client in Eclipse) on helping developers to answer common questions related to software evolution.ResultsThe experiment shows that Replay leads to a decrease in completion time with respect to a set of software evolution comprehension tasks.ConclusionWe conclude that there are benefits in using Replay over the state of the practice tools for answering questions that require fine-grained change information and those related to recent changes.     

Event-driven web application testing based on model-based mutation testing

ContextEvent-Driven Software (EDS) is a class of software whose behavior is driven by incoming events. Web and desktop applications that respond to user-initiated events on their Graphical User Interface (GUI), or embedded software responding to events and signals received from the equipment in its operating environment are examples of EDS. Testing EDS poses great challenges to software testers. One of these challenges is the need to generate a huge number of possible event sequences that could sufficiently cover the EDS’s state space.ObjectiveIn this paper, we introduce a new six-stage testing procedure for event-driven web applications to overcome EDS testing challenges.MethodThe stages of the testing procedure include dividing the application based on its structure, creating functional graphs for each section, creating mutants from functional graphs, choosing coverage criteria to produce test paths, merging event sequences to make longer ones, and deriving and running test cases. We have analyzed our proposed testing procedure with the help of four metrics consisting of Fault Detection Density (FDD), Fault Detection Effectiveness (FDE), Mutation Score, and Unique Fault.ResultsUsing this procedure, we have prepared prioritized test cases and also discovered a list of unique faults by running the suggested test cases on a sample real-world web application called Academic E-mail System.ConclusionWe propose that our suggested testing procedure has some advantages such as creating functional graphs with requirements document, resolving the problem of removing infeasible test cases with these graphs and conditions on the “add edge” operator before creating mutants. But the suggested testing procedure, like any other method, had some drawbacks. Because most of the stages in the approach were performed manually, the testing time was increased.     

Designing multi-agent unit tests using systematic test design patterns-(extended version)

Software agents are the basic building blocks in many software systems especially those based on artificial intelligence methods, e.g., reinforcement learning based multi-agent systems (MASs). However, testing software agents is considered a challenging problem. This is due to the special characteristics of agents which include its autonomy, distributed nature, intelligence, and heterogeneous communication protocols. Following the test-driven development (TDD) paradigm, we present a framework that allows MAS developers to write test scenarios that test each agent individually. The framework relies on the concepts of building mock agents and testing common agent interaction design patterns. We analyze the most common agent interaction patterns including pair and mediation patterns in order to provide stereotype implementation for their corresponding test cases. These implementations serve as test building blocks and are provided as a set of ready-for-reuse components in our repository. This way, the developer can concentrate on testing the business logic itself and spare him/her the burden of implementing tests for the underlying agent interaction patterns. Our framework is based on standard components such as the JADE agent platform, the JUnit framework, and the eclipse plug-in architecture. In this paper, we present in details the design and function of the framework. We demonstrate how we can use the proposed framework to define more stereotypes in the code repository and provide a detailed analysis of the code coverage for our designed stereotype test code implementations.     

KSAP: An approach to bug report assignment using KNN search and heterogeneous proximity

ContextBug report assignment, namely, to assign new bug reports to developers for timely and effective bug resolution, is crucial for software quality assurance. However, with the increasing size of software system, it is difficult to assign bugs to appropriate developers for bug managers.ObjectiveThis paper propose an approach, called KSAP (K-nearest-neighbor search and heterogeneous proximity), to improve automatic bug report assignment by using historical bug reports and heterogeneous network of bug repository.MethodWhen a new bug report was submitted to the bug repository, KSAP assigns developers for the bug report by using a two-phase procedure. The first phase is to search historically-resolved similar bug reports to the new bug report by K-nearest-neighbor (KNN) method. The second phase is to rank the developers who contributed to those similar bug reports by heterogeneous proximity.ResultsWe collected bug repositories of Mozilla, Eclipse, Apache Ant and Apache Tomcat6 projects to investigate the performance of the proposed KSAP approach. Experimental results demonstrate that KSAP can improve the recall of bug report assignment between 7.5–32.25% in comparison with the state of art techniques. When there is only a small number of developer collaborations on common bug reports, KSAP has shown its excellence over other sate of art techniques. When we tune the parameters of the number of historically-resolved similar bug reports (K) and the number of developers (Q) for recommendation, KSAP keeps its superiority steadily.ConclusionThis is the first paper to demonstrate how to automatically build heterogeneous network of a bug repository and extract meta-paths of developer collaborations from the heterogeneous network for bug report assignment.     

Why software repositories are not used for defect-insertion circumstance analysis more often: A case study

ContextRoot-cause analysis is a data-driven technique for developing software process improvements in mature software organizations. The search for individual process correlates of high defect densities, which we call defect insertion circumstance analysis (DICA), is potentially both effective and cost-efficient as one approach to be used when attempting a general defect root cause analysis. In DICA, data from existing repositories (version archive, bug tracker) is evaluated largely automatically in order to determine conditions (such as the people, roles, components, or time-periods involved) that correlate with higher-than-normal defect insertion frequencies. Nevertheless, no reports of industrial use of DICA have been published.ObjectiveDetermine the reasons why DICA is not used more often by practitioners.MethodWe use a single-case, typical-case, revelatory-type case study to evaluate in parallel the importance of six plausible reasons (R1–R6). The case is based on 11 years of repository data from a small but mature software company building a product in the high-end content management system domain and describes a four person-months effort to make use of these data.ResultsWhile DICA required non-negligible effort (R3) and some degree of inventiveness (R2), the most relevant roadblock was insufficient reliability of the results (R6) combined with the difficulty of assessing this reliability (R5). We identify three difficulties that led to this outcome.ConclusionCurrent repository mining methods are too immature for successful DICA. Gradual improvements are unlikely to help; different principles of operation will be required. Even with such different techniques, issues with input data quality may continue to make good results difficult-to-have.     

Analyzing the relationships between inspections and testing to provide a software testing focus

ContextQuality assurance effort, especially testing effort, is frequently a major cost factor during software development. Consequently, one major goal is often to reduce testing effort. One promising way to improve the effectiveness and efficiency of software quality assurance is the use of data from early defect detection activities to provide a software testing focus. Studies indicate that using a combination of early defect data and other product data to focus testing activities outperforms the use of other product data only. One of the key challenges is that the use of data from early defect detection activities (such as inspections) to focus testing requires a thorough understanding of the relationships between these early defect detection activities and testing. An aggravating factor is that these relationships are highly context-specific and need to be evaluated for concrete environments.ObjectiveThe underlying goal of this paper is to help companies get a better understanding of these relationships for their own environment, and to provide them with a methodology for finding relationships in their own environments.MethodThis article compares three different strategies for evaluating assumed relationships between inspections and testing. We compare a confidence counter, different quality classes, and the F-measure including precision and recall.ResultsOne result of this case-study-based comparison is that evaluations based on the aggregated F-measures are more suitable for industry environments than evaluations based on a confidence counter. Moreover, they provide more detailed insights about the validity of the relationships.ConclusionWe have confirmed that inspection results are suitable data for controlling testing activities. Evaluated knowledge about relationships between inspections and testing can be used in the integrated inspection and testing approach In²Test to focus testing activities. Product data can be used in addition. However, the assumptions have to be evaluated in each new context.     

Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption

ContextA software product line is a family of related software products, typically created from a set of common assets. Users select features to derive a product that fulfills their needs. Users often expect a product to have specific non-functional properties, such as a small footprint or a bounded response time. Because a product line may have an exponential number of products with respect to its features, it is usually not feasible to generate and measure non-functional properties for each possible product.ObjectiveOur overall goal is to derive optimal products with respect to non-functional requirements by showing customers which features must be selected.MethodWe propose an approach to predict a product’s non-functional properties based on the product’s feature selection. We aggregate the influence of each selected feature on a non-functional property to predict a product’s properties. We generate and measure a small set of products and, by comparing measurements, we approximate each feature’s influence on the non-functional property in question. As a research method, we conducted controlled experiments and evaluated prediction accuracy for the non-functional properties footprint and main-memory consumption. But, in principle, our approach is applicable for all quantifiable non-functional properties.ResultsWith nine software product lines, we demonstrate that our approach predicts the footprint with an average accuracy of 94%, and an accuracy of over 99% on average if feature interactions are known. In a further series of experiments, we predicted main memory consumption of six customizable programs and achieved an accuracy of 89% on average.ConclusionOur experiments suggest that, with only few measurements, it is possible to accurately predict non-functional properties of products of a product line. Furthermore, we show how already little domain knowledge can improve predictions and discuss trade-offs between accuracy and required number of measurements. With this technique, we provide a basis for many reasoning and product-derivation approaches.     

Higher order mutation testing: A Systematic Literature Review

Mutation testing is the process whereby a fault is deliberately inserted into a software system, in order to assess the quality of test data, in terms of its ability to find this fault. Mutation testing is also used as a way to drive the test data development process. Traditionally, faults were inserted one by one into a software system, but more recently there has been an upsurge of interest by the area of higher-order mutation, in which multiple faults are inserted into the system at once. Originally, this was thought to be too expensive, as there was already a concern that the size of the pool of mutants for traditional mutation was already too large to handle. However, following a seminal publication in 2008, it was realized that the space of higher-order mutants (HOMs) could be searched for useful mutants that drive testing harder, and to reduce the overall test effort, by clever combination of first-order mutants. As a result, many authors examined the way in which HOM testing could find subtle hard to kill faults, capture partial fault masking, reduce equivalent mutants problem, reduce test effort while increasing effectiveness, and capture more realistic faults than those captured by simple insertion of first-order mutants. Because of the upsurge of interest in the previous issues, this paper presents the first Systematic Literature Review research specifically targeted at a higher-order mutation. This Systematic Literature Review analyzes the results of more than one hundred sixty research articles in this area. The current paper presents qualitative results and bibliometric analysis for the surveyed articles. In addition, it augments these results with scientific findings and quantitative results from the primary literature. As a result of this work, this SLR presents an outline for many future work.     

An event‐flow model of GUI‐based applications for testing

Graphical user interfaces (GUIs) are by far the most popular means used to interact with today's software. The functional correctness of a GUI is required to ensure the safety, robustness and usability of an entire software system. GUI testing techniques used in practice are resource intensive; model‐based automated techniques are rarely employed. A key reason for the reluctance in the adoption of model‐based solutions proposed by researchers is their limited applicability; moreover, the models are expensive to create. Over the past few years, the present author has been developing different models for various aspects of GUI testing. This paper consolidates all of the models into one scalable event‐flow model and outlines algorithms to semi‐automatically reverse‐engineer the model from an implementation. Earlier work on model‐based test‐case generation, test‐oracle creation, coverage evaluation, and regression testing is recast in terms of this model by defining event‐space exploration strategies (ESESs) and creating an end‐to‐end GUI testing process. Three such ESESs are described: for checking the event‐flow model, test‐case generation, and test‐oracle creation. Two demonstrational scenarios show the application of the model and the three ESESs for experimentation and application in GUI testing. Copyright © 2007 John Wiley & Sons, Ltd.