Answering software evolution questions: An empirical evaluation

ContextDevelopers often need to find answers to questions regarding the evolution of a system when working on its code base. While their information needs require data analysis pertaining to different repository types, the source code repository has a pivotal role for program comprehension tasks. However, the coarse-grained nature of the data stored by commit-based software configuration management systems often makes it challenging for a developer to search for an answer.ObjectiveWe present Replay, an Eclipse plug-in that allows developers to explore the change history of a system by capturing the changes at a finer granularity level than commits, and by replaying the past changes chronologically inside the integrated development environment, with the source code at hand.MethodWe conducted a controlled experiment to empirically assess whether Replay outperforms a baseline (SVN client in Eclipse) on helping developers to answer common questions related to software evolution.ResultsThe experiment shows that Replay leads to a decrease in completion time with respect to a set of software evolution comprehension tasks.ConclusionWe conclude that there are benefits in using Replay over the state of the practice tools for answering questions that require fine-grained change information and those related to recent changes.     

An empirical study using task assignment patterns to improve theaccuracy of software effort estimation

In most software development organizations, there is seldom a one-to-one mapping between software developers and development tasks. It is frequently necessary to concurrently assign individuals to multiple tasks and to assign more than one individual to work cooperatively on a single task. A principal goal in making such assignments should be to minimize the effort required to complete each task. But what impact does the manner in which developers are assigned to tasks have on the effort requirements? This paper identifies four task assignment factors: team size, concurrency, intensity, and fragmentation. These four factors are shown to improve the predictive ability of the well-known intermediate COCOMO cost estimation model. A parsimonious effort estimation model is also derived that utilizes a subset of the task assignment factors and unadjusted function points. For the data examined, this parsimonious model is shown to have goodness of fit and quality of estimation superior to that of the COCOMO model, while utilizing fewer cost factors     

Studying software evolution using topic models

Topic models are generative probabilistic models which have been applied to information retrieval to automatically organize and provide structure to a text corpus. Topic models discover topics in the corpus, which represent real world concepts by frequently co-occurring words. Recently, researchers found topics to be effective tools for structuring various software artifacts, such as source code, requirements documents, and bug reports. This research also hypothesized that using topics to describe the evolution of software repositories could be useful for maintenance and understanding tasks. However, research has yet to determine whether these automatically discovered topic evolutions describe the evolution of source code in a way that is relevant or meaningful to project stakeholders, and thus it is not clear whether topic models are a suitable tool for this task.In this paper, we take a first step towards evaluating topic models in the analysis of software evolution by performing a detailed manual analysis on the source code histories of two well-known and well-documented systems, JHotDraw and jEdit. We define and compute various metrics on the discovered topic evolutions and manually investigate how and why the metrics evolve over time. We find that the large majority (87%–89%) of topic evolutions correspond well with actual code change activities by developers. We are thus encouraged to use topic models as tools for studying the evolution of a software system.     

How changes affect software entropy: an empirical study

Context Software systems continuously change for various reasons, such as adding new features, fixing bugs, or refactoring. Changes may either increase the source code complexity and disorganization, or help to reducing it. Aim This paper empirically investigates the relationship of source code complexity and disorganization—measured using source code change entropy—with four factors, namely the presence of refactoring activities, the number of developers working on a source code file, the participation of classes in design patterns, and the different kinds of changes occurring on the system, classified in terms of their topics extracted from commit notes. Method We carried out an exploratory study on an interval of the life-time span of four open source systems, namely ArgoUML, Eclipse-JDT, Mozilla, and Samba, with the aim of analyzing the relationship between the source code change entropy and four factors: refactoring activities, number of contributors for a file, participation of classes in design patterns, and change topics. Results The study shows that (i) the change entropy decreases after refactoring, (ii) files changed by a higher number of developers tend to exhibit a higher change entropy than others, (iii) classes participating in certain design patterns exhibit a higher change entropy than others, and (iv) changes related to different topics exhibit different change entropy, for example bug fixings exhibit a limited change entropy while changes introducing new features exhibit a high change entropy. Conclusions Results provided in this paper indicate that the nature of changes (in particular changes related to refactorings), the software design, and the number of active developers are factors related to change entropy. Our findings contribute to understand the software aging phenomenon and are preliminary to identifying better ways to contrast it.     

On the relationship between similar requirements and similar software

Recommender systems for requirements are typically built on the assumption that similar requirements can be used as proxies to retrieve similar software. When a stakeholder proposes a new requirement, natural language processing (NLP)-based similarity metrics can be exploited to retrieve existing requirements, and in turn, identify previously developed code. Several NLP approaches for similarity computation between requirements are available. However, there is little empirical evidence on their effectiveness for code retrieval. This study compares different NLP approaches, from lexical ones to semantic, deep-learning techniques, and correlates the similarity among requirements with the similarity of their associated software. The evaluation is conducted on real-world requirements from two industrial projects from a railway company. Specifically, the most similar pairs of requirements across two industrial projects are automatically identified using six language models. Then, the trace links between requirements and software are used to identify the software pairs associated with each requirements pair. The software similarity between pairs is then automatically computed with JPLag. Finally, the correlation between requirements similarity and software similarity is evaluated to see which language model shows the highest correlation and is thus more appropriate for code retrieval. In addition, we perform a focus group with members of the company to collect qualitative data. Results show a moderately positive correlation between requirements similarity and software similarity, with the pre-trained deep learning-based BERT language model with preprocessing outperforming the other models. Practitioners confirm that requirements similarity is generally regarded as a proxy for software similarity. However, they also highlight that additional aspect comes into play when deciding software reuse, e.g., domain/project knowledge, information coming from test cases, and trace links. Our work is among the first ones to explore the relationship between requirements and software similarity from a quantitative and qualitative standpoint. This can be useful not only in recommender systems but also in other requirements engineering tasks in which similarity computation is relevant, such as tracing and change impact analysis.

     

Deriving thresholds of software metrics to predict faults on open source software: Replicated case studies

Object-oriented metrics aim to exhibit the quality of source code and give insight to it quantitatively. Each metric assesses the code from a different aspect. There is a relationship between the quality level and the risk level of source code. The objective of this paper is to empirically examine whether or not there are effective threshold values for source code metrics. It is targeted to derive generalized thresholds that can be used in different software systems. The relationship between metric thresholds and fault-proneness was investigated empirically in this study by using ten open-source software systems. Three types of fault-proneness were defined for the software modules: non-fault-prone, more-than-one-fault-prone, and more-than-three-fault-prone. Two independent case studies were carried out to derive two different threshold values. A single set was created by merging ten datasets and was used as training data by the model. The learner model was created using logistic regression and the Bender method. Results revealed that some metrics have threshold effects. Seven metrics gave satisfactory results in the first case study. In the second case study, eleven metrics gave satisfactory results. This study makes contributions primarily for use by software developers and testers. Software developers can see classes or modules that require revising; this, consequently, contributes to an increment in quality for these modules and a decrement in their risk level. Testers can identify modules that need more testing effort and can prioritize modules according to their risk levels.     

Assessing the cost-effectiveness of software reuse: A model for planned reuse

Information systems development is typically acknowledged as an expensive and lengthy process, often producing code that is of uneven quality and difficult to maintain. Software reuse has been advocated as a means of revolutionizing this process. The claimed benefits from software reuse are reduction in development cost and time, improvement in software quality, increase in programmer productivity, and improvement in maintainability. Software reuse entails undeniable costs of creating, populating, and maintaining a library of reusable components. There is anecdotal evidence to suggest that some organizations benefit from reuse. However, many software developers practicing reuse claim these benefits without formal demonstration thereof. There is little research to suggest when the benefits are expected and to what extent they will be realized. For example, does a larger library of reusable components lead to increased savings? What is the impact of component size on the effectiveness of reuse? This research seeks to address some of these questions. It represents the first step in a series wherein the effects of software reuse on overall development effort and costs are modeled with a view to understanding when it is most effective.     

Communication and organization: an empirical study of discussion ininspection meetings

This paper describes an empirical study that addresses the issue of communication among members of a software development organization. In particular, data was collected concerning code inspections in one software development project. The question of interest is whether or not organizational structure (the network of relationships between developers) has an effect on the amount of effort expended on communication between developers. The independent variables in this study are various attributes of the organizational structure in which the inspection participants work. The dependent variables are measures of the communication effort expended in various parts of the code inspection process, focusing on the inspection meeting. Both quantitative and qualitative methods were used, including participant observation, structured interviews, generation of hypotheses from field notes, statistical tests of relationships, and interpretation of results with qualitative anecdotes. The study results show that past and present working relationships between inspection participants affect the amount of meeting time spent in different types of discussion, thus affecting the overall inspection meeting length. Reporting relationships and physical proximity also have an effect. The contribution of the study is a set of well-supported hypotheses for further investigation     

An experiment to assess the cost-benefits of code inspections inlarge scale software development

We conducted a long term experiment to compare the costs and benefits of several different software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development effort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. The article has several goals: (1) to describe the experiment's design and show how we used simulation techniques to optimize it; (2) to present our results and discuss their implications for both software practitioners and researchers; and (3) to discuss several new questions raised by our findings. For each inspection, we randomly assigned three independent variables: (1) the number of reviewers on each inspection team (1, 2, or 4); (2) the number of teams inspecting the code unit (1 or 2); and (3) the requirement that defects be repaired between the first and second team's inspections. The reviewers for each inspection were randomly selected without replacement from a pool of 11 experienced software developers. The dependent variables for each inspection included inspection interval (elapsed time), total effort, and the defect detection rate. Our results showed that these treatments did not significantly influence the defect detection effectiveness, but that certain combinations of changes dramatically increased the inspection interval     

Understanding the gap between software process practices and actual practice in very small companies

This paper reports on a grounded theory to study into software developers’ use of software development processes in actual practice in the specific context of very small companies. This study was conducted in three very small software product companies located in Ecuador. The data collection was based on semi-structured qualitative interviews with software project managers, focus group with software developers and was supplemented by the literature and document studies. We interviewed two types of participants (managers and developers), so as to ensure that we elicited a holistic perspective of how they approached the software development process in actual practice. The goal was to study what practices are actually used and their opinion and attitude toward the potential adopting of an international standard (ISO/IEC 29110) specifically designed for very small companies. With the collected data, we performed an analysis utilizing grounded theory coding techniques, as this methodology promotes the focus on uncovering the real concerns of the participants. This study highlighted three areas of concern: customer, software product and development tasks coordination and tracking. The findings in this study give an insight toward the work products as they relate to software development process practices in very small companies and the important factors that must be considered to assist project success.     

Understanding reuse of software examples: A case study of prejudice in a community of practice

ContextThe context of this research is software developers’ perceptions about the use of code examples in professional software development.ObjectiveThe primary objective of this paper is to identify the human factors that dominate example usage among professional software developers, and to provide a theory that explains these factors.MethodTo achieve this goal, we analyzed the perceptions of professional software developers as manifested on LinkedIn online community. We analyzed the data qualitatively using adapted grounded theory research procedures.ResultsThe research yields an initial framework of key factors that dominate professional developers’ perception regarding example usage. We use the theoretical lens of prejudice theory to put these factors in a broader context, and outline initial recommendations to address these factors in professional organizational context.ConclusionThe results of this work, in particular the use of qualitative techniques – allowed us to obtain rich insight into key human factors that affect professional software developers, and enrich the body of literature on the issues of reuse. These factors need to be taken into account as part of an organizational reuse strategy.     

Improving software security with static automated code analysis in an industry setting

Software security can be improved by identifying and correcting vulnerabilities. In order to reduce the cost of rework, vulnerabilities should be detected as early and efficiently as possible. Static automated code analysis is an approach for early detection. So far, only few empirical studies have been conducted in an industrial context to evaluate static automated code analysis. A case study was conducted to evaluate static code analysis in industry focusing on defect detection capability, deployment, and usage of static automated code analysis with a focus on software security. We identified that the tool was capable of detecting memory related vulnerabilities, but few vulnerabilities of other types. The deployment of the tool played an important role in its success as an early vulnerability detector, but also the developers perception of the tools merit. Classifying the warnings from the tool was harder for the developers than to correct them. The correction of false positives in some cases created new vulnerabilities in previously safe code. With regard to defect detection ability, we conclude that static code analysis is able to identify vulnerabilities in different categories. In terms of deployment, we conclude that the tool should be integrated with bug reporting systems, and developers need to share the responsibility for classifying and reporting warnings. With regard to tool usage by developers, we propose to use multiple persons (at least two) in classifying a warning. The same goes for making the decision of how to act based on the warning. Copyright © 2012 John Wiley & Sons, Ltd.     

Change Analysis with Evolizer and ChangeDistiller

Changes are the heartbeat of a software system. Software must change to reflect new business, pro­cess, and technology requirements, or it will become progressively less useful. Software typically grows and becomes more complex, inducing more time and effort for performing changes. Software archives such as source code version-control systems and issue-tracking systems (for bugs and change requests) are rich sources to examine what changes have what impact on the software. A software evolution analysis platform called Evolizer analyzes change histories and potential support for evolution. Change types, a core part of the analysis, help discover significant changes and change patterns. A tool called ChangeDistiller enables fine-grained change type extraction and analysis to reason about coding conventions, control or exception flow, and even code and comment coevolution. An investigation of open source and commercial software systems contributed to a deeper understanding of how researchers can actively support software evolution in an integrated development environment.     

Hipikat: a project memory for software development

Sociological and technical difficulties, such as a lack of informal encounters, can make it difficult for new members of noncollocated software development teams to learn from their more experienced colleagues. To address this situation, we have developed a tool, named Hipikat that provides developers with efficient and effective access to the group memory for a software development project that is implicitly formed by all of the artifacts produced during the development. This project memory is built automatically with little or no change to existing work practices. After describing the Hipikat tool, we present two studies investigating Hipikat's usefulness in software modification tasks. One study evaluated the usefulness of Hipikat's recommendations on a sample of 20 modification tasks performed on the Eclipse Java IDE during the development of release 2.1 of the Eclipse software. We describe the study, present quantitative measures of Hipikat's performance, and describe in detail three cases that illustrate a range of issues that we have identified in the results. In the other study, we evaluated whether software developers who are new to a project can benefit from the artifacts that Hipikat recommends from the project memory. We describe the study, present qualitative observations, and suggest implications of using project memory as a learning aid for project newcomers.     

Change profile analysis of open-source software systems to understand their evolutionary behavior

Source code management systems (such as git) record changes to code repositories of Open-Source Software (OSS) projects. The metadata about a change includes a change message to record the intention of the change. Classification of changes, based on change messages, into different change types has been explored in the past to understand the evolution of software systems from the perspective of change size and change density only. However, software evolution analysis based on change classification with a focus on change evolution patterns is still an open research problem. This study examines change messages of 106 OSS projects, as recorded in the git repository, to explore their evolutionary patterns with respect to the types of changes performed over time. An automated keyword-based classifier technique is applied to the change messages to categorize the changes into various types (corrective, adaptive, perfective, preventive, and enhancement). Cluster analysis helps to uncover distinct change patterns that each change type follows. We identify three categories of 106 projects for each change type: high activity, moderate activity, and low activity. Evolutionary behavior is different for projects of different categories. The projects with high and moderate activity receive maximum changes during 76–81 months of the project lifetime. The project attributes such as the number of committers, number of files changed, and total number of commits seem to contribute the most to the change activity of the projects. The statistical findings show that the change activity of a project is related to the number of contributors, amount of work done, and total commits of the projects irrespective of the change type. Further, we explored languages and domains of projects to correlate change types with domains and languages of the projects. The statistical analysis indicates that there is no significant and strong relation of change types with domains and languages of the 106 projects.     

Automatically propagating changes from reference implementations to code generation templates

ContextCode generators can automatically perform some tedious and error-prone implementation tasks, increasing productivity and quality in the software development process. Most code generators are based on templates, which are fundamentally composed of text expansion statements. To build templates, the code of an existing, tested and validated implementation may serve as reference, in a process known as templatization. With the dynamics of software evolution/maintenance and the need for performing changes in the code generation templates, there is a loss of synchronism between the templates and this reference code. Additional effort is required to keep them synchronized.ObjectiveThis paper proposes automation as a way to reduce the extra effort needed to keep templates and reference code synchronized.MethodA mechanism was developed to semi-automatically detect and propagate changes from reference code to templates, keeping them synchronized with less effort. The mechanism was also submitted to an empirical evaluation to analyze its effects in terms of effort reduction during maintenance/evolution templatization.ResultsIt was observed that the developed mechanism can lead to a 50% reduction in the effort needed to perform maintenance/evolution templatization, when compared to a manual approach. It was also observed that this effect depends on the nature of the evolution/maintenance task, since for one of the tasks there was no observable advantage in using the mechanism. However, further studies are needed to better characterize these tasks.ConclusionAlthough there is still room for improvement, the results indicate that automation can be used to reduce effort and cost in the maintenance and evolution of a template-based code generation infrastructure.     

Assessing the capability of code smells to explain maintenance problems: an empirical study combining quantitative and qualitative data

Code smells are indicators of deeper design problems that may cause difficulties in the evolution of a software system. This paper investigates the capability of twelve code smells to reflect actual maintenance problems. Four medium-sized systems with equivalent functionality but dissimilar design were examined for code smells. Three change requests were implemented on the systems by six software developers, each of them working for up to four weeks. During that period, we recorded problems faced by developers and the associated Java files on a daily basis. We developed a binary logistic regression model, with “problematic file” as the dependent variable. Twelve code smells, file size, and churn constituted the independent variables. We found that violation of the Interface Segregation Principle (a.k.a. ISP violation) displayed the strongest connection with maintenance problems. Analysis of the nature of the problems, as reported by the developers in daily interviews and think-aloud sessions, strengthened our view about the relevance of this code smell. We observed, for example, that severe instances of problems relating to change propagation were associated with ISP violation. Based on our results, we recommend that code with ISP violation should be considered potentially problematic and be prioritized for refactoring.     

An empirical study on the impact of static typing on software maintainability

Static type systems play an essential role in contemporary programming languages. Despite their importance, whether static type systems impact human software development capabilities remains open. One frequently mentioned argument in favor of static type systems is that they improve the maintainability of software systems—an often-used claim for which there is little empirical evidence. This paper describes an experiment that tests whether static type systems improve the maintainability of software systems, in terms of understanding undocumented code, fixing type errors, and fixing semantic errors. The results show rigorous empirical evidence that static types are indeed beneficial to these activities, except when fixing semantic errors. We further conduct an exploratory analysis of the data in order to understand possible reasons for the effect of type systems on the three kinds of tasks used in this experiment. From the exploratory analysis, we conclude that developers using a dynamic type system tend to look at different files more frequently when doing programming tasks—which is a potential reason for the observed differences in time.     

Evaluating software design processes by analyzing change data overtime

An analysis is presented of early design and code change data from the software cost reduction (SCR) project, a well-reported effort conducted at the US Naval Research Laboratory from 1978 to 1988. The analyses are mostly time-based studies of the change data and relationships between the data and SCR personnel activity data. Some analyses of the change data show patterns consistent with a major goal of the SCR project: the design and development of easy-to-change software. Specifically, most changes took a day or less to uncover and resolve; the majority of changes updated at most one module. Moreover, these percentages remained fairly stable. No positive relationship appeared between error-correction effort and the number of days that an error remained in the SCR design documentation. Other analyses suggest that consistency may have been temporary. For example, the analyses suggest a stepwise growth in average change effort, and an increasing percentage of changes resulted in module interface updates. Certain specific ratios between SCR change data and personnel activity data may be possible indicators of design incompleteness     

Process quality costing adapted to software development

The Prevention-Appraisal-Failure and Process models of measuring quality costs, as described in BS6413, are discussed in the context of software quality costs. The Process model is selected as the better base for constructing a software quality cost model because of its suitability for application to software engineering processes. A seven-step method for modelling software quality costs is proposed and illustrated by application to a software maintenance project. Key processes in software maintenance and its quality tasks are derived from the most recently available standards-related publications. Although no results are available, a qualitative evaluation in terms of accuracy, improvement issues, measurement issues and quality costs to management concludes that the model is valid and that a greater consensus is required for identification of software quality cost elements. Implementation of quality cost measurement is not discussed save noting that there are still complex issues to be addressed.