Using metrics to manage software projects

Five years ago, Bull's Enterprise Servers Operation in Phoenix, Arizona, used a software process that, although understandable, was unpredictable in terms of product quality and delivery schedule. The process generated products with unsatisfactory quality levels and required significant extra effort to avoid major schedule slips. All but the smallest software projects require metrics for effective project management. Hence, as part of a program designed to improve the quality, productivity, and predictability of software development projects, the Phoenix operation launched a series of improvements in 1989. One improvement based software project management on additional software measures. Another introduced an inspection program, since inspection data was essential to project management improvements. Project sizes varied from several thousand lines of code (KLOC) to more than 300 KLOC. The improvement projects enhanced quality and productivity. In essence, Bull now has a process that is repeatable and manageable, and that delivers higher quality products at lower cost. We describe the metrics we selected and implemented, illustrating with examples drawn from several development projects     

Software process evolution at the SEL

The Software Engineering Laboratory of the National Aeronautics and Space Administration's Goddard Space Flight Center has been adapting, analyzing, and evolving software processes for the last 18 years (1976-94). Their approach is based on the Quality Improvement Paradigm, which is used to evaluate process effects on both product and people. The authors explain this approach as it was applied to reduce defects in code. In examining and adapting reading techniques, we go through a systematic process of evaluating the candidate process and refining its implementation through lessons learned from previous experiments and studies. As a result of this continuous, evolutionary process, we determined that we could successfully apply key elements of the cleanroom development method in the SEL environment, especially for projects involving fewer than 50000 lines of code (all references to lines of code refer to developed, not delivered, lines of code). We saw indications of lower error rates, higher productivity, a more complete and consistent set of code comments, and a redistribution of developer effort. Although we have not seen similar reliability and cost gains for larger efforts, we continue to investigate the cleanroom method's effect on them     

Improving Software Development through Three Stages

In three stages covering 12 years and 16 projects, a software development group worked on attaining visible schedule control and usable software design documentation. The first stage identified two major problems: unstable code and schedule delays. The group used code inspection to address the immediate issue of code construction. In the second stage, the group formulated and implemented a software development process that addressed the entire software development life cycle. The third stage improved that process. The major factor in process improvement was the group's willingness to identify its mistakes and problems.     

The increase of productivity over time—an industrial case study

Introducing new and specialized technology is often seen as a way of meeting increasing non-functional requirements. An example of such a technology is a software platform that provides high performance and availability. The novelty of such a platform and lack of related experience and competence among the staff may affect initial development productivity. The competence problems should disappear with time. In this paper, we present a study, which we conducted at Ericsson. The purpose of the study was to assess the impact of experience and maturity on productivity in software development on the specialized platform. We quantify the impact by comparing productivity of two projects. One represents an initial development stage while the other represents a subsequent and thus more matured development stage. Both projects resulted in large commercial products. We reveal a factor of four difference in productivity. The difference was caused by a higher code delivery rate and a lower number of code lines per functionality in the latter project. We assess the impact of both these issues on productivity and explain their nature. Based on our findings, we suggest a number of improvement suggestions and guidelines for the process of introducing a new technology.     

Using a Line of Code Metric to Understand Software Rework

Developers should factor rework into sizing and productivity calculations when estimating software effort. Reworked code is software created during development that doesn't exist in the final build. Using lines of code as a sizing metric is helpful when estimating projects with similar domains, platforms, processes, development teams, and coding constraints. A simple method can measure new effective lines of code (nLOC)—that is, added, changed, or deleted nonblank, noncomment lines of code—during development. Analysis of three projects shows that between 19 and 40 percent of the code written wasn't in the final release, with the nLOC measured biweekly and accumulated over each project's development.     

Engineers don't build [software development]

Most of us never approach a construction project with a software development methodology. However, construction projects exhibit much higher success rates than software development projects. The author identifies the “master-builder” approach taken by most software development projects where the developers assume the role of architect, engineer, builder, and inspector. Most large-scale construction projects abandoned the approach nearly 100 years ago. Why do we do it for large-scale software projects? Engineers design systems or buildings, and programmers or builders implement them. That approach yields greater success rates in terms of quality, time, and budget     

The effectiveness of test-driven development: an industrial case study

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an industrial environment. Three comparable medium-sized projects were observed during their development cycle. Two projects were driven without TDD practice, while the third one introduced TDD into the development process. The effectiveness of TDD was expressed in terms of external code quality, productivity, and maintainability. Our results indicate that the TDD developers produced higher quality code that is easier to maintain, although we did observe a reduction in productivity.     

Using visualization to maintain large computer systems

The Year 2000 software conversion effort dramatically illustrates how time consuming and costly maintaining large computer systems can become, especially when each system consists of millions of lines of source code. Understanding how a system's components interact is a key factor in implementing portfolio-wide changes, adding new features, and providing ongoing system maintenance. Any help that software developers can get in making existing software systems easier to understand improves developer productivity, enhances software quality, and reduces development cycles, all leading to faster time to market. The paper discusses the Visual Insights code viewer, a visualization application developed specifically to address the problem of working with large amounts of source code. Used within the Lucent Technologies 5ESS Switch development environment, the code viewer has resulted in increased software developer productivity. In addition, a systems integrator currently uses this tool to help understand and correct the Year 2000 date references in customer software     

From Aristotle to Ringelmann: a large-scale analysis of team productivity and coordination in Open Source Software projects

Complex software development projects rely on the contribution of teams of developers, who are required to collaborate and coordinate their efforts. The productivity of such development teams, i.e., how their size is related to the produced output, is an important consideration for project and schedule management as well as for cost estimation. The majority of studies in empirical software engineering suggest that - due to coordination overhead - teams of collaborating developers become less productive as they grow in size. This phenomenon is commonly paraphrased as Brooks’ law of software project management, which states that “adding manpower to a software project makes it later”. Outside software engineering, the non-additive scaling of productivity in teams is often referred to as the Ringelmann effect, which is studied extensively in social psychology and organizational theory. Conversely, a recent study suggested that in Open Source Software (OSS) projects, the productivity of developers increases as the team grows in size. Attributing it to collective synergetic effects, this surprising finding was linked to the Aristotelian quote that “the whole is more than the sum of its parts”. Using a data set of 58 OSS projects with more than 580,000 commits contributed by more than 30,000 developers, in this article we provide a large-scale analysis of the relation between size and productivity of software development teams. Our findings confirm the negative relation between team size and productivity previously suggested by empirical software engineering research, thus providing quantitative evidence for the presence of a strong Ringelmann effect. Using fine-grained data on the association between developers and source code files, we investigate possible explanations for the observed relations between team size and productivity. In particular, we take a network perspective on developer-code associations in software development teams and show that the magnitude of the decrease in productivity is likely to be related to the growth dynamics of co-editing networks which can be interpreted as a first-order approximation of coordination requirements.     

基于概念传播的软件项目代码注释生成方法

软件代码注释生成是软件工程领域近期研究的一个重要问题.目前很多研究工作已经在包含大量<代码片段,注释语句>对的开源数据集上取得了较好效果.但在企业应用中,待注释的代码往往是一个软件项目库,其必须首先决策在哪些代码行上生成注释更好,而且待注释的代码片段大小、粒度各不相同,需要研究提出一种注释决策和生成一体化的、抗噪音的代码注释生成方法.针对这个问题,提出一个面向软件项目的代码自动注释生成方法CoComment.所提方法能够自动抽取软件项目文档中的领域基本概念,并基于代码解析与文本匹配进行概念传播和扩展.在此基础上,通过定位概念相关的代码行/段进行自动注释决策,最终利用模板融合概念和上下文生成具有高可读性的自然语言代码注释.目前CoComment已经在3个企业软件项目、超过4.6万条人工代码注释数据上进行了对比试验.结果表明,所提方法不仅能够有效地进行代码注释决策,其注释内容与现有方法相比也能够提供更多有益于理解代码的信息,从而为软件项目代码的注释决策和注释生成问题提供了一种一体化的解决方案.     

Evaluating the effect of a lightweight formal technique in industry

We evaluate the effect of applying the commercial formal technique Analytical Software Design (ASD) to an industrial project. In ASD, interfaces and software designs are modelled using a formal tabular notation. The ASD tool set supports formal checks of these models, such as deadlock freedom and interface compliance. In addition, full code can be generated from design models. ASD has been applied at Philips Healthcare to develop parts of the software of interventional X-ray systems. We report about the experiences with the embedding of ASD into the development processes. The quality of the resulting code and the productivity has been analysed and compared to code developed with other techniques. We observe that the use of ASD leads to a strong reduction of the number of defects and an increase in productivity. The results are also compared to the literature about standards and related projects at other companies.     

Test-driven modeling for model-driven development

A new software development process called test-driven modeling applies the Extreme Programming test-driven paradigm in a model-driven development environment. (The basis of this article is a project in Motorola's iDEN division that is extending and migrating a large legacy telecommunication system to new platforms using TDM.) This process involves automatic testing through simulation and using executable models as living software system architecture documents. In TDM, we use the same message sequence charts (MSCs) for both system analysis (or design documents) and unit test cases. Similarly, we use the same high-level modeling diagrams for both automatic code generation and living software architecture documents to guide the system's detailed implementation in later phases. Practical results show that developers can effectively apply TDM to large projects with high productivity and quality in terms of the number of code defects.     

Comparing observed bug and productivity rates for Java and C++

An experiment was conducted to compare programmer productivity and defect rates for Java and C++. A modified version of the Personal Software Process (PSP) was used to gather defect rate, bug rate, and productivity data on C++ and Java during two real world development projects. A bug is defined to be a problem detected during testing or deployment. A defect is either a bug, or an error detected during compile time. A typical C++ program had two to three times as many bugs per line of code as a typical Java program. C++ also generated between 15 per cent and 50 per cent more defects per line, and perhaps took six times as long to debug. Java was between 30 per cent and 200 per cent more productive, in terms of lines of code per minute. When defects were measured against development time, Java and C++ showed no difference, but C++ had two to three times as many bugs per hour. Statistics were generated using Student's t‐test at a 95 per cent confidence level. Some discussion of why the differences occurred is included, but the reasons offered have not been tested experimentally. The study is limited to one programmer over two projects, so it is not a definitive experimental result. The programmer was experienced in C++, but only learning Java, so the results would probably favour Java more strongly for equally‐experienced programmers. The experiment shows that it is possible to experimentally measure the fitness of a programming language. Copyright © 1999 John Wiley & Sons, Ltd.     

Factors that motivate software engineering teams: A four country empirical study

Motivation, although difficult to quantify, is considered to be the single largest factor in developer productivity; there are also suggestions that low motivation is an important factor in software development project failure. We investigate factors that motivate software engineering teams using survey data collected from software engineering practitioners based in Australia, Chile, USA and Vietnam. We also investigate the relationship between team motivation and project outcome, identifying whether the country in which software engineering practitioners are based affects this relationship. Analysis of 333 questionnaires indicates that failed projects are associated with low team motivation. We found a set of six common team motivational factors that appear to be culturally independent (project manager has good communication with project staff, project risks reassessed, controlled and managed during the project, customer has confidence in the project manager and the development team, the working environment is good, the team works well together, and the software engineer had a pleasant experience). We also found unique groupings of team motivational factors for each of the countries investigated. This indicates that there are cultural differences that project managers need to consider when working in a global environment.     

ERP Customization as Software Engineering: Knowledge Sharing and Cooperation

Enterprise resource planning (ERP) vendors provide multiple configuration possibilities ranging from module selection to master data provision to steer access rights for different users. These configuration possibilities cover anticipated variability. If the customer requires adaptation beyond what's anticipated, the source code of the product must be adapted. Customizations in this article's context are source code based adaptations of software products. The size and complexity of customizations range from simple report generation to developing independent add-ons that support specific businesses, for example, solutions for flight carriers. The size and lead time of such projects can compare to a full-size software development project. Enterprise resource planning (ERP) systems must be configured and customized to fit a specific company. The authors discuss cooperation with regard to ERP systems customization.     

A statistical analysis of the effects of Scrum and Kanban on software development projects

Traditionally, software development processes have relied on the use of the “Waterfall” and “Vee” models. Later, Agile methodologies were used to handle the challenges of managing complex projects during the development phase. Agile methodologies are a group of incremental and iterative methods that are more effective, and have been used in project management. Kanban and Scrum are two powerful Agile project management approaches in software development. The objective of Scrum and Kanban is achieved by optimizing the development process by identifying the tasks, managing time more effectively, and setting-up teams. A review of the literature reveals that there is a lack of statistical evidence to conclude which methodology is more effective in dealing with the traditional project management factors of budget handling, risk control, quality of the project, available resources, having clear project scope, and schedule handling. This research statistically compares the effectiveness of the Scrum and Kanban methods in terms of their effects on the project management factors for software development projects. Numerical analysis is performed based on survey responses from those with experience in the Scrum and Kanban methods. Results suggest that both Scrum and Kanban lead to the development of successful projects, and that the Kanban method can be better than the Scrum method in terms of managing project schedule.     

The Apache HTTP Server Project

Most reports of Internet collaboration refer to small scale operations among a few authors or designers. However, several projects have shown that the Internet can also be the locus for large scale collaboration. In these projects, contributors from around the world combine their individual forces and develop a product that rivals those of multibillion dollar corporations. The Apache HTTP Server Project is a case in point. This collaborative software development effort has created a robust, feature-rich HTTP server software package that currently dominates the public Internet market (46 percent compared with 16 percent for Microsoft and 12 percent for Netscape, according to a June 1997 survey published by Netcraft). The software and its source code are free, but Apache's popularity is more often attributed to performance than price. The project is managed by the Apache Group, a geographically distributed group of volunteers who use the Internet and Web to communicate, develop, and distribute the server and its related documentation. In addition, hundreds of users have contributed ideas, code, and documentation to the project     

Software effort models for early estimation of process controlapplications

Models are developed to estimate lines of code and function counts directly from user application features of process control systems early in the software development lifecycle. Since the application features are known with reasonable degree of confidence during early stages of development, it is possible to extend the use of the constructive cost model (COCOMO) and function-points-based approach for early software cost estimation. Alternative feature-based models that estimate size and effort using application features and productivity factors are developed. The feature-based models have been shown to estimate software effort with the least error     

Studying the impact of social interactions on software quality

Correcting software defects accounts for a significant amount of resources in a software project. To make best use of testing efforts, researchers have studied statistical models to predict in which parts of a software system future defects are likely to occur. By studying the mathematical relations between predictor variables used in these models, researchers can form an increased understanding of the important connections between development activities and software quality. Predictor variables used in past top-performing models are largely based on source code-oriented metrics, such as lines of code or number of changes. However, source code is the end product of numerous interlaced and collaborative activities carried out by developers. Traces of such activities can be found in the various repositories used to manage development efforts. In this paper, we develop statistical models to study the impact of social interactions in a software project on software quality. These models use predictor variables based on social information mined from the issue tracking and version control repositories of two large open-source software projects. The results of our case studies demonstrate the impact of metrics from four different dimensions of social interaction on post-release defects. Our findings show that statistical models based on social information have a similar degree of explanatory power as traditional models. Furthermore, our results demonstrate that social information does not substitute, but rather augments traditional source code-based metrics used in defect prediction models.     

A statistical study of the relevance of lines of code measures in software projects

Lines of code metrics are routinely used as measures of software system complexity, programmer productivity, and defect density, and are used to predict both effort and cost. The guidelines for using a direct metric, such as lines of code, as a proxy for a quality factor such as complexity or defect density, or in derived metrics such as cost and effort are clear. Amongst other criteria, the direct metric must be linearly related to, and accurately predict, the quality factor and these must be validated through statistical analysis following a rigorous validation methodology. In this paper, we conduct such an analysis to determine the validity and utility of lines of code as a measure using the ISBGS-10 data set. We find that it fails to meet the specified validity tests and, therefore, has limited utility in derived measures.