Efficient management of inspections in software development projects

During the last two decades a universal agreement has been established on the fact that software inspections play a fundamental role in improving software quality. The number of software organizations that have incorporated formal reviews in their development process is constantly increasing and the belief that efficient inspections can not only detect defects but also reduce cycle time and lower costs is spreading. However, despite the importance of the inspections in a software development project, scheduling of inspections has not been given the necessary attention so far. As a result, inspections tend to accumulate towards internal project deadlines, possibly leading to excess overtime costs, quality degradation and difficulties in meeting milestones. In this paper, data from a major telecommunications software project is analyzed in an effort to illustrate the problems that can arise from inefficient planning of inspections and their related activities.     

Fully employing software inspections data

Software inspections provide a proven approach to quality assurance for software products of all kinds, including requirements, design, code, test plans, among others. Common to all inspections is the aim of finding and fixing defects as early as possible, and thereby providing cost savings by minimizing the amount of rework necessary later in the life cycle. Measurement data, such as the number and type of found defects and the effort spent by the inspection team, provide not only direct feedback about the software product to the project team, but are also valuable for process improvement activities. In this paper, we discuss NASA??s use of software inspections and the rich set of data that has resulted. In particular, we present results from analysis of inspection data that illustrate the benefits of fully utilizing that data for process improvement at several levels. Examining such data across multiple inspections or projects allows team members to monitor and trigger cross project improvements. Such improvements may focus on the software development processes of the whole organization as well as improvements to the applied inspection process itself.     

State‐of‐the‐art: software inspections after 25 years

Software inspections, which were originally developed by Michael Fagan in 1976, are an important means to verify and achieve sufficient quality in many software projects today. Since Fagan's initial work, the importance of software inspections has been long recognized by software developers and many organizations. Various proposals have been made by researchers in the hope of improving Fagan's inspection method. The proposals include structural changes to the process and several types of support for the inspection process. Most of the proposals have been empirically investigated in different studies. This is a review paper focusing on the software inspection process in the light of Fagan's inspection method and it summarizes and reviews other types of software inspection processes that have emerged in the last 25 years. This paper also addresses important issues related to the inspection process and examines experimental studies and their findings that are of interest with the purpose of identifying future avenues of research in software inspection. Copyright © 2002 John Wiley & Sons, Ltd.     

Simplified software inspection process in compliance with international standards

A significant amount of software is developed all over the world by small and medium size software organizations. These organizations do not have enough infrastructures and resources to implement an austere quality plan. Software inspection is a fundamental component of the software quality assurance process. Formal review methods are rigorous and their implementation is cumbersome for small and medium enterprises. In this paper, we have presented a new simplified inspection process which is easy to implement, requires fewer resource and almost no documentation. Also, people who are conducting this inspection need not be present at the same place during most stages of the inspection process. We have also compared this process with IEEE and NASA standards for software inspection and found that it meets almost 99% of both standards. While there has been much research on inspection, little attention is paid towards compliance with international standards. These results could be used as a basis for further research in software inspection and process towards aligning with international standards. This process has been successfully implemented in a CMM level 3 software development organization which is striving to accomplish higher maturity levels to establish at the international level.     

Applying sampling to improve software inspections

The main objective of software inspections is to find faults in software documents. The benefits of inspections are reported from researchers as well as software organizations. However, inspections are time consuming and the resources may not be sufficient to inspect all documents. Sampling of documents in inspections provides a systematic solution to select what to be inspected in the case resources are not sufficient to inspect everything. The method presented in this paper uses sampling, inspection and resource scheduling to increase the efficiency of an inspection session. A pre-inspection phase is used in order to determine which documents need most inspection time, i.e. which documents contain most faults. Then, the main inspection is focused on these documents. We describe the sampling method and provide empirical evidence, which indicates that the method is appropriate to use. A Monte Carlo simulation is used to evaluate the proposed method and a case study using industrial data is used to validate the simulation model. Furthermore, we discuss the results and important future research in the area of sampling of software inspections.     

Issues in software inspection

In manufacturing, agriculture and transportation, inspections are a routine part of business, making our lives safer and more comfortable. Yet in software, we generally leave inspections to companies that are quite advanced in their process maturity. Why do most companies skip inspections when their is so much to gain? In 1991, the author began the National Software Quality Experiment, an ongoing database of software defects and inspections practice. In his study, participants from dozens of organizations described defects and these were entered into the experiment's database. In this article, the author makes a case for implementing inspections at all levels of software development practice, based on what he has learned thus far     

Source code review systems

Inspections are a well-established, cost-effective way to find defects. But they are not universally used. There are many reasons, including the lack of training on how to do inspections well, the need for project managers to move resources away from testing into inspections, and the large amount of paperwork that formal inspections require. Because of these problems, the software inspections program at Soluris had withered away. When the software engineering group at Soluris decided to relaunch inspection process, they did two things. First, they purchased Peer Reviews in Software by Karl Wiegers. It helped them fix inspection process by describing how to have an inspection meeting, how much to limit the length of the items under inspection, and what data to collect to justify the inspections over the long term. Second, they selected a software tool to automate our inspections process and thus eliminate the paper forms. This article compares two open source tools ugzilla and Codestriker - and three commercial tools - the CodeReview add-on for Visual Studio .Net, CodeReviewer, and ReviewPro.     

The role of inspection in software quality assurance

Due to the complexity of the code, software is released with many errors. In response, both software practitioners and software researchers need to improve the reputation of the software. Inspection is the only way to improve the quality of software. Inspection methods can be more effective but success depends on having a sound and systematic procedure for conducting the inspection. The Workshop on Inspection in Software Engineering (WISE), a satellite event of the 2001 Computer Aided Verification (CAV '01) Conference, brought together researchers, practitioners, and regulators in the hope of finding effective approaches to software inspection. The workshop included invited lectures and paper presentations in the form of panel discussions on all aspects of software inspection. Submissions explained how practitioners and researchers were performing inspections, discussed the relevance of inspections, provided evidence of how inspections could be improved through refinement of the inspection process and computer aided tool support and explained how careful design of software could make inspections easier or more effective.     

Evaluation of Usage-Based Reading—Conclusions after Three Experiments

Software inspections have been introduced in software engineering in order to detect faults before testing is performed. Reading techniques provide reviewers in software inspections with guidelines on how they should check the documents under inspection. Several reading techniques with different purposes have been introduced and empirically evaluated. In this paper, we describe a reading technique with the special aim to detect faults that are severe from a user’s point of view. The reading technique is named usage-based reading (UBR) and it can be used to inspect all software artefacts. In the series of experiments, a high-level design document is used. The main focus of the paper is on the third experiment, which investigates the information needed for UBR in the individual preparation and the meeting of software inspections. Hence, the paper discusses (1) the series of three experiments of UBR, (2) the individual preparation of the third experiment, and (3) the meeting part of the third experiment. For each of these three parts, results are produced. The main results are (1) UBR is an efficient and effective reading technique that can be used for user-focused software inspections, (2) UBR is more efficient and effective if the information used for UBR is developed prior to, instead of during the individual preparation, and (3) the meeting affects the UBR inspection in terms of increased effectiveness and decreased efficiency. In summary, the empirical evidence shows that UBR is an efficient and effective reading technique to be used by software organizations that produce software for which the user perceived quality is important.     

Defect propagation at the project-level: results and a post-hoc analysis on inspection efficiency

Inspections are increasingly utilized to enhance software quality. While the effectiveness of inspections in uncovering defects is widely accepted, there is a lack of research that takes a more holistic approach by considering defect counts from initial phases of the development process (requirements, design, and coding) and examining defect propagation where defect counts are aggregated to the project-level (i.e., application-level). Using inspection data collected from a large software development firm, this paper investigates the extent of defect propagation at the project-level during early lifecycle phases. I argue that defect propagation can be observed from the relationship between defects in the prior phase and the defects in the subsequent phase. Both Ordinary Least Squares and 3-Stage Least Squares analyses support the hypotheses on defect propagation. Moreover, results show that the inspection efficiency (defects per unit inspection time) decreases as the software product progresses from requirements to design to coding. A post-hoc analysis revealed further insights into inspection efficiency. In each phase, as the inspection time increased, efficiency reached an optimal point and then dropped off. In addition, a project’s inspection efficiency generally tends to remain stable from one phase to another. These insights offer managers means to assess inspections, their efficiency, and make adjustments to the time allotted to inspect project’s artifacts in both the current and the subsequent phase. Implications for managers and future research directions are discussed.     

Key lessons in achieving widespread inspection use

Hewlett-Packard has distilled its experience in promoting software inspections into a model of how technology adoption occurs and a metric of where it stands. Its managers know when and how to accelerate efforts to adopt inspections and other best practices. Experience has shown that the return on investment in technology adoption efforts can be huge. At HP, inspection technology was adopted across the entire company in four recognizable stages, which are defined as: experimental, initial guidelines, widespread belief and adoption, and standardization. HP's inspections program has progressed through three of these stages. We address several key questions: What characterizes the four stages? What were the most important lessons learned? What situations led to failure or success? And most important, how can we apply what we learned to speed the adoption of other proven practices? The lessons related will give many who are directly or indirectly responsible for software process improvement more confidence that inspections apply to all software-development organizations     

A family of experiments to investigate the effects of groupware for software inspection

It is widely accepted that the inspection of software artifacts can find defects early in the development process and gather information on the quality of the evolving product. However, the inspection process is resource-intensive and involves tedious tasks, such as searching, sorting, and checking. Tool support for inspections can help accelerating these tasks and allows inspectors to concentrate on tasks particularly needing human attention. Only few tools are available for inspections. We have thus developed a set of groupware tools for both individual defect detection and inspection meetings to lower the effort of inspections and to increase their efficiency. This paper presents the Groupware-supported Inspection Process (GrIP) and describes tools for inspecting software requirements. As only little empirical work exists that directly compares paper-based and tool-based software inspection, we conducted a family of experiments in an academic environment to empirically investigate the effect of tool support regarding defect detection and inspection meetings. The main results of our family of experiments regarding individual defect detection are promising: The effectiveness of inspectors and teams is comparable to paper-based inspection without tool support; the inspection effort and defect overlap decreases significantly with tool support, while the efficiency of inspection teams increases considerably. Regarding tool support for inspection meetings the main findings of the experiments are that tool support considerably lowers the meeting effort, supports inspectors in identifying false positives, and reduces the number of true defects lost during a meeting. The number of unidentified false positives is still quite high.     

Optimized allocation of defect inspection capacity with a dynamic sampling strategy

In semiconductor manufacturing, in-line inspections are necessary to monitor processes, products and tools in order to reduce excursions and achieve high yields of final products. However, capacity is limited and inspections directly impact the cycle times of products. Sampling strategies are used to improve product yields while limiting the number of inspections, and thus the impact on the cycle times of the inspected lots. Dynamic sampling has been recently introduced and new models are required to estimate the associated inspection capacity. In this paper, we focus on micro-defect inspections and the risk on process tools in terms of Wafers at Risk (W@R), which is the number of wafers processed on a tool since its latest defect inspection. A linear programming model that estimates the required defect inspection capacity to satisfy the W@R limits on process tools is proposed. Our model can be used at different decision levels. At the tactical level, it shows if W@R limits can be satisfied when the product mix changes and/or if planned W@R reductions can be met with the available inspection capacity. At the strategic level, the model helps to justify capacity investments if the objectives in terms of W@R reduction cannot be achieved with the available inspection capacity. Numerical experiments on industrial data are performed and discussed.     

A comprehensive evaluation of capture-recapture models forestimating software defect content

An important requirement to control the inspection of software artifacts is to be able to decide, based on more objective information, whether the inspection can stop or whether it should continue to achieve a suitable level of artifact quality. A prediction of the number of remaining defects in an inspected artifact can be used for decision making. Several studies in software engineering have considered capture-recapture models to make a prediction. However, few studies compare the actual number of remaining defects to the one predicted by a capture-recapture model on real software engineering artifacts. The authors focus on traditional inspections and estimate, based on actual inspections data, the degree of accuracy of relevant state-of-the-art capture-recapture models for which statistical estimators exist. In order to assess their robustness, we look at the impact of the number of inspectors and the number of actual defects on the estimators' accuracy based on actual inspection data. Our results show that models are strongly affected by the number of inspectors, and therefore one must consider this factor before using capture-recapture models. When the number of inspectors is too small, no model is sufficiently accurate and underestimation may be substantial. In addition, some models perform better than others in a large number of conditions and plausible reasons are discussed. Based on our analyses, we recommend using a model taking into account that defects have different probabilities of being detected and the corresponding Jackknife Estimator. Furthermore, we calibrate the prediction models based on their relative error, as previously computed on other inspections. We identified theoretical limitations to this approach which were then confirmed by the data     

Analyzing the relationships between inspections and testing to provide a software testing focus

ContextQuality assurance effort, especially testing effort, is frequently a major cost factor during software development. Consequently, one major goal is often to reduce testing effort. One promising way to improve the effectiveness and efficiency of software quality assurance is the use of data from early defect detection activities to provide a software testing focus. Studies indicate that using a combination of early defect data and other product data to focus testing activities outperforms the use of other product data only. One of the key challenges is that the use of data from early defect detection activities (such as inspections) to focus testing requires a thorough understanding of the relationships between these early defect detection activities and testing. An aggravating factor is that these relationships are highly context-specific and need to be evaluated for concrete environments.ObjectiveThe underlying goal of this paper is to help companies get a better understanding of these relationships for their own environment, and to provide them with a methodology for finding relationships in their own environments.MethodThis article compares three different strategies for evaluating assumed relationships between inspections and testing. We compare a confidence counter, different quality classes, and the F-measure including precision and recall.ResultsOne result of this case-study-based comparison is that evaluations based on the aggregated F-measures are more suitable for industry environments than evaluations based on a confidence counter. Moreover, they provide more detailed insights about the validity of the relationships.ConclusionWe have confirmed that inspection results are suitable data for controlling testing activities. Evaluated knowledge about relationships between inspections and testing can be used in the integrated inspection and testing approach In²Test to focus testing activities. Product data can be used in addition. However, the assumptions have to be evaluated in each new context.     

An internally replicated quasi-experimental comparison of checklistand perspective based reading of code documents

The basic premise of software inspections is that they detect and remove defects before they propagate to subsequent development phases where their detection and correction cost escalates. To exploit their full potential, software inspections must call for a close and strict examination of the inspected artifact. For this, reading techniques for defect detection may be helpful since these techniques tell inspection participants what to look for and, more importantly, how to scrutinize a software artifact in a systematic manner. Recent research efforts investigated the benefits of scenario-based reading techniques. A major finding has been that these techniques help inspection teams find more defects than existing state-of-the-practice approaches, such as, ad-hoc or checklist-based reading (CBR). We experimentally compare one scenario-based reading technique, namely, perspective-based reading (PBR), for defect detection in code documents with the more traditional CBR approach. The comparison was performed in a series of three studies, as a quasi experiment and two internal replications, with a total of 60 professional software developers at Bosch Telecom GmbH. Meta-analytic techniques were applied to analyze the data     

In-process evaluation for software inspection and test

The goal of software inspection and test is to reduce the expected cost of software failure over the life of a product. The authors extend the use of defect triggers, the events that cause defects to be discovered, to help evaluate the effectiveness of inspections and test scenarios. In the case of inspections, the defect trigger is defined as a set of values that associate the skills of the inspector with the discovered defect. Similarly, for test scenarios, the defect trigger values embody the deferring strategies being used in creating these scenarios. The usefulness of triggers in evaluating the effectiveness of software inspections and tests is demonstrated by evaluating the inspection and test activities of some software products. These evaluations are used to point to deficiencies in inspection and test strategies, and to progress made in improving such strategies. The trigger distribution of the entire inspection or test series may then be used to highlight areas for further investigation, with the aim of improving the design, implementation, and test processes     

Software inspections: an effective verification process

The authors explain how to perform software inspections to locate defects. They present metrics for inspection and examples of its effectiveness. The authors contend, on the basis of their experiences and those reported in the literature, that inspections can detect and eliminate faults more cheaply than testing     

A Pattern-Based Framework for Software Anomaly Detection

This paper presents a pattern-based framework for developing tool support to detect software anomalies. The use of a pattern-based approach is important because it provides the flexibility needed to address domain-specific needs, with respect to the types of problems the tools detect and the strategies used to inspect and adapt the code. Patterns can be used to detect a variety of problems, ranging from simple syntactic issues to difficult semantic problems requiring global analysis. Patterns can also be used to describe transformations of the software, used to rectify problems detected through software inspection, and to support interactive inspection and adaptation when full automation is impractical. This paper describes a part of the Knowledge Centric Software (KCS) framework that embodies the pattern-based approach and provides capabilities for addressing different languages and different application domains. While only the part of the framework relevant to code inspections is addressed in this paper, in future, we also expect to address UML analysis and design models. As an application of the research, we present an overview of an inspection tool being developed for high assurance software for avionics systems.     

Vessel Inspection: A Micro-Aerial Vehicle-based Approach

Vessel maintenance entails periodic visual inspections of the internal and external parts of the hull in order to detect the typical defective situations affecting metallic structures, such as coating breakdown, corrosion, cracks, etc. The main goal of project MINOAS is the automation of the inspection process, currently undertaken by human surveyors, by means of a fleet of robotic agents. This paper overviews an approach to the inspection problem based on an autonomous Micro Aerial Vehicle (MAV) which, as part of this fleet, is in charge of regularly supplying images that can teleport the surveyor from a base station to the areas of the hull to be inspected. The control software approach adopted for the MAV is fully described, with a special emphasis on the self-localization capabilities of the vehicle. Experimental results showing the suitability of the platform to the application are as well reported and discussed.