What do we know about defect detection methods? [software testing]

A survey of defect detection studies comparing inspection and testing techniques yields practical recommendations: use inspections for requirements and design defects, and use testing for code. Evidence-based software engineering can help software practitioners decide which methods to use and for what purpose. EBSE involves defining relevant questions, surveying and appraising avail able empirical evidence, and integrating and evaluating new practices in the target environment. This article helps define questions regarding defect detection techniques and presents a survey of empirical studies on testing and inspection techniques. We then interpret the findings in terms of practical use. The term defect always relates to one or more underlying faults in an artifact such as code. In the context of this article, defects map to single faults.     

Further investigations of reading techniques for object-oriented design inspection

This paper describes an experimental evaluation of two reading techniques, namely Checklist-based reading (CBR) and Perspective-based reading (PBR) for Object-Oriented (OO) design document, written using the notation of Unified Modelling Language, inspection. Inspections are usually applied for defect detection in requirement documents or code modules, and there is a significant lack of information how inspections should be applied to OO design documents.The paper reports on a controlled experiment with 59 subject students to compare CBR and PBR techniques with respect to individual time spent on inspection, cost per defect, effectiveness, and 3-person simulated team effectiveness.     

A family of experiments to investigate the effects of groupware for software inspection

It is widely accepted that the inspection of software artifacts can find defects early in the development process and gather information on the quality of the evolving product. However, the inspection process is resource-intensive and involves tedious tasks, such as searching, sorting, and checking. Tool support for inspections can help accelerating these tasks and allows inspectors to concentrate on tasks particularly needing human attention. Only few tools are available for inspections. We have thus developed a set of groupware tools for both individual defect detection and inspection meetings to lower the effort of inspections and to increase their efficiency. This paper presents the Groupware-supported Inspection Process (GrIP) and describes tools for inspecting software requirements. As only little empirical work exists that directly compares paper-based and tool-based software inspection, we conducted a family of experiments in an academic environment to empirically investigate the effect of tool support regarding defect detection and inspection meetings. The main results of our family of experiments regarding individual defect detection are promising: The effectiveness of inspectors and teams is comparable to paper-based inspection without tool support; the inspection effort and defect overlap decreases significantly with tool support, while the efficiency of inspection teams increases considerably. Regarding tool support for inspection meetings the main findings of the experiments are that tool support considerably lowers the meeting effort, supports inspectors in identifying false positives, and reduces the number of true defects lost during a meeting. The number of unidentified false positives is still quite high.     

An experiment to assess the cost-benefits of code inspections inlarge scale software development

We conducted a long term experiment to compare the costs and benefits of several different software inspection methods. These methods were applied by professional developers to a commercial software product they were creating. Because the laboratory for this experiment was a live development effort, we took special care to minimize cost and risk to the project, while maximizing our ability to gather useful data. The article has several goals: (1) to describe the experiment's design and show how we used simulation techniques to optimize it; (2) to present our results and discuss their implications for both software practitioners and researchers; and (3) to discuss several new questions raised by our findings. For each inspection, we randomly assigned three independent variables: (1) the number of reviewers on each inspection team (1, 2, or 4); (2) the number of teams inspecting the code unit (1 or 2); and (3) the requirement that defects be repaired between the first and second team's inspections. The reviewers for each inspection were randomly selected without replacement from a pool of 11 experienced software developers. The dependent variables for each inspection included inspection interval (elapsed time), total effort, and the defect detection rate. Our results showed that these treatments did not significantly influence the defect detection effectiveness, but that certain combinations of changes dramatically increased the inspection interval     

An experimental comparison of usage-based and checklist-based reading

Software quality can be defined as the customers' perception of how a system works. Inspection is a method to monitor and control the quality throughout the development cycle. Reading techniques applied to inspections help reviewers to stay focused on the important parts of an artifact when inspecting. However, many reading techniques focus on finding as many faults as possible, regardless of their importance. Usage-based reading helps reviewers to focus on the most important parts of a software artifact from a user's point of view. We present an experiment, which compares usage-based and checklist-based reading. The results show that reviewers applying usage-based reading are more efficient and effective in detecting the most critical faults from a user's point of view than reviewers using checklist-based reading. Usage-based reading may be preferable for software organizations that utilize or start utilizing use cases in their software development.     

A comprehensive evaluation of capture-recapture models forestimating software defect content

An important requirement to control the inspection of software artifacts is to be able to decide, based on more objective information, whether the inspection can stop or whether it should continue to achieve a suitable level of artifact quality. A prediction of the number of remaining defects in an inspected artifact can be used for decision making. Several studies in software engineering have considered capture-recapture models to make a prediction. However, few studies compare the actual number of remaining defects to the one predicted by a capture-recapture model on real software engineering artifacts. The authors focus on traditional inspections and estimate, based on actual inspections data, the degree of accuracy of relevant state-of-the-art capture-recapture models for which statistical estimators exist. In order to assess their robustness, we look at the impact of the number of inspectors and the number of actual defects on the estimators' accuracy based on actual inspection data. Our results show that models are strongly affected by the number of inspectors, and therefore one must consider this factor before using capture-recapture models. When the number of inspectors is too small, no model is sufficiently accurate and underestimation may be substantial. In addition, some models perform better than others in a large number of conditions and plausible reasons are discussed. Based on our analyses, we recommend using a model taking into account that defects have different probabilities of being detected and the corresponding Jackknife Estimator. Furthermore, we calibrate the prediction models based on their relative error, as previously computed on other inspections. We identified theoretical limitations to this approach which were then confirmed by the data     

Defect propagation at the project-level: results and a post-hoc analysis on inspection efficiency

Inspections are increasingly utilized to enhance software quality. While the effectiveness of inspections in uncovering defects is widely accepted, there is a lack of research that takes a more holistic approach by considering defect counts from initial phases of the development process (requirements, design, and coding) and examining defect propagation where defect counts are aggregated to the project-level (i.e., application-level). Using inspection data collected from a large software development firm, this paper investigates the extent of defect propagation at the project-level during early lifecycle phases. I argue that defect propagation can be observed from the relationship between defects in the prior phase and the defects in the subsequent phase. Both Ordinary Least Squares and 3-Stage Least Squares analyses support the hypotheses on defect propagation. Moreover, results show that the inspection efficiency (defects per unit inspection time) decreases as the software product progresses from requirements to design to coding. A post-hoc analysis revealed further insights into inspection efficiency. In each phase, as the inspection time increased, efficiency reached an optimal point and then dropped off. In addition, a project’s inspection efficiency generally tends to remain stable from one phase to another. These insights offer managers means to assess inspections, their efficiency, and make adjustments to the time allotted to inspect project’s artifacts in both the current and the subsequent phase. Implications for managers and future research directions are discussed.     

Evaluation of Usage-Based Reading—Conclusions after Three Experiments

Software inspections have been introduced in software engineering in order to detect faults before testing is performed. Reading techniques provide reviewers in software inspections with guidelines on how they should check the documents under inspection. Several reading techniques with different purposes have been introduced and empirically evaluated. In this paper, we describe a reading technique with the special aim to detect faults that are severe from a user’s point of view. The reading technique is named usage-based reading (UBR) and it can be used to inspect all software artefacts. In the series of experiments, a high-level design document is used. The main focus of the paper is on the third experiment, which investigates the information needed for UBR in the individual preparation and the meeting of software inspections. Hence, the paper discusses (1) the series of three experiments of UBR, (2) the individual preparation of the third experiment, and (3) the meeting part of the third experiment. For each of these three parts, results are produced. The main results are (1) UBR is an efficient and effective reading technique that can be used for user-focused software inspections, (2) UBR is more efficient and effective if the information used for UBR is developed prior to, instead of during the individual preparation, and (3) the meeting affects the UBR inspection in terms of increased effectiveness and decreased efficiency. In summary, the empirical evidence shows that UBR is an efficient and effective reading technique to be used by software organizations that produce software for which the user perceived quality is important.     

Investigating the defect detection effectiveness and cost benefit of nominal inspection teams

Inspection is an effective but also expensive quality assurance activity to find defects early during software development. The defect detection process, team size, and staff hours invested can have a considerable impact on the defect detection effectiveness and cost-benefit of an inspection. In this paper, we use empirical data and a probabilistic model to estimate this impact for nominal (noncommunicating) inspection teams in an experiment context. Further, the analysis investigates how cutting off the inspection after a certain time frame would influence inspection performance. Main findings of the investigation are: 1) Using combinations of different reading techniques in a team is considerably more effective than using the best single technique only (regardless of the observed level of effort). 2) For optimizing the inspection performance, determining the optimal process mix in a team is more important than adding an inspector (above a certain team size) in our model. 3) A high level of defect detection effectiveness is much more costly to achieve than a moderate level since the average cost for the defects found by the inspector last added to a team increases more than linearly with growing effort investment. The work provides an initial baseline of inspection performance with regard to process diversity and effort in inspection teams. We encourage further studies on the topic of time usage with defect detection techniques and its effect on inspection effectiveness in a variety of inspection contexts to support inspection planning with limited resources.     

In-process evaluation for software inspection and test

The goal of software inspection and test is to reduce the expected cost of software failure over the life of a product. The authors extend the use of defect triggers, the events that cause defects to be discovered, to help evaluate the effectiveness of inspections and test scenarios. In the case of inspections, the defect trigger is defined as a set of values that associate the skills of the inspector with the discovered defect. Similarly, for test scenarios, the defect trigger values embody the deferring strategies being used in creating these scenarios. The usefulness of triggers in evaluating the effectiveness of software inspections and tests is demonstrated by evaluating the inspection and test activities of some software products. These evaluations are used to point to deficiencies in inspection and test strategies, and to progress made in improving such strategies. The trigger distribution of the entire inspection or test series may then be used to highlight areas for further investigation, with the aim of improving the design, implementation, and test processes     

Analyzing the relationships between inspections and testing to provide a software testing focus

ContextQuality assurance effort, especially testing effort, is frequently a major cost factor during software development. Consequently, one major goal is often to reduce testing effort. One promising way to improve the effectiveness and efficiency of software quality assurance is the use of data from early defect detection activities to provide a software testing focus. Studies indicate that using a combination of early defect data and other product data to focus testing activities outperforms the use of other product data only. One of the key challenges is that the use of data from early defect detection activities (such as inspections) to focus testing requires a thorough understanding of the relationships between these early defect detection activities and testing. An aggravating factor is that these relationships are highly context-specific and need to be evaluated for concrete environments.ObjectiveThe underlying goal of this paper is to help companies get a better understanding of these relationships for their own environment, and to provide them with a methodology for finding relationships in their own environments.MethodThis article compares three different strategies for evaluating assumed relationships between inspections and testing. We compare a confidence counter, different quality classes, and the F-measure including precision and recall.ResultsOne result of this case-study-based comparison is that evaluations based on the aggregated F-measures are more suitable for industry environments than evaluations based on a confidence counter. Moreover, they provide more detailed insights about the validity of the relationships.ConclusionWe have confirmed that inspection results are suitable data for controlling testing activities. Evaluated knowledge about relationships between inspections and testing can be used in the integrated inspection and testing approach In²Test to focus testing activities. Product data can be used in addition. However, the assumptions have to be evaluated in each new context.     

Usage-based reading—an experiment to guide reviewers with use cases

Reading methods for software inspections are used for aiding reviewers to focus on special aspects in a software artefact. Many experiments were conducted for checklist-based reading and scenario-based reading concluding that the focus is important for software reviewers. This paper describes and evaluates a reading technique called usage-based reading (UBR). UBR utilises prioritised use cases to guide reviewers through an inspection. More importantly, UBR drives the reviewers to focus on the software parts that are most important for a user. An experiment was conducted on 27 third year Bachelor's software engineering students, where one group used use cases sorted in a prioritised order and the control group used randomly ordered use cases. The main result is that reviewers in the group with prioritised use cases are significantly more efficient and effective in detecting the most critical faults from a user's point of view. Consequently, UBR has the potential to become an important reading technique. Future extensions to the reading technique are suggested and experiences gained from the experiment to support replications are provided.     

Source code review systems

Inspections are a well-established, cost-effective way to find defects. But they are not universally used. There are many reasons, including the lack of training on how to do inspections well, the need for project managers to move resources away from testing into inspections, and the large amount of paperwork that formal inspections require. Because of these problems, the software inspections program at Soluris had withered away. When the software engineering group at Soluris decided to relaunch inspection process, they did two things. First, they purchased Peer Reviews in Software by Karl Wiegers. It helped them fix inspection process by describing how to have an inspection meeting, how much to limit the length of the items under inspection, and what data to collect to justify the inspections over the long term. Second, they selected a software tool to automate our inspections process and thus eliminate the paper forms. This article compares two open source tools ugzilla and Codestriker - and three commercial tools - the CodeReview add-on for Visual Studio .Net, CodeReviewer, and ReviewPro.     

Evaluating capture-recapture models with two inspectors

Capture-recapture (CR) models have been proposed as an objective method for controlling software inspections. CR models were originally developed to estimate the size of animal populations. In software, they have been used to estimate the number of defects in an inspected artifact. This estimate can be another source of information for deciding whether the artifact requires a reinspection to ensure that a minimal inspection effectiveness level has been attained. Little evaluative research has been performed thus far on the utility of CR models for inspections with two inspectors. We report on an extensive Monte Carlo simulation that evaluated capture-recapture models suitable for two inspectors assuming a code inspections context. We evaluate the relative error of the CR estimates as well as the accuracy of the reinspection decision made using the CR model. Our results indicate that the most appropriate capture-recapture model for two inspectors is an estimator that allows for inspectors with different capabilities. This model always produces an estimate (i.e., does not fail), has a predictable behavior (i.e., works well when its assumptions are met), will have a relatively high decision accuracy, and will perform better than the default decision of no reinspections. Furthermore, we identify the conditions under which this estimator will perform best     

Empirical Evaluations of Usage-Based Reading and Fault Content Estimation for Software Inspections

This thesis describes six empirical investigations of two techniques applied to software inspections, namely reading techniques and fault content estimation techniques. The first part of the thesis presents a series of experiments of a novel reading technique called usage-based reading. The second part investigates fault content estimation applied to the software inspection process. The estimation methods used are capture-recapture and curve fitting.     

How perspective-based reading can improve requirements inspections

Because defects constitute an unavoidable aspect of software development, discovering and removing them early is crucial. Overlooked defects (like faults in the software system requirements, design, or code) propagate to subsequent development phases where detecting and correcting them becomes more difficult. At best, developers will eventually catch the defects, but at the expense of schedule delays and additional product-development costs. At worst, the defects will remain, and customers will receive a faulty product. The authors explain their perspective based reading (PBR) technique that provides a set of procedures to help developers solve software requirements inspection problems. PBR reviewers stand in for specific stakeholders in the document to verify the quality of requirements specifications. The authors show how PBR leads to improved defect detection rates for both individual reviewers and review teams working with unfamiliar application domains     

Software inspections: an effective verification process

The authors explain how to perform software inspections to locate defects. They present metrics for inspection and examples of its effectiveness. The authors contend, on the basis of their experiences and those reported in the literature, that inspections can detect and eliminate faults more cheaply than testing     

Are the Perspectives Really Different? – Further Experimentation on Scenario-Based Reading of Requirements

Perspective-BasedReading (PBR) is a scenario-based inspection technique whereseveral reviewers read a document from different perspectives(e.g. user, designer, tester). The reading is made accordingto a special scenario, specific for each perspective. The basicassumption behind PBR is that the perspectives find differentdefects and a combination of several perspectives detects moredefects compared to the same amount of reading with a singleperspective. This paper presents a study which analyses the differencesin perspectives. The study is a partial replication of previousstudies. It is conducted in an academic environment using graduatestudents as subjects. Each perspective applies a specific modellingtechnique: use case modelling for the user perspective, equivalencepartitioning for the tester perspective and structured analysisfor the design perspective. A total of 30 subjects were dividedinto 3 groups, giving 10 subjects per perspective. The analysisresults show that (1) there is no significant difference amongthe three perspectives in terms of defect detection rate andnumber of defects found per hour, (2) there is no significantdifference in the defect coverage of the three perspectives,and (3) a simulation study shows that 30 subjects is enough todetect relatively small perspective differences with the chosenstatistical test. The results suggest that a combination of multipleperspectives may not give higher coverage of the defects comparedto single-perspective reading, but further studies are neededto increase the understanding of perspective difference.     

Case-based reasoning for interpretation of data from non-destructive testing

Non-destructive testing (NDT) is a name for a range of methods and procedures used to determine fitness of industrial products for further use. The use of NDT testing techniques results in data in the form of signals, images, or sequences of these, which have to be analysed in order to determine if they contain any indications of defects in the inspected objects. This analysis is often quite complex. In the past, systems have been built which used neural networks (and other statistical classifiers) as well as expert systems to interpret NDT data; however, successful uses of these systems in inspection practice are rare. This article presents how the case-based reasoning methodology (where interpretation of new data is based on previous data-interpretation cases) can be used to tackle the problem of NDT data interpretation. The article presents the characteristics of CBR, which make it an interesting alternative to statistical classifiers and to expert systems. Suitability of CBR for NDT data interpretation is illustrated based on examples of two applications: a CBR system for ultrasonic rail inspection and a CBR system for eddy-current inspection of heat exchangers.     

轨道病害视觉检测：背景、方法与趋势

铁路作为国家重要基础设施、国民经济大动脉和大众化运输方式,对社会经济发展起着不可替代的支撑作用。轨道是铁路系统的重要组件,轨道病害检测是铁路工务部门的核心业务。传统的人工巡检不仅费时费力,而且检测结果容易受到各种主观因素的影响。因此,自动化轨道病害检测对维护铁路运输安全具有重要的现实意义。考虑到视觉检测在速度、成本和可视化等方面的优势,本文聚焦于轨道病害视觉检测技术。首先以广泛应用的无砟轨道为例介绍轨道的基本结构,对常见的轨道表观病害进行样例展示、成因分析和影响评价;简要梳理常见的自动化轨道检测技术的基本原理和应用场景,对轨道病害视觉检测面临的图像质量不均、可用特征较少和模型更新困难等主要挑战进行归纳;然后,依照前景模型、背景模型、盲源分离模型及深度学习模型的分类逻辑对轨道病害视觉检测领域的研究现状进行综述,简要介绍了各类方法的代表性工作,总结了各类方法的技术特点与应用局限性;最后,针对智能化铁路的发展需求,展望了未来轨道病害视觉检测技术的研究趋势,即利用小样本/零样本学习、多任务学习与多源异构数据融合等技术手段来解决当前视觉检测系统中的鲁棒性弱、虚警率高等问题。