Studying just-in-time defect prediction using cross-project models

Unlike traditional defect prediction models that identify defect-prone modules, Just-In-Time (JIT) defect prediction models identify defect-inducing changes. As such, JIT defect models can provide earlier feedback for developers, while design decisions are still fresh in their minds. Unfortunately, similar to traditional defect models, JIT models require a large amount of training data, which is not available when projects are in initial development phases. To address this limitation in traditional defect prediction, prior work has proposed cross-project models, i.e., models learned from other projects with sufficient history. However, cross-project models have not yet been explored in the context of JIT prediction. Therefore, in this study, we empirically evaluate the performance of JIT models in a cross-project context. Through an empirical study on 11 open source projects, we find that while JIT models rarely perform well in a cross-project context, their performance tends to improve when using approaches that: (1) select models trained using other projects that are similar to the testing project, (2) combine the data of several other projects to produce a larger pool of training data, and (3) combine the models of several other projects to produce an ensemble model. Our findings empirically confirm that JIT models learned using other projects are a viable solution for projects with limited historical data. However, JIT models tend to perform best in a cross-project context when the data used to learn them are carefully selected.     

A transfer cost-sensitive boosting approach for cross-project defect prediction

Software defect prediction has been regarded as one of the crucial tasks to improve software quality by effectively allocating valuable resources to fault-prone modules. It is necessary to have a sufficient set of historical data for building a predictor. Without a set of sufficient historical data within a company, cross-project defect prediction (CPDP) can be employed where data from other companies are used to build predictors. In such cases, a transfer learning technique, which extracts common knowledge from source projects and transfers it to a target project, can be used to enhance the prediction performance. There exists the class imbalance problem, which causes difficulties for the learner to predict defects. The main impacts of imbalanced data under cross-project settings have not been investigated in depth. We propose a transfer cost-sensitive boosting method that considers both knowledge transfer and class imbalance for CPDP when given a small amount of labeled target data. The proposed approach performs boosting that assigns weights to the training instances with consideration of both distributional characteristics and the class imbalance. Through comparative experiments with the transfer learning and the class imbalance learning techniques, we show that the proposed model provides significantly higher defect detection accuracy while retaining better overall performance. As a result, a combination of transfer learning and class imbalance learning is highly effective for improving the prediction performance under cross-project settings. The proposed approach will help to design an effective prediction model for CPDP. The improved defect prediction performance could help to direct software quality assurance activities and reduce costs. Consequently, the quality of software can be managed effectively.     

Value-cognitive boosting with a support vector machine for cross-project defect prediction

It is well-known that software defect prediction is one of the most important tasks for software quality improvement. The use of defect predictors allows test engineers to focus on defective modules. Thereby testing resources can be allocated effectively and the quality assurance costs can be reduced. For within-project defect prediction (WPDP), there should be sufficient data within a company to train any prediction model. Without such local data, cross-project defect prediction (CPDP) is feasible since it uses data collected from similar projects in other companies. Software defect datasets have the class imbalance problem increasing the difficulty for the learner to predict defects. In addition, the impact of imbalanced data on the real performance of models can be hidden by the performance measures chosen. We investigate if the class imbalance learning can be beneficial for CPDP. In our approach, the asymmetric misclassification cost and the similarity weights obtained from distributional characteristics are closely associated to guide the appropriate resampling mechanism. We performed the effect size A-statistics test to evaluate the magnitude of the improvement. For the statistical significant test, we used Wilcoxon rank-sum test. The experimental results show that our approach can provide higher prediction performance than both the existing CPDP technique and the existing class imbalance technique.     

An empirical study of factors affecting cross-project aging-related bug prediction with TLAP

Software aging is a phenomenon in which long-running software systems show an increasing failure rate and/or progressive performance degradation. Due to their nature, Aging-Related Bugs (ARBs) are hard to discover during software testing and are also challenging to reproduce. Therefore, automatically predicting ARBs before software release can help developers reduce ARB impact or avoid ARBs. Many bug prediction approaches have been proposed, and most of them show effectiveness in within-project prediction settings. However, due to the low presence and reproducing difficulty of ARBs, it is usually hard to collect sufficient training data to build an accurate prediction model. A recent work proposed a method named Transfer Learning based Aging-related bug Prediction (TLAP) for performing cross-project ARB prediction. Although this method considerably improves cross-project ARB prediction performance, it has been observed that its prediction result is affected by several key factors, such as the normalization methods, kernel functions, and machine learning classifiers. Therefore, this paper presents the first empirical study to examine the impact of these factors on the effectiveness of cross-project ARB prediction in terms of single-factor pattern, bigram pattern, and triplet pattern and validates the results with the Scott-Knott test technique. We find that kernel functions and classifiers are key factors affecting the effectiveness of cross-project ARB prediction, while normalization methods do not show statistical influence. In addition, the order of values in three single-factor patterns is maintained in three bigram patterns and one triplet pattern to a large extent. Similarly, the order of values in the three bigram patterns is also maintained in the triplet pattern.

     

An effort prediction framework for software defect correction

This article tackles the problem of predicting effort (in person–hours) required to fix a software defect posted on an Issue Tracking System. The proposed method is inspired by the Nearest Neighbour Approach presented by the pioneering work of Weiss et al. (2007) [1]. We propose four enhancements to Weiss et al. (2007) [1]: Data Enrichment, Majority Voting, Adaptive Threshold and Binary Clustering. Data Enrichment infuses additional issue information into the similarity-scoring procedure, aiming to increase the accuracy of similarity scores. Majority Voting exploits the fact that many of the similar historical issues have repeating effort values, which are close to the actual. Adaptive Threshold automatically adjusts the similarity threshold to ensure that we obtain only the most similar matches. We use Binary Clustering if the similarity scores are very low, which might result in misleading predictions. This uses common properties of issues to form clusters (independent of the similarity scores) which are then used to produce the predictions. Numerical results are presented showing a noticeable improvement over the method proposed in Weiss et al. (2007) [1].     

基于缺陷分类和缺陷预测的软件缺陷预防

为降低软件缺陷率,对现有的缺陷预测模型进行了优化,同时引入正交缺陷分类方法,并对该方法加以改进,使其能够支持缺陷的原因分析,将缺陷预测与改进的正交缺陷分类方法结合起来,形成一套软件缺陷预防流程并应用在实际项目中.实验结果表明,该成果可以在软件生命周期的各个阶段有效预防缺陷,最大限度地提高软件质量.     

An empirical study on software defect prediction with a simplified metric set

ContextSoftware defect prediction plays a crucial role in estimating the most defect-prone components of software, and a large number of studies have pursued improving prediction accuracy within a project or across projects. However, the rules for making an appropriate decision between within- and cross-project defect prediction when available historical data are insufficient remain unclear.ObjectiveThe objective of this work is to validate the feasibility of the predictor built with a simplified metric set for software defect prediction in different scenarios, and to investigate practical guidelines for the choice of training data, classifier and metric subset of a given project.MethodFirst, based on six typical classifiers, three types of predictors using the size of software metric set were constructed in three scenarios. Then, we validated the acceptable performance of the predictor based on Top-k metrics in terms of statistical methods. Finally, we attempted to minimize the Top-k metric subset by removing redundant metrics, and we tested the stability of such a minimum metric subset with one-way ANOVA tests.ResultsThe study has been conducted on 34 releases of 10 open-source projects available at the PROMISE repository. The findings indicate that the predictors built with either Top-k metrics or the minimum metric subset can provide an acceptable result compared with benchmark predictors. The guideline for choosing a suitable simplified metric set in different scenarios is presented in Table 12.ConclusionThe experimental results indicate that (1) the choice of training data for defect prediction should depend on the specific requirement of accuracy; (2) the predictor built with a simplified metric set works well and is very useful in case limited resources are supplied; (3) simple classifiers (e.g., Naïve Bayes) also tend to perform well when using a simplified metric set for defect prediction; and (4) in several cases, the minimum metric subset can be identified to facilitate the procedure of general defect prediction with acceptable loss of prediction precision in practice.     

An empirical investigation on the challenges of creating custom static analysis rules for defect localization

Software Quality Journal - Custom static analysis rules, i.e., rules specific for one or more applications, have been successfully applied to perform corrective and preventive software maintenance....     

基于复杂性的软件缺陷预测

为解决软件缺陷预测问题引入了最小二乘支持向量机算法(LS-SVM),加速了超参数的选择过程,给出了逐个加入新的样本用以模型校正的快捷方法,以软件复杂性度量为线索,建立了基于FLS-SVM的软件缺陷预测模型。通过具体实例阐明了模型的执行过程及小样本情况下比神经网络更为出色的预测能力,并根据回归方程指出了对软件缺陷影响显著的复杂性度量。     

基于生命周期的软件缺陷预测技术

为保证软件可靠性和软件质量,在基于软件开发周期的基础上,提出了一种利用PCA-BP模糊神经网络的软件缺陷预计方法.针对影响软件可靠性的各种因素,依据相关的标准,结合工程实践,选取了影响软件可靠性的度量元.收集了实际工程中的一类飞行控制软件的度量数据,利用提出的模型进行缺陷预测,并将预测结果与传统的BP神经网络模型计算的结果进行了对比.对比结果表明,与基于BP神经网络的预测方法相比较,结合了主成分分析方法的PCA-BP神经网络预测方法具有更快的收敛速度和更高的预测准确度.     

Software defect association mining and defect correction effort prediction

Much current software defect prediction work focuses on the number of defects remaining in a software system. In this paper, we present association rule mining based methods to predict defect associations and defect correction effort. This is to help developers detect software defects and assist project managers in allocating testing resources more effectively. We applied the proposed methods to the SEL defect data consisting of more than 200 projects over more than 15 years. The results show that, for defect association prediction, the accuracy is very high and the false-negative rate is very low. Likewise, for the defect correction effort prediction, the accuracy for both defect isolation effort prediction and defect correction effort prediction are also high. We compared the defect correction effort prediction method with other types of methods - PART, C4.5, and Naive Bayes - and show that accuracy has been improved by at least 23 percent. We also evaluated the impact of support and confidence levels on prediction accuracy, false-negative rate, false-positive rate, and the number of rules. We found that higher support and confidence levels may not result in higher prediction accuracy, and a sufficient number of rules is a precondition for high prediction accuracy.     

基于两种统计模型的软件缺陷预测

采集软件研发过程中可能与缺陷有关的过程数据或产品数据,对软件缺陷数量进行预测,达到对软件质量的把控。采用LASSO进行特征值选择确定最佳影响因子集合,采用线性模型和贝叶斯网络模型分别对样本数据进行预测,说明两种模型的因子分析过程和模型构建过程,采用R语言进行编码实现。通过预测结果的对比验证了当数据经过二次主观加工后,采用线性模型的预测结果比贝叶斯网络预测结果更准确。     

基于AODE和再抽样的软件缺陷预测模型

为了确切地估计软件缺陷分布,提出了基于AODE和再抽样的软件缺陷预测模型。分析了几种常用贝叶斯分类器的优缺点,以及软件缺陷度量元和再抽样方法,设计了多AODE分类器,该分类器是由多个AODE二分类器组成的。在以上基础上,建立了采用多AODE分类器和再抽样方法的软件缺陷预测模型,实验结果表明,该分类器的分类准确度较常用贝叶斯分类器高。通过收集到的缺陷数据样本比较结果表明,该模型比一些常用贝叶斯模型具有更好的预测准确性和稳定性。     

Software defect prediction using Bayesian networks

There are lots of different software metrics discovered and used for defect prediction in the literature. Instead of dealing with so many metrics, it would be practical and easy if we could determine the set of metrics that are most important and focus on them more to predict defectiveness. We use Bayesian networks to determine the probabilistic influential relationships among software metrics and defect proneness. In addition to the metrics used in Promise data repository, we define two more metrics, i.e. NOD for the number of developers and LOCQ for the source code quality. We extract these metrics by inspecting the source code repositories of the selected Promise data repository data sets. At the end of our modeling, we learn the marginal defect proneness probability of the whole software system, the set of most effective metrics, and the influential relationships among metrics and defectiveness. Our experiments on nine open source Promise data repository data sets show that response for class (RFC), lines of code (LOC), and lack of coding quality (LOCQ) are the most effective metrics whereas coupling between objects (CBO), weighted method per class (WMC), and lack of cohesion of methods (LCOM) are less effective metrics on defect proneness. Furthermore, number of children (NOC) and depth of inheritance tree (DIT) have very limited effect and are untrustworthy. On the other hand, based on the experiments on Poi, Tomcat, and Xalan data sets, we observe that there is a positive correlation between the number of developers (NOD) and the level of defectiveness. However, further investigation involving a greater number of projects is needed to confirm our findings.     

一种基于关联规则的网络软件缺陷预测方法

针对网络软件缺陷预测中缺陷之间的显性关联关系和隐性关联关系,为提高网络软件缺陷预测的精确度和效率,基于现有的软件缺陷数据集,提出一种基于关联规则的网络软件缺陷预测方法。首先采用随机方法从美国国家航空航天局NASA的软件缺陷数据库中提取用于分类和测试的数据集,利用关联规则方法中Apriori算法对数据集进行关联规则生成和关联分类器的构建,并与BP神经网络方法的预测结果进行对比。结果表明,基于关联规则方法能够在小样本数据集中提高网络软件缺陷预测的精确度和有效性。     

The impact of tangled code changes on defect prediction models

When interacting with source control management system, developers often commit unrelated or loosely related code changes in a single transaction. When analyzing version histories, such tangled changes will make all changes to all modules appear related, possibly compromising the resulting analyses through noise and bias. In an investigation of five open-source Java projects, we found between 7 % and 20 % of all bug fixes to consist of multiple tangled changes. Using a multi-predictor approach to untangle changes, we show that on average at least 16.6 % of all source files are incorrectly associated with bug reports. These incorrect bug file associations seem to not significantly impact models classifying source files to have at least one bug or no bugs. But our experiments show that untangling tangled code changes can result in more accurate regression bug prediction models when compared to models trained and tested on tangled bug datasets—in our experiments, the statistically significant accuracy improvements lies between 5 % and 200 %. We recommend better change organization to limit the impact of tangled changes.     

A critique of software defect prediction models

Many organizations want to predict the number of defects (faults) in software systems, before they are deployed, to gauge the likely delivered quality and maintenance effort. To help in this numerous software metrics and statistical models have been developed, with a correspondingly large literature. We provide a critical review of this literature and the state-of-the-art. Most of the wide range of prediction models use size and complexity metrics to predict defects. Others are based on testing data, the “quality” of the development process, or take a multivariate approach. The authors of the models have often made heroic contributions to a subject otherwise bereft of empirical studies. However, there are a number of serious theoretical and practical problems in many studies. The models are weak because of their inability to cope with the, as yet, unknown relationship between defects and failures. There are fundamental statistical and data quality problems that undermine model validity. More significantly many prediction models tend to model only part of the underlying problem and seriously misspecify it. To illustrate these points the Goldilock's Conjecture, that there is an optimum module size, is used to show the considerable problems inherent in current defect prediction approaches. Careful and considered analysis of past and new results shows that the conjecture lacks support and that some models are misleading. We recommend holistic models for software defect prediction, using Bayesian belief networks, as alternative approaches to the single-issue models used at present. We also argue for research into a theory of “software decomposition” in order to test hypotheses about defect introduction and help construct a better science of software engineering     

An investigation of latency prediction for NoC-based communication architectures using machine learning techniques

The Journal of Supercomputing - Due to the increasing number of cores in Systems on Chip (SoCs), bus architectures have suffered with limitations regarding performance. As applications demand...     

An investigation of artificial neural networks based prediction systems in software project management

A critical issue in software project management is the accurate estimation of size, effort, resources, cost, and time spent in the development process. Underestimates may lead to time pressures that may compromise full functional development and the software testing process. Likewise, overestimates can result in noncompetitive budgets. In this paper, artificial neural network and stepwise regression based predictive models are investigated, aiming at offering alternative methods for those who do not believe in estimation models. The results presented in this paper compare the performance of both methods and indicate that these techniques are competitive with the APF, SLIM, and COCOMO methods.