A simulation study of the model evaluation criterion MMRE

The mean magnitude of relative error, MMRE, is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. One purpose of MMRE is to assist us to select the best model. In this paper, we have performed a simulation study demonstrating that MMRE does not always select the best model. Our findings cast some doubt on the conclusions of any study of competing software prediction models that use MMRE as a basis of model comparison. We therefore recommend not using MMRE to evaluate and compare prediction models. At present, we do not have any universal replacement for MMRE. Meanwhile, we therefore recommend using a combination of theoretical justification of the models that are proposed together with other metrics proposed in this paper.     

A systematic review of machine learning techniques for software fault prediction

BackgroundSoftware fault prediction is the process of developing models that can be used by the software practitioners in the early phases of software development life cycle for detecting faulty constructs such as modules or classes. There are various machine learning techniques used in the past for predicting faults.MethodIn this study we perform a systematic review of studies from January 1991 to October 2013 in the literature that use the machine learning techniques for software fault prediction. We assess the performance capability of the machine learning techniques in existing research for software fault prediction. We also compare the performance of the machine learning techniques with the statistical techniques and other machine learning techniques. Further the strengths and weaknesses of machine learning techniques are summarized.ResultsIn this paper we have identified 64 primary studies and seven categories of the machine learning techniques. The results prove the prediction capability of the machine learning techniques for classifying module/class as fault prone or not fault prone. The models using the machine learning techniques for estimating software fault proneness outperform the traditional statistical models.ConclusionBased on the results obtained from the systematic review, we conclude that the machine learning techniques have the ability for predicting software fault proneness and can be used by software practitioners and researchers. However, the application of the machine learning techniques in software fault prediction is still limited and more number of studies should be carried out in order to obtain well formed and generalizable results. We provide future guidelines to practitioners and researchers based on the results obtained in this work.     

Comparative analysis of statistical and machine learning methods for predicting faulty modules

The demand for development of good quality software has seen rapid growth in the last few years. This is leading to increase in the use of the machine learning methods for analyzing and assessing public domain data sets. These methods can be used in developing models for estimating software quality attributes such as fault proneness, maintenance effort, testing effort. Software fault prediction in the early phases of software development can help and guide software practitioners to focus the available testing resources on the weaker areas during the software development. This paper analyses and compares the statistical and six machine learning methods for fault prediction. These methods (Decision Tree, Artificial Neural Network, Cascade Correlation Network, Support Vector Machine, Group Method of Data Handling Method, and Gene Expression Programming) are empirically validated to find the relationship between the static code metrics and the fault proneness of a module. In order to assess and compare the models predicted using the regression and the machine learning methods we used two publicly available data sets AR1 and AR6. We compared the predictive capability of the models using the Area Under the Curve (measured from the Receiver Operating Characteristic (ROC) analysis). The study confirms the predictive capability of the machine learning methods for software fault prediction. The results show that the Area Under the Curve of model predicted using the Decision Tree method is 0.8 and 0.9 (for AR1 and AR6 data sets, respectively) and is a better model than the model predicted using the logistic regression and other machine learning methods.     

软件缺陷预测模型间的比较实验: 问题、进展与挑战

近年来,研究者提出了大量的软件缺陷预测模型,新模型往往通过与过往模型进行比较实验来表明其有效性.然而,研究者在设计新旧模型间的比较实验时并没有达成共识,不同的工作往往采用不完全一致的比较实验设置,这可能致使在对比模型时得到误导性结论,最终错失提升缺陷预测能力的机会.对近年来国内外学者所做的缺陷预测模型间的比较实验进行系统性的总结:首先,阐述缺陷预测模型间的比较实验的研究问题;然后,分别从缺陷数据集、数据集划分、基线模型、性能指标、分类阈值这5个方面对现有的比较实验进行总结;最后,指出目前在进行缺陷预测模型间比较实验时面临的挑战,并给出建议的研究方向.     

Comparing cost prediction models by resampling techniques

The accurate software cost prediction is a research topic that has attracted much of the interest of the software engineering community during the latest decades. A large part of the research efforts involves the development of statistical models based on historical data. Since there are a lot of models that can be fitted to certain data, a crucial issue is the selection of the most efficient prediction model. Most often this selection is based on comparisons of various accuracy measures that are functions of the model’s relative errors. However, the usual practice is to consider as the most accurate prediction model the one providing the best accuracy measure without testing if this superiority is in fact statistically significant. This policy can lead to unstable and erroneous conclusions since a small change in the data is able to turn over the best model selection. On the other hand, the accuracy measures used in practice are statistics with unknown probability distributions, making the testing of any hypothesis, by the traditional parametric methods, problematic. In this paper, the use of statistical simulation tools is proposed in order to test the significance of the difference between the accuracy of two prediction methods: regression and estimation by analogy. The statistical simulation procedures involve permutation tests and bootstrap techniques for the construction of confidence intervals for the difference of measures. Four known datasets are used for experimentation in order to validate the results and make comparisons between the simulation methods and the traditional parametric and non-parametric procedures.     

Dimensionality reduction methods for machine translation quality estimation

Quality estimation (QE) for machine translation is usually addressed as a regression problem where a learning model is used to predict a quality score from a (usually highly-redundant) set of features that represent the translation. This redundancy hinders model learning, and thus penalizes the performance of quality estimation systems. We propose different dimensionality reduction methods based on partial least squares regression to overcome this problem, and compare them against several reduction methods previously used in the QE literature. Moreover, we study how the use of such methods influence the performance of different learning models. Experiments carried out on the English-Spanish WMT12 QE task showed that it is possible to improve prediction accuracy while significantly reducing the size of the feature sets.     

软件可靠性预测的相关向量机模型

相关向量机是一种解决回归问题非常有效的方法,针对软件失效时间及其之前的m个失效时间数据使用相关向量机进行学习,以建立失效时间之间内在的依赖关系,由此构建新的基于相关向量机的软件可靠性预测模型.在4个数据集上的实验结果表明,新模型在预测能力上较之广泛使用的基于支持向量机或人工神经网络的软件可靠性预测模型有明显的提高,同时也表明现时失效数据的预测能力比很久之前观测的失效数据更强,最后通过实验对合理的m值及不同数据集上核函数参数取值进行研究.     

基于拟似然估计方法的软件失效预测模型

软件缺陷预测是软件可靠性研究的一个重要方向。由于影响软件失效的因素有很多,相互之间关联关系复杂,在分析建模中常用联合分布函数来描述,而实际应用中难以确定,直接影响软件失效预测。基于拟似然估计提出一种软件失效预测方法,通过主成分分析筛选影响软件失效的主要影响因素,建立多因素软件失效预测模型,利用这些影响因素的数字特征(均值函数和方差函数)以及采用拟似然估计方法估计出模型参数,进而对软件失效进行预测分析。基于两个真实数据集Eclipse JDT和Eclipse PDE,与经典Logistic回归和Probit回归预测模型进行实验对比分析,结果表明采用拟似然估计对软件缺陷预测具有可行性,且预测精度均优于这两种经典回归预测模型。     

Robust feedforward and recurrent neural network based dynamic weighted combination models for software reliability prediction

Traditional parametric software reliability growth models (SRGMs) are based on some assumptions or distributions and none such single model can produce accurate prediction results in all circumstances. Non-parametric models like the artificial neural network (ANN) based models can predict software reliability based on only fault history data without any assumptions. In this paper, initially we propose a robust feedforward neural network (FFNN) based dynamic weighted combination model (PFFNNDWCM) for software reliability prediction. Four well-known traditional SRGMs are combined based on the dynamically evaluated weights determined by the learning algorithm of the proposed FFNN. Based on this proposed FFNN architecture, we also propose a robust recurrent neural network (RNN) based dynamic weighted combination model (PRNNDWCM) to predict the software reliability more justifiably. A real-coded genetic algorithm (GA) is proposed to train the ANNs. Predictability of the proposed models are compared with the existing ANN based software reliability models through three real software failure data sets. We also compare the performances of the proposed models with the models that can be developed by combining three or two of the four SRGMs. Comparative studies demonstrate that the PFFNNDWCM and PRNNDWCM present fairly accurate fitting and predictive capability than the other existing ANN based models. Numerical and graphical explanations show that PRNNDWCM is promising for software reliability prediction since its fitting and prediction error is much less relative to the PFFNNDWCM.     

Nonlinear Poisson regression using neural networks: a simulation study

We describe a novel extension of the Poisson regression model to be based on a multi-layer perceptron, a type of neural network. This relaxes the assumptions of the traditional Poisson regression model, while including it as a special case. In this paper, we describe neural network regression models with six different schemes and compare their performances in three simulated data sets, namely one linear and two nonlinear cases. From the simulation study it is found that the Poisson regression models work well when the linearity assumption is correct, but the neural network models can largely improve the prediction in nonlinear situations.     

A novel composite model approach to improve software quality prediction

Context:How can quality of software systems be predicted before deployment? In attempting to answer this question, prediction models are advocated in several studies. The performance of such models drops dramatically, with very low accuracy, when they are used in new software development environments or in new circumstances.ObjectiveThe main objective of this work is to circumvent the model generalizability problem. We propose a new approach that substitutes traditional ways of building prediction models which use historical data and machine learning techniques.MethodIn this paper, existing models are decision trees built to predict module fault-proneness within the NASA Critical Mission Software. A genetic algorithm is developed to combine and adapt expertise extracted from existing models in order to derive a “composite” model that performs accurately in a given context of software development. Experimental evaluation of the approach is carried out in three different software development circumstances.ResultsThe results show that derived prediction models work more accurately not only for a particular state of a software organization but also for evolving and modified ones.ConclusionOur approach is considered suitable for software data nature and at the same time superior to model selection and data combination approaches. It is then concluded that learning from existing software models (i.e., software expertise) has two immediate advantages; circumventing model generalizability and alleviating the lack of data in software-engineering.     

Predictive accuracy comparison between neural networks and statistical regression for development effort of software projects

To get a better prediction of costs, schedule, and the risks of a software project, it is necessary to have a more accurate prediction of its development effort. Among the main prediction techniques are those based on mathematical models, such as statistical regressions or machine learning (ML). The ML models applied to predicting the development effort have mainly based their conclusions on the following weaknesses: (1) using an accuracy criterion which leads to asymmetry, (2) applying a validation method that causes a conclusion instability by randomly selecting the samples for training and testing the models, (3) omitting the explanation of how the parameters for the neural networks were determined, (4) generating conclusions from models that were not trained and tested from mutually exclusive data sets, (5) omitting an analysis of the dependence, variance and normality of data for selecting the suitable statistical test for comparing the accuracies among models, and (6) reporting results without showing a statistically significant difference. In this study, these six issues are addressed when comparing the prediction accuracy of a radial Basis Function Neural Network (RBFNN) with that of a regression statistical (the model most frequently compared with ML models), to feedforward multilayer perceptron (MLP, the most commonly used in the effort prediction of software projects), and to general regression neural network (GRNN, a RBFNN variant). The hypothesis tested is the following: the accuracy of effort prediction for RBFNN is statistically better than the accuracy obtained from a simple linear regression (SLR), MLP and GRNN when adjusted function points data, obtained from software projects, is used as the independent variable. Samples obtained from the International Software Benchmarking Standards Group (ISBSG) Release 11 related to new and enhanced projects were used. The models were trained and tested from a leave-one-out cross-validation method. The criteria for evaluating the models were based on Absolute Residuals and by a Friedman statistical test. The results showed that there was a statistically significant difference in the accuracy among the four models for new projects, but not for enhanced projects. Regarding new projects, the accuracy for RBFNN was better than for a SLR at the 99% confidence level, whereas the MLP and GRNN were better than for a SLR at the 90% confidence level.     

基于GMDH因果关系的软件缺陷预测模型

软件缺陷预测是软件可靠性研究的一个重要方向。基于自组织数据挖掘(GMDH)网络与因果关系检验理论提出了一种软件缺陷预测模型,借鉴Granger检验思想,利用GMDH网络选择与软件失效具有因果关系的度量指标,建立软件缺陷预测模型。该方法从复杂系统建模角度研究软件度量指标与软件缺陷之间的因果关系,可以检验多变量之间在非线性意义上的因果关系。最后基于两组真实软件失效数据集,将所提出的方法与基于Granger因果检验的软件缺陷预测模型进行比较分析。结果表明,基于GMDH因果关系的软件缺陷预测模型比Granger因果检验方法具有更为显著的预测效果。     

构件软件的回归测试复杂性度量

基于构件的软件构建方法目前被广泛使用在软件开发中,用于减少软件开发的工程成本和加快软件开发进度.在软件维护过程中,由于构件更新或者新版本的发布,基于构件的系统会受到影响,需要进行回归测试.对于指定的软件修改需求,维护者可以实施不同的修改手段.不同的修改手段会导致不同的回归测试复杂性,这种复杂性是软件维护成本和有效性的重要因素.目前的研究没有强调构件软件的回归测试复杂性问题.基于修改影响复杂性模型和度量,提出一种回归测试的复杂性度量框架.该度量框架包括两个部分:基于图的模型和形式化度量计算.该度量可以有效表示构件软件分别在构件和系统层面的回归测试复杂性因素,可视化地体现复杂性变化.然后根据模型,提出具体的度量计算方式.最后,通过实验研究,针对同一个构件软件的相同修改需求,利用若干个实验组进行独立修改实施,然后比较回归测试的复杂性.实验结果表明,所提出的度量方式是可行和有效的.     

Modeling and Simulation of Entrepreneur Individual Based on Dynamic and Complex System Computing

There are disadvantages such as lack of resources and experience in college students’ entrepreneurship and the current research belong to the investigation and research, lacking the prediction simulation model research. Based on the theory of individual learning and the theory of complex systems, this study analyzes the mechanism of college students’ entrepreneurial process through dynamic learning theory, establishes the model of college students’ entrepreneurial subject, studies the different learning styles of college students, and discusses the influence of environmental dynamics on college students’ chance recognition. Through simulation and practice analysis, it is concluded that college students’ entrepreneurship is the process of learning and development of an individual and enterprise subject. Simultaneously, this study finds that the improvement of learning efficiency, learning channels and vision can effectively promote the development of enterprises, and can provide new ideas for the theoretical research of college students’ entrepreneurship.     

Model based bootstrap methods for interval censored data

The performance of model based bootstrap methods for constructing point-wise confidence intervals around the survival function with interval censored data is investigated. It is shown that bootstrapping from the nonparametric maximum likelihood estimator of the survival function is inconsistent for the current status model. A model based smoothed bootstrap procedure is proposed and proved to be consistent. In fact, a general framework for proving the consistency of any model based bootstrap scheme in the current status model is established. In addition, simulation studies are conducted to illustrate the (in)-consistency of different bootstrap methods in mixed case interval censoring. The conclusions in the interval censoring model would extend more generally to estimators in regression models that exhibit non-standard rates of convergence.     

A probabilistic model for predicting software development effort

Recently, Bayesian probabilistic models have been used for predicting software development effort. One of the reasons for the interest in the use of Bayesian probabilistic models, when compared to traditional point forecast estimation models, is that Bayesian models provide tools for risk estimation and allow decision-makers to combine historical data with subjective expert estimates. In this paper, we use a Bayesian network model and illustrate how a belief updating procedure can be used to incorporate decision-making risks. We develop a causal model from the literature and, using a data set of 33 real-world software projects, we illustrate how decision-making risks can be incorporated in the Bayesian networks. We compare the predictive performance of the Bayesian model with popular nonparametric neural-network and regression tree forecasting models and show that the Bayesian model is a competitive model for forecasting software development effort.     

基于极限学习机的组合软件可靠性模型研究

针对单一软件可靠性模型适应性不强和数据驱动模型稳定性较差的问题,本文选取3种典型软件可靠性模型作为基模型,利用极限学习机对基模型的预测结果进行加权优化,得到组合软件可靠性模型,实现经典软件可靠性模型和人工智能算法的有机结合。通过对3组失效数据进行仿真实验,并与单一模型、基于其他神经网络算法的组合模型以及数据驱动模型的预测结果进行对比,验证了本文模型能够有效地提升预测精度和模型的适应性。     

Software reliability prediction model based on support vector regression with improved estimation of distribution algorithms

Software reliability prediction plays a very important role in the analysis of software quality and balance of software cost. The data during software lifecycle is used to analyze and predict software reliability. However, predicting the variability of software reliability with time is very difficult. Recently, support vector regression (SVR) has been widely applied to solve nonlinear predicting problems in many fields and has obtained good performance in many situations; however it is still difficult to optimize SVR's parameters. Previously, some optimization algorithms have been used to find better parameters of SVR, but these existing algorithms usually are not fully satisfactory. In this paper, we first improve estimation of distribution algorithms (EDA) in order to maintain the diversity of the population, and then a hybrid improved estimation of distribution algorithms (IEDA) and SVR model, called IEDA-SVR model, is proposed. IEDA is used to optimize parameters of SVR, and IEDA-SVR model is used to predict software reliability. We compare IEDA-SVR model with other software reliability models using real software failure datasets. The experimental results show that the IEDA-SVR model has better prediction performance than the other models.     

Empirical validation of object-oriented metrics for predicting fault proneness models

Empirical validation of software metrics used to predict software quality attributes is important to ensure their practical relevance in software organizations. The aim of this work is to find the relation of object-oriented (OO) metrics with fault proneness at different severity levels of faults. For this purpose, different prediction models have been developed using regression and machine learning methods. We evaluate and compare the performance of these methods to find which method performs better at different severity levels of faults and empirically validate OO metrics given by Chidamber and Kemerer. The results of the empirical study are based on public domain NASA data set. The performance of the predicted models was evaluated using Receiver Operating Characteristic (ROC) analysis. The results show that the area under the curve (measured from the ROC analysis) of models predicted using high severity faults is low as compared with the area under the curve of the model predicted with respect to medium and low severity faults. However, the number of faults in the classes correctly classified by predicted models with respect to high severity faults is not low. This study also shows that the performance of machine learning methods is better than logistic regression method with respect to all the severities of faults. Based on the results, it is reasonable to claim that models targeted at different severity levels of faults could help for planning and executing testing by focusing resources on fault-prone parts of the design and code that are likely to cause serious failures.