Does software reliability growth behavior follow a non-homogeneous Poisson process

It is widely believed in software reliability community that software reliability growth behavior follows a non-homogeneous Poisson process (NHPP) based on analyzing the behavior of the mean of the cumulative number of observed software failures. In this paper we present two controlled software experiments to examine this belief. The behavior of the mean of the cumulative number of observed software failures and that of the corresponding variance are examined simultaneously. Both empirical observations and statistical hypothesis testing suggest that software reliability behavior does not follow a non-homogeneous Poisson process in general, and does not fit the Goel–Okumoto NHPP model in particular. Although this new finding should be further tested on other software experiments, it is reasonable to cast doubt on the validity of the NHPP framework for software reliability modeling. The importance of the work presented in this paper is not only for the new finding which is distinctly different from existing popular belief of software reliability modeling, but also for the adopted research approach which is to examine the behavior of the mean and that of the corresponding variance simultaneously on basis of controlled software experiments.     

A time/structure based software reliability model

The past 20 years have seen the formulation of numerous analytical software reliability models for estimating the reliability growth of a software product. The predictions obtained by applying these models tend to be optimistic due to the inaccuracies in the operational profile, and saturation effect of testing. Incorporating knowledge gained about some structural attribute of the code, such as test coverage, into the time-domain models can help alleviate this optimistic trend. In this paper we present an enhanced non-homogeneous Poisson process (ENHPP) model which incorporates explicitly the time-varying test-coverage function in its analytical formulation, and provides for defective fault detection and test coverage during the testing and operational phases. It also allows for a time varying fault detection rate. The ENHPP model offers a unifying framework for all the previously reported finite failure NHPP models via test coverage. We also propose the log-logistic coverage function which can capture an increasing/decreasing failure detection rate per fault, which cannot be accounted for by the previously reported finite failure NHPP models. We present a methodology based on the ENHPP model for reliability prediction earlier in the testing phase. Expressions for predictions in the operational phase of the software, software availability, and optimal software release times subject to various constraints such as cost, reliability, and availability are developed based on the ENHPP model. We also validate the ENHPP model based on four different coverage functions using five failure data sets. This revised version was published online in June 2006 with corrections to the Cover Date.     

Nonparametric dynamically weighted combination model to determine when to stop testing

Software manufacturers need to minimize the number of their software failures in their production environments. So, software reliability becomes a critical factor for these manufacturers to focus on. Software Reliability Growth Models (SRGMs) are used as indicators of the number of failures that may be faced after the shipping of the software and thus are indicators of the readiness of the software for shipping. SRGMs to handle varying operational profiles have been proposed by researchers earlier. However, as it is difficult to predict the nature of the project in advance, the reliability engineer has to try out each model one at a time before zeroing in on the model to be used in the project. We have derived a combination model, called dynamically weighted infinite NHPP combination, using the existing models for determining the release time. The nonparametric dynamically weighted combination model that we propose was validated and was found to be effective.

     

A unified scheme of some Nonhomogenous Poisson process models for software reliability estimation

In this paper, we describe how several existing software reliability growth models based on Nonhomogeneous Poisson processes (NHPPs) can be comprehensively derived by applying the concept of weighted arithmetic, weighted geometric, or weighted harmonic mean. Furthermore, based on these three weighted means, we thus propose a more general NHPP model from the quasi arithmetic viewpoint. In addition to the above three means, we formulate a more general transformation that includes a parametric family of power transformations. Under this general framework, we verify the existing NHPP models and derive several new NHPP models. We show that these approaches cover a number of well-known models under different conditions.     

NHPP models with Markov switching for software reliability

We describe the use of a latent Markov process governing the parameters of a nonhomogeneous Poisson process (NHPP) model for characterizing the software development defect discovery process. Use of a Markov switching process allows us to characterize non-smooth variations in the rate at which defects are found, better reflecting the industrial software development environment in practice. Additionally, we propose a multivariate model for characterizing changes in the distribution of defect types that are found over time, conditional on the total number of defects. A latent Markov chain governs the evolution of probabilities of the different types. Bayesian methods via Markov chain Monte Carlo facilitate inference. We illustrate the efficacy of the methods using simulated data, then apply them to model reliability growth in a large operating system software component-based on defects discovered during the system testing phase of development.     

结合故障相关的软件构件NHPP可靠性模型研究

传统的NHPP类模型是评价软件可靠性的重要模型之一。通过将故障关联融入到NHPP模型中,改进了传统的GO模型。并分析了时间域模型和基于构件模型的不足,提出一种结合两种模型的方法,同时考虑了故障排除和软件体系的方面的问题,使软件可靠性模型更加接近于实际的软件系统。     

一个基于响铃形故障检测率函数的软件可靠性增长模型

非齐次泊松过程类软件可靠性增长模型是评价软件产品可靠性指标的有效工具．影响软件可靠性增长模型评估和预测准确性的最重要的两个因素是软件中隐藏的初始故障数和故障检测率．一些非齐次泊松过程类模型假设故障检测率是不随测试时间变化的常量,有些模型假设故障检测率是增函数或减函数．这些假设或忽略了测试者的学习过程,或忽略了越迟被检测到的故障的概率就可能越低的特点．该文将测试者的学习过程和软件固有故障检测率的变化特征相结合,提出了一个铃形的故障检测率函数,建立了一个非齐次泊松过程类软件可靠性增长模型——Bbell—SRGM．在一组失效数据上的实验分析表明：对这组失效数据,Bbell—SRGM模型比G-O模型等的拟合效果更好．     

自动化空中交通管制系统软件可靠性评估研究

文章提出了评估自动化空中交通管制系统(AATCS)软件可靠性的一种新方法—基于PCR的软件可靠性评估。经操作员和软件支持业务人员(SSF)现场测试,对发现、过滤的软件故障进行确认,生成S2级的软件系统问题变更报告(PCR)。采用CROW-AMSSA(NHPP)模型进行系统的可靠性增长计算,通过极大似然估计法确定模型的参数,经过计算得到AATCS的S2级失效强度和可靠性预测趋势。     

考虑故障相关的软件可靠性增长模型研究

软件可靠性增长模型是用来评估和预测软件可靠性的重要工具.目前,绝大多数的软件可靠性增长模型并没有考虑故障之间的相关性,也没有考虑测试环境和运行环境的区别.文中提出了一种随机过程类非齐次泊松过程(NHPP)中的考虑故障相关性、测试环境和运行环境差别的模型.在两组失效数据上的实验分析表明:对这两组失效数据,文中提出的模型比其他一些非齐次泊松过程类模型的拟合效果和预测效果更好.     

考虑测试工作量与覆盖率的软件可靠性模型

为了进一步提升现有非齐次泊松过程类软件可靠性增长模型的拟合与预计精度,首先,提出一个同时考虑测试工作量与测试覆盖率的NHPP类软件可靠性建模框架.在此基础上,将变形S型测试工作量函数(IS-TEF)以及Logistic测试覆盖率函数(LO-TCF)带入该建模框架,建立了一个新的软件可靠性增长模型,即IS-LO-SRGM.同时,还对利用该框架进行建模过程中的两个重要问题进行了描述与分析,即如何确定具体的TEF和TCF以及模型参数估计.然后,在两组真实的失效数据集上,利用该建模框架建立了最为合适的增长模型,即IS-LO-SRGM,并将该模型与8种经典NHPP模型进行对比.实例验证结果表明,所提出的IS-LO-SRGM模型具有最为优秀的拟合与预计性能,从而证明新建模框架的有效性和实用性.最后,对不完美排错情况进行了初步的讨论与建模分析.     

考虑测试环境和实际运行环境的软件可靠性增长模型

软件可靠性增长模型中测试阶段和操作运行阶段环境的不同导致了两个阶段故障检测率的不同.非齐次泊松过程类软件可靠性增长模型是评价软件产品可靠性指标的有效工具.在一些非齐次泊松过程类模型中,有些学者提出了常量的环境因子,用来描述测试环境和运行环境的差别.实际上,环境因子应该是随时间变化的变量.考虑了运行阶段和测试阶段环境的不同,根据实测数据得到了变化的环境因子,并且根据测试阶段的故障检测率和变化的环境因子,转化得到了操作运行阶段的故障检测率.考虑到故障的排除效率和故障引入率,从而建立了一个既考虑运行环境和测试环境差别,又考虑故障排除效率和故障引入率的非齐次泊松过程类软件可靠性增长模型（PTEO-SRGM）.在两组失效数据上的实验分析表明,对这组失效数据,PTEO-SRGM模型比G-O模型等模型的拟合效果和预测能力更好.     

一个NHPP类软件可靠性增长模型框架

NHPP类软件可靠性增长模型已经成为软件可靠性工程实践中非常成功的工具,从某些模型的一些共同特征出发,研究了NHPP类软件可靠性增长模型的有限通用框架,提出了一 个既考虑软件测试的不完美性、故障检测率随时间的变化,又考虑了故障改正效率随时间变化的NHPP类软件可靠性增长模型框架。一些已经存在的NHPP类软件可靠性增长模型型是这个框架的特例。     

基于神经网络集成的软件故障预测及实验分析

软件系统故障预测是软件测试过程中软件可靠性研究的重点之一。利用软件系统测试过程中前期的故障相关信息进行建模,预测后期的软件故障信息,以便于后期测试和验证资源的合理分配。根据软件测试过程中已知的软件故障时间序列,利用非齐次泊松分布过程、神经网络、神经网络集成等方法对其进行建模。通过对三个实例分别建模,其预测平均相对误差G-O模型依次为3.02%、5.88%和6.58%,而神经网络集成模型为0.19%、1.88%和1.455%,实验结果表明神经网络集成模型具有更精确的预测能力。     

A mathematical modeling framework for software reliability testing†

Software reliability testing refers to various software testing activities that are driven to achieve a quantitative reliability goal given a priori or lead to a quantitative reliability assessment for the software under test. In this paper we develop a modeling framework for the software reliability testing process, comprising a simplifying model and a generalized model. In both models the software testing action selection process and the defect removal mechanism are explicitly described. Both the discrete-time domain and the continuous-time domain are involved. The generalized model is more accurate or realistic than the simplifying model since the former avoids the assumption that defects are equally detectable and the assumption that defects are removed upon being detected. However simulation examples show that the simplifying model really captures some of essential features of the software testing process after a short initial testing stage. The modeling framework is practically realistic, mathematically rigorous, and quantitatively precise. It demonstrates that the relationship between software testing and delivered software reliability, which was poor understood, can well be formulated and quantified. Rigorous examinations show that several common assumptions adopted in software reliability modeling, including the independence assumption, the exponentiality assumption, and the NHPP assumption, are theoretically false in general. This paper sets a good starting point to further formalize and quantify the software testing process and its relation to delivered software reliability.     

一个考虑多种排错延迟的NHPP类软件可靠性增长模型

软件可靠性增长模型通常假设软件的测试环境与软件实际运行的现场环境相同,期望利用测试阶段获得的失效数据评估软件在现场运行时的失效行为。多数非齐次泊松过程类软件可靠性增长模型假设软件故障被发现后立即被排除,这点假设无论是在测试环境还是在现场环境下都很难实现。根据故障对测试过程的影响,故障的排错时间可被分为多种。提出了一个考虑多种排错延迟的软件可靠性增长模型,讨论了基于这个模型的故障排除效率函数,指出从用户角度出发讨论软件可靠性时必须考虑重复性故障。     

潜伏故障点不完美覆盖的软件可靠性增长模型

现有的基于测试覆盖率的非齐次泊松过程（NHPP）类软件可靠性增长模型绝大多数都没有考虑到潜伏故障点不完美覆盖的情况。提出了一种考虑潜伏故障点不完美覆盖的软件可靠性NHPP增长模型,称之为UPNHPP模型。在一组失效数据上的实验分析表明,对这组数据,UPNHPP模型与其他模型相比有更好的拟合效果。     

On the use of Bayesian belief networks for the prediction of software productivity

In spite of numerous methods proposed, software cost estimation remains an open issue and in most situations expert judgment is still being used. In this paper, we propose the use of Bayesian belief networks (BBNs), already applied in other software engineering areas, to support expert judgment in software cost estimation. We briefly present BBNs and their advantages for expert opinion support and we propose their use for productivity estimation. We illustrate our approach by giving two examples, one based on the COCOMO81 cost factors and a second one, dealing with productivity in ERP system localization.