Optimal testing resource allocation during module testing considering cost, testing effort and reliability

Software reliability is one of the most important quality attributes of commercial software. During software testing, software reliability growth models (SRGMs) are commonly used to describe the phenomenon of failure occurrence and/or fault removal which consequently enhancements software reliability. Large software systems are developed by integrating a number of relatively small and independent modules, which are tested independently during module testing phase. The amount of testing resource available is limited which is desired to be consumed judiciously so as to optimize the testing process. In this paper we formulate a resource allocation problem of minimizing the cost of software testing under available amount of testing resource, given a reliability constraint. We use a flexible SRGM considering testing effort which, depending upon the values of parameters, can describe either exponential or S-shaped failure pattern of software modules. A systematic and sequential Algorithm is proposed to solve the optimization problem formulated. Numerical examples are given to illustrate the formulation and solution procedures. Sensitivity analysis is performed to examine the behavior of some parameters of SRGM with most significant influence.     

Considering the fault dependency concept with debugging time lag in software reliability growth modeling using a power function of testing time

Since the early 1970s tremendous growth has been seen in the research of software reliability growth modeling.In general, software reliability growth models (SRGMs) are applicable to the late stages of testing in software development and they can provide useful information about how to improve the reliability of software products.A number of SRGMs have been proposed in the literature to represent time-dependent fault identification/removal phenomenon;still new models are being proposed that could fit a greater number of reliability growth curves.Often,it is assumed that detected faults axe immediately corrected when mathematical models are developed.This assumption may not be realistic in practice because the time to remove a detected fault depends on the complexity of the fault,the skill and experience of the personnel,the size of the debugging team,the technique,and so on.Thus,the detected fault need not be immediately removed,and it may lag the fault detection process by a delay effect factor.In this paper,we first review how different software reliability growth models have been developed,where fault detection process is dependent not only on the number of residual fault content but also on the testing time,and see how these models can be reinterpreted as the delayed fault detection model by using a delay effect factor.Based on the power function of the testing time concept,we propose four new SRGMs that assume the presence of two types of faults in the software:leading and dependent faults.Leading faults are those that can be removed upon a failure being observed.However,dependent faults are masked by leading faults and can only be removed after the corresponding leading fault has been removed with a debugging time lag.These models have been tested on real software error data to show its goodness of fit,predictive validity and applicability.     

Modeling optimal release policy under fuzzy paradigm in imperfect debugging environment

ContextIn this study, a software optimal release time with cost-reliability criteria has been discussed in an imperfect debugging environment.ObjectiveThe motive of this study is to model uncertainty involved in estimated parameters of the software reliability growth model (SRGM).MethodInitially the reliability parameters of SRGM are estimated using least square estimation (LSE). Considering the uncertainty involved in the estimated parameters due to human behavior being subjective in nature and the dynamism of the testing environment, the concept of fuzzy set theory is applicable in developing SRGM. Finally, using arithmetic operations on fuzzy numbers, the reliability and total software cost are calculated.ResultsVarious reliability measures have been computed at different levels of uncertainties, and a comparison has been made with the existing results reported in the literature.ConclusionIt is evident from the results that a better prediction of reliability measures, namely, software reliability and total software cost can be made under the fuzzy paradigm.     

Debugging‐workflow‐aware software reliability growth analysis

Software reliability growth models support the prediction/assessment of product quality, release time, and testing/debugging cost. Several software reliability growth model extensions take into account the bug correction process. However, their estimates may be significantly inaccurate when debugging fails to fully fit modelling assumptions. This paper proposes debugging‐workflow‐aware software reliability growth method (DWA‐SRGM), a method for reliability growth analysis leveraging the debugging data usually managed by companies in bug tracking systems. On the basis of a characterization of the debugging workflow within the software project under consideration (in terms of bug features and treatment phases), DWA‐SRGM pinpoints the factors impacting the estimates and to spot bottlenecks, thus supporting process improvement decisions. Two industrial case studies are presented, a customer relationship management system and an enterprise resource planning system, whose defects span a period of about 17 and 13 months, respectively. DWA‐SRGM revealed effective to obtain more realistic estimates and to capitalize on the awareness of critical factors for improving debugging.     

软件可靠性增长模型研究综述

软件可靠性增长模型SRGM（Software Reliability and Growth Model）是目前建模可靠性及其过程提高的重要数学工具,对可靠性的评测、保证以及测试资源管控和最优发布研究具有重要作用,文中对SRGM研究进行阐述和分析.对SRGM的核心研究内容与建模流程进行分析,给出了SRGM基本功用.同时,梳理了SRGM的发展演变历程,进而对当前研究现状进行深入剖析,给出当前研究特征.从软件中总的故障个数、故障检测率FDR（Fault Detection Rate）和测试工作量TE（Testing-Effort）三个方面对影响SRGM的因素进行了分析.文中基于作者前期研究中提出的统一性框架模型,对当前典型的解析模型进行了分类比较和分析;对基于有限与无限服务队列模型的SRGM进行分析与讨论;对以率驱动事件过程RDEP（Rate-Driven Event Processes）为重点的仿真方法进行剖析.进一步,为了验证与分析不同模型的差异,对26个典型的模型在公开发表的16个数据集上进行了实验.结果表明,SRGM的性能差异取决于失效数据集的客观性以及研究人员对测试过程进行不同假设下所建立的数学模型的主观性.最后,指出了SRGM面临的挑战、发展趋势和亟待解决的问题.     

Software fault prediction using Nonlinear Autoregressive with eXogenous Inputs (NARX) network

This paper explores a new approach for predicting software faults by means of NARX neural network. Also, a careful analysis has been carried out to determine the applicability of NARX network in software reliability. The validation of the proposed approach has been performed using two real software failure data sets. Comparison has been made with some existing parametric software reliability models as well as some neural network (Elman net and TDNN) based SRGM. The results computed shows that the proposed approach outperformed the other existing parametric and neural network based software reliability models with a reasonably good predictive accuracy.     

一个考虑多种排错延迟的NHPP类软件可靠性增长模型

软件可靠性增长模型通常假设软件的测试环境与软件实际运行的现场环境相同,期望利用测试阶段获得的失效数据评估软件在现场运行时的失效行为。多数非齐次泊松过程类软件可靠性增长模型假设软件故障被发现后立即被排除,这点假设无论是在测试环境还是在现场环境下都很难实现。根据故障对测试过程的影响,故障的排错时间可被分为多种。提出了一个考虑多种排错延迟的软件可靠性增长模型,讨论了基于这个模型的故障排除效率函数,指出从用户角度出发讨论软件可靠性时必须考虑重复性故障。     

Parameter optimization of software reliability growth model with S-shaped testing-effort function using improved swarm intelligent optimization

Software reliability growth model (SRGM) with testing-effort function (TEF) is very helpful for software developers and has been widely accepted and applied. However, each SRGM with TEF (SRGMTEF) contains some undetermined parameters. Optimization of these parameters is a necessary task. Generally, these parameters are estimated by the Least Square Estimation (LSE) or the Maximum Likelihood Estimation (MLE). We found that the MLE can be used only when the software failure data to satisfy some assumptions such as to satisfy a certain distribution. However, the software failure data may not satisfy such a distribution. In this paper, we investigate the improvement and application of a swarm intelligent optimization algorithm, namely quantum particle swarm optimization (QPSO) algorithm, to optimize these parameters of SRGMTEF. The performance of the proposed SRGMTEF model with optimized parameters is also compared with other existing models. The experiment results show that the proposed parameter optimization approach using QPSO is very effective and flexible, and the better software reliability growth performance can be obtained based on SRGMTEF on the different software failure datasets.     

Using Software Reliability Growth Models in Practice

The amount of software in consumer electronics has grown from thousands to millions of lines of source code over the past decade. Up to a million of these products are manufactured each month for a successful mobile phone or television. Development organizations must meet two challenging requirements at the same time: be predictable to meet market windows and provide nearly fault-free software. Software reliability is the probability of failure-free operation for a specified period of time in a specified environment. The process of finding and removing faults to improve the software reliability can be described by a mathematical relationship called a software reliability growth model (SRGM). Our goal is to assess the practical application of SRGMs during integration and test and compare them with other estimation methods. We empirically validated SRGMs' usability in a software development environment. During final test phases for three embedded software projects, software reliability growth models predicted remaining faults in the software, supporting management's decisions.     

考虑测试工作量与覆盖率的软件可靠性模型

为了进一步提升现有非齐次泊松过程类软件可靠性增长模型的拟合与预计精度,首先,提出一个同时考虑测试工作量与测试覆盖率的NHPP类软件可靠性建模框架.在此基础上,将变形S型测试工作量函数(IS-TEF)以及Logistic测试覆盖率函数(LO-TCF)带入该建模框架,建立了一个新的软件可靠性增长模型,即IS-LO-SRGM.同时,还对利用该框架进行建模过程中的两个重要问题进行了描述与分析,即如何确定具体的TEF和TCF以及模型参数估计.然后,在两组真实的失效数据集上,利用该建模框架建立了最为合适的增长模型,即IS-LO-SRGM,并将该模型与8种经典NHPP模型进行对比.实例验证结果表明,所提出的IS-LO-SRGM模型具有最为优秀的拟合与预计性能,从而证明新建模框架的有效性和实用性.最后,对不完美排错情况进行了初步的讨论与建模分析.     

Generalized Discrete Software Reliability Modeling With Effect of Program Size

Generalized methods for software reliability growth modeling have been proposed so far. But, most of them are on continuous-time software reliability growth modeling. Many discrete software reliability growth models (SRGM) have been proposed to describe a software reliability growth process depending on discrete testing time such as the number of days (or weeks); the number of executed test cases. In this paper, we discuss generalized discrete software reliability growth modeling in which the software failure-occurrence times follow a discrete probability distribution. Our generalized discrete SRGMs enable us to assess software reliability in consideration of the effect of the program size, which is one of the influential factors related to the software reliability growth process. Specifically, we develop discrete SRGMs in which the software failure-occurrence times follow geometric and discrete Rayleigh distributions, respectively. Moreover, we derive software reliability assessment measures based on a unified framework for discrete software reliability growth modeling. Additionally, we also discuss optimal software release problems based on our generalized discrete software reliability growth modeling. Finally, we show numerical examples of software reliability assessment by using actual fault-counting data     

考虑故障相关的软件可靠性增长模型研究

软件可靠性增长模型是用来评估和预测软件可靠性的重要工具.目前,绝大多数的软件可靠性增长模型并没有考虑故障之间的相关性,也没有考虑测试环境和运行环境的区别.文中提出了一种随机过程类非齐次泊松过程(NHPP)中的考虑故障相关性、测试环境和运行环境差别的模型.在两组失效数据上的实验分析表明:对这两组失效数据,文中提出的模型比其他一些非齐次泊松过程类模型的拟合效果和预测效果更好.     

Reliability growth modeling and optimal release policy under fuzzy environment of an N-version programming system incorporating the effect of fault removal efficiency

Failure of a safety critical system can lead to big losses.Very high software reliability is required for automating the working of systems such as aircraft controller and nuclear reactor controller software systems.Fault-tolerant softwares are used to increase the overall reliability of software systems.Fault tolerance is achieved using the fault-tolerant schemes such as fault recovery (recovery block scheme),fault masking (N-version programming (NVP)) or a combination of both (Hybrid scheme).These softwares incorporate the ability of system survival even on a failure.Many researchers in the field of software engineering have done excellent work to study the reliability of fault-tolerant systems.Most of them consider the stable system reliability.Few attempts have been made in reliability modeling to study the reliability growth for an NVP system.Recently,a model was proposed to analyze the reliability growth of an NVP system incorporating the effect of fault removal efficiency.In this model,a proportion of the number of failures is assumed to be a measure of fault generation while an appropriate measure of fault generation should be the proportion of faults removed.In this paper,we first propose a testing efficiency model incorporating the effect of imperfect fault debugging and error generation.Using this model,a software reliability growth model (SRGM) is developed to model the reliability growth of an NVP system.The proposed model is useful for practical applications and can provide the measures of debugging effectiveness and additional workload or skilled professional required.It is very important for a developer to determine the optimal release time of the software to improve its performance in terms of competition and cost.In this paper,we also formulate the optimal software release time problem for a 3VP system under fuzzy environment and discuss a the fuzzy optimization technique for solving the problem with a numerical illustration.     

可靠性模型中故障检测率研究述评

故障检测率FDR（Fault Detection Rate）是可靠性研究的关键要素,对于测试环境构建、故障检测效率提升、可靠性建模和可靠性增长具有重要作用,对于提高系统可靠性与确定发布时间具有重要现实意义.首先,对基于NHPP（Non-Homogeneous Poisson Process,非齐次泊松过程）类的软件可靠性增长模型SRGM（Software Reliability Growth Mode）进行概述,给出了建模本质、功用与流程.基于此,引出可靠性建模与研究中的关键参数——FDR,给出定义,对测试环境描述能力进行分析,展示不同模型的差异.着重剖析了FDR与失效强度、冒险率（风险率）的区别,得出三者之间的关联性表述.全面梳理了FDR的大类模型,分别从测试覆盖函数视角、直接设定角度、测试工作量函数参与构成方式三个方面进行剖析,继而提出统一的FDR相关的可靠性模型.考虑到对真实测试环境描述能力需要,建立不完美排错框架模型,衍生出不完美排错下多个不同FDR参与的可靠性增长模型.进一步,在12个真实描述应用场景与公开发表的失效数据集上进行实验,验证不同FDR模型相关的可靠性模型效用,对差异性进行分析与讨论.结果表明,FDR模型自身的性能可以支撑可靠性模型性能的提升.最后,指出了未来研究趋势和需要解决的问题.     

一个基于响铃形故障检测率函数的软件可靠性增长模型

非齐次泊松过程类软件可靠性增长模型是评价软件产品可靠性指标的有效工具．影响软件可靠性增长模型评估和预测准确性的最重要的两个因素是软件中隐藏的初始故障数和故障检测率．一些非齐次泊松过程类模型假设故障检测率是不随测试时间变化的常量,有些模型假设故障检测率是增函数或减函数．这些假设或忽略了测试者的学习过程,或忽略了越迟被检测到的故障的概率就可能越低的特点．该文将测试者的学习过程和软件固有故障检测率的变化特征相结合,提出了一个铃形的故障检测率函数,建立了一个非齐次泊松过程类软件可靠性增长模型——Bbell—SRGM．在一组失效数据上的实验分析表明：对这组失效数据,Bbell—SRGM模型比G-O模型等的拟合效果更好．     

Forecasting of Software Reliability Using Neighborhood Fuzzy Particle Swarm Optimization Based Novel Neural Network

This paper proposes an artificial neural network (ANN) based software reliability model trained by novel particle swarm optimization (PSO) algorithm for enhanced forecasting of the reliability of software. The proposed ANN is developed considering the fault generation phenomenon during software testing with the fault complexity of different levels. We demonstrate the proposed model considering three types of faults residing in the software. We propose a neighborhood based fuzzy PSO algorithm for competent learning of the proposed ANN using software failure data. Fitting and prediction performances of the neighborhood fuzzy PSO based proposed neural network model are compared with the standard PSO based proposed neural network model and existing ANN based software reliability models in the literature through three real software failure data sets. We also compare the performance of the proposed PSO algorithm with the standard PSO algorithm through learning of the proposed ANN. Statistical analysis shows that the neighborhood fuzzy PSO based proposed neural network model has comparatively better fitting and predictive ability than the standard PSO based proposed neural network model and other ANN based software reliability models. Faster release of software is achievable by applying the proposed PSO based neural network model during the testing period.      

考虑环境因素的软件可靠性增长模型

软件可靠性增长模型中由于测试阶段和实际运行阶段环境的不同导致了失效强度函数的判断偏差。在Musa执行时间模型中的经典模型M-O对数泊松执行时间模型基础上,提出考虑环境因素的对数泊松模型,该模型能较好的刻画失效强度函数变化规律,并给出参数估计公式。通过对失效数据集的实验,结果表明该模型具有较好的拟合效果。     

Imperfect debugging models with fault introduction rate for software reliability assessment

In general it is considered to be unrealistic in software reliability modelling to assume that the faults detected by software testing are perfectly removed without introducing new faults. In this paper we propose two software reliability assessment models with imperfect debugging by assuming that new faults are sometimes introduced when the faults originally latent in a software system are corrected and removed during the testing phase. It is assumed that the fault detection rate is proportional to the sum of the numbers of faults remaining originally in the system and faults introduced by imperfect debugging. These two models are described by a nonhomogeneous Poisson process. Several quantitative measures for reliability assessment are derived, and the maximum likelihood estimations of unknown model parameters are presented. Finally, numerical examples of software reliability analysis based on these two models are shown.