On allocation of resources during testing phase of a modular software

To provide reliable software, it is tested over a wide range of testing environment. In the process, testing resources such as time, testing personnel etc. are used. These resources are not infinitely large and therefore need to be used judiciously. In this article, we discuss the testing resource allocation problem among modules to maximize the total fault removal from software consisting of several independent components (modules). For the resulting optimization problem, we define marginal testing effort function (MTEF), where the testing resource consumption is represented in terms of fault removal. The three MTEFs proposed in this article account for both exponential and S-shaped growth curves, which are commonly used in software reliability analysis. Results are illustrated numerically using different data sets.     

考虑故障相关的软件可靠性增长模型研究

软件可靠性增长模型是用来评估和预测软件可靠性的重要工具.目前,绝大多数的软件可靠性增长模型并没有考虑故障之间的相关性,也没有考虑测试环境和运行环境的区别.文中提出了一种随机过程类非齐次泊松过程(NHPP)中的考虑故障相关性、测试环境和运行环境差别的模型.在两组失效数据上的实验分析表明:对这两组失效数据,文中提出的模型比其他一些非齐次泊松过程类模型的拟合效果和预测效果更好.     

基于聚类的软件失效数据预处理

应用失效数据进行软件可靠性参数的评估与预测是软件可靠性工程的主要内容之一。利用分层聚类算法对失效数据中的扰动和异常数据点进行隔离,并将以故障密度为相似性度量进行聚类处理后的失效数据集进行SRGM建模与软件可靠性参数估计,以较好的曲线拟合度提高SRGM参数估计与可靠性预测的精确性和抗干扰性,从而得到良好的软件可靠性分析与预测结果。     

Discrete software reliability growth modeling for errors of different severity incorporating change-point concept

Several software reliability growth models (SRGM) have been developed to monitor the reliability growth during the testing phase of software development.In most of the existing research available in the literatures,it is considered that a similar testing effort is required on each debugging effort.However,in practice,different types of faults may require different amounts of testing efforts for their detection and removal.Consequently,faults are classified into three categories on the basis of severity:simple,hard and complex.This categorization may be extended to (?) type of faults on the basis of severity.Although some existing research in the literatures has incorporated this concept that fault removal rate (FRR) is different for different types of faults,they assume that the FRR remains constant during the overall testing period.On the contrary,it has been observed that as testing progresses,FRR changes due to changing testing strategy,skill,environment and personnel resources.In this paper,a general discrete SRGM is proposed for errors of different severity in software systems using the change-point concept.Then,the models are formulated for two particular environments.The models were validated on two real-life data sets.The results show better fit and wider applicability of the proposed models as to different types of failure datasets.     

Modeling optimal release policy under fuzzy paradigm in imperfect debugging environment

ContextIn this study, a software optimal release time with cost-reliability criteria has been discussed in an imperfect debugging environment.ObjectiveThe motive of this study is to model uncertainty involved in estimated parameters of the software reliability growth model (SRGM).MethodInitially the reliability parameters of SRGM are estimated using least square estimation (LSE). Considering the uncertainty involved in the estimated parameters due to human behavior being subjective in nature and the dynamism of the testing environment, the concept of fuzzy set theory is applicable in developing SRGM. Finally, using arithmetic operations on fuzzy numbers, the reliability and total software cost are calculated.ResultsVarious reliability measures have been computed at different levels of uncertainties, and a comparison has been made with the existing results reported in the literature.ConclusionIt is evident from the results that a better prediction of reliability measures, namely, software reliability and total software cost can be made under the fuzzy paradigm.     

Parameter optimization of software reliability growth model with S-shaped testing-effort function using improved swarm intelligent optimization

Software reliability growth model (SRGM) with testing-effort function (TEF) is very helpful for software developers and has been widely accepted and applied. However, each SRGM with TEF (SRGMTEF) contains some undetermined parameters. Optimization of these parameters is a necessary task. Generally, these parameters are estimated by the Least Square Estimation (LSE) or the Maximum Likelihood Estimation (MLE). We found that the MLE can be used only when the software failure data to satisfy some assumptions such as to satisfy a certain distribution. However, the software failure data may not satisfy such a distribution. In this paper, we investigate the improvement and application of a swarm intelligent optimization algorithm, namely quantum particle swarm optimization (QPSO) algorithm, to optimize these parameters of SRGMTEF. The performance of the proposed SRGMTEF model with optimized parameters is also compared with other existing models. The experiment results show that the proposed parameter optimization approach using QPSO is very effective and flexible, and the better software reliability growth performance can be obtained based on SRGMTEF on the different software failure datasets.     

SRGM下失效数据集效用与验证分析

针对软件可靠性增长模型SRGM研究中的参数拟合与性能评测对失效数据集FDS的依赖,对FDS在SRGM中的效用以及其对SRGM的影响进行深入研究,并给出FDS的不足与发布建议。首先给出了基于FDS的SRGM性能评测流程,提出一般化的不完美排错框架模型,对收集到的FDS进行结构化描述与归类分析。对7个典型的不完美排错相关的SRGM在公开发表的9个真实计算机工程系统FDSs上进行实验,从拟合与预测角度分析FDS与SRGM的关系及影响。从发布方与科研人员视角对当前FDS的不足进行分析,并据此给出了FDS的发布建议。研究结果表明,科研人员尚需要充分挖掘、分析FDS中待发布的更多测试信息,用以建立更为准确的SRGM。最后指出,描述新型软件结构以及含有更多数据量的FDS的缺乏已成为制约SRGM发展的主要客观事实。     

Generalized Discrete Software Reliability Modeling With Effect of Program Size

Generalized methods for software reliability growth modeling have been proposed so far. But, most of them are on continuous-time software reliability growth modeling. Many discrete software reliability growth models (SRGM) have been proposed to describe a software reliability growth process depending on discrete testing time such as the number of days (or weeks); the number of executed test cases. In this paper, we discuss generalized discrete software reliability growth modeling in which the software failure-occurrence times follow a discrete probability distribution. Our generalized discrete SRGMs enable us to assess software reliability in consideration of the effect of the program size, which is one of the influential factors related to the software reliability growth process. Specifically, we develop discrete SRGMs in which the software failure-occurrence times follow geometric and discrete Rayleigh distributions, respectively. Moreover, we derive software reliability assessment measures based on a unified framework for discrete software reliability growth modeling. Additionally, we also discuss optimal software release problems based on our generalized discrete software reliability growth modeling. Finally, we show numerical examples of software reliability assessment by using actual fault-counting data     

Debugging‐workflow‐aware software reliability growth analysis

Software reliability growth models support the prediction/assessment of product quality, release time, and testing/debugging cost. Several software reliability growth model extensions take into account the bug correction process. However, their estimates may be significantly inaccurate when debugging fails to fully fit modelling assumptions. This paper proposes debugging‐workflow‐aware software reliability growth method (DWA‐SRGM), a method for reliability growth analysis leveraging the debugging data usually managed by companies in bug tracking systems. On the basis of a characterization of the debugging workflow within the software project under consideration (in terms of bug features and treatment phases), DWA‐SRGM pinpoints the factors impacting the estimates and to spot bottlenecks, thus supporting process improvement decisions. Two industrial case studies are presented, a customer relationship management system and an enterprise resource planning system, whose defects span a period of about 17 and 13 months, respectively. DWA‐SRGM revealed effective to obtain more realistic estimates and to capitalize on the awareness of critical factors for improving debugging.     

基于NHPP类模型的Web软件可靠性分析

结合Web软件运行的实际情况,提出一种基于BurrX测试工作量函数(TEF)的非齐次泊松过程类软件可靠性增长模型(SRGM),并将其应用到Web软件可靠性分析中。TEF能直接体现Web软件工作量与日历时间之间的关系,解决Web软件因工作量高度不均衡导致的问题。实验结果表明,与G-O模型相比,SRGM具有较好的Web软件可靠性评估效果,能较准确地描述Web软件运行的失效过程。     

软件可靠性增长模型研究综述

软件可靠性增长模型SRGM（Software Reliability and Growth Model）是目前建模可靠性及其过程提高的重要数学工具,对可靠性的评测、保证以及测试资源管控和最优发布研究具有重要作用,文中对SRGM研究进行阐述和分析.对SRGM的核心研究内容与建模流程进行分析,给出了SRGM基本功用.同时,梳理了SRGM的发展演变历程,进而对当前研究现状进行深入剖析,给出当前研究特征.从软件中总的故障个数、故障检测率FDR（Fault Detection Rate）和测试工作量TE（Testing-Effort）三个方面对影响SRGM的因素进行了分析.文中基于作者前期研究中提出的统一性框架模型,对当前典型的解析模型进行了分类比较和分析;对基于有限与无限服务队列模型的SRGM进行分析与讨论;对以率驱动事件过程RDEP（Rate-Driven Event Processes）为重点的仿真方法进行剖析.进一步,为了验证与分析不同模型的差异,对26个典型的模型在公开发表的16个数据集上进行了实验.结果表明,SRGM的性能差异取决于失效数据集的客观性以及研究人员对测试过程进行不同假设下所建立的数学模型的主观性.最后,指出了SRGM面临的挑战、发展趋势和亟待解决的问题.     

可靠性模型中故障检测率研究述评

故障检测率FDR（Fault Detection Rate）是可靠性研究的关键要素,对于测试环境构建、故障检测效率提升、可靠性建模和可靠性增长具有重要作用,对于提高系统可靠性与确定发布时间具有重要现实意义.首先,对基于NHPP（Non-Homogeneous Poisson Process,非齐次泊松过程）类的软件可靠性增长模型SRGM（Software Reliability Growth Mode）进行概述,给出了建模本质、功用与流程.基于此,引出可靠性建模与研究中的关键参数——FDR,给出定义,对测试环境描述能力进行分析,展示不同模型的差异.着重剖析了FDR与失效强度、冒险率（风险率）的区别,得出三者之间的关联性表述.全面梳理了FDR的大类模型,分别从测试覆盖函数视角、直接设定角度、测试工作量函数参与构成方式三个方面进行剖析,继而提出统一的FDR相关的可靠性模型.考虑到对真实测试环境描述能力需要,建立不完美排错框架模型,衍生出不完美排错下多个不同FDR参与的可靠性增长模型.进一步,在12个真实描述应用场景与公开发表的失效数据集上进行实验,验证不同FDR模型相关的可靠性模型效用,对差异性进行分析与讨论.结果表明,FDR模型自身的性能可以支撑可靠性模型性能的提升.最后,指出了未来研究趋势和需要解决的问题.     

ISTREM软件测试可靠性评估方法研究

在航空电子系统软件测试的背景上,对ARINC653平台燃油模块在Testbed测试平台上的测试用例进行统计,得出实际测试案例分布服从特殊泊松的数学特征。根据软件测试的时间依赖性提出一种新的可靠性评估模型ISTREM,并且求得软件测试用例的失效率,从而对软件进行可靠性评估,解决了软件测试可靠性评估过程复杂且计算量较大并且忽略时间依赖性的问题。在Matlab平台上对软件系统(燃油模块)的测试可靠性进行评估,得出实验结果,并且提出不足。     

Determining an optimal time interval for testing and debuggingsoftware

A decision-theoretic procedure for determining an optimal time interval for testing software prior to its release is proposed. The approach is based on the principles of decision-making under uncertainty and involves a maximization of expected utility. Two plausible forms for the utility function, one based on costs and the other involving the realized reliability of the software, are described. Using previous results on probabilistic models for software failure, the ensuing optimization problem (which can be addressed using numerical techniques) is outlined for the case of single-state testing. The sensitivity of the results to the various input parameters is discussed, and some directions for future research are outlined     

Establishing flight software reliability: testing,model checking,constraint-solving,monitoring and learning

In this paper we discuss the application of a range of techniques to the verification of mission-critical flight software at NASA’s Jet Propulsion Laboratory. For this type of application we want to achieve a higher level of confidence than can be achieved through standard software testing. Unfortunately, given the current state of the art, especially when efforts are constrained by the tight deadlines and resource limitations of a flight project, it is not feasible to produce a rigorous formal proof of correctness of even a well-specified stand-alone module such as a file system (much less more tightly coupled or difficult-to-specify modules). This means that we must look for a practical alternative in the area between traditional testing and proof, as we attempt to optimize rigor and coverage. The approaches we describe here are based on testing, model checking, constraint-solving, monitoring, and finite-state machine learning, in addition to static code analysis. The results we have obtained in the domain of file systems are encouraging, and suggest that for more complex properties of programs with complex data structures, it is possibly more beneficial to use constraint solvers to guide and analyze execution (i.e., as in testing, even if performed by a model checking tool) than to translate the program and property into a set of constraints, as in abstraction-based and bounded model checkers. Our experience with non-file-system flight software modules shows that methods even further removed from traditional static formal methods can be assisted by formal approaches, yet readily adopted by test engineers and software developers, even as the key problem shifts from test generation and selection to test evaluation.     

Optimal and adaptive testing for software reliability assessment

Optimal software testing is concerned with how to test software such that the underlying testing goal is achieved in an optimal manner. Our previous work shows that the optimal testing problem for software reliability growth can be treated as closed-loop or feedback control problem, where the software under test serves as a controlled object and the software testing strategy serves as the corresponding controller. More specifically, the software under test is modeled as controlled Markov chains (CMCs) and the control theory of Markov chains is used to synthesize the required optimal testing strategy. In this paper, we show that software reliability assessment can be treated as a feedback control problem and the CMC approach is also applicable to dealing with the optimal testing problem for software reliability assessment. In this problem, the code of the software under test is frozen and the software testing process is optimized in the sense that the variance of the software reliability estimator is minimized. An adaptive software testing strategy is proposed that uses the testing data collected on-line to estimate the required parameters and selects next test cases. Simulation results show that the proposed adaptive software testing strategy can really work in the sense that the resulting variance of the software reliability estimate is much smaller than that resulting from the random testing strategies. The work presented in this paper is a contribution to the new area of software cybernetics that explores the interplay between software and control.     

一个基于响铃形故障检测率函数的软件可靠性增长模型

非齐次泊松过程类软件可靠性增长模型是评价软件产品可靠性指标的有效工具．影响软件可靠性增长模型评估和预测准确性的最重要的两个因素是软件中隐藏的初始故障数和故障检测率．一些非齐次泊松过程类模型假设故障检测率是不随测试时间变化的常量,有些模型假设故障检测率是增函数或减函数．这些假设或忽略了测试者的学习过程,或忽略了越迟被检测到的故障的概率就可能越低的特点．该文将测试者的学习过程和软件固有故障检测率的变化特征相结合,提出了一个铃形的故障检测率函数,建立了一个非齐次泊松过程类软件可靠性增长模型——Bbell—SRGM．在一组失效数据上的实验分析表明：对这组失效数据,Bbell—SRGM模型比G-O模型等的拟合效果更好．     

考虑测试工作量与覆盖率的软件可靠性模型

为了进一步提升现有非齐次泊松过程类软件可靠性增长模型的拟合与预计精度,首先,提出一个同时考虑测试工作量与测试覆盖率的NHPP类软件可靠性建模框架.在此基础上,将变形S型测试工作量函数(IS-TEF)以及Logistic测试覆盖率函数(LO-TCF)带入该建模框架,建立了一个新的软件可靠性增长模型,即IS-LO-SRGM.同时,还对利用该框架进行建模过程中的两个重要问题进行了描述与分析,即如何确定具体的TEF和TCF以及模型参数估计.然后,在两组真实的失效数据集上,利用该建模框架建立了最为合适的增长模型,即IS-LO-SRGM,并将该模型与8种经典NHPP模型进行对比.实例验证结果表明,所提出的IS-LO-SRGM模型具有最为优秀的拟合与预计性能,从而证明新建模框架的有效性和实用性.最后,对不完美排错情况进行了初步的讨论与建模分析.     

Search based software testing of object-oriented containers

Automatic software testing tools are still far from ideal for real world object-oriented (OO) software. The use of nature inspired search algorithms for this problem has been investigated recently. Testing complex data structures (e.g., containers) is very challenging since testing software with simple states is already hard. Because containers are used in almost every type of software, their reliability is of utmost importance. Hence, this paper focuses on the difficulties of testing container classes with nature inspired search algorithms. We will first describe how input data can be automatically generated for testing Java containers. Input space reductions and a novel testability transformation are presented to aid the search algorithms. Different search algorithms are then considered and studied in order to understand when and why a search algorithm is effective for a testing problem. In our experiments, these nature inspired search algorithms seem to give better results than the traditional techniques described in literature. Besides, the problem of minimising the length of the test sequences is also addressed. Finally, some open research questions are given.     

Reliability growth modeling and optimal release policy under fuzzy environment of an N-version programming system incorporating the effect of fault removal efficiency

Failure of a safety critical system can lead to big losses.Very high software reliability is required for automating the working of systems such as aircraft controller and nuclear reactor controller software systems.Fault-tolerant softwares are used to increase the overall reliability of software systems.Fault tolerance is achieved using the fault-tolerant schemes such as fault recovery (recovery block scheme),fault masking (N-version programming (NVP)) or a combination of both (Hybrid scheme).These softwares incorporate the ability of system survival even on a failure.Many researchers in the field of software engineering have done excellent work to study the reliability of fault-tolerant systems.Most of them consider the stable system reliability.Few attempts have been made in reliability modeling to study the reliability growth for an NVP system.Recently,a model was proposed to analyze the reliability growth of an NVP system incorporating the effect of fault removal efficiency.In this model,a proportion of the number of failures is assumed to be a measure of fault generation while an appropriate measure of fault generation should be the proportion of faults removed.In this paper,we first propose a testing efficiency model incorporating the effect of imperfect fault debugging and error generation.Using this model,a software reliability growth model (SRGM) is developed to model the reliability growth of an NVP system.The proposed model is useful for practical applications and can provide the measures of debugging effectiveness and additional workload or skilled professional required.It is very important for a developer to determine the optimal release time of the software to improve its performance in terms of competition and cost.In this paper,we also formulate the optimal software release time problem for a 3VP system under fuzzy environment and discuss a the fuzzy optimization technique for solving the problem with a numerical illustration.