星载软件可靠性测试实践

为验证和评估软件可靠性水平,阐述了开展软件可靠性测试的方法和过程,针对星载嵌入式软件的特点,介绍了进行软件可靠性测试的关键步骤的实现途径和方法,包括:失效的定义、测试环境的搭建、使用剖面的构造、测试过程的执行以及失效数据的收集,最后给出了该软件的可靠性定量评估结果,表明了该方法和流程的实用性和可行性,为后续开展类似的软件可靠性测试提供实践经验.     

基于活动图模型测试剖面的测试用例生成方法

给出了测试用例的定义和如何根据UML活动图模型来生成测试用例的基本方法，包括基于活动图模型控制流结构的测试场景生成和针对活动的输入量的测试数据生成。根据活动图模型的层次型特点，引入了针对话动的层次化的测试剖面(Test Profiles)概念和输入输出数据描述规范，用以支持用户在活动图上分层次地提供有关测试数据生成的约束条件。同时，给出了基于测试剖面的基本测试数据的生成方法，以及基于测试场景和基本测试数据的组合来生成一组测试用例的方法。     

基于正交试验设计的软件可靠性测试

提出基于正交试验设计法构建操作剖面，通过对操作输入集进行处理，即可优化测试输入集，进而进行软件可靠性测试。实例表明，该方法简化了操作剖面的构建，大大降低了软件可靠性测试的代价。     

基于故障树分析的软件安全性测试研究

针对软件安全性测试的本质特征在于快速降低由于软件失效而导致系统事故的风险, 结合基于Baye-sian统计理论的测试方法, 建立一套构建安全性测试剖面, 并由此产生测试用例的测试方法。该方法运用故障树分析技术, 对各模块发生故障对系统安全性的影响进行分析, 找出影响较大的关键性模块, 然后利用分析结果构建安全性测试剖面。最后给出了测试停止的标准。通过对例子的分析可知, 本方法在快速降低软件事故风险方面比现有软件测试方法更有效。     

FPGA软件操作序列剖面驱动的可靠性测试技术研究

操作剖面及其构造方法在软件可靠性测试中已经被广泛应用,然而,在FPGA软件测试的场景下,由于操作的时序性、操作之间的约束性,以及操作的连续性,使得其传统的操作剖面及构造方法存在不能反映操作之间的约束关系、时序关系和随机性的缺点,难以满足测试用例的合理性和覆盖性。提出一种基于操作序列剖面驱动的可靠性测试方法,根据不同的阶段,采用“六步构造法”构造相互独立的操作序列剖面,同时,以相机控制FPGA软件为例进行了可靠性用例生成并进行了测试可靠度评估。评估结果表明,该方法生成的测试用例效率高、代价低、通用性强,能够对FPGA软件的可靠性进行有效的验证。提出的方法将为FPGA软件可靠性测试提供一种具有较高实用价值的技术手段,以满足FPGA复杂系统软件的越来越高的测试需求。     

软件可靠性测试中最小测试量的确定方法

分析了软件可靠性测试中确定最小测试量的重要意义,阐述了基于操作剖面的软件可靠性测试数据生成方法和软件可靠性测试数据统计特征原理下的确定最小测试量的理论依据和具体方法,提出了该理论指导下的测试数据生成流程,通过实例给出了方法的具体应用过程.     

基于失效聚集度改进自适应随机测试算法

对于现有的自适应随机测试(ART)算法针对点状失效模式普遍存在有效性和效率均比随机测试(RT)差的问题,提出一种基于失效聚集度的自适应随机测试(CLART)算法,对传统的ART——固定候选集(FSCS)、区域排除随机测试(RRT)等算法进行改进。首先,根据被测程序的输入域估计主失效聚集度,确定局部搜索区域;然后,在区域内使用传统ART算法生成若干测试用例(TC)进行测试;若未发现错误,重新选择局部区域生成TC;重复这一过程直至发现错误。仿真实验显示在点状失效模式和块状失效模式下CLART算法的有效性比FSCS算法提高约20%,效率比FSCS算法提高约60%。实验结果表明CLART算法利用多个局部区域依次搜索可以快速锁定引发失效输入分布密集高的失效区域,从而提高测试的有效性和效率。     

基于故障剖面的安全关键系统可靠性测试与评估

可靠性测试是安全关键系统可靠性评估的重要手段。论文结合在某电信系统的工程实践,介绍一种基于故障剖面的可靠性测试和评估的方法：通过逆向工程从已有的安全关键系统的失效事件中分析提取出故障概率数据,结合故障注入测试对系统的可靠性进行评估。该方法直接从故障入手,不受缺乏缺陷引发故障概率数据问题的困扰,并通过故障模式的双层模型明确测试范围,简化了评估过程。     

远程多管火箭炮火控系统的软件安全性测试分析

简单介绍了远程多管火箭炮火控系统的软件组成、功能、典型任务剖面和软件安全性测试的基本内涵;然后根据火控系统的典型任务剖面分析了不同阶段的软件安全性测试,系统解决了远程多管火箭炮火控系统软件安全性测试“难”“杂”“多”问题,有效提高了测试效率和质量,进一步确保了远程多管火箭炮火控系统的安全性。     

软件可靠性测试方法新探

针对传统软件可靠性测试方法在对软件因长期使用软件性能下降,甚至完全失效这种严重影响软件可靠性的测试存在不足的现状,通过对软件自身特性以及软件可靠性估算面临问题的深入分析和研究,结合传统的软件可靠性测试方法,提出了基于操作剖面的软件可靠性压力测试思想和操作剖面、压力测试点相互结合、互为补充的软件可靠性测试方法,并给出了软件可靠性测试新方法实施的技术途径.该思想与方法既是对传统软件可靠性测试方法的一个大胆探索,也是对软件可靠性测试方法的一个有益补充.     

在测试用例不放回时比较随机测试和分割测试

在测试用例放回的情况下,关于随机测试和分割测试的比较,许多研究者做了大量的工作,取得了显著成果.在测试用例不放回的情况下,类似的比较工作在国内外文献中尚未见到.然而在实际工作中,尤其是在软件测试早期和模块测试阶段,测试用例是不放回的.因此,在测试用例不放回的情况下,对随机测试和分割测试进行了比较,得出4个结论.与Chen和Yu在测试用例放回情况下的研究成果相比,一个不同的发现是,在平分子域、错误数、测试次数时,分割测试不如随机测试效果好.另外还发现,如何利用各种信息分割出错误集中的区域,然后着重测试,这是分割测试的核心.     

Proportional sampling strategy: guidelines for software testing practitioners

Recently, several sufficient conditions have been developed that guarantee partition testing to have a higher probability of detecting at least one failure than random testing. One of these conditions is that the number of test cases selected from each partition is proportional to the size of the partition. We call such a method of allocating test cases the proportional sampling strategy. Although this condition is not the most general one, it is the most easily and practically applicable one. In this paper, we discuss how the proportional sampling strategy can be applied effectively in practice. Some practical issues that need to be attended are identified and guidelines to deal with these issues are suggested.     

On comparisons of random, partition, and proportional partitiontesting

Early studies of random versus partition testing used the probability of detecting at least one failure as a measure of test effectiveness and indicated that partition testing is not significantly more effective than random testing. More recent studies have focused on proportional partition testing because a proportional allocation of the test cases (according to the probabilities of the subdomains) can guarantee that partition testing will perform at least as well as random testing. We show that this goal for partition testing is not a worthwhile one. Guaranteeing that partition testing has at least as high a probability of detecting a failure comes at the expense of decreasing its relative advantage over random testing. We then discuss other problems with previous studies and show that failure to include important factors (cost, relative effectiveness) can lead to misleading results     

The Economics of Unit Testing

Conventional wisdom and anecdote suggests that testing takes between 30 to 50% of a project's effort. However testing is not a monolithic activity as it consists of a number of different phases such as unit testing, integration testing and finally system and acceptance test. Unit testing has received a lot of criticism in terms of the amount of time that it is perceived to take and its perceived costs. However it still remains an important verification activity being an effective means to test individual software components for boundary value behavior and ensure that all code has been exercised adequately. We examine the available data from three safety-related, industrial software projects that have made use of unit testing. Using this information we argue that the perceived costs of unit testing may be exaggerated and that the likely benefits in terms of defect detection are quite high in relation to those costs. We also discuss the different issues that have been found applying the technique at different phases of the development and using different methods to generate those tests. We also compare results we have obtained with empirical results from the literature and highlight some possible weakness of research in this area.     

Test case selection for black-box regression testing of database applications

ContextThis paper presents an approach for selecting regression test cases in the context of large-scale database applications. We focus on a black-box (specification-based) approach, relying on classification tree models to model the input domain of the system under test (SUT), in order to obtain a more practical and scalable solution. We perform an experiment in an industrial setting where the SUT is a large database application in Norway’s tax department.ObjectiveWe investigate the use of similarity-based test case selection for supporting black box regression testing of database applications. We have developed a practical approach and tool (DART) for functional black-box regression testing of database applications. In order to make the regression test approach scalable for large database applications, we needed a test case selection strategy that reduces the test execution costs and analysis effort. We used classification tree models to partition the input domain of the SUT in order to then select test cases. Rather than selecting test cases at random from each partition, we incorporated a similarity-based test case selection, hypothesizing that it would yield a higher fault detection rate.MethodAn experiment was conducted to determine which similarity-based selection algorithm was the most suitable in selecting test cases in large regression test suites, and whether similarity-based selection was a worthwhile and practical alternative to simpler solutions.ResultsThe results show that combining similarity measurement with partition-based test case selection, by using similarity-based test case selection within each partition, can provide improved fault detection rates over simpler solutions when specific conditions are met regarding the partitions.ConclusionsUnder the conditions present in the experiment the improvements were marginal. However, a detailed analysis concludes that the similarity-based selection strategy should be applied when a large number of test cases are contained in each partition and there is significant variability within partitions. If these conditions are not present, incorporating similarity measures is not worthwhile, since the gain is negligible over a random selection within each partition.     

基于Markov决策过程用交叉熵方法优化软件测试

研究了待测软件某些参数已知的条件下,以最小化平均测试费用为目标的软件测试优化问题.将软件测试过程处理成马尔可夫(Markov)决策过程,给出了软件测试的马尔可夫决策模型,运用交叉熵方法,通过一种学习策略获得软件测试的最优测试剖面,用于优化软件测试.模拟结果表明,学习策略给出的测试剖面要优于随机测试策略,检测和排除相同数目的软件缺陷,学习策略比随机测试能够显著地减少测试用例数,降低测试成本,提高缺陷检测效率.     

Partition testing vs. random testing: the influence of uncertainty

This paper compares partition testing and random testing on the assumption that program failure rates are not known with certainty before testing and are, therefore, modeled by random variables. It is shown that under uncertainty, partition testing compares more favorably to random testing than suggested by prior investigations concerning the deterministic case: the restriction to failure rates that are known with certainty systematically favors random testing. In particular, we generalize a result by Weyuker and Jeng (1991) stating equal fault detection probabilities for partition testing and random testing in the case where the failure rates in the subdomains defined by the partition are equal. It turns out that for independent random failure rates with equal expectation, the case above is a boundary case (the worst case for partition testing), and the fault detection probability of partition testing can be up to k times higher than that of random testing, where k is the number of subdomains. Also in a related model for dependent failure rates, partition testing turns out to be consistently better than random testing. The dominance can also be verified for the expected (weighted) number of detected faults as an alternative comparison criterion     

Comparison of adaptive random testing and random testing under various testing and debugging scenarios

Adaptive random testing is an enhancement of random testing. Previous studies on adaptive random testing assumed that once a failure is detected, testing is terminated and debugging is conducted immediately. It has been shown that adaptive random testing normally uses fewer test cases than random testing for detecting the first software failure. However, under many practical situations, testing should not be withheld after the detection of a failure. Thus, it is important to investigate the effectiveness with respect to the detection of multiple failures. In this paper, we compare adaptive random testing and random testing under various scenarios and examine whether adaptive random testing is still able to use fewer test cases than random testing to detect multiple software failures. Our study delivers some interesting results and highlights a number of promising research projects. Copyright © 2011 John Wiley & Sons, Ltd.     

Adaptive random testing based on distribution metrics

Random testing (RT) is a fundamental software testing technique. Adaptive random testing (ART), an enhancement of RT, generally uses fewer test cases than RT to detect the first failure. ART generates test cases in a random manner, together with additional test case selection criteria to enforce that the executed test cases are evenly spread over the input domain. Some studies have been conducted to measure how evenly an ART algorithm can spread its test cases with respect to some distribution metrics. These studies observed that there exists a correlation between the failure detection capability and the evenness of test case distribution. Inspired by this observation, we aim to study whether failure detection capability of ART can be enhanced by using distribution metrics as criteria for the test case selection process. Our simulations and empirical results show that the newly proposed algorithms not only improve the evenness of test case distribution, but also enhance the failure detection capability of ART.     

Distributing test cases more evenly in adaptive random testing

Adaptive random testing (ART) has recently been proposed to enhance the failure-detection capability of random testing. In ART, test cases are not only randomly generated, but also evenly spread over the input domain. Various ART algorithms have been developed to evenly spread test cases in different ways. Previous studies have shown that some ART algorithms prefer to select test cases from the edge part of the input domain rather than from the centre part, that is, inputs do not have equal chance to be selected as test cases. Since we do not know where the failure-causing inputs are prior to testing, it is not desirable for inputs to have different chances of being selected as test cases. Therefore, in this paper, we investigate how to enhance some ART algorithms by offsetting the edge preference, and propose a new family of ART algorithms. A series of simulations have been conducted and it is shown that these new algorithms not only select test cases more evenly, but also have better failure detection capabilities.