关于蜕变测试和特殊用例测试的实例研究

提出了基于蜕变测试方法的集成测试环境MTest,进而为检验蜕变测试方法的能力和效率,以稀疏矩阵乘法程序为例设计了一组实验.该实验基于变异分析技术,分别以mutation score和错误发现率为度量指标,定量地分析和对比了特殊用例测试,以特殊测试用例和随机测试用例为源测试用例的蜕变测试这3种方法的测试能力和效率.该实验可在MTest测试环境下自动完成.实验结果表明,蜕变测试与特殊用例测试之间是互补的,而且就蜕变测试的源测试用例而言,随机测试用例在测试能力和效率上优于特殊测试用例.     

面向对象程序蜕变关系构造方法

针对面向对象软件类级测试中蜕变关系构造不充分的问题,提出一种由错误类型指导面向对象程序蜕变关系构造的方法。首先分析类方法中包含的基本操作,根据错误发生位置和作用效果将错误分成3类;其次 按照执行效果的不同 将类中方法分成两类;再针对错误类型对每个方法分别构造蜕变关系;最后通过Rectangle类的实验对比该方法与其他方法。实验证明提出的由错误类型指导蜕变关系构造的方法的检错率有所提高,并且有助于错误定位。     

一种采用接口错误注入的构件安全性测试方法

构件特别是第三方构件的安全性是影响构件技术发展的重要因素.当前甚少将构件安全性问题作为一个专门的课题来研究,也没有确实可行的方法和技术用来检测构件安全漏洞.构件的安全性问题仍然没有得到较好的解决.提出一种基于接口错误注入的构件安全性测试方法.方法首先给出构件脆弱性错误注入算子、断言规则库及其脆弱因子,然后基于错误注入算子和断言规则库给出一种构件安全测试算法.算法根据错误注入算子有针对性地产生测试用例并执行测试,动态监测机制能实时监测构件运行状态及异常信息.算法根据断言规则库、脆弱性因子及异常信息给出构件安全异常报告及构件安全等级.最后在一集成测试平台中实现了该方法,同时做了大量实验,并和FUZZ法进行了对比分析.实验结果表明所提出的测试方法可操作性强,且对构件脆弱性具有较好的检测效果.     

面向整数错误检测的蜕变测试方法研究

整型错误在安全性和功能性方面都是影响软件质量的重要因素,但是由于测试人员往往面临Oracle问题,无法判断程序是否出现整型错误,除非程序输出一个明显的异常值。本文利用蜕变测试技术发现由于整型错误产生的隐错,实验结果显示基于蜕变关系的整型错误检测方法可检测出平时发现不了的隐式非预期输出,有效地提高了检测整型错误的效率。     

用于黑盒测试的测试用例优先级改进算法

为了在源代码不可见的黑盒环境下提高软件测试效率,研究了通过调整测试用例执行顺序的测试用例优先级方法在黑盒测试中的应用.针对已有的用于黑盒测试优先级方法的不足,提出了改进的基于黑盒测试的优先级方法.以提高错误检测率为目标,该方法通过结合两种用于不同情况下的优先级调整策略和使用动态方法代替静态方法生成优先级步调调整值对原方法提出了两处改进.仿真实验结果表明,该算法有效且可行,两处改进均能有效地提高测试集的错误检测率,同时使用改进效果尤为明显.     

面向条件筛选搜索系统的蜕变测试方法

为缓解拥有庞大数据信息的条件筛选搜索系统搜索结果时带来的Oracle问题,提出蜕变测试方法.通过识别程序多次输入输出之间的关系是否违反蜕变关系,可有效缓解Oracle问题.因此,有效识别蜕变关系是完成蜕变测试的前提.根据条件筛选搜索系统的自定义条件,设计两种蜕变关系模式帮助简化蜕变关系的识别,提高蜕变测试的故障检测质量.实验结果表明,利用改进的蜕变关系模式在条件筛选搜索系统中执行蜕变测试提高了14％的精确率,验证了该方法识别蜕变关系的简洁性和有效性.     

机器学习图像分类程序的蜕变测试框架

机器学习在计算机视觉、语音识别和自然语言处理等实际应用中已经取得了显著的成功。图像分类作为计算机视觉的一个主要分支。不久的将来,许多的图像分类程序会以机器学习的方式呈现。然而,由于机器学习图像分类程序的测试面临着测试预言难题,这使得在测试的过程中将需要大量的人力及物力。为了缓解测试预言难题,使用了蜕变测试技术。为了规范测试流程、提高测试效率,提出了一种适用于机器学习图像分类程序的蜕变测试框架。并且通过测试基于SVM和VGGNet图像分类程序,验证了该测试框架的合理性和有效性。     

基于影响分析的回归测试优先级错误定位方法

基于程序行为特征的错误定位方法由于只孤立地看待每个程序实体,使其错误定位的效率受到影响,而回归测试错误定位又由于需要执行全部测试用例将大大增加开发和测试成本。针对以上问题,提出一种基于影响分析的回归测试优先级错误定位方法,该方法将联合依赖图、基于程序行为特征的错误定位方法和回归测试优先级进行有机结合。实验结果表明,与Ochiai,Tarantula,PPDG,CP和Naish等经典方法相比,该方法可更加有效地定位软件错误。     

基于程序路径分析的有效蜕变测试

蜕变测试对于预期输出难以构造的程序是实用和高效的.作者在系统研究已有蜕变测试方法和路径分析技术的基础上,首先针对使用二元蜕变关系的测试提出了一组蜕变测试准则,以在多个不同的层次上定义蜕变测试用例集的充分性;然后给出了3种能够生成相应测试用例集的算法;最后通过变异分析的方法证实这些算法的有效性.实验结果表明,蜕变关系和测试准则的选取直接影响到测试的效果,另外,使用蜕变关系全路径覆盖可满足性算法(APCEMsT)可以快速准确地发现待测程序中的错误,而生成的测试用例的数量却比传统技术要少.     

基于提交排序和预测模型的测试套件选择方法

为在持续集成（CI）环境下减少回归测试集、提升回归测试的效率,提出一种适用于CI环境的回归测试套件选择方法。首先,根据每个提交的测试套件历史失败率和执行率信息,进行提交排序;然后,采用机器学习方法,对提交涉及的测试套件进行失败率预测,并选择具有较高失败率的测试套件。该方法综合使用提交排序技术和测试套件选择技术,从而保证既提高故障检测率又能在一定程度上降低测试成本。在Google的开源数据集上进行的实验结果表明：与同样采用提交排序的方法和采用测试套件选择的方法相比,所提方法的开销感知平均故障检测率APFDc提高了1%~27%;在相同的测试时间成本下,所提方法的测试召回提高了33.33~38.16个百分点,变更召回提高了15.67~24.52个百分点,测试套件选择率降低了约6个百分点。     

Linear complexity assertions for sorting

Correctness of the execution of sorting programs can be checked by two assertions: the order assertion and the permutation assertion. The order assertion checks if the sorted data is in ascending or descending order. The permutation assertion checks if the output data produced by sorting is a permutation of the original input data. Permutation and order assertions are sufficient for the detection of errors in the execution of sorting programs; however, in terms of execution time these assertions cost the same as sorting programs. An assertion, called the order-sum assertion, that has lower execution cost than sorting programs is derived from permutation and order assertions. The reduction in cost is achieved at the expense of incomplete checking. Some metrics are derived to quantify the effectiveness of order-sum assertion under various error models. A natural connection between the effectiveness of the order-sum assertion and the partition theory of numbers is shown. Asymptotic formulae for partition functions are derived     

General test result checking with log file analysis

We describe and apply a lightweight formal method for checking test results. The method assumes that the software under test writes a text log file; this log file is then analyzed by a program to see if it reveals failures. We suggest a state-machine-based formalism for specifying the log file analyzer programs and describe a language and implementation based on that formalism. We report on empirical studies of the application of log file analysis to random testing of units. We describe the results of experiments done to compare the performance and effectiveness of random unit testing with coverage checking and log file analysis to other unit testing procedures. The experiments suggest that writing a formal log file analyzer and using random testing is competitive with other formal and informal methods for unit testing.     

A history-based cost-cognizant test case prioritization technique in regression testing

Software testing is typically used to verify whether the developed software product meets its requirements. From the result of software testing, developers can make an assessment about the quality or the acceptability of developed software. It is noted that during testing, the test case is a pair of input and expected output, and a number of test cases will be executed either sequentially or randomly. The techniques of test case prioritization usually schedule test cases for regression testing in an order that attempts to increase the effectiveness. However, the cost of test cases and the severity of faults are usually varied. In this paper, we propose a method of cost-cognizant test case prioritization based on the use of historical records. We gather the historical records from the latest regression testing and then propose a genetic algorithm to determine the most effective order. Some controlled experiments are performed to evaluate the effectiveness of our proposed method. Evaluation results indicate that our proposed method has improved the fault detection effectiveness. It can also been found that prioritizing test cases based on their historical information can provide high test effectiveness during testing.     

Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites. In particular, Java and the JUnit testing framework are being used extensively to build software systems in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm. To investigate the practical implications of these results, we present a set of cost-benefits models for test case prioritization, and show how the effectiveness differences observed can result in savings in practice, but vary substantially with the cost factors associated with particular testing processes.     

Empirical studies of a prediction model for regression testselection

Regression testing is an important activity that can account for a large proportion of the cost of software maintenance. One approach to reducing the cost of regression testing is to employ a selective regression testing technique that: chooses a subset of a test suite that was used to test the software before the modifications; then uses this subset to test the modified software. Selective regression testing techniques reduce the cost of regression testing if the cost of selecting the subset from the test suite together with the cost of running the selected subset of test cases is less than the cost of rerunning the entire test suite. Rosenblum and Weyuker (1997) proposed coverage-based predictors for use in predicting the effectiveness of regression test selection strategies. Using the regression testing cost model of Leung and White (1989; 1990), Rosenblum and Weyuker demonstrated the applicability of these predictors by performing a case study involving 31 versions of the KornShell. To further investigate the applicability of the Rosenblum-Weyuker (RW) predictor, additional empirical studies have been performed. The RW predictor was applied to a number of subjects, using two different selective regression testing tools, Deja vu and TestTube. These studies support two conclusions. First, they show that there is some variability in the success with which the predictors work and second, they suggest that these results can be improved by incorporating information about the distribution of modifications. It is shown how the RW prediction model can be improved to provide such an accounting     

基于Markov决策过程用交叉熵方法优化软件测试

研究了待测软件某些参数已知的条件下,以最小化平均测试费用为目标的软件测试优化问题.将软件测试过程处理成马尔可夫(Markov)决策过程,给出了软件测试的马尔可夫决策模型,运用交叉熵方法,通过一种学习策略获得软件测试的最优测试剖面,用于优化软件测试.模拟结果表明,学习策略给出的测试剖面要优于随机测试策略,检测和排除相同数目的软件缺陷,学习策略比随机测试能够显著地减少测试用例数,降低测试成本,提高缺陷检测效率.     

Generating a checking sequence with a minimum number of reset transitions

Given a finite state machine M, a checking sequence is an input sequence that is guaranteed to lead to a failure if the implementation under test is faulty and has no more states than M. There has been much interest in the automated generation of a short checking sequence from a finite state machine. However, such sequences can contain reset transitions whose use can adversely affect both the cost of applying the checking sequence and the effectiveness of the checking sequence. Thus, we sometimes want a checking sequence with a minimum number of reset transitions rather than a shortest checking sequence. This paper describes a new algorithm for generating a checking sequence, based on a distinguishing sequence, that minimises the number of reset transitions used.     

On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general     

A learning algorithm for optimizing continuous integration development and testing practice

Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large-scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault-detection effectiveness of continuous integration tests using historical test records and combining this information with coverage-based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest-all approach. The results further demonstrate no significant reduction in fault-detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.