软件测试理论初步框架

软件测试是软件开发中不可或缺的部分,也是软件工程化方法中的重要环节。目前各种软件测试技术日趋成熟,但相关的测试基本原理框架还有待开发。在前人经验的基础上,试图提出一个初步理论框架来定义软件测试的样本空间,引入反映软件某种情况(比如缺陷)的随机变量,概括白盒测试和黑盒测试的概率测度及数学期望描述。这样的构建不仅能够加深对软件缺陷存在的理论根源的理解,从而进一步提出更好的测试方法,还对发展软件测试的科学理论有所帮助。     

基于故障检测上下文的等价变异体识别算法

等价变异体识别一直是阻碍变异测试在工业界得以广泛应用的一个关键难题.为此提出了一种基于故障检测上下文的等价变异体识别算法.该算法通过静态分析技术抽取程序中与故障检测条件相关的代码上下文信息,以构造故障检测上下文;接着,故障检测上下文被转换为文档模型,经过一个文档表示学习网络进行编码;最后通过机器学习模型将变异体分类为等价或非等价变异.在包含了22个C程序和118 000个变异体样本的训练集上,该算法取得91%的分类精准度和82%的召回率;同时在跨项目交叉验证中,机器学习模型取得了77%的精准度和78%的召回率.该结果表明基于故障检测上下文的识别技术能够有效地提高等价变异体分类的精准性和泛用性,为提高变异测试技术的有效性提供了技术支持.     

Analyzing clusters of class characteristics in OO applications

The transition from Java 1.4 to Java 1.5 has provided the programmer with more flexibility due to the inclusion of several new language constructs, such as parameterized types. This transition is expected to increase the number of class clusters exhibiting different combinations of class characteristics. In this paper we investigate how the number and distribution of clusters are expected to change during this transition. We present the results of an empirical study were we analyzed applications written in both Java 1.4 and 1.5. In addition, we show how the variability of the combinations of class characteristics may affect the testing of class members.     

A set of complexity metrics for guiding the software test process

Software verification and validation is a domain which is covered by many dynamic test, static analysis, and formal verification techniques. This presents a problem to practitioners with respect to selecting those suitable techniques which can be used successfully. The basic idea of the methodology presented here is to select test techniques which fit the software under test. A dynamic test technique requires that certain program elements are covered, will be sensitive to errors associated with these elements, because executing an error location is a precondition for revealing the error. Furthermore, it is likely that the probability of errors increases with complexity. Complexity can be characterized in terms of several properties which can be used to suggest various testing strategies. The complexity of the various software properties can be measured using appropriate complexity metrics. Properties with unusual high complexity measures should be tested very throughly. The approach described in this paper permits the selection of test techniques based on the values of the metrics with respect to a particular software product.     

软件测试不确定性研究及解决途径

从分析软件测试的不确定性产生的原因,分析软件测试中存在的问题,提出加强软件测试工程化管理、测试方法模板的复用、采用测试用例的知识共享作为解决问题的途径,介绍了一种软件测试管理平台的实现。     

A case study of tracking software development using quality metrics

This paper presents a case history of Mentor Graphics using a set of quality metrics to track development progress for a recent major software release. It provides background on how Mentor Graphics originally began using software metrics to measure product quality, how this became accepted, and how these metrics later fell out of favour. To restore these metrics to effective use, process changes were required for setting quality and metric targets, and for the way the metrics are used for tracking development progress. With these process changes in place, and the addition of a new metric, the case history demonstrates that the metric set could be used effectively to indicate problems in this release and help manage changes to the plan for completion of the release. The lessons learned in this case history are presented, along with subsequent data that further validates these metrics.     

浅析工程应用软件的可靠性测试技术实施

本文通过对软件可靠性模型的分析探讨,对工程应用软件的样本测试提出了检验性测试和验证性测试的思路。根据该思路的测试步骤,论述了测试中案例的选取和运行轨道,以及与测试相关的测试策略和评判标准,并借助实际工程应用软件样本,对测试结果进行了综合评定。     

基于软件可靠性工程的测试模型

软件可靠性工程是软件工程的一个重要分支,主要建立在操作剖面、软件可靠性模型、概率论和软件测试等理论的基础之上。该文结合软件可靠性、软件可靠性工程和软件测试的相关理论,设计了可靠性测试模型,并在市政交通一卡通工程系统中进行了实际验证。     

Class noise detection based on software metrics and ROC curves

Noise detection for software measurement datasets is a topic of growing interest. The presence of class and attribute noise in software measurement datasets degrades the performance of machine learning-based classifiers, and the identification of these noisy modules improves the overall performance. In this study, we propose a noise detection algorithm based on software metrics threshold values. The threshold values are obtained from the Receiver Operating Characteristic (ROC) analysis. This paper focuses on case studies of five public NASA datasets and details the construction of Naive Bayes-based software fault prediction models both before and after applying the proposed noise detection algorithm. Experimental results show that this noise detection approach is very effective for detecting the class noise and that the performance of fault predictors using a Naive Bayes algorithm with a logNum filter improves if the class labels of identified noisy modules are corrected.     

基于规则与学习的变异技术对比研究

变异测试是一种有效的软件测试技术,通过生成变异体来模拟软件缺陷,帮助提升现有测试套件的缺陷检测能力.变异体的质量对于变异测试的有效性具有显著影响.传统的变异测试方法通常采用人工设计的基于语法规则的变异算子生成变异体,并已取得一定的研究成果.近年来,许多研究开始结合深度学习技术,通过学习开源项目历史代码生成变异体.目前,该新方法在变异体生成方面取得了初步的成果.基于语法规则和基于学习的两种变异技术,其机理不同,但其目标均是通过生成变异体来提高测试套件的缺陷检测能力,因此,全面比较这两种变异技术对于变异测试及其下游任务至关重要.针对这一问题,设计实现了一项针对基于语法规则和基于学习的变异技术的实证研究,旨在了解不同机理的变异技术在变异测试任务上的性能以及生成的变异体在程序语义上的差异性.具体地,以Defect4J v1.2.0数据集为实验对象,比较以MAJOR和PIT为代表的基于语法规则的变异技术和以DeepMutation、μBERT和LEAM为代表的基于深度学习的变异技术.实验结果表明:基于规则与学习的变异技术均可有效支持变异测试实践,但MAJOR的测试效果最优,能够检测85.4%的真实缺陷.在语义表示上, MAJOR具有最强的语义代表能力,基于其构造的测试套件能够杀死其余变异技术生成的超过95%占比的变异体.在缺陷表征上,两类技术均具有独特性.     

基于测试充分性准则的非死锁并发缺陷定位方法

并发程序的非确定性使得其调试工作异常困难。基于程序谱的软件缺陷定位方法虽然能够缓解该情况,但其定位结果依赖于调试信息。针对在此过程中难以获得调试信息及如何选择利用调试信息的问题,提出了一种基于测试充分性准则的缺陷定位方法,该方法包括3个部分:预测满足测试充分性准则的条件;制定相应的测试方案;将收集到的调试信息用于缺陷定位分析。依据此方法,用C#语言实现了缺陷定位工具——ConFinder。在含有实际并发缺陷的程序上进行实验,结果表明该方法可以有效找出引起程序失效的原因并且所得结果具有很好的稳定性。     

Identification of defect-prone classes in telecommunication software systems using design metrics

The goal of this paper is to investigate the relation between object-oriented design choices and defects in software systems, with focus on a real-time telecommunication domain. The design choices are measured using the widely accepted metrics suite proposed by Chidamber and Kemerer for object oriented languages [S.R. Chidamber, C.F. Kemerer, A metrics suite for object oriented design, IEEE Transactions on Software Engineering 20 (6) (1994) 476-493].This paper reports the results of an extensive case study, which strongly reinforces earlier, mainly anecdotal, evidence that design aspects related to communication between classes can be used as indicators of the most defect-prone classes.Statistical models applicable for the non-normally distributed count data are used, such as Poisson regression, negative binomial regression, and zero-inflated negative binomial regression. The performances of the models are assessed using correlations, dispersion coefficients and Alberg diagrams.The zero-inflated negative binomial regression model based on response for a class shows the best overall ability to describe the variability of the number of defects in classes.     

试析软件开发过程中的软件测试

对软件测试进行了介绍,阐述了软件测试的分类、特性及准则步骤,并对软件测试的未来进行了展望,提出了关于软件测试的方法和见解。     

Improving Fault Detection in Modified Code ——- A Study from the Telecommunication Industry

Many software systems are developed in a number of consecutive releases. In each release not only new code is added but also existing code is often modified. In this study we show that the modified code can be an important source of faults. Faults are widely recognized as one of the major cost drivers in software projects. Therefore, we look for methods that improve the fault detection in the modified code. We propose and evaluate a number of prediction models that increase the efficiency of fault detection. To build and evaluate our models we use data collected from two large telecommunication systems produced by Ericsson. We evaluate the performance of our models by applying them both to a different release of the system than the one they are built on and to a different system. The performance of our models is compared to the performance of the theoretical best model, a simple model based on size, as well as to analyzing the code in a random order （not using any model）. We find that the use of our models provides a significant improvement over not using any model at all and over using a simple model based on the class size. The gain offered by our models corresponds to 38-57% of the theoretical maximum gain.     

Applying machine learning to software fault-proneness prediction

The importance of software testing to quality assurance cannot be overemphasized. The estimation of a module’s fault-proneness is important for minimizing cost and improving the effectiveness of the software testing process. Unfortunately, no general technique for estimating software fault-proneness is available. The observed correlation between some software metrics and fault-proneness has resulted in a variety of predictive models based on multiple metrics. Much work has concentrated on how to select the software metrics that are most likely to indicate fault-proneness. In this paper, we propose the use of machine learning for this purpose. Specifically, given historical data on software metric values and number of reported errors, an Artificial Neural Network (ANN) is trained. Then, in order to determine the importance of each software metric in predicting fault-proneness, a sensitivity analysis is performed on the trained ANN. The software metrics that are deemed to be the most critical are then used as the basis of an ANN-based predictive model of a continuous measure of fault-proneness. We also view fault-proneness prediction as a binary classification task (i.e., a module can either contain errors or be error-free) and use Support Vector Machines (SVM) as a state-of-the-art classification method. We perform a comparative experimental study of the effectiveness of ANNs and SVMs on a data set obtained from NASA’s Metrics Data Program data repository.     

A static API birthmark for Windows binary executables

A software birthmark is the inherent characteristics of a program extracted from the program itself. By comparing birthmarks, we can detect whether a program is a copy of another program or not. We propose a static API birthmark for Windows executables that utilizes sets of API calls identified by a disassembler statically. By comparing 49 Windows executables, we show that our birthmark can distinguish similar programs and detect copies. By comparing binaries generated by various compilers, we also demonstrate that our birthmark is resilient. We compare our birthmark with a previous Windows dynamic birthmark to show that it is more appropriate for GUI applications.     

A relationship between software coverage metrics and reliability

Software engineers have tacitly assumed that there exists a link between coverage and reliability. An argument is presented which leads to an inverse logarithmic relationship which depends upon the structure of the software and on a parameter called the strength of the coverage metric. The relationship is developed by considering reliability as a function of an idealized coverage metric and then interpreting the result for a range of actual coverage metrics.