GUICop: Approach and toolset for specification‐based GUI testing

Oracles used for testing graphical user interface (GUI) programmes are required to take into consideration complicating factors such as variations in screen resolution or colour scheme when comparing observed GUI elements with expected GUI elements. Researchers proposed fuzzy comparison rules and computationally expensive image processing techniques to tame the comparison process because otherwise the naïve matching comparison would be too constraining and consequently impractical. Alternatively, this paper proposes GUICop, a novel approach with a supporting toolset that takes (1) a GUI programme and (2) user‐defined GUI specifications characterizing the rendering behaviour of the GUI elements and checks whether the execution traces of the programme satisfy the specifications. GUICop comprises the following: (1) a GUI Specification Language; (2) a Driver; (3) Instrumented GUI Libraries; 4) a Solver; and (5) a Code Weaver. The user defines the specifications of the subject GUI programme using the GUI Specification Language. The Driver traverses the GUI structure of the programme and generates events that drive its execution. The Instrumented GUI Libraries capture the GUI execution trace, ie, information about the positions and visibility of the GUI elements. And the Solver, enabled by code injected by the Code Weaver, checks whether the traces satisfy the specifications. GUICop was successfully evaluated using 4 open source GUI applications that included 8 defects, namely, Jajuk, Gason, JEdit, and TerpPaint.     

A systematic review of approaches for testing concurrent programs

Concurrent programs are replacing the sequential programs as they utilize the true capabilities of multicore architecture. The extensive use of multicore systems and multithreaded paradigms warrants more attention to the testing of the concurrent programs. The testing concurrent program is not a new field as it has been more than 40 years because the first problem related to the testing concurrent program was addressed by the researchers. The field covers various domains, which include concurrency problems, testing approaches, techniques, graphical representations, tools, and subject systems. This paper aims at providing an overview of research in the domain of testing concurrent programs by classifying it into eight categories: (a) reachability testing, (b) structural testing, (c) model‐based testing, (d) mutation‐based testing, (e) slicing‐based testing, (f) formal methods, (g) random testing, and (h) search‐based testing. The survey is focused on the techniques applied, methodologies followed, and tools used in these aforementioned approaches. Furthermore, the gaps are also identified in different approaches. The paper concludes with the consolidation of various testing parameters along with the future directions. Copyright © 2015 John Wiley & Sons, Ltd.     

A combinatorial testing strategy for concurrent programs

One approach to testing concurrent programs is called reachability testing, which derives test sequences automatically and on‐the‐fly, without constructing a static model. Existing reachability testing algorithms are exhaustive in that they are intended to exercise all possible synchronization sequences of a concurrent program with a given input. In this paper, we present a new testing strategy, called t‐way reachability testing, that adopts the dynamic framework of reachability testing but selectively exercises a subset of synchronization sequences. The selection of the synchronization sequences is based on a combinatorial testing strategy called t‐way testing. We present an algorithm that implements t‐way reachability testing, and report the results of several case studies that were conducted to evaluate its effectiveness. The results indicate that t‐way reachability testing can substantially reduce the number of synchronization sequences exercised during reachability testing while still effectively detecting faults. Copyright © 2007 John Wiley & Sons, Ltd.     

A transformation‐based approach to testing concurrent programs using UML activity diagrams

Unified Modeling Language (UML) activity diagrams are widely used to model concurrent interaction among multiple objects. In this paper, we propose a transformation‐based approach to generating scenario‐oriented test cases for applications modeled by UML activity diagrams. Using a set of transformation rules, the proposed approach first transforms a UML activity diagram specification into an intermediate representation, from which it then constructs test scenarios with respect to the given concurrency coverage criteria. The approach then finally derives a set of test cases for the constructed test scenarios. The approach resolves the difficulties associated with fork and join concurrency in the UML activity diagram and enables control over the number of the resulting test cases. We further implemented a tool to automate the proposed approach and studied its feasibility and effectiveness using a case study. Experimental results show that the approach can generate test cases on demand to satisfy a given concurrency coverage criterion and can detect up to 76.5% of seeded faults when a weak coverage criterion is used. With the approach, testers can not only schedule the software test process earlier, but can also better allocate the testing resources for testing concurrent applications. Copyright © 2015 John Wiley & Sons, Ltd.     

Structural testing for message‐passing concurrent programs: an extended test model

Developing high‐quality, error‐free message‐passing concurrent programs is not trivial. Although a number of different primitives with associated semantics are available to assist such development, they often increase the complexity of the testing process. In this paper, we extend our previous test model for message‐passing programs and present new structural testing criteria, taking into account additional features used in this paradigm, such as collective communication, non‐blocking sends, distinct semantics for non‐blocking receives, and persistent operations. Our new model also recognizes that sender primitives cannot always be matched with every receive primitive. This improvement allows us to remove statically a significant number of infeasible synchronization edges that would otherwise have to be analyzed later by the tester. In this paper, the test model is presented using the Message‐Passing Interface standard; however, our new model has been designed to be flexible, and it can be configured to support a range of different message‐passing environments or languages. We have carried out case studies showing the applicability of the new test model to represent message‐passing programs and also to reveal errors, mainly those errors related to inter‐process communication. In addition to increasing the number of features supported by the test model, we have also reduced the overall cost of testing significantly. Our case studies suggest that the number of synchronization edges can be reduced by up to 93%, mainly by eliminating infeasible edges between unmatchable communication primitives. The main contribution of the paper is to present a more flexible test model that provides improved coverage for message‐passing programs and at the same time reduces the cost of testing significantly. Copyright © 2012 John Wiley & Sons, Ltd.     

蜕变测试研究进展及其在并行程序测试中的研究展望

在软件测试过程中,待测程序的预期输出是判断软件是否存在缺陷的重要因素.蜕变测试技术是利用被测软件的属性来检查程序输出,从而有效地解决程序预期输出难以构造的问题.近年来,蜕变测试在软件测试领域取得了蓬勃的发展,许多研究人员将蜕变测试技术进行优化,将其运用到各个领域,有效提高了软件质量.从原理、过程及其优化,应用领域3个方面,总结蜕变测试的研究工作,着重分析了近5年的研究进展,进一步展望了蜕变测试用于并行程序时,可能的研究主题.首先,介绍蜕变测试的基本概念和蜕变测试过程;接着,从蜕变关系、测试用例、测试执行过程以及蜕变测试工具4个角度,总结蜕变测试优化技术;然后,汇总了蜕变测试的应用领域;最后,基于已有研究成果,讨论蜕变测试在并行程序测试领域面临的问题,为蜕变技术在并行程序测试领域的研究提供可能的思路.     

Finding failures from passed test cases: improving the pattern classification approach to the testing of mesh simplification programs

Mesh simplification programs create three‐dimensional polygonal models similar to an original polygonal model, and yet use fewer polygons. They produce different graphics even though they are based on the same original polygonal model. This results in a test oracle problem. To address the problem, our previous work has developed a technique that uses a reference model of the program under test to train a classifier. Using such an approach may mistakenly mark a failure‐causing test case as passed. It lowers the testing effectiveness of revealing failures. This paper suggests piping the test cases marked as passed by a statistical pattern classification module to an analytical metamorphic testing (MT) module. We evaluate our approach empirically using three subject programs with over 2700 program mutants. The result shows that, using a resembling reference model to train a classifier, the integrated approach can significantly improve the failure detection effectiveness of the pattern classification approach. We also explain how MT in our design trades specificity for sensitivity. Copyright © 2009 John Wiley & Sons, Ltd.     

坡度坡向量算程序蜕变测试方法

坡度、坡向量算是地理信息系统的主要功能之一。然而,由于截断误差、舍入误差以及误差传播的影响,导致测试判定难以获取。基于此,提出将蜕变测试应用于坡度、坡向量算程序的测试中,通过分析坡度、坡向量算程序功能的几何属性、数值计算特性以及具体实现算法等提出蜕变关系,并通过分析蜕变关系的适用范围,形成坡度坡向量算程序蜕变测试方法。最后,通过实例研究,验证了提出的蜕变关系在消除不同类型变异方面的有效性。实验结果表明,该方法可有效解决坡度、坡向量算程序的测试判定问题,同时也为解决地理信息系统中其他空间度量程序的测试判定问题提供了借鉴,进一步拓展了蜕变测试技术的应用范围。     

面向服务软件的蜕变测试方法

在面向服务软件的测试过程中,由于在服务发现之前不可知的交互对象和同一个服务可能会有不同实现,往往出现程序执行结果不能提前预知的Oracle问题。为了有效地解决面向服务软件测试中的Oracle问题,基于面向服务架构(SOA)的特点,提出将蜕变测试方法用于面向服务软件的单元测试和集成测试过程中,依据面向服务软件每个服务的自身性质构造蜕变关系,设计蜕变测试类执行测试用例并验证蜕变关系是否保持,如果蜕变关系被违反了,则发现和报告缺陷,从而有效地支持面向服务软件的测试。     

FAST: a framework for automating statistics-based testing

To achieve software quality, testing is an essential component in all software development. It involves the execution of a deterministic software system with test data and a comparison of the results with the expected output, which must satisfy the users' requirements. This accounts for over 25% of the cost of a software development. Therefore, automation has considerable potential. The quality programming introduced by Cho can automatically generate data for testing, based on a so-called SIAD tree which is used to represent the hierarchical and network relation between input elements and also incorporates rules into the tree for using the inputs. However, it lacks a clear framework which would show how automated testing can be achieved. To address this problem, we present a Framework for Automating Statistics-based Testing (FAST), which is an extension of the testing concept in quality programming to achieve automated testing. In FAST, we propose a SOAD tree, which is similar to the structure of the SIAD tree, to describe the syntactic structure of the product unit and its defectiveness. Based on this tool, the inspection of test results can be automatically achieved by lexical and syntax analysis. The implementation of automated software testing or Command File Interpreter (CFI) software which incorporates the framework is also described.     

Integration testing of Object-Oriented and Aspect-Oriented programs: A structural pairwise approach for Java

Several testing approaches focus on finding faults in software units of implementation. A problem not addressed by unit testing is the interaction among units, with respect to the correctness of their interfaces. In this paper a structural integration testing approach for Object-Oriented (OO) and Aspect-Oriented (AO) Java programs is presented. To make the activity more feasible, we address the testing of pairs of units (i.e., methods and pieces of advice). A model called (PairWise Def-Use) graph to represent the flow of control and data between pairs of units is proposed. Based on the , the following family of testing criteria is defined: all-pairwise-integrated-nodes (control-flow based), all-pairwise-integrated-edges (control-flow based), and all-pairwise-integrated-uses (data-flow based). To evaluate the proposed approach, an implementation of the criteria in a testing tool is presented along with an example of usage and an exploratory study. The study with 7 AO programs that are also OO was conducted to investigate the cost of application and usefulness of the approach. Results provided evidence that the criteria are practical and useful for integration testing of OO and AO programs.     

Distributed reachability testing of concurrent programs

Reachability testing is an approach to verifying concurrent programs. During reachability testing, every partially ordered synchronization sequence of a program with a given input is exercised exactly once. In this paper, we present the design and implementation of a distributed reachability testing algorithm for a cluster of workstations. This algorithm allows different test sequences to be exercised concurrently by different workstations without any synchronization, and without any duplication of sequences among workstations. Dynamic load balancing is performed using a work‐stealing scheme. A novel aspect of this scheme is that work‐stealing requests progress in rounds. This round‐based structure identifies overloaded workstations to target for work stealing. Empirical studies show good speedup for four benchmark Java programs and one Lotos specification. Copyright © 2010 John Wiley & Sons, Ltd.     

PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs

Graphics applications often need to manipulate numerous graphical objects stored as polygonal models. Mesh simplification is an approach to vary the levels of visual details as appropriate, thereby improving on the overall performance of the applications. Different mesh simplification algorithms may cater for different needs, producing diversified types of simplified polygonal model as a result. Testing mesh simplification implementations is essential to assure the quality of the graphics applications. However, it is very difficult to determine the oracles (or expected outcomes) of mesh simplification for the verification of test results.A reference model is an implementation closely related to the program under test. Is it possible to use such reference models as pseudo-oracles for testing mesh simplification programs? If so, how effective are they?This paper presents a fault-based pattern classification methodology called PAT, to address the questions. In PAT, we train the C4.5 classifier using black-box features of samples from a reference model and its fault-based versions, in order to test samples from the subject program. We evaluate PAT using four implementations of mesh simplification algorithms as reference models applied to 44 open-source three-dimensional polygonal models. Empirical results reveal that the use of a reference model as a pseudo-oracle is effective for testing the implementations of resembling mesh simplification algorithms. However, the results also show a tradeoff: When compared with a simple reference model, the use of a resembling but sophisticated reference model is more effective and accurate but less robust.     

Variable strength combinatorial testing of concurrent programs

Reachability testing is an important approach to testing concurrent programs. It generates and exercises synchronization sequences automatically and on-the-fly without saving any test history. Existing reachability testing can be classified into exhaustive and t-way testing. Exhaustive testing is impractical in many cases while t-way testing may decrease the capability of fault detection in some cases. In this paper, we present a variable strength reachability testing strategy, which adopts the dynamic framework of reachability testing and uses a variable strength combinatorial strategy. Different parameter groups are provided with different covering strength. Variable strength testing covers no t-way combinations but the necessary combinations of parameters having mutual interactions in a concurrent program. It is more reasonable than t-way testing because uniform interactions between parameters do not often exist in concurrent systems. We propose a merging algorithmthat implements the variable strength combinatorial testing strategy and conduct our experiment on several concurrent programs. The experimental results indicate that our variable strength reachability testing reaches a good tradeoff between the effectiveness and efficiency. It can keep the same capability of fault detection as exhaustive reachability testing while substantially reducing the number of synchronization sequences and decreasing the execution time in most cases.     

GPS导航软件的一种功能自动化测试框架

GPS导航软件已经大量涌入市场,对于GPS导航软件本身的质量越来越受到重视。针对GPS导航软件的应用特点,提出了一种GPS导航软件功能自动测试框架,并介绍了相关实现技术,它能自动模拟软件的使用环境和GPS测试数据来验证软件的正确性。在实践中证明此框架行之有效,能有效构建测试环境,及时发现软件的缺陷,并可提高其回归测试的效率。     

方面单元测试框架的设计与实现

缺乏成熟的方面测试和方面冲突检测技术是阻碍面向方面软件编程大规模应用的瓶颈性问题。针对方面单元测试,提出了无需与基模块编织的方面单元的独立测试框架AJUnit。该框架运用Java注释和源码与字节码映射等技术实现编译细节的透明化。基于控制流和数据流分析,AJUnit综合运用随机测试数据生成、相对测试充分性准则的优化和指定路径的测试数据生成等技术实现测试过程的自动化。文中也介绍了以AJUnit为基础的方面单元测试工具的设计和实现。     

Structural testing criteria for message‐passing parallel programs

Parallel programs present some features such as concurrency, communication and synchronization that make the test a challenging activity. Because of these characteristics, the direct application of traditional testing is not always possible and adequate testing criteria and tools are necessary. In this paper we investigate the challenges of validating message‐passing parallel programs and present a set of specific testing criteria. We introduce a family of structural testing criteria based on a test model. The model captures control and data flow of the message‐passing programs, by considering their sequential and parallel aspects. The criteria provide a coverage measure that can be used for evaluating the progress of the testing activity and also provide guidelines for the generation of test data. We also describe a tool, called ValiPar, which supports the application of the proposed testing criteria. Currently, ValiPar is configured for parallel virtual machine (PVM) and message‐passing interface (MPI). Results of the application of the proposed criteria to MPI programs are also presented and analyzed. Copyright © 2008 John Wiley & Sons, Ltd.     

Systematic testing: A means for validating reactive systems

The validity of the first (formal) model of a system to be developed is crucial for the whole development process. Systematically checking this validity helps avoid costs that could arise if it were discovered too late that the system does not satisfy the customer's needs and expectations. This paper addresses how to validate synchronous reactive programs using the technique of systematic testing. Testing reactive systems differs from testing sequential systems: instead of checking simple pairs of inputs and outputs, sequences of inputs and outputs have to be checked. Thus, testing cannot be based on a simple function model, mapping input values onto output values nor on a control flow graph model (where a path from the start node to the final node represents one execution through the represented program). The model widely used instead is that of a finite-state machine. A systematic testing approach is presented that is both effective and efficient for validating reactive systems. It uses an additional specification based on a finite-state machine model. The approach is demonstrated for the well-known lift example. It is shown how to use the specification for carefully choosing a set of test criteria that address different types of fault; a procedure for selecting test cases and test data that satisfy the chosen criteria is presented.     

面向GUI软件测试的交互图生成及其实现*

从控件的信息入手,研究了界面的控件间功能交互关系,提出了基于交互图的GUI测试方法,并开发了一个基于交互图的GUI测试工具GUITester.它能自动生成交互图和测试用例,大大提高了GUI软件测试的效率和覆盖率.