A threat model‐based approach to security testing

Software security issues have been a major concern in the cyberspace community, so a great deal of research on security testing has been performed, and various security testing techniques have been developed. Threat modeling provides a systematic way to identify threats that might compromise security, and it has been a well‐accepted practice by the industry, but test case generation from threat models has not been addressed yet. Thus, in this paper, we propose a threat model‐based security testing approach that automatically generates security test sequences from threat trees and transforms them into executable tests. The security testing approach we consider consists of three activities in large: building threat models with threat trees; generating security test sequences from threat trees; and creating executable test cases by considering valid and invalid inputs. To support our approach, we implemented security test generation techniques, and we also conducted an empirical study to assess the effectiveness of our approach. The results of our study show that our threat tree‐based approach is effective in exposing vulnerabilities. Copyright © 2012 John Wiley & Sons, Ltd.     

Generating transition probabilities to support model‐based software testing

Markov chain usage models support test planning, test automation, and analysis of test results. In practice, transition probabilities for Markov chain usage models are often specified using a cycle of assigning, verifying, and revising specific values for individual transition probabilities. For large systems, such an approach can be difficult for a variety of reasons. We describe an improved approach that represents transition probabilities by explicitly preserving the information concerning test objectives and the relationships between transition probabilities in a format that is easy to maintain and easy to analyze. Using mathematical programming, transition probabilities are automatically generated to satisfy test management objectives and constraints. A more mathematical treatment of this approach is given in References [ 1 ] (Poore JH, Walton GH, Whittaker JA. A constraint‐based approach to the representation of software usage models. Information and SoftwareTechnology 2000; at press) and [ 2 ] (Walton GH. Generating transition probabilities for Markov chain usage models. PhD Thesis, University of Tennessee, Knoxville, TN, May 1995.). In contrast, this paper is targeted at the software engineering practitioner, software development manager, and test manager. This paper also adds to the published literature on Markov chain usage modeling and model‐based testing by describing and illustrating an iterative process for usage model development and optimization and by providing some recommendations for embedding model‐based testing activities within an incremental development process. Copyright © 2000 John Wiley & Sons, Ltd.     

金融软件功能自动化测试的分析及应用

在金融软件测试中,面时大规模复杂的金融算法以及大量的回归测试,自动化测试有着相当的重要的作用.通过介绍自动化测试的相关知识,结合金融软件的特点,搭建相应的金融软件自动化测试环境,并使用一个实例来展示功能自动化测试在金融软件测试中的应用.实验结果表明,自动化测试显著提高金融软件测试的效率和准确率,节省人力和成本.讨论了自动化测试实施的难点和有待改进的地方.     

A holistic approach to model‐based testing of Web service compositions

The behavior of composed Web services depends on the results of the invoked services; unexpected behavior of one of the invoked services can threat the correct execution of an entire composition. This paper proposes an event‐based approach to black‐box testing of Web service compositions based on event sequence graphs, which are extended by facilities to deal not only with service behavior under regular circumstances (i.e., where cooperating services are working as expected) but also with their behavior in undesirable situations (i.e., where cooperating services are not working as expected). Furthermore, the approach can be used independently of artifacts (e.g., Business Process Execution Language) or type of composition (orchestration/choreography). A large case study, based on a commercial Web application, demonstrates the feasibility of the approach and analyzes its characteristics. Test generation and execution are supported by dedicated tools. Especially, the use of an enterprise service bus for test execution is noteworthy and differs from other approaches. The results of the case study encourage to suggest that the new approach has the power to detect faults systematically, performing properly even with complex and large compositions. Copyright © 2012 John Wiley & Sons, Ltd.     

On the use of a similarity function for test case selection in the context of model‐based testing

Test case selection in model‐based testing is discussed focusing on the use of a similarity function. Automatically generated test suites usually have redundant test cases. The reason is that test generation algorithms are usually based on structural coverage criteria that are applied exhaustively. These criteria may not be helpful to detect redundant test cases as well as the suites are usually impractical due to the huge number of test cases that can be generated. Both problems are addressed by applying a similarity function. The idea is to keep in the suite the less similar test cases according to a goal that is defined in terms of the intended size of the test suite. The strategy presented is compared with random selection by considering transition‐based and fault‐based coverage. The results show that, in most of the cases, similarity‐based selection can be more effective than random selection when applied to automatically generated test suites. Copyright © 2009 John Wiley & Sons, Ltd.     

A case study in model‐based testing of specifications and implementations

Despite the existence of a number of animation tools for a variety of languages, methods for employing these tools for specification testing have not been adequately explored. Similarly, despite the close correspondence between specification testing and implementation testing, the two processes are often treated independently, and relatively little investigation has been performed to explore their relationship. This paper presents the results of applying a framework and method for the systematic testing of specifications and their implementations. This framework exploits the close correspondence between specification testing and implementation testing. The framework is evaluated on a sizable case study of the Global System for Mobile Communications 11.11 Standard, which has been developed towards use in a commercial application. The evaluation demonstrates that the framework is of similar cost‐effectiveness to the BZ‐Testing‐Tools framework and more cost‐effective than manual testing. A mutation analysis detected more than 95% of non‐equivalent specification and implementation mutants. Copyright © 2010 John Wiley & Sons, Ltd.     

A model‐based development approach for the verification of real‐time Java code

Many real‐time systems are safety‐and security‐critical systems and, as a result, tools and techniques for verifying them are extremely important. Simulation and testing such systems can be exceedingly time‐consuming and these techniques provide only probabilistic measures of correctness. There are a number of model‐checking tools for real‐time systems. Although they provide formal verification for models, we still need to implement these models. To increase the confidence in real‐time programs written in real‐time Java, this paper proposes a model‐based approach to the development of such programs. First, models can be mechanically verified, to check whether they satisfy particular properties, by using current real‐time model‐checking tools. Then, programs can be derived from the model by following a systematic approach. We introduce a timed automata to RTSJ Tool (TART), a prototype tool to automatically generate real‐time Java code from the model. Finally, we show the applicability of our approach by means of four examples: a gear controller, an audio/video protocol, a producer/consumer and the Fischer protocol. Copyright © 2011 John Wiley & Sons, Ltd.     

Testing‐based process for component substitutability

Software components have emerged to ease the assembly of software systems. However, updates of systems by substitution or upgrades of components demand careful management due to stability risks of deployed systems. Replacement components must be properly evaluated to identify if they provide the expected behaviour affected by substitution. To address this problem, this paper proposes a substitutability assessment process in which the regular compatibility analysis is complemented with the use of black‐box testing criteria. The purpose is to observe the components' behaviour by analysing their internal functions of data transformation, which fulfils the observability testing metric. The approach is conceptually based on the technique Back‐to‐Back testing. When a component should be replaced, a specific Test Suite TS is built in order to represent its behavioural facets, viz. a Component Behaviour TS. This TS is later exercised on candidate upgrades or replacement components with the purpose of identifying the required compatibility. Automation of the process is supported through the testooj tool, which constrains the conditions and steps of the whole process in order to provide a rigorous and reliable approach. Copyright © 2010 John Wiley & Sons, Ltd.     

Killing strategies for model‐based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Back's action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright © 2014 John Wiley & Sons, Ltd.     

A dynamic stochastic model for automatic grammar‐based test generation

Grammar‐based test generation provides a systematic approach to producing test cases from a given context‐free grammar. Unfortunately, naive grammar‐based test generation is problematic because of the fact that exhaustive random test case production is often explosive, and grammar‐based test generation with explicit annotation controls often causes unbalanced testing coverage. In this paper, we present an automatic grammar‐based test generation approach, which takes a symbolic grammar as input, requires zero control input from users, and produces well‐distributed test cases. Our approach utilizes a novel dynamic stochastic model where each variable is associated with a tuple of probability distributions, which are dynamically adjusted along the derivation. We further present a coverage tree illustrating the distribution of generated test cases and their detailed derivations. More importantly, the coverage tree supports various implicit derivation control mechanisms. We implemented this approach in a Java‐based system, named Gena. Each test case generated by Gena automatically comes with a set of structural features, which can play an important and effective role on automated failure causes localization. Experimental results demonstrate the effectiveness of our approach, the well‐balanced distribution of generated test cases over grammatical structures, and a case study on grammar‐based failure causes localization. Copyright © 2014 John Wiley & Sons, Ltd.     

Generating test data from state‐based specifications

Although the majority of software testing in industry is conducted at the system level, most formal research has focused on the unit level. As a result, most system‐level testing techniques are only described informally. This paper presents formal testing criteria for system level testing that are based on formal specifications of the software. Software testing can only be formalized and quantified when a solid basis for test generation can be defined. Formal specifications represent a significant opportunity for testing because they precisely describe what functions the software is supposed to provide in a form that can be automatically manipulated. This paper presents general criteria for generating test inputs from state‐based specifications. The criteria include techniques for generating tests at several levels of abstraction for specifications (transition predicates, transitions, pairs of transitions and sequences of transitions). These techniques provide coverage criteria that are based on the specifications and are made up of several parts, including test prefixes that contain inputs necessary to put the software into the appropriate state for the test values. The test generation process includes several steps for transforming specifications to tests. These criteria have been applied to a case study to compare their ability to detect seeded faults. Copyright © 2003 John Wiley & Sons, Ltd.     

Seeding strategies in search‐based unit test generation

Search‐based techniques have been applied successfully to the task of generating unit tests for object‐oriented software. However, as for any meta‐heuristic search, the efficiency heavily depends on many factors; seeding, which refers to the use of previous related knowledge to help solve the testing problem at hand, is one such factor that may strongly influence this efficiency. This paper investigates different seeding strategies for unit test generation, in particular seeding of numerical and string constants derived statically and dynamically, seeding of type information and seeding of previously generated tests. To understand the effects of these seeding strategies, the results of a large empirical analysis carried out on a large collection of open‐source projects from the SF110 corpus and the Apache Commons repository are reported. These experiments show with strong statistical confidence that, even for a testing tool already able to achieve high coverage, the use of appropriate seeding strategies can further improve performance. © 2016 The Authors. Software Testing, Verification and Reliability Published by John Wiley & Sons Ltd.     

GUICop: Approach and toolset for specification‐based GUI testing

Oracles used for testing graphical user interface (GUI) programmes are required to take into consideration complicating factors such as variations in screen resolution or colour scheme when comparing observed GUI elements with expected GUI elements. Researchers proposed fuzzy comparison rules and computationally expensive image processing techniques to tame the comparison process because otherwise the naïve matching comparison would be too constraining and consequently impractical. Alternatively, this paper proposes GUICop, a novel approach with a supporting toolset that takes (1) a GUI programme and (2) user‐defined GUI specifications characterizing the rendering behaviour of the GUI elements and checks whether the execution traces of the programme satisfy the specifications. GUICop comprises the following: (1) a GUI Specification Language; (2) a Driver; (3) Instrumented GUI Libraries; 4) a Solver; and (5) a Code Weaver. The user defines the specifications of the subject GUI programme using the GUI Specification Language. The Driver traverses the GUI structure of the programme and generates events that drive its execution. The Instrumented GUI Libraries capture the GUI execution trace, ie, information about the positions and visibility of the GUI elements. And the Solver, enabled by code injected by the Code Weaver, checks whether the traces satisfy the specifications. GUICop was successfully evaluated using 4 open source GUI applications that included 8 defects, namely, Jajuk, Gason, JEdit, and TerpPaint.     

Observers in nonlinear model‐based predictive control

A closed‐loop reformulation of the dual‐mode paradigm affords significant advantages in model‐based predictive control of systems subject to uncertainty and/or disturbances and/or nonlinear dynamics. This paper considers earlier results based on ellipsoidal invariant sets and proposes extensions to the output feedback nonlinear case by defining invariant sets both for the system state and the error dynamics of a nonlinear observer. Particular attention is paid to the class of systems with separable nonlinearities but the results carry over to the general case. Copyright © 2000 John Wiley & Sons, Ltd.     

Search‐based software test data generation: a survey

The use of metaheuristic search techniques for the automatic generation of test data has been a burgeoning interest for many researchers in recent years. Previous attempts to automate the test generation process have been limited, having been constrained by the size and complexity of software, and the basic fact that, in general, test data generation is an undecidable problem. Metaheuristic search techniques offer much promise in regard to these problems. Metaheuristic search techniques are high‐level frameworks, which utilize heuristics to seek solutions for combinatorial problems at a reasonable computational cost. To date, metaheuristic search techniques have been applied to automate test data generation for structural and functional testing; the testing of grey‐box properties, for example safety constraints; and also non‐functional properties, such as worst‐case execution time. This paper surveys some of the work undertaken in this field, discussing possible new future directions of research for each of its different individual areas. Copyright © 2004 John Wiley & Sons, Ltd.     

A novel model‐driven approach for seamless integration

This research mentions integration problems and describes a novel model‐driven approach that intend to reach a higher degree of interoperability among different software development tools coming from different technological spaces (TSs) by representing data of tools through the models. The proposed concept introduce a way to integrate various software related tools and aim to provide a modular syntax for tool integration that leverage the collaboration of different tools. In this work, the proposed approach has been tested through a case–study by demonstrating a single aspect of the model‐driven tool integration because the model‐driven tool integration has a wide scope and it is difficult to show all aspects of it in one research. It is proved that the model‐driven tool integration is possible based on the proposed concept between different TSs, and the formulation of the proposed approach is provided. As the results indicate, the proposed system integrates selected software‐related tools coming from different TSs and enables them to use each other's capabilities. This work paves the way to contribute for the standardization efforts of the model‐driven tool integration. Finally, further research opportunities are provided. Copyright © 2016 John Wiley & Sons, Ltd.     

Fault detection model‐based controller for process systems

This paper develops a model‐based control system for fault detection and controller reconfiguration using stochastic model predictive control (MPC). The system can determine online the optimal control actions, detect faults quickly, and reconfigure the controller accordingly. Such a system can perform its function correctly in the presence of internal faults. A fault detection model based (FDMB) controller consists of two main parts: the first is fault detection and diagnosis (FDD) and the second is controller reconfiguration (CR). Systems subject to such abrupt failures are modeled as stochastic hybrid systems with variable‐structure. This paper deals with three challenging issues: design of the fault‐model set; estimation of hybrid multiple models; and stochastic MPC of hybrid multiple models. For the first issue, we propose a simple scheme for designing a fault model set based on random variables. For the second issue, we consider and select a fast and reliable FDD system applied to the above model set. Finally, we develop a stochastic MPC scheme for multiple model CR with soft switching signals based on the weighted probabilities of the outputs of different models. Simulations for the proposed FDMB controller are illustrated and analyzed. Copyright © 2011 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Using component metadata to regression test component‐based software

Increasingly, modern‐day software systems are being built by combining externally‐developed software components with application‐specific code. For such systems, existing program‐analysis‐based software engineering techniques may not directly apply, due to lack of information about components. To address this problem, the use of component metadata has been proposed. Component metadata are metadata and metamethods provided with components, that retrieve or calculate information about those components. In particular, two component‐metadata‐based approaches for regression test selection are described: one using code‐based component metadata and the other using specification‐based component metadata. The results of empirical studies that illustrate the potential of these techniques to provide savings in re‐testing effort are provided. Copyright © 2006 John Wiley & Sons, Ltd.     

Model‐based security testing: a taxonomy and systematic classification

Model‐based security testing relies on models to test whether a software system meets its security requirements. It is an active research field of high relevance for industrial applications, with many approaches and notable results published in recent years. This article provides a taxonomy for model‐based security testing approaches. It comprises filter criteria (i.e. model of system security, security model of the environment and explicit test selection criteria) as well as evidence criteria (i.e. maturity of evaluated system, evidence measures and evidence level). The taxonomy is based on a comprehensive analysis of existing classification schemes for model‐based testing and security testing. To demonstrate its adequacy, 119 publications on model‐based security testing are systematically extracted from the five most relevant digital libraries by three researchers and classified according to the defined filter and evidence criteria. On the basis of the classified publications, the article provides an overview of the state of the art in model‐based security testing and discusses promising research directions with regard to security properties, coverage criteria and the feasibility and return on investment of model‐based security testing. Copyright © 2015 John Wiley & Sons, Ltd.     

Fault‐driven stress testing of distributed real‐time software based on UML models

In a previous article, a stress testing methodology was reported to detect network traffic‐related Real‐Time (RT) faults in distributed RT systems based on the design UML model of a System Under Test (SUT). The stress methodology, referred to as Test LOcation‐driven Stress Testing (TLOST), aimed at increasing the chances of RT failures (violations in RT constraints) associated with a given stress test location (an network or a node under test). As demonstrated and experimented in this article, although TLOST is useful in stress testing different test locations (nodes and network, it does not guarantee to target (test) all RT constraints in an SUT. This is because the durations of message sequences bounded by some RT constraints might never be exercised (covered) by TLOST. A complementary stress test methodology is proposed in this article, which guarantees to target (cover) all RT constraints in an SUT and detect their potential RT faults (if any). Using a case study, this article shows that the new complementary methodology is capable of targeting the RT faults not detected by the previous test methodology. Copyright © 2009 John Wiley & Sons, Ltd.