Fault-Based Conformance Testing in Practice

Conforming to protocol speci.cations is a critical issue in modern distributed software systems. Nowadays, complex service infrastructures, such as Voice-over-IP systems, are usually built by combining components of di.erent vendors. If the components do not correctly implement the various protocol speci.cations, failures will certainly occur. In the case of emergency calls this may be even life-threatening. Functional black-box conformance testing, where one checks the conformance of the implemented protocol to a speci.cation becomes therefore a major issue. In this work, we report on our experiences and .ndings when applying fault-based conformance testing to an industrial application. Besides a discussion on modeling and simpli.cations we present a technique that prevents an application from implementing particular faults. Faults are modeled at the level of the speci.cation. We show how such a technique can be adapted to speci.cations with large state spaces and present results obtained when applying our technique to the Session Initiation Protocol and to the Conference Protocol. Finally, we compare our results to random and scenario based testing.     

Killing strategies for model‐based mutation testing

This article presents the techniques and results of a novel model‐based test case generation approach that automatically derives test cases from UML state machines. The main contribution of this article is the fully automated fault‐based test case generation technique together with two empirical case studies derived from industrial use cases. Also, an in‐depth evaluation of different fault‐based test case generation strategies on each of the case studies is given and a comparison with plain random testing is conducted. The test case generation methodology supports a wide range of UML constructs and is grounded on the formal semantics of Back's action systems and the well‐known input–output conformance relation. Mutation operators are employed on the level of the specification to insert faults and generate test cases that will reveal the faults inserted. The effectiveness of this approach is shown and it is discussed how to gain a more expressive test suite by combining cheap but undirected random test case generation with the more expensive but directed mutation‐based technique. Finally, an extensive and critical discussion of the lessons learnt is given as well as a future outlook on the general usefulness and practicability of mutation‐based test case generation. Copyright © 2014 John Wiley & Sons, Ltd.