Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion?, an embedded system providing helicopter landing assistance.