Practical minimization of pairwise-covering test configurations using constraint programming

Context: Testing highly-configurable software systems is challenging due to a large number of test configurations that have to be carefully selected in order to reduce the testing effort as much as possible, while maintaining high software quality. Finding the smallest set of valid test configurations that ensure sufficient coverage of the system’s feature interactions is thus the objective of validation engineers, especially when the execution of test configurations is costly or time-consuming. However, this problem is NP-hard in general and approximation algorithms have often been used to address it in practice.Objective: In this paper, we explore an alternative exact approach based on constraint programming that will allow engineers to increase the effectiveness of configuration testing while keeping the number of configurations as low as possible.Method: Our approach consists in using a (time-aware) minimization algorithm based on constraint programming. Given the amount of time, our solution generates a minimized set of valid test configurations that ensure coverage of all pairs of feature values (a.k.a. pairwise coverage). The approach has been implemented in a tool called PACOGEN.Results: PACOGEN was evaluated on 224 feature models in comparison with the two existing tools that are based on a greedy algorithm. For 79% of 224 feature models, PACOGEN generated up to 60% fewer test configurations than the competitor tools. We further evaluated PACOGEN in the case study of an industrial video conferencing product line with a feature model of 169 features, and found 60% fewer configurations compared with the manual approach followed by test engineers. The set of test configurations generated by PACOGEN decreased the time required by test engineers in manual test configuration by 85%, increasing the feature-pairs coverage at the same time.Conclusion: Our experimental evaluation concluded that optimal time-aware minimization of pairwise-covering test configurations is efficiently addressed using constraint programming techniques.     

Automated analysis of feature models 20 years later: A literature review

Software product line engineering is about producing a set of related products that share more commonalities than variabilities. Feature models are widely used for variability and commonality management in software product lines. Feature models are information models where a set of products are represented as a set of features in a single model. The automated analysis of feature models deals with the computer-aided extraction of information from feature models. The literature on this topic has contributed with a set of operations, techniques, tools and empirical results which have not been surveyed until now. This paper provides a comprehensive literature review on the automated analysis of feature models 20 years after of their invention. This paper contributes by bringing together previously disparate streams of work to help shed light on this thriving area. We also present a conceptual framework to understand the different proposals as well as categorise future contributions. We finally discuss the different studies and propose some challenges to be faced in the future.     

Assessing the maintainability of software product line feature models using structural metrics

A software product line is a unified representation of a set of conceptually similar software systems that share many common features and satisfy the requirements of a particular domain. Within the context of software product lines, feature models are tree-like structures that are widely used for modeling and representing the inherent commonality and variability of software product lines. Given the fact that many different software systems can be spawned from a single software product line, it can be anticipated that a low-quality design can ripple through to many spawned software systems. Therefore, the need for early indicators of external quality attributes is recognized in order to avoid the implications of defective and low-quality design during the late stages of production. In this paper, we propose a set of structural metrics for software product line feature models and theoretically validate them using valid measurement-theoretic principles. Further, we investigate through controlled experimentation whether these structural metrics can be good predictors (early indicators) of the three main subcharacteristics of maintainability: analyzability, changeability, and understandability. More specifically, a four-step analysis is conducted: (1) investigating whether feature model structural metrics are correlated with feature model maintainability through the employment of classical statistical correlation techniques; (2) understanding how well each of the structural metrics can serve as discriminatory references for maintainability; (3) identifying the sufficient set of structural metrics for evaluating each of the subcharacteristics of maintainability; and (4) evaluating how well different prediction models based on the proposed structural metrics can perform in indicating the maintainability of a feature model. Results obtained from the controlled experiment support the idea that useful prediction models can be built for the purpose of evaluating feature model maintainability using early structural metrics. Some of the structural metrics show significant correlation with the subjective perception of the subjects about the maintainability of the feature models.     

Automated product line test case selection: industrial case study and controlled experiment

Automated test case selection for a new product in a product line is challenging due to several reasons. First, the variability within the product line needs to be captured in a systematic way; second, the reusable test cases from the repository are required to be identified for testing a new product. The objective of such automated process is to reduce the overall effort for selection (e.g., selection time), while achieving an acceptable level of the coverage of testing functionalities. In this paper, we propose a systematic and automated methodology using a feature model for testing (FM_T) to capture commonalities and variabilities of a product line and a component family model for testing (CFM_T) to capture the overall structure of test cases in the repository. With our methodology, a test engineer does not need to manually go through the repository to select a relevant set of test cases for a new product. Instead, a test engineer only needs to select a set of relevant features using FM_T at a higher level of abstraction for a product and a set of relevant test cases will be selected automatically. We evaluated our methodology via three different ways: (1) We applied our methodology to a product line of video conferencing systems called Saturn developed by Cisco, and the results show that our methodology can reduce the selection effort significantly; (2) we conducted a questionnaire-based study to solicit the views of test engineers who were involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology and models (FM_T and CFM_T) in their current practice; (3) we conducted a controlled experiment with 20 graduate students to assess the performance (i.e., cost, effectiveness and efficiency) of our automated methodology as compared to the manual approach. The results showed that our methodology is cost-effective as compared to the manual approach, and at the same time, its efficiency is not affected by the increased complexity of products.     

A systematic test case selection methodology for product lines: results and insights from an industrial case study

In the context of product lines, test case selection aims at obtaining a set of relevant test cases for a product from the entire set of test cases available for a product line. While working on a research-based innovation project on automated testing of product lines of Video Conferencing Systems (VCSs) developed by Cisco, we felt the need to devise a cost-effective way of selecting relevant test cases for a product. To fulfill such need, we propose a systematic and automated test selection methodology using: 1) Feature Model for Testing (FM_T) to capture commonalities and variabilities of a product line; 2) Component Family Model for Testing (CFM_T) to model the structure of test case repository; 3) A tool to automatically build restrictions from CFM_T to FM_T and traces from CFM_T to the actual test cases. Using our methodology, a test engineer is only required to select relevant features through FM_T at a higher level of abstraction for a product and the corresponding test cases will be obtained automatically. We evaluate our methodology by applying it to a VCS product line called Saturn with seven commercial products and the results show that our methodology can significantly reduce cost measured as test selection time and at the same time achieves higher effectiveness (feature coverage, feature pairwise coverage and fault detection) as compared with the current manual process. Moreover, we conduct a questionnaire-based study to solicit the views of test engineers who are involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology in their current practice. Finally, we present a set of lessons learnt while applying product line engineering at Cisco for test case selection.     

Mapping feature models onto domain models: ensuring consistency of configured domain models

We present an approach to model-driven software product line engineering which is based on feature models and domain models. A feature model describes both common and varying properties of the instances of a software product line. The domain model is composed of a structural model (package and class diagrams) and a behavioral model (story diagrams). Features are mapped onto the domain model by annotating elements of the domain model with features. An element of a domain model is specific to the features included in its feature annotation. An instance of the product line is defined by a set of selected features (a feature configuration). A configuration of the domain model is built by excluding all elements whose feature set is not included in the feature configuration. To ensure consistency of the configured domain model, we define constraints on the annotations of inter-dependent domain model elements. These constraints guarantee that a model element may be selected only when the model elements are also included on which it depends. Violations of dependency constraints may be removed automatically with the help of an error repair tool which propagates features to dependent model elements.     

Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion?, an embedded system providing helicopter landing assistance.     

Heuristic and exact algorithms for product configuration in software product lines

Software product line (SPL) is a set of software applications that share a common set of features satisfying the specific needs of a particular market segment. SPL engineering is a paradigm to develop software applications that commonly use a feature model to capture and document common and variable features, and their relationships. A big challenge is to derive one product among all possible products in the SPL, which satisfies the business and customer requirements. This task is known as product configuration. Although product configuration has been extensively investigated in the literature, customer's preferences are frequently neglected. In this paper, we propose a novel approach to configure a product that considers both qualitative and quantitative feature properties. We model the product configuration task as a combinatorial optimization problem, and heuristic and exact algorithms are proposed. As far as we are concerned, this proposal is the first work in the literature that considers feature properties in both leaf and nonleaf features. Computational experiments showed that the best of our heuristics found optimal solutions for all instances where those are known. For the instances where optimal solutions are not known, our heuristic outperformed the best solution obtained by a one‐hour run of the exact algorithm by up to 67.89%.     

The effects of visualization and interaction techniques on feature model configuration

A Software Product Line is a set of software systems of a domain, which share some common features but also have significant variability. A feature model is a variability modeling artifact which represents differences among software products with respect to variability relationships among their features. Having a feature model along with a reference model developed in the domain engineering lifecycle, a concrete product of the family is derived by selecting features in the feature model (referred to as the configuration process) and by instantiating the reference model. However, feature model configuration can be a cumbersome task because: 1) feature models may consist of a large number of features, which are hard to comprehend and maintain; and 2) many factors including technical limitations, implementation costs, stakeholders’ requirements and expectations must be considered in the configuration process. Recognizing these issues, a significant amount of research efforts has been dedicated to different aspects of feature model configuration such as automating the configuration process. Several approaches have been proposed to alleviate the feature model configuration challenges through applying visualization and interaction techniques. However, there have been limited empirical insights available into the impact of visualization and interaction techniques on the feature model configuration process. In this paper, we present a set of visualization and interaction interventions for representing and configuring feature models, which are then empirically validated to measure the impact of the proposed interventions. An empirical study was conducted by following the principles of control experiments in software engineering and by applying the well-known software quality standard ISO 9126 to operationalize the variables investigated in the experiment. The results of the empirical study revealed that the employed visualization and interaction interventions significantly improved completion time of comprehension and changing of the feature model configuration. Additionally, according to results, the proposed interventions are easy-to-use and easy-to-learn for the participants.     

Software product lines: a case study

A software product line is a family of products that share common features to meet the needs of a market area. Systematic processes have been developed to dramatically reduce the cost of a product line. Such product‐line engineering processes have proven practical and effective in industrial use, but are not widely understood. The Family‐Oriented Abstraction, Specification and Translation (FAST) process has been used successfully at Lucent Technologies in over 25 domains, providing productivity improvements of as much as four to one. In this paper, we show how to use FAST to document precisely the key abstractions in a domain, exploit design patterns in a generic product‐line architecture, generate documentation and Java code, and automate testing to reduce costs. The paper is based on a detailed case study covering all aspects from domain analysis through testing. Copyright © 2000 John Wiley & Sons, Ltd.     

产品线成本模型的比较与分析

为给软件产品线决策者应用软件产品线模型提供理论上的参考,综合分析比较了近年来的20种软件产品线模型,在对软件产品线模型的投资循环、重用方式、货币时间价值、经济函数、成本因子和重用成本等方面因素进行细致分析的基础上提出了软件产品线模型的比较框架,在该框架内着重分析了其中5种典型的软件产品线模型,对应用软件产品线开发方式的成本估算和投资分析做了细致分析,并对当前软件产品线模型时存在的问题和发展方向进行了探讨.     

Automating Test Case Selection in Model-Based Software Product Line Development

We address the problem of how to select test cases for products in a controlled model-based software product line development process. CVL, the common variability language, gives a framework for materialisation of product models from a given base model, variability model and resolution model. From such product models, software products can be derived. In practise, test case development for the product line often is independent from the product development. Therefore, the problem arises which test cases can be applied to which products. In particular, the question is whether a test case for one speci c product can be also used for a "similar" product. In this paper, we show how the expected outcome of a test case to a product in a model-based software product line development can be determined. That is, we give a procedure for assigning the outcome of a given test case on an arbitrary member of a software product line. We recall the relevant de nitions for software product line engineering, describe our approach, and demonstrate it with the example of a product line of super-automatic espresso machines.     

Effective product-line testing using similarity-based product prioritization

Software and Systems Modeling - A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible...     

A novel model-based testing approach for software product lines

Model-based testing relies on a model of the system under test. FineFit is a framework for model-based testing of Java programs. In the FineFit approach, the model is expressed by a set of tables based on Parnas tables. A software product line is a family of programs (the products) with well-defined commonalities and variabilities that are developed by (re)using common artifacts. In this paper, we address the issue of using the FineFit approach to support the development of correct software product lines. We specify a software product line as a specification product line where each product is a FineFit specification of the corresponding software product. The main challenge is to concisely specify the software product line while retaining the readability of the specification of a single system. To address this, we used delta-oriented programming, a recently proposed flexible approach for implementing software product lines, and developed: (1) delta tables as a means to apply the delta-oriented programming idea to the specification of software product lines; and (2) DeltaFineFit as a novel model-based testing approach for software product lines.     

A holistic approach to feature modeling for product line requirements engineering

Requirements engineering (RE) offers the means to discover, model, and manage the requirements of the products that comprise a product line, while software product line engineering (SPLE) offers the means of realizing the products’ requirements from a common base of software assets. In practice, however, RE and SPLE have proven to be less complementary than they should. While some RE techniques, particularly goal modeling, support the exploration of alternative solutions, the appropriate solution is typically conditional on context and a large product line may have many product-defining contexts. Thus, scalability and traceability through into product line features are key challenges for RE. Feature modeling, by contrast, has been widely accepted as a way of modeling commonality and variability of products of a product line that may be very complex. In this paper, we propose a goal-driven feature modeling approach that separates a feature space in terms of problem space and solution space features, and establish explicit mappings between them. This approach contributes to reducing the inherent complexity of a mixed-view feature model, deriving key engineering drivers for developing core assets of a product line, and facilitating the quality-based product configuration.     

Model-based pairwise testing for feature interaction coverage in software product line engineering

Testing software product lines (SPLs) is very challenging due to a high degree of variability leading to an enormous number of possible products. The vast majority of today??s testing approaches for SPLs validate products individually using different kinds of reuse techniques for testing. Because of their reusability and adaptability capabilities, model-based approaches are suitable to describe variability and are therefore frequently used for implementation and testing purposes of SPLs. Due to the enormous number of possible products, individual product testing becomes more and more infeasible. Pairwise testing offers one possibility to test a subset of all possible products. However, according to the best of our knowledge, there is no contribution discussing and rating this approach in the SPL context. In this contribution, we provide a mapping between feature models describing the common and variable parts of an SPL and a reusable test model in the form of statecharts. Thereby, we interrelate feature model-based coverage criteria and test model-based coverage criteria such as control and data flow coverage and are therefore able to discuss the potentials and limitations of pairwise testing. We pay particular attention to test requirements for feature interactions constituting a major challenge in SPL engineering. We give a concise definition of feature dependencies and feature interactions from a testing point of view, and we discuss adequacy criteria for SPL coverage under pairwise feature interaction testing and give a generalization to the T-wise case. The concept and implementation of our approach are evaluated by means of a case study from the automotive domain.     

Scalable prediction of non-functional properties in software product lines: Footprint and memory consumption

ContextA software product line is a family of related software products, typically created from a set of common assets. Users select features to derive a product that fulfills their needs. Users often expect a product to have specific non-functional properties, such as a small footprint or a bounded response time. Because a product line may have an exponential number of products with respect to its features, it is usually not feasible to generate and measure non-functional properties for each possible product.ObjectiveOur overall goal is to derive optimal products with respect to non-functional requirements by showing customers which features must be selected.MethodWe propose an approach to predict a product’s non-functional properties based on the product’s feature selection. We aggregate the influence of each selected feature on a non-functional property to predict a product’s properties. We generate and measure a small set of products and, by comparing measurements, we approximate each feature’s influence on the non-functional property in question. As a research method, we conducted controlled experiments and evaluated prediction accuracy for the non-functional properties footprint and main-memory consumption. But, in principle, our approach is applicable for all quantifiable non-functional properties.ResultsWith nine software product lines, we demonstrate that our approach predicts the footprint with an average accuracy of 94%, and an accuracy of over 99% on average if feature interactions are known. In a further series of experiments, we predicted main memory consumption of six customizable programs and achieved an accuracy of 89% on average.ConclusionOur experiments suggest that, with only few measurements, it is possible to accurately predict non-functional properties of products of a product line. Furthermore, we show how already little domain knowledge can improve predictions and discuss trade-offs between accuracy and required number of measurements. With this technique, we provide a basis for many reasoning and product-derivation approaches.     

Reducing the verification cost of evolving product families using static analysis techniques

Software product line engineering enables proactive reuse among a set of related products through explicit modeling of commonalities and differences among them. Software product lines are intended to be used in a long period of time. As a result, they evolve over time, due to the changes in the requirements. Having several individual products in a software family, verification of the entire family may take a considerable effort. In this paper we aim to decrease this cost by reducing the number of verified products using static analysis techniques. Furthermore, to reduce model checking costs after product line evolution, we restrict the number of products that should be re-verified by reusing the previous verification result. All proposed techniques are based on static analysis of the product family model with respect to the property and can be automated. To show the effectiveness of these techniques we apply them on a set of case studies and present the results.     

Toward automated feature model configuration with optimizing non-functional requirements

ContextA software product line is a family of software systems that share some common features but also have significant variabilities. A feature model is a variability modeling artifact, which represents differences among software products with respect to the variability relationships among their features. Having a feature model along with a reference model developed in the domain engineering lifecycle, a concrete product of the family is derived by binding the variation points in the feature model (called configuration process) and by instantiating the reference model.ObjectiveIn this work we address the feature model configuration problem and propose a framework to automatically select suitable features that satisfy both the functional and non-functional preferences and constraints of stakeholders. Additionally, interdependencies between various non-functional properties are taken into account in the framework.MethodThe proposed framework combines Analytical Hierarchy Process (AHP) and Fuzzy Cognitive Maps (FCM) to compute the non-functional properties weights based on stakeholders’ preferences and interdependencies between non-functional properties. Afterwards, Hierarchical Task Network (HTN) planning is applied to find the optimal feature model configuration.ResultOur approach improves state-of-art of feature model configuration by considering positive or negative impacts of the features on non-functional properties, the stakeholders’ preferences, and non-functional interdependencies. The approach presented in this paper extends earlier work presented in [1] from several distinct perspectives including mechanisms handling interdependencies between non-functional properties, proposing a novel tooling architecture, and offering visualization and interaction techniques for representing functional and non-functional aspects of feature models.Conclusionour experiments show the scalability of our configuration approach when considering both functional and non-functional requirements of stakeholders.     

两阶段过程模型下的产品线复用策略优化

考虑了软件产品线工程中各软件模块复用策略的选择问题. 在建立软件产品线开发两阶段过程模型的基础上, 归纳出6种典型的复用策略及其实现方式, 提出考虑开发成本、工时、故障率的情况下进行复用策略的选择优化模型, 解决在满足可获得预算、开发周期、系统可靠性需求的约束下产品线质量最优问题. 并基于贝叶斯理论构建易测试函数, 为产品线开发中领域测试强度的估算提供了方法. 最后, 以邮箱服务系统为实例说明该模型的有效 性.