Testing variability-intensive systems using automated analysis: an application to Android

Software product lines are used to develop a set of software products that, while being different, share a common set of features. Feature models are used as a compact representation of all the products (e.g., possible configurations) of the product line. The number of products that a feature model encodes may grow exponentially with the number of features. This increases the cost of testing the products within a product line. Some proposals deal with this problem by reducing the testing space using different techniques. However, a daunting challenge is to explore how the cost and value of test cases can be modeled and optimized in order to have lower-cost testing processes. In this paper, we present TESting vAriAbiLity Intensive Systems (TESALIA), an approach that uses automated analysis of feature models to optimize the testing of variability-intensive systems. We model test value and cost as feature attributes, and then we use a constraint satisfaction solver to prune, prioritize and package product line tests complementing prior work in the software product line testing literature. A prototype implementation of TESALIA is used for validation in an Android example showing the benefits of maximizing the mobile market share (the value function) while meeting a budgetary constraint.     

From extended feature models to constraint logic programming

Since feature models for realistic product families may be quite complicated, the automated analysis of feature models is desirable. Although several approaches reported in the literature address this issue, complex cross-tree relationships involving attributes in extended feature models have not been handled. In this article, we introduce a mapping from extended feature models to constraint logic programming over finite domains. This mapping is used to translate into constraint logic programs; basic, cardinality-based and extended feature models, which can include complex cross-tree relationships involving attributes. This translation enables the use of off-the-shelf constraint solvers for the automated analysis of extended feature models involving such complex relationships. We also present the performance results of some well-known analysis operations on an example translated model.     

Automated metamorphic testing on the analyses of feature models

Context

A feature model (FM) represents the valid combinations of features in a domain. The automated extraction of information from FMs is a complex task that involves numerous analysis operations, techniques and tools. Current testing methods in this context are manual and rely on the ability of the tester to decide whether the output of an analysis is correct. However, this is acknowledged to be time-consuming, error-prone and in most cases infeasible due to the combinatorial complexity of the analyses, this is known as the oracle problem.

Objective

In this paper, we propose using metamorphic testing to automate the generation of test data for feature model analysis tools overcoming the oracle problem. An automated test data generator is presented and evaluated to show the feasibility of our approach.

Method

We present a set of relations (so-called metamorphic relations) between input FMs and the set of products they represent. Based on these relations and given a FM and its known set of products, a set of neighbouring FMs together with their corresponding set of products are automatically generated and used for testing multiple analyses. Complex FMs representing millions of products can be efficiently created by applying this process iteratively.

Results

Our evaluation results using mutation testing and real faults reveal that most faults can be automatically detected within a few seconds. Two defects were found in FaMa and another two in SPLOT, two real tools for the automated analysis of feature models. Also, we show how our generator outperforms a related manual suite for the automated analysis of feature models and how this suite can be used to guide the automated generation of test cases obtaining important gains in efficiency.

Conclusion

Our results show that the application of metamorphic testing in the domain of automated analysis of feature models is efficient and effective in detecting most faults in a few seconds without the need for a human oracle.     

Reasoning about product-line evolution using complex feature model differences

Features define common and variable parts of the members of a (software) product line. Feature models are used to specify the set of all valid feature combinations. Feature models not only enjoy an intuitive tree-like graphical syntax, but also a precise formal semantics, which can be denoted as propositional formulae over Boolean feature variables. A product line usually constitutes a long-term investment and, therefore, has to undergo continuous evolution to meet ever-changing requirements. First of all, product-line evolution leads to changes of the feature model due to its central role in the product-line paradigm. As a result, product-line engineers are often faced with the problems that (1) feature models are changed in an ad-hoc manner without proper documentation, and (2) the semantic impact of feature diagram changes is unclear. In this article, we propose a comprehensive approach to tackle both challenges. For (1), our approach compares the old and new version of the diagram representation of a feature model and specifies the changes using complex edit operations on feature diagrams. In this way, feature model changes are automatically detected and formally documented. For (2), we propose an approach for reasoning about the semantic impact of diagram changes. We present a set of edit operations on feature diagrams, where complex operations are primarily derived from evolution scenarios observed in a real-world case study, i.e., a product line from the automation engineering domain. We evaluated our approach to demonstrate its applicability with respect to the case study, as well as its scalability concerning experimental data sets.     

机械产品的特征信息处理

本文讨论了单纯采用CSG模型和单纯采用B-Rep模型来表达机械产品的特征信息时所遇到的一些实际困难,根据机械产品设计与制造一体化的特殊要求,提出了基于CSG+B-Rep数据结构的特征信息表达方案。其主要特点是以面为特征定义的基本单位,利用FSMTS中CSG与B-ReP的一致对应关系,将产品的特征信息PIT链入CSG和B-Rep组成的循环链中,其优点在于既可通过CSG摸型建立参数化的产品模型,又可通过交互方式定义和修改产品的特征信息。     

Feature Nets: behavioural modelling of software product lines

Software product lines (SPLs) are diverse systems that are developed using a dual engineering process: (a) family engineering defines the commonality and variability among all members of the SPL, and (b) application engineering derives specific products based on the common foundation combined with a variable selection of features. The number of derivable products in an SPL can thus be exponential in the number of features. This inherent complexity poses two main challenges when it comes to modelling: firstly, the formalism used for modelling SPLs needs to be modular and scalable. Secondly, it should ensure that all products behave correctly by providing the ability to analyse and verify complex models efficiently. In this paper, we propose to integrate an established modelling formalism (Petri nets) with the domain of software product line engineering. To this end, we extend Petri nets to Feature Nets. While Petri nets provide a framework for formally modelling and verifying single software systems, Feature Nets offer the same sort of benefits for software product lines. We show how SPLs can be modelled in an incremental, modular fashion using Feature Nets, provide a Feature Nets variant that supports modelling dynamic SPLs, and propose an analysis method for SPL modelled as Feature Nets. By facilitating the construction of a single model that includes the various behaviours exhibited by the products in an SPL, we make a significant step towards efficient and practical quality assurance methods for software product lines.     

A systematic test case selection methodology for product lines: results and insights from an industrial case study

In the context of product lines, test case selection aims at obtaining a set of relevant test cases for a product from the entire set of test cases available for a product line. While working on a research-based innovation project on automated testing of product lines of Video Conferencing Systems (VCSs) developed by Cisco, we felt the need to devise a cost-effective way of selecting relevant test cases for a product. To fulfill such need, we propose a systematic and automated test selection methodology using: 1) Feature Model for Testing (FM_T) to capture commonalities and variabilities of a product line; 2) Component Family Model for Testing (CFM_T) to model the structure of test case repository; 3) A tool to automatically build restrictions from CFM_T to FM_T and traces from CFM_T to the actual test cases. Using our methodology, a test engineer is only required to select relevant features through FM_T at a higher level of abstraction for a product and the corresponding test cases will be obtained automatically. We evaluate our methodology by applying it to a VCS product line called Saturn with seven commercial products and the results show that our methodology can significantly reduce cost measured as test selection time and at the same time achieves higher effectiveness (feature coverage, feature pairwise coverage and fault detection) as compared with the current manual process. Moreover, we conduct a questionnaire-based study to solicit the views of test engineers who are involved in developing FM_T and CFM_T. The results show that test engineers are positive about adapting our methodology in their current practice. Finally, we present a set of lessons learnt while applying product line engineering at Cisco for test case selection.     

Automated generation of computationally hard feature models using evolutionary algorithms

A feature model is a compact representation of the products of a software product line. The automated extraction of information from feature models is a thriving topic involving numerous analysis operations, techniques and tools. Performance evaluations in this domain mainly rely on the use of random feature models. However, these only provide a rough idea of the behaviour of the tools with average problems and are not sufficient to reveal their real strengths and weaknesses. In this article, we propose to model the problem of finding computationally hard feature models as an optimization problem and we solve it using a novel evolutionary algorithm for optimized feature models (ETHOM). Given a tool and an analysis operation, ETHOM generates input models of a predefined size maximizing aspects such as the execution time or the memory consumption of the tool when performing the operation over the model. This allows users and developers to know the performance of tools in pessimistic cases providing a better idea of their real power and revealing performance bugs. Experiments using ETHOM on a number of analyses and tools have successfully identified models producing much longer executions times and higher memory consumption than those obtained with random models of identical or even larger size.     

Verifying feature models using OWL

Feature models are widely used in domain engineering to capture common and variant features among systems in a particular domain. However, the lack of a formal semantics and reasoning support of feature models has hindered the development of this area. Industrial experiences also show that methods and tools that can support feature model analysis are badly appreciated. Such reasoning tool should be fully automated and efficient. At the same time, the reasoning tool should scale up well since it may need to handle hundreds or even thousands of features a that modern software systems may have. This paper presents an approach to modeling and verifying feature diagrams using Semantic Web OWL ontologies. We use OWL DL ontologies to precisely capture the inter-relationships among the features in a feature diagram. OWL reasoning engines such as FaCT++ are deployed to check for the inconsistencies of feature configurations fully automatically. Furthermore, a general OWL debugger has been developed to tackle the disadvantage of lacking debugging aids for the current OWL reasoner and to complement our verification approach. We also developed a CASE tool to facilitate visual development, interchange and reasoning of feature diagrams in the Semantic Web environment.     

A formal framework for software product lines

ContextA Software Product Line is a set of software systems that are built from a common set of features. These systems are developed in a prescribed way and they can be adapted to fit the needs of customers. Feature models specify the properties of the systems that are meaningful to customers. A semantics that models the feature level has the potential to support the automatic analysis of entire software product lines.ObjectiveThe objective of this paper is to define a formal framework for Software Product Lines. This framework needs to be general enough to provide a formal semantics for existing frameworks like FODA (Feature Oriented Domain Analysis), but also to be easily adaptable to new problems.MethodWe define an algebraic language, called SPLA, to describe Software Product Lines. We provide the semantics for the algebra in three different ways. The approach followed to give the semantics is inspired by the semantics of process algebras. First we define an operational semantics, next a denotational semantics, and finally an axiomatic semantics. We also have defined a representation of the algebra into propositional logic.ResultsWe prove that the three semantics are equivalent. We also show how FODA diagrams can be automatically translated into SPLA. Furthermore, we have developed our tool, called AT, that implements the formal framework presented in this paper. This tool uses a SAT-solver to check the satisfiability of an SPL.ConclusionThis paper defines a general formal framework for software product lines. We have defined three different semantics that are equivalent; this means that depending on the context we can choose the most convenient approach: operational, denotational or axiomatic. The framework is flexible enough because it is closely related to process algebras. Process algebras are a well-known paradigm for which many extensions have been defined.     

语义特征造型中特征修改优先准则的研究

特征造型技术是新一代CAD系统的关键技术,是产品模型技术的核心。该文讨论了以细胞元模型为基础的语义特征造型技术。在语义特征造型中用特征依赖图（FDG）的数据模型来保存和维护设计中的各种特征信息及其之间的关系,在此基础上分析了依据特征依赖图中特征间的关系动态修改特征优先级的方法,确保了模型修改的有效性。最后给出了一个实例。     

Constructing local information feature for spatial image steganalysis

Feature is a key part for steganalysis. In this paper we propose a spatial feature set for image steganalysis, named Local Information Feature (LIF), to increase the diversity of spatial steganalysis feature and improve its performance. It also provide a heuristic framework for designing steganalysis feature through 3 steps. It first collects local information from its local region consisting of adjacent pixels. Then according to certain rules, it maps each pixel to its corresponding local type by its local information. Finally, the feature set is formed by adaptive weighted statistical histograms of local types. We design two schemes for LIF, each of which can generate different feature sets using different methods of local information computing. Experimental results show that our feature is effective for detecting stego images embedded by adaptive steganography. We also discussed some possible method to extent the feature designing based on LIF.     

基于多视图的特征模型定制(英文)

特征模型是捕获特定领域可复用软件需求的一种重要技术手段。对特征模型的复用通常采用定制的方式,通过定制使其适应领域内不同系统的特定需求。特征模型定制的一个难点在于定制过程往往涉及众多的利益相关者,而这些利益相关者的关注点各有不同。由于特征模型是针对整个领域的需求而组织的,特定利益相关者的关注点可能散布在整个特征模型中且可能与其他利益相关者的关注点相互交织,这为特定利益相关者的定制工作带来一定的困难。为此,文章提出一种基于多视图的特征模型定制方法,针对不同的利益相关者将其所关注的散布在整个特征模型中的信息聚类到不同的视图中,并提供一种约束一致性维护机制,方便相应的利益相关者的定制活动。     

基于特征切片的软件产品线模型检测

特征模型是一种描述软件产品线中共性和可变性特征的通用形式。特征模型象征着所有可能的应用程序配置空间,是实现个性化产品定制的基础。随着软件产品线的规模和复杂程度的增加,如何有效支持以用户需求为基础以及根据特定需求和利益相关者的目标进行个性定制开发是亟待解决的实际问题。提出一种根据用户需求对特征模型进行切片,进一步结合三值逻辑对行为模型进行抽象,最后利用模型检测技术对软件产品线进行验证的方法。实验结果证实了该方法的有效性。     

Multi-purpose,multi-level feature modeling of large-scale industrial software systems

Feature models are frequently used to capture the knowledge about configurable software systems and product lines. However, feature modeling of large-scale systems is challenging as models are needed for diverse purposes. For instance, feature models can be used to reflect the perspectives of product management, technical solution architecture, or product configuration. Furthermore, models are required at different levels of granularity. Although numerous approaches and tools are available, it remains hard to define the purpose, scope, and granularity of feature models. This paper first reports results and experiences of an exploratory case study on developing feature models for two large-scale industrial automation software systems. We report results on the characteristics and modularity of the feature models, including metrics about model dependencies. Based on the findings from the study, we developed FORCE, a modeling language, and tool environment that extends an existing feature modeling approach to support models for different purposes and at multiple levels, including mappings to the code base. We demonstrate the expressiveness and extensibility of our approach by applying it to the well-known Pick and Place Unit example and an injection molding subsystem of an industrial product line. We further show how our approach supports consistency between different feature models. Our results and experiences show that considering the purpose and level of features is useful for modeling large-scale systems and that modeling dependencies between feature models is essential for developing a system-wide perspective.     

Interactive Inconsistency Fixing in Feature Modeling

Feature models have been widely adopted to reuse the requirements of a set of similar products in a domain. In feature models’ construction, one basic task is to ensure the consistency of feature models, which often involves detecting and fixing of inconsistencies in feature models. While many approaches have been proposed, most of them focus on detecting inconsistencies rather than fixing inconsistencies. In this paper, we propose a novel dynamic-priority based approach to interactively fixing inconsistencies in feature models, and report an implementation of a system that not only automatically recommends a solution to fixing inconsistencies but also supports domain analysts to gradually reach the desirable solution by dynamically adjusting priorities of constraints. The key technical contribution is, as far as we are aware, the first application of the constraint hierarchy theory to feature modeling, where the degree of domain analysts’ confidence on constraints is expressed by using priority and inconsistencies are resolved by deleting one or more lower-priority constraints. Two case studies demonstrate the usability and scalability (efficiency) of our new approach.     

面向对象影像信息提取软件Feature Analyst和eCognition的分析与比较

介绍了目前最为先进的两种面向对象的影像信息提取软件Feature Analyst和eCognition。通过对两种软件设计思路、工作流程和软件特殊性的对比分析,探讨了影像信息自动提取的发展趋势。     

Feature-based ontological framework for semantic interoperability in product development

An essential requirement in integrating tasks in product development is to have a seamless exchange of product information through the entire product lifecycle. A key challenge in the integration is the exchange of shape semantics in terms of understandable labels and representations. A unified taxonomy is proposed to represent, classify, and extract shape features. This taxonomy is built using the Domain-Independent Form Feature (DIFF) model as the representation of features. All the shape features in a product model are classified under three main classes, namely, volumetric features, deformation features and free-form surface features. Shape feature ontology is developed using the unified taxonomy, which brings the shape features under a single reasoning framework. One-to-many reasoning framework is presented for mapping semantically equivalent information (label and representation) of the feature to be exchanged to target applications, and the reconstruction of the shape model automatically in that target application. An algorithm has been developed to extract the semantics of shape features and construct the model in the target application. The algorithm developed has been tested for shape models taken from literature and test cases are selected based on variations of topology and geometry. Results of exchanging product information are presented and discussed. Finally, the limitations of the proposed method for exchanging product information are explained.     

Heuristic and exact algorithms for product configuration in software product lines

Software product line (SPL) is a set of software applications that share a common set of features satisfying the specific needs of a particular market segment. SPL engineering is a paradigm to develop software applications that commonly use a feature model to capture and document common and variable features, and their relationships. A big challenge is to derive one product among all possible products in the SPL, which satisfies the business and customer requirements. This task is known as product configuration. Although product configuration has been extensively investigated in the literature, customer's preferences are frequently neglected. In this paper, we propose a novel approach to configure a product that considers both qualitative and quantitative feature properties. We model the product configuration task as a combinatorial optimization problem, and heuristic and exact algorithms are proposed. As far as we are concerned, this proposal is the first work in the literature that considers feature properties in both leaf and nonleaf features. Computational experiments showed that the best of our heuristics found optimal solutions for all instances where those are known. For the instances where optimal solutions are not known, our heuristic outperformed the best solution obtained by a one‐hour run of the exact algorithm by up to 67.89%.