Testing variability-intensive systems using automated analysis: an application to Android

Software product lines are used to develop a set of software products that, while being different, share a common set of features. Feature models are used as a compact representation of all the products (e.g., possible configurations) of the product line. The number of products that a feature model encodes may grow exponentially with the number of features. This increases the cost of testing the products within a product line. Some proposals deal with this problem by reducing the testing space using different techniques. However, a daunting challenge is to explore how the cost and value of test cases can be modeled and optimized in order to have lower-cost testing processes. In this paper, we present TESting vAriAbiLity Intensive Systems (TESALIA), an approach that uses automated analysis of feature models to optimize the testing of variability-intensive systems. We model test value and cost as feature attributes, and then we use a constraint satisfaction solver to prune, prioritize and package product line tests complementing prior work in the software product line testing literature. A prototype implementation of TESALIA is used for validation in an Android example showing the benefits of maximizing the mobile market share (the value function) while meeting a budgetary constraint.     

Quality-aware analysis in product line engineering with the orthogonal variability model

Software product line engineering is about producing a set of similar products in a certain domain. A variability model documents the variability amongst products in a product line. The specification of variability can be extended with quality information, such as measurable quality attributes (e.g., CPU and memory consumption) and constraints on these attributes (e.g., memory consumption should be in a range of values). However, the wrong use of constraints may cause anomalies in the specification which must be detected (e.g., the model could represent no products). Furthermore, based on such quality information, it is possible to carry out quality-aware analyses, i.e., the product line engineer may want to verify whether it is possible to build a product that satisfies a desired quality. The challenge for quality-aware specification and analysis is threefold. First, there should be a way to specify quality information in variability models. Second, it should be possible to detect anomalies in the variability specification associated with quality information. Third, there should be mechanisms to verify the variability model to extract useful information, such as the possibility to build a product that fulfils certain quality conditions (e.g., is there any product that requires less than 512?MB of memory?). In this article, we present an approach for quality-aware analysis in software product lines using the orthogonal variability model (OVM) to represent variability. We propose to map variability represented in the OVM associated with quality information to a constraint satisfaction problem and to use an off-the-shelf constraint programming solver to automatically perform the verification task. To illustrate our approach, we use a product line in the automotive domain which is an example that was created in a national project by a leading car company. We have developed a prototype tool named FaMa-OVM, which works as a proof of concepts. We were able to identify void models, dead and false optional elements, and check whether the product line example satisfies quality conditions.     

Extending the reflexion method for consolidating software variants into product lines

Software variants emerge from ad-hoc copying in-the-large with adaptations to a specific context. As the number of variants increases, maintaining such software variants becomes more and more difficult and expensive. In contrast to such ad-hoc reuse, software product lines offer organized ways of reuse, taking advantage of similarities of different products. To re-gain control, software variants may be consolidated as organized software product lines. In this paper, we describe a method and supporting tools to compare software variants at the architectural level extending the reflexion method to software variants. Murphy’s reflexion method allows one to reconstruct the module view, a static architectural view describing the static components, their interfaces and dependencies and their grouping as layers and subsystems. The method consists of the specification of the module view and the mapping of implementation components onto the module view. An automatic analysis determines differences between the module view and its implementation. We extend the reflexion method from single systems to software variants. Because software variants share a very large amount of code, we can expect components of one variant to re-occur in the other variant either identically or at least similarly. We use similarity metrics to determine this type of correspondence among implementation components between two variants. Because the similarity metrics are expensive to compute, we use clone detection techniques to reduce the number of candidate pairs of implementation components for which the similarity is computed. The correspondence is then used to transfer as much as possible of the mapping for the analyzed variants to the next variant to be analyzed. Furthermore, we describe how to unify the individual product architectures into a software product line architecture.     

Automated analysis of feature models 20 years later: A literature review

Software product line engineering is about producing a set of related products that share more commonalities than variabilities. Feature models are widely used for variability and commonality management in software product lines. Feature models are information models where a set of products are represented as a set of features in a single model. The automated analysis of feature models deals with the computer-aided extraction of information from feature models. The literature on this topic has contributed with a set of operations, techniques, tools and empirical results which have not been surveyed until now. This paper provides a comprehensive literature review on the automated analysis of feature models 20 years after of their invention. This paper contributes by bringing together previously disparate streams of work to help shed light on this thriving area. We also present a conceptual framework to understand the different proposals as well as categorise future contributions. We finally discuss the different studies and propose some challenges to be faced in the future.     

Requirements engineering in software product line engineering

Many attempts have been made to increase the productivity and quality of software products based on software reuse. Software product line practice is one such approach, one that focuses on developing a family of products which have a majority of features in common. Hence, there are numerous requirements that are common across the family, but others are unique to individual products. Traditional requirements engineering methods were conceived to deal with single product requirements and are usually not flexible enough to address the needs arising from reusing requirements for a family of products. There is also the additional burden of correctly identifying and engineering both product-line-wide requirements and product-specific requirements as well as evolving them. Therefore, in this special issue, we want to highlight the importance and the role of requirements engineering for product line development as well as to provide insights into the state of the art in the field.     

Effective product-line testing using similarity-based product prioritization

Software and Systems Modeling - A software product line comprises a family of software products that share a common set of features. Testing an entire product-line product-by-product is infeasible...     

Feature Nets: behavioural modelling of software product lines

Software product lines (SPLs) are diverse systems that are developed using a dual engineering process: (a) family engineering defines the commonality and variability among all members of the SPL, and (b) application engineering derives specific products based on the common foundation combined with a variable selection of features. The number of derivable products in an SPL can thus be exponential in the number of features. This inherent complexity poses two main challenges when it comes to modelling: firstly, the formalism used for modelling SPLs needs to be modular and scalable. Secondly, it should ensure that all products behave correctly by providing the ability to analyse and verify complex models efficiently. In this paper, we propose to integrate an established modelling formalism (Petri nets) with the domain of software product line engineering. To this end, we extend Petri nets to Feature Nets. While Petri nets provide a framework for formally modelling and verifying single software systems, Feature Nets offer the same sort of benefits for software product lines. We show how SPLs can be modelled in an incremental, modular fashion using Feature Nets, provide a Feature Nets variant that supports modelling dynamic SPLs, and propose an analysis method for SPL modelled as Feature Nets. By facilitating the construction of a single model that includes the various behaviours exhibited by the products in an SPL, we make a significant step towards efficient and practical quality assurance methods for software product lines.     

A family of standards for software and systems product lines

There is a drastic rise and evolution of Software and Systems Product Lines (SSPL) as a potential solution to deliver quality software products in a shorter time and with less effort. This has led to a plethora of diversified tools across the entire life cycle of SSPL, making it difficult for product line organizations to select the right set of tools for product line realization and tool vendors to develop interoperable tools. In this paper, we motivate the need for standardization of tools and methods for SSPL. The ISO/IEC JTC1/SC7 (Software and Systems Engineering) of the International Organization for Standardization (ISO) has proposed a set of interrelated standards in the area of SSPL to cater to the emerging demands of software and systems industry. We present the foundation of SSPL family of standards that are driven by market needs and discuss the core patterns and processes of SSPL. We then present an overview of ISO/IEC 26550, which is the reference model for SSPL family of standards and provide examples from current and ongoing standards. We highlight a few examples of industry adoption of SSPL and standards and emphasize the need for further research for standards in SSPL. We see this paper as a first step to bring standards knowledge into the academic world in the field of SSPL.     

Evolution support mechanisms for software product line process

Software product family process evolution needs specific support for incremental change. Product line process evolution involves in addition to identifying new requirements the building of a meta-process describing the migration from the old process to the new one. This paper presents basic mechanisms to support software product line process evolution. These mechanisms share four strategies - change identification, change impact, change propagation, and change validation. It also examines three kinds of evolution processes - architecture, product line, and product. In addition, change management mechanisms are identified. Specifically we propose support mechanisms for static local entity evolution and complex entity evolution including transient evolution process. An evolution model prototype based on dependency relationships structure of the various product line artifacts is developed.     

Modular product platform configuration and co-planning of assembly lines using assembly and disassembly

Formation of products platforms is carried out during the planning stage and very often separately from the planning of corresponding assembly lines. There is a dearth of literature which considers the different aspects of fully integrating platform design, product family formation, assembly line design, delayed product differentiation, and new concepts of mass customization. A Modular Product Platform Configuration model which uses assembly and disassembly for configuring product variants and Co-Planning of products platforms (MPCC) and their assembly Lines is presented. It is used to co-plan the common platform components and the associated product families simultaneously with the planning of its corresponding mixed-model assembly line. Using both assembly and disassembly to customize the product family platform in order to generate product variants is not commonly discussed in literature. It is defined as the formation of platforms for use to derive multiple products by including many components not shared by every product. The platform is then customized by assembling or disassembling components to form different product variants. The model is formulated using mixed integer mathematical programming to minimize the number of assembly stations and cycle time. Two case studies are used for verification and demonstration. They illustrated the ability of the MPCC model to integrate the planning of product platform, product families and the number of assembly stations required to assemble and disassemble components from mass-assembled product platforms to derive new product variants.     

Software performance tuning of software product family architectures: Two case studies in the real-time embedded systems domain

Software performance is an important non-functional quality attribute and software performance evaluation is an essential activity in the software development process. Especially in embedded real-time systems, software design and evaluation are driven by the needs to optimize the limited resources, to respect time deadlines and, at the same time, to produce the best experience for end-users. Software product family architectures add additional requirements to the evaluation process. In this case, the evaluation includes the analysis of the optimizations and tradeoffs for the whole products in the family. Performance evaluation of software product family architectures requires knowledge and a clear understanding of different domains: software architecture assessments, software performance and software product family architecture. We have used a scenario-driven approach to evaluate performance and dynamic memory management efficiency in one Nokia software product family architecture. In this paper we present two case studies. Furthermore, we discuss the implications and tradeoffs of software performance against evolvability and maintenability in software product family architectures.     

Efficient software product-line model checking using induction and a SAT solver

Software product line (SPL) engineering is increasingly being adopted in safety-critical systems. It is highly desirable to rigorously show that these systems are designed correctly. However, formal analysis for SPLs is more difficult than for single systems because an SPL may contain a large number of individual systems. In this paper, we propose an efficient model-checking technique for SPLs using induction and a SAT (Boolean satisfiability problem) solver. We show how an induction-based verification method can be adapted to the SPLs, with the help of a SAT solver. To combat the state space explosion problem, a novel technique that exploits the distinguishing characteristics of SPLs, called feature cube enlargement, is proposed to reduce the verification efforts. The incremental SAT mechanism is applied to further improve the efficiency. The correctness of our technique is proved. Experimental results show dramatic improvement of our technique over the existing binary decision diagram (BDD)-based techniques.     

Towards Testing Product Line Architectures

A product line architecture is a single specification capturing the overall architecture of a series of closely related products. Its structure consists of a set of mandatory elements and a set of variation points. Whereas mandatory elements are part of the architecture of every product in the product line architecture, variation points precisely define the dimensions along which the architectures of individual products differ from each other.The increased use of product line architectures in today's software development projects poses several challenges for existing testing techniques. In this paper we discuss those challenges and discuss what we believe are opportunities for addressing them.     

The effects of visualization and interaction techniques on feature model configuration

A Software Product Line is a set of software systems of a domain, which share some common features but also have significant variability. A feature model is a variability modeling artifact which represents differences among software products with respect to variability relationships among their features. Having a feature model along with a reference model developed in the domain engineering lifecycle, a concrete product of the family is derived by selecting features in the feature model (referred to as the configuration process) and by instantiating the reference model. However, feature model configuration can be a cumbersome task because: 1) feature models may consist of a large number of features, which are hard to comprehend and maintain; and 2) many factors including technical limitations, implementation costs, stakeholders’ requirements and expectations must be considered in the configuration process. Recognizing these issues, a significant amount of research efforts has been dedicated to different aspects of feature model configuration such as automating the configuration process. Several approaches have been proposed to alleviate the feature model configuration challenges through applying visualization and interaction techniques. However, there have been limited empirical insights available into the impact of visualization and interaction techniques on the feature model configuration process. In this paper, we present a set of visualization and interaction interventions for representing and configuring feature models, which are then empirically validated to measure the impact of the proposed interventions. An empirical study was conducted by following the principles of control experiments in software engineering and by applying the well-known software quality standard ISO 9126 to operationalize the variables investigated in the experiment. The results of the empirical study revealed that the employed visualization and interaction interventions significantly improved completion time of comprehension and changing of the feature model configuration. Additionally, according to results, the proposed interventions are easy-to-use and easy-to-learn for the participants.     

Statistical prioritization for software product line testing: an experience report

Software product lines (SPLs) are families of software systems sharing common assets and exhibiting variabilities specific to each product member of the family. Commonalities and variabilities are often represented as features organized in a feature model. Due to combinatorial explosion of the number of products induced by possible features combinations, exhaustive testing of SPLs is intractable. Therefore, sampling and prioritization techniques have been proposed to generate sorted lists of products based on coverage criteria or weights assigned to features. Solely based on the feature model, these techniques do not take into account behavioural usage of such products as a source of prioritization. In this paper, we assess the feasibility of integrating usage models into the testing process to derive statistical testing approaches for SPLs. Usage models are given as Markov chains, enabling prioritization of probable/rare behaviours. We used featured transition systems, compactly modelling variability and behaviour for SPLs, to determine which products are realizing prioritized behaviours. Statistical prioritization can achieve a significant reduction in the state space, and modelling efforts can be rewarded by better automation. In particular, we used MaTeLo, a statistical test cases generation suite developed at ALL4TEC. We assess feasibility criteria on two systems: Claroline, a configurable course management system, and Sferion?, an embedded system providing helicopter landing assistance.     

A set of complexity metrics for guiding the software test process

Software verification and validation is a domain which is covered by many dynamic test, static analysis, and formal verification techniques. This presents a problem to practitioners with respect to selecting those suitable techniques which can be used successfully. The basic idea of the methodology presented here is to select test techniques which fit the software under test. A dynamic test technique requires that certain program elements are covered, will be sensitive to errors associated with these elements, because executing an error location is a precondition for revealing the error. Furthermore, it is likely that the probability of errors increases with complexity. Complexity can be characterized in terms of several properties which can be used to suggest various testing strategies. The complexity of the various software properties can be measured using appropriate complexity metrics. Properties with unusual high complexity measures should be tested very throughly. The approach described in this paper permits the selection of test techniques based on the values of the metrics with respect to a particular software product.     

办公自动化软件产品线的研究与设计

软件产品线是一组具有可管理的公共特性的产品集合。选取网络办公自动化（OA）系统作为特定的研究领域，在分析此领域中产品的共性后，给出了OA系统软件产品线的实现方法。具体描述了OA系统产品线的需求工程，OA系统产品线核心资产的提取，并以科技项目管理系统为例说明了如何将核心资产应用到具体的产品开发中。     

Project management in a software product line organization

In traditional software engineering project management, managers provide focused guidance to a team responsible for producing a specific result in a specified amount of time. Today, however, organizations are increasingly taking a product line approach to software to exploit product commonalities. Software product line organizations have unique practices and project definitions. These unconventional features offer new challenges and directions for traditional project management. How does the traditional concept of a project - a temporary endeavor aimed at creating a unique product or service - hold up under this new paradigm? In this article, we discuss this question, along with how the idea of a "project" and project management techniques must expand to fit a product line context. In particular, we show how the "overall guidelines, policies, and procedures" that Thayer and Pyster spoke of some years ago remain crucially important in product line organizations today.     

An architecture process maturity model of software product line engineering

Software architecture has been a key research area in the software engineering community due to its significant role in creating high-quality software. The trend of developing product lines rather than single products has made the software product line a viable option in the industry. Software product line architecture (SPLA) is regarded as one of the crucial components in the product lines, since all of the resulting products share this common architecture. The increased popularity of software product lines demands a process maturity evaluation methodology. Consequently, this paper presents an architecture process maturity model for software product line engineering to evaluate the current maturity of the product line architecture development process in an organization. Assessment questionnaires and a rating methodology comprise the framework of this model. The objective of the questionnaires is to collect information about the SPLA development process. Thus, in general this work contributes towards the establishment of a comprehensive and unified strategy for the process maturity evaluation of software product line engineering. Furthermore, we conducted two case studies and reported the assessment results, which show the maturity of the architecture development process in two organizations.     

从领域需求到产品线体系结构的映射——一种面向特征的方法

领域需求之间的依赖关系对软件产品线的体系结构有很大的影响,在已有的面向特征的管理产品线需求依赖的方法中很少有研究从需求到产品线体系结构的映射.基于一种特征依赖的分类方法,提出了从领域需求到特征,以及从特征到产品线体系结构的映射规则.通过这些映射规则,一致的需求通过映射得到一致的产品线核心资产,从而减少产品线中核心资产的不一致性并增加产品线的复用程度.用金融领域的现货交易产品线作为实例说明这个方法的实用性.