A systematic mapping study of software product lines testing

Context

In software development, Testing is an important mechanism both to identify defects and assure that completed products work as specified. This is a common practice in single-system development, and continues to hold in Software Product Lines (SPL). Even though extensive research has been done in the SPL Testing field, it is necessary to assess the current state of research and practice, in order to provide practitioners with evidence that enable fostering its further development.

Objective

This paper focuses on Testing in SPL and has the following goals: investigate state-of-the-art testing practices, synthesize available evidence, and identify gaps between required techniques and existing approaches, available in the literature.

Method

A systematic mapping study was conducted with a set of nine research questions, in which 120 studies, dated from 1993 to 2009, were evaluated.

Results

Although several aspects regarding testing have been covered by single-system development approaches, many cannot be directly applied in the SPL context due to specific issues. In addition, particular aspects regarding SPL are not covered by the existing SPL approaches, and when the aspects are covered, the literature just gives brief overviews. This scenario indicates that additional investigation, empirical and practical, should be performed.

Conclusion

The results can help to understand the needs in SPL Testing, by identifying points that still require additional investigation, since important aspects regarding particular points of software product lines have not been addressed yet.     

Investigations about replication of empirical studies in software engineering: A systematic mapping study

ContextTwo recent mapping studies which were intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994 and 2012) and a second group of studies that are concerned with definitions, classifications, processes, guidelines, and other research topics or themes about replication work in empirical software engineering research (published between 1996 and 2012).ObjectiveIn this current article, our goal is to analyze and discuss the contents of the second set of studies about replications to increase our understanding of the current state of the work on replication in empirical software engineering research.MethodWe applied the systematic literature review method to build a systematic mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996–2012 complemented by manual and automatic search procedures that collected articles published in 2013.ResultsWe analyzed 37 papers reporting studies about replication published in the last 17 years. These papers explore different topics related to concepts and classifications, presented guidelines, and discuss theoretical issues that are relevant for our understanding of replication in our field. We also investigated how these 37 papers have been cited in the 135 replication papers published between 1994 and 2012.ConclusionsReplication in SE still lacks a set of standardized concepts and terminology, which has a negative impact on the replication work in our field. To improve this situation, it is important that the SE research community engage on an effort to create and evaluate taxonomy, frameworks, guidelines, and methodologies to fully support the development of replications.     

Model-based testing using UML activity diagrams: A systematic mapping study

Context:The Unified Modeling Language (UML) has become the de facto standard for software modeling. UML models are often used to visualize, understand, and communicate the structure and behavior of a system. UML activity diagrams (ADs) are often used to elaborate and visualize individual use cases. Due to their higher level of abstraction and process-oriented perspective, UML ADs are also highly suitable for model-based test generation. In the last two decades, different researchers have used UML ADs for test generation. Despite the growing use of UML ADs for model-based testing, there are currently no comprehensive and unbiased studies on the topic.Objective:To present a comprehensive and unbiased overview of the state-of-the-art on model-based testing using UML ADs.Method:We review and structure the current body of knowledge on model-based testing using UML ADs by performing a systematic mapping study using well-known guidelines. We pose nine research questions, outline our selection criteria, and develop a classification scheme.Results:The results comprise 41 primary studies analyzed against nine research questions. We also highlight the current trends and research gaps in model-based testing using UML ADs and discuss some shortcomings for researchers and practitioners working in this area. The results show that the existing approaches on model-based testing using UML ADs tend to rely on intermediate formats and formalisms for model verification and test generation, employ a multitude of graph-based coverage criteria, and use graph search algorithms.Conclusion:We present a comprehensive overview of the existing approaches on model-based testing using UML ADs. We conclude that (1) UML ADs are not being used for non-functional testing, (2) only a few approaches have been validated against realistic, industrial case studies, (3) most approaches target very restricted application domains, and (4) there is currently a clear lack of holistic approaches for model-based testing using UML ADs.     

Forty years of research on personality in software engineering: A mapping study

In this article, we present a systematic mapping study of research on personality in software engineering. The goal is to plot the landscape of current published empirical and theoretical studies that deal with the role of personality in software engineering. We applied the systematic review method to search and select published articles, and to extract and synthesize data from the selected articles that reported studies about personality. Our search retrieved more than 19,000 articles, from which we selected 90 articles published between 1970 and 2010. Nearly 72% of the studies were published after 2002 and 83% of the studies reported empirical research findings. Data extracted from the 90 studies showed that education and pair programming were the most recurring research topics, and that MBTI was the most used test. Research related to pair programming, education, team effectiveness, software process allocation, software engineer personality characteristics, and individual performance concentrated over 88% of the studies, while team process, behavior and preferences, and leadership performance were the topics with the smallest number of studies. We conclude that the number of articles has grown in the last few years, but contradictory evidence was found that might have been caused by differences in context, research method, and versions of the tests used in the studies. While this raises a warning for practitioners that wish to use personality tests in practice, it shows several opportunities for the research community to improve and extend findings in this field.     

Testing in Service Oriented Architectures with dynamic binding: A mapping study

Context

Service Oriented Architectures (SOA) have emerged as a new paradigm to develop interoperable and highly dynamic applications.

Objective

This paper aims to identify the state of the art in the research on testing in Service Oriented Architectures with dynamic binding.

Method

A mapping study has been performed employing both manual and automatic search in journals, conference/workshop proceedings and electronic databases.

Results

A total of 33 studies have been reviewed in order to extract relevant information regarding a previously defined set of research questions. The detection of faults and the decision making based on the information gathered from the tests have been identified as the main objectives of these studies. To achieve these goals, monitoring and test case generation are the most proposed techniques testing both functional and non-functional properties. Furthermore, different stakeholders have been identified as participants in the tests, which are performed in specific points in time during the life cycle of the services. Finally, it has been observed that a relevant group of studies have not validated their approach yet.

Conclusions

Although we have only found 33 studies that address the testing of SOA where the discovery and binding of the services are performed at runtime, this number can be considered significant due to the specific nature of the reviewed topic. The results of this study have contributed to provide a body of knowledge that allows identifying current gaps in improving the quality of the dynamic binding in SOA using testing approaches.     

Investigating software testing and maintenance reports: Case study

ContextAlthough many papers have been published on software development and defect prediction techniques, problem reports in real projects quite often differ from those described in the literature. Hence, there is still a need for deeper exploration of case studies from industry.ObjectiveThe aim of this study is to present the impact of fine-grained problem reports on improving evaluation of testing and maintenance processes. It is targeted at projects involving several releases and complex schemes of problem handling. This is based on our experience gained while monitoring several commercial projects.MethodExtracting certain features from detailed problem reports, we derive various measures and present analysis models which characterize and visualize the effectiveness of testing and problem resolution processes. The considered reports describe types of problems (e.g. defects), their locations in project versions and software modules, ways of their resolution, etc. The performed analysis is related to eleven projects developed in the same company. This study is an exploratory research with some explanatory features. Moreover, having identified some drawbacks, we present extensions of problem reports and their analysis which have been verified in another industrial case study project.ResultsFine-grained (accurate) problem handling reports provide a wider scope of possible measures to assess the relevant development processes. This is helpful in controlling single projects (local perspective) as well as in managing these processes in the whole company (global perspective).ConclusionDetailed problem handling reports extend the space and quality of statistical analysis, they provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.     

Analogy-based software development effort estimation: A systematic mapping and review

ContextAnalogy-based Software development Effort Estimation (ASEE) techniques have gained considerable attention from the software engineering community. However, existing systematic map and review studies on software development effort prediction have not investigated in depth several issues of ASEE techniques, to the exception of comparisons with other types of estimation techniques.ObjectiveThe objective of this research is twofold: (1) to classify ASEE studies which primary goal is to propose new or modified ASEE techniques according to five criteria: research approach, contribution type, techniques used in combination with ASEE methods, and ASEE steps, as well as identifying publication channels and trends and (2) to analyze these studies from five perspectives: estimation accuracy, accuracy comparison, estimation context, impact of the techniques used in combination with ASEE methods, and ASEE tools.MethodWe performed a systematic mapping of studies for which the primary goal is to develop or to improve ASEE techniques published in the period 1990–2012, and reviewed them based on an automated search of four electronic databases.ResultsIn total, we identified 65 studies published between 1990 and 2012, and classified them based on our predefined classification criteria. The mapping study revealed that most researchers focus on addressing problems related to the first step of an ASEE process, that is, feature and case subset selection. The results of our detailed analysis show that ASEE methods outperform the eight techniques with which they were compared, and tend to yield acceptable results especially when combining ASEE techniques with Fuzzy Logic (FL) or Genetic Algorithms (GA).ConclusionBased on the findings of this study, the use of other techniques such FL and GA in combination with an ASEE method is promising to generate more accurate estimates. However, the use of ASEE techniques by practitioners is still limited: developing more ASEE tools may facilitate the application of these techniques and then lead to increasing the use of ASEE techniques in industry.     

一种新的软件测试方法的研究

提出基于蒙特卡罗的软件可靠性测试方法。它是软件工程中的一个新的研究领域,主要探讨蒙特卡罗方法和软件工程的交叉。一方面借鉴一些传统的软件测试方法,另一方面借助于蒙特卡罗方法,提出的一套全新的测试技术。     

Thematic mapping on microcomputers: the hardware and software environments

This paper considers the mapping tasks that can be accomplished using microcomputers and the power and limitations of the present generation of machines. The range of peripheral devices that need to be driven by a microcomputer in mapping applications is discussed. It is argued that in a rapidly changing environment, with hardware regularly offering more power for less money, portable machine-independent software needs to be developed for mapping applications. This needs to take full advantage of the powerful interactive capabilities of microcomputers to provide both skilled and naive users with opportunities for interacting with maps, both at the design and the end-user stages, and in new forms. Such interactive facilities hitherto have been too expensive or too difficult to implement on mainframe computers.     

Virtual Battery: A testing tool for power-aware software

Virtualization is an inexpensive and convenient method for setting up software test environments. Thus it is being widely used as a test tool for software products requiring high reliability such as mission critical cyber-physical systems. However, existing virtualization platforms do not fully virtualize the battery subsystem. Therefore, it is difficult to test battery-related features of guest systems. In this paper, we propose Virtual Battery, a battery virtualization scheme for type II full virtualization platforms. Virtual Battery takes the form of an ACPI-compatible battery device driver dedicated to each virtual machine, which virtualizes a target system. Through Virtual Battery, developers can easily manipulate the charging and battery status of each virtual machine (VM), regardless of the existence or current status of the host system’s battery. In addition, Virtual Battery emulates the behavior of batteries by discharging the virtual batteries according to the resource usages of their VMs. This feature enables VMs to act as battery resource containers. Three case studies demonstrate the effectiveness of the proposed scheme.     

An experimental study of adaptive testing for software reliability assessment

Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previous work has shown that adaptive testing can be formulated and guided in theory to minimize the variance of an unbiased software reliability estimator and to achieve optimal software reliability assessment. In this paper, we present an experimental study of adaptive testing for software reliability assessment, where the adaptive testing strategy, the random testing strategy and the operational profile based testing strategy were applied to the Space program in four experiments. The experimental results demonstrate that the adaptive testing strategy can really work in practice and may noticeably outperform the other two. Therefore, the adaptive testing strategy can serve as a preferable alternative to the random testing strategy and the operational profile based testing strategy if high confidence in the reliability estimates is required or the real-world operational profile of the software under test cannot be accurately identified.     

基于本体的软件测试知识管理模型研究

本文针对知识管理在软件测试领域的具体应用，提出了一个基于本体的软件测试知识管理模型，其关注的主要对象是软件测试过程中产生的具有高知识密集性的创造性文档。该模型的主要思想是：将软件测试知识分为个人知识和组织知识，并以本体索引的自由文本为软件测试知识的方式得以表示和组织，然后根据SWEBOK软件测试领域本体得到分类保存，最终目的是协助软件测试知识以“知识螺旋”的方式得以共享和重用。     

面向软件测试过程的知识管理方法的研究与实现

在软件测试组织中,对测试过程中的知识进行有效的管理,是提升组织整体测试水平的关键.一个重要的研究问题是怎样将知识管理过程与软件测试过程有效集成,从而促进知识资产在软件测试组织中的传播与重用.给出了软件测试领域知识管理子系统的体系结构、主要功能以及使用流程,对一些关键技术进行了探讨.最后,给出了知识地图模块的设计与实现,验证了本文所实现的软件测试领域知识管理系统的合理性和有效性.     

On the reliability of mapping studies in software engineering

Background

Systematic literature reviews and systematic mapping studies are becoming increasingly common in software engineering, and hence it becomes even more important to better understand the reliability of such studies.

Objective

This paper presents a study of two systematic mapping studies to evaluate the reliability of mapping studies and point out some challenges related to this type of study in software engineering.

Method

The research is based on an in-depth case study of two published mapping studies on software product line testing.

Results

We found that despite the fact that the two studies are addressing the same topic, there are quite a number of differences when it comes to papers included and in terms of classification of the papers included in the two mapping studies.

Conclusions

From this we conclude that although mapping studies are important, their reliability cannot simply be taken for granted. Based on the findings we also provide four conjectures that further research has to address to make secondary studies (systematic mapping studies and systematic literature reviews) even more valuable to both researchers and practitioners.     

一种软件可靠性测试模型研究

在马尔可夫链模型的基础上,将测试问题转化为一个数学问题。通过建立软件的使用链,根据使用链进行序列抽样,产生测试用例,将软件测试结果的分析问题转化为一个经典概率问题。运用实例证明,这种技术具有一定的实用性和有效性。     

Software product line testing - A systematic mapping study

Context

Software product lines (SPL) are used in industry to achieve more efficient software development. However, the testing side of SPL is underdeveloped.

Objective

This study aims at surveying existing research on SPL testing in order to identify useful approaches and needs for future research.

Method

A systematic mapping study is launched to find as much literature as possible, and the 64 papers found are classified with respect to focus, research type and contribution type.

Results

A majority of the papers are of proposal research types (64%). System testing is the largest group with respect to research focus (40%), followed by management (23%). Method contributions are in majority.

Conclusions

More validation and evaluation research is needed to provide a better foundation for SPL testing.     

参数依赖型软件参数更动测试策略和方法

参数依赖型软件是指初始化时读取并解析配置参数,并据此进行任务处理的软件,航天测控软件是典型的参数依赖型软件;航天测控软件具有明显的领域软件特征,多采用领域工程分析技术,实现业务处理逻辑和具体任务参数的分离,达到仅通过修改任务配置参数而适应高强度型号任务的目的;通过对参数依赖型软件架构、应用模式的分析,提出一种对参数依赖特性进行验收测试、参数更动测试的流程、策略和方法;并基于该方法,对远程数据交互软件进行了参数依赖特性测试,测试结果表明,该方法具有测试覆盖性强、测试重点突出、测试效率高的特点。     

一种软件可靠性测试方法研究

为了缩短软件测试周期,本文把马尔可夫链模型运用于软件可靠性测试中,提出了这一技术进行软件可靠性测试的方法。在测试过程中使用了新的评判准则分析测试结果,通过实例证明了该评判准则的实用性和有效性。     

Software evolution visualization: A systematic mapping study

BackgroundSoftware evolution is an important topic in software engineering. It generally deals with large amounts of data, as one must look at whole project histories as opposed to their current snapshot. Software visualization is the field of software engineering that aims to help people to understand software through the use of visual resources. It can be effectively used to analyze and understand the large amount of data produced during software evolution.ObjectiveThis study investigates Software Evolution Visualization (SEV) approaches, collecting evidence about how SEV research is structured, synthesizing current evidence on the goals of the proposed approaches and identifying key challenges for its use in practice.MethodsA mapping study was conducted to analyze how the SEV area is structured. Selected primary studies were classified and analyzed with respect to nine research questions.ResultsSEV has been used for many different purposes, especially for change comprehension, change prediction and contribution analysis. The analysis identified gaps in the studies with respect to their goals, strategies and approaches. It also pointed out to a widespread lack of empirical studies in the area.ConclusionResearchers have proposed many SEV approaches during the past years, but some have failed to clearly state their goals, tie them back to concrete problems, or formally validate their usefulness. The identified gaps indicate that there still are many opportunities to be explored in the area.