Testing scientific software: A systematic literature review

ContextScientific software plays an important role in critical decision making, for example making weather predictions based on climate models, and computation of evidence for research publications. Recently, scientists have had to retract publications due to errors caused by software faults. Systematic testing can identify such faults in code.ObjectiveThis study aims to identify specific challenges, proposed solutions, and unsolved problems faced when testing scientific software.MethodWe conducted a systematic literature survey to identify and analyze relevant literature. We identified 62 studies that provided relevant information about testing scientific software.ResultsWe found that challenges faced when testing scientific software fall into two main categories: (1) testing challenges that occur due to characteristics of scientific software such as oracle problems and (2) testing challenges that occur due to cultural differences between scientists and the software engineering community such as viewing the code and the model that it implements as inseparable entities. In addition, we identified methods to potentially overcome these challenges and their limitations. Finally we describe unsolved challenges and how software engineering researchers and practitioners can help to overcome them.ConclusionsScientific software presents special challenges for testing. Specifically, cultural differences between scientist developers and software engineers, along with the characteristics of the scientific software make testing more difficult. Existing techniques such as code clone detection can help to improve the testing process. Software engineers should consider special challenges posed by scientific software such as oracle problems when developing testing techniques.     

Software architecture awareness in long-term software product evolution

Software architecture has been established in software engineering for almost 40 years. When developing and evolving software products, architecture is expected to be even more relevant compared to contract development. However, the research results seem not to have influenced the development practice around software products very much. The architecture often only exists implicitly in discussions that accompany the development. Nonetheless many of the software products have been used for over 10, or even 20 years. How do development teams manage to accommodate changing needs and at the same time maintain the quality of the product? In order to answer this question, grounded theory study based on 15 semi-structured interviews was conducted in order to find out about the wide spectrum of architecture practices in software product developing organisations. Our results indicate that a chief architect or central developer acts as a ‘walking architecture’ devising changes and discussing local designs while at the same time updating his own knowledge about problematic aspects that need to be addressed. Architecture documentation and representations might not be used, especially if they replace the feedback from on-going developments into the ‘architecturing’ practices. Referring to results from Computer Supported Cooperative Work, we discuss how explicating the existing architecture needs to be complemented by social protocols to support the communication and knowledge sharing processes of the ‘walking architecture’.     

Controlling software architecture erosion: A survey

Software architectures capture the most significant properties and design constraints of software systems. Thus, modifications to a system that violate its architectural principles can degrade system performance and shorten its useful lifetime. As the potential frequency and scale of software adaptations increase to meet rapidly changing requirements and business conditions, controlling such architecture erosion becomes an important concern for software architects and developers. This paper presents a survey of techniques and technologies that have been proposed over the years either to prevent architecture erosion or to detect and restore architectures that have been eroded. These approaches, which include tools, techniques and processes, are primarily classified into three generic categories that attempt to minimise, prevent and repair architecture erosion. Within these broad categories, each approach is further broken down reflecting the high-level strategies adopted to tackle erosion. These are: process-oriented architecture conformance, architecture evolution management, architecture design enforcement, architecture to implementation linkage, self-adaptation and architecture restoration techniques consisting of recovery, discovery and reconciliation. Some of these strategies contain sub-categories under which survey results are presented.We discuss the merits and weaknesses of each strategy and argue that no single strategy can address the problem of erosion. Further, we explore the possibility of combining strategies and present a case for further work in developing a holistic framework for controlling architecture erosion.     

Using CMMI together with agile software development: A systematic review

BackgroundThe search for adherence to maturity levels by using lightweight processes that require low levels of effort is regarded as a challenge for software development organizations.ObjectiveThis study seeks to evaluate, synthesize, and present results on the use of the Capability Maturity Model Integration (CMMI) in combination with agile software development, and thereafter to give an overview of the topics researched, which includes a discussion of their benefits and limitations, the strength of the findings, and the implications for research and practice.MethodsThe method applied was a Systematic Literature Review on studies published up to (and including) 2011.ResultsThe search strategy identified 3193 results, of which 81 included studies on the use of CMMI together with agile methodologies. The benefits found were grouped into two main categories: those related to the organization in general and those related to the development process, and were organized into subcategories, according to the area to which they refer. The limitations were also grouped into these categories. Using the criteria defined, the strength of the evidence found was considered low. The implications of the results for research and practice are discussed.ConclusionAgile methodologies can be used by companies to reduce efforts in getting to levels 2 and 3 of CMMI, there even being reports of applying agile practices that led to achieving level 5. However, agile methodologies alone, according to the studies, were not sufficient to obtain a rating at a given level, it being necessary to resort to additional practices to do so.     

Software process modeling languages: A systematic literature review

ContextOrganizations working in software development are aware that processes are very important assets as well as they are very conscious of the need to deploy well-defined processes with the goal of improving software product development and, particularly, quality. Software process modeling languages are an important support for describing and managing software processes in software-intensive organizations.ObjectiveThis paper seeks to identify what software process modeling languages have been defined in last decade, the relationships and dependencies among them and, starting from the current state, to define directions for future research.MethodA systematic literature review was developed. 1929 papers were retrieved by a manual search in 9 databases and 46 primary studies were finally included.ResultsSince 2000 more than 40 languages have been first reported, each of which with a concrete purpose. We show that different base technologies have been used to define software process modeling languages. We provide a scheme where each language is registered together with the year it was created, the base technology used to define it and whether it is considered a starting point for later languages. This scheme is used to illustrate the trend in software process modeling languages. Finally, we present directions for future research.ConclusionThis review presents the different software process modeling languages that have been developed in the last ten years, showing the relevant fact that model-based SPMLs (Software Process Modeling Languages) are being considered as a current trend. Each one of these languages has been designed with a particular motivation, to solve problems which had been detected. However, there are still several problems to face, which have become evident in this review. This let us provide researchers with some guidelines for future research on this topic.     

Combining service-orientation and software product line engineering: A systematic mapping study

ContextService-Orientation (SO) is a rapidly emerging paradigm for the design and development of adaptive and dynamic software systems. Software Product Line Engineering (SPLE) has also gained attention as a promising and successful software reuse development paradigm over the last decade and proven to provide effective solutions to deal with managing the growing complexity of software systems.ObjectiveThis study aims at characterizing and identifying the existing research on employing and leveraging SO and SPLE.MethodWe conducted a systematic mapping study to identify and analyze related literature. We identified 81 primary studies, dated from 2000–2011 and classified them with respect to research focus, types of research and contribution.ResultThe mapping synthesizes the available evidence about combining the synergy points and integration of SO and SPLE. The analysis shows that the majority of studies focus on service variability modeling and adaptive systems by employing SPLE principles and approaches.In particular, SPLE approaches, especially feature-oriented approaches for variability modeling, have been applied to the design and development of service-oriented systems. While SO is employed in software product line contexts for the realization of product lines to reconcile the flexibility, scalability and dynamism in product derivations thereby creating dynamic software product lines.ConclusionOur study summarizes and characterizes the SO and SPLE topics researchers have investigated over the past decade and identifies promising research directions as due to the synergy generated by integrating methods and techniques from these two areas.     

Analysis of architecture evaluation data

The output of 18 software architecture evaluations is analyzed. The goal of the analysis is to find patterns in the important quality attributes and risk themes identified in the evaluations. The major results are

•

A categorization of risk themes.

•

The observation that twice as many risk themes are risks of “omission” as are risks of “commission”.

•

A failure to find a relationship between the business and mission goals of a system and the risk themes from an evaluation of that system.

•

A failure to find a correlation between the domain of a system being evaluated and the important quality attributes for that system.

•

A wide diversity of names used for various quality attributes.

The results of this investigation have application to practitioners by suggesting activities on which developers should put greater focus. They also have application to researchers by suggesting further areas of investigation.     

Software architecture review by association

During the process of software design, software architects have their reasons to choose certain software components to address particular software requirements and constraints. However, existing software architecture review techniques often rely on the design reviewers’ knowledge and experience, and perhaps using some checklists, to identify design gaps and issues, without questioning the reasoning behind the decisions made by the architects. In this paper, we approach design reviews from a design reasoning perspective. We propose to use an association-based review procedure to identify design issues by first associating all the relevant design concerns, problems and solutions systematically; and then verifying if the causal relationships between these design elements are valid. Using this procedure, we discovered new design issues in all three industrial cases, despite their internal architecture reviews and one of the three systems being operational. With the newly found design issues, we derive eight general design reasoning failure scenarios.     

Using software evolution to focus architectural recovery

Ideally, a software project commences with requirements gathering and specification, reaches its major milestone with system implementation and delivery, and then continues, possibly indefinitely, into an operation and maintenance phase. The software system's architecture is in many ways the linchpin of this process: it is supposed to be an effective reification of the system's technical requirements and to be faithfully reflected in the system's implementation. Furthermore, the architecture is meant to guide system evolution, while also being updated in the process. However, in reality developers frequently deviate from the architecture, causing architectural erosion, a phenomenon in which the initial, “as documented' architecture of an application is (arbitrarily) modified to the point where its key properties no longer hold. Architectural recovery is a process frequently used to cope with architectural erosion whereby the current, “as implemented” architecture of a software system is extracted from the system's implementation. In this paper we propose a light-weight approach to architectural recovery, called Focus, which has three unique facets. First, Focus uses a system's evolution requirements to isolate and incrementally recover only the fragment of the system's architecture affected by the evolution. In this manner, Focus allows engineers to direct their primary attention to the part of the system that is immediately impacted by the desired change; subsequent changes will incrementally uncover additional parts of the system's architecture. Secondly, in addition to software components, which are the usual target of existing recovery approaches, Focus also recovers the key architectural notions of software connector and architectural style. Finally, Focus does not only recover a system's architecture, but may in fact rearchitect the system. We have applied and evaluated Focus in the context of several off-the-shelf applications and architectural styles to date. We discuss its key strengths and point out several open issues that will frame our future work.     

Scientific research ontology to support systematic review in software engineering

The term systematic review is used to refer to a specific methodology of research, developed in order to gather and evaluate the available evidence pertaining to a focused topic. It represents a secondary study that depends on primary study results to be accomplished. Several primary studies have been conducted in the field of Software Engineering in the last years, determining an increasing improvement in methodology. However, in most cases software is built with technologies and processes for which developers have insufficient evidence to confirm their suitability, limits, qualities, costs, and inherent risks. Conducting systematic reviews in Software Engineering consists in a major methodological tool to scientifically improve the validity of assertions that can be made in the field and, as a consequence, the reliability degree of the methods that are employed for developing software technologies and supporting software processes. This paper aims at discussing the significance of experimental studies, particularly systematic reviews, and their use in supporting software processes. A template designed to support systematic reviews in Software Engineering is presented, and the development of ontologies to describe knowledge regarding such experimental studies is also introduced.     

Barriers in the selection of offshore software development outsourcing vendors: An exploratory study using a systematic literature review

Context

Software development outsourcing is a contract-based relationship between client and vendor organisations in which a client contracts out all or part of its software development activities to a vendor, who provides agreed services for remuneration.

Objective

The objective is to identify various barriers that have a negative impact on software outsourcing clients in the selection process of offshore software development outsourcing vendors.

Method

We have performed a systematic literature review (SLR) process for the identification of barriers. We have performed all the SLR steps such as the protocol development, initial selection, final selection, quality assessment, data extraction and data synthesis.

Results

We have identified barriers such as ‘language and cultural barriers’, ‘country instability’, ‘lack of project management’, ‘lack of protection for intellectual property rights’ and ‘lack of technical capability’ that generally have a negative impact on outsourcing clients. We have identified only one common frequently cited barrier in three types of organisations (i.e. small, medium and large) which is ‘language and cultural barriers’. We did not identify any common frequently cited barrier in three continents (Asia, North America and Europe) and in two decades (1990-1999 and 2000-mid 2008). The results also reveal the similarities and differences in the barriers identified through different study strategies.

Conclusions

Vendors should address frequently cited barriers such as ‘language and cultural barriers’, ‘country instability’, ‘lack of project management’, ‘lack of protection for intellectual property rights’ and ‘lack of technical capability’ in order to compete in the offshore outsourcing business.     

Process models for service-based applications: A systematic literature review

Context

Service-Oriented Computing (SOC) is a promising computing paradigm which facilitates the development of adaptive and loosely coupled service-based applications (SBAs). Many of the technical challenges pertaining to the development of SBAs have been addressed, however, there are still outstanding questions relating to the processes required to develop them.

Objective

The objective of this study is to systematically identify process models for developing service-based applications (SBAs) and review the processes within them. This will provide a useful starting point for any further research in the area. A secondary objective of the study is to identify process models which facilitate the adaptation of SBAs.

Method

In order to achieve this objective a systematic literature review (SLR) of the existing software engineering literature is conducted.

Results

During this research 722 studies were identified using a predefined search strategy, this number was narrowed down to 57 studies based on a set of strict inclusion and exclusion criteria. The results are reported both quantitatively in the form of a mapping study, as well as qualitatively in the form of a narrative summary of the key processes identified.

Conclusion

There are many process models reported for the development of SBAs varying in detail and maturity, this review has identified and categorised the processes within those process models. The review has also identified and evaluated process models which facilitate the adaptation of SBAs.     

A classification and comparison of model checking software architecture techniques

Software architecture specifications are used for many different purposes, such as documenting architectural decisions, predicting architectural qualities before the system is implemented, and guiding the design and coding process. In these contexts, assessing the architectural model as early as possible becomes a relevant challenge. Various analysis techniques have been proposed for testing, model checking, and evaluating performance based on architectural models. Among them, model checking is an exhaustive and automatic verification technique, used to verify whether an architectural specification conforms to expected properties. While model checking is being extensively applied to software architectures, little work has been done to comprehensively enumerate and classify these different techniques.The goal of this paper is to investigate the state-of-the-art in model checking software architectures. For this purpose, we first define the main activities in a model checking software architecture process. Then, we define a classification and comparison framework and compare model checking software architecture techniques according to it.     

A systematic review of evaluation of variability management approaches in software product lines

Context

Variability management (VM) is one of the most important activities of software product-line engineering (SPLE), which intends to develop software-intensive systems using platforms and mass customization. VM encompasses the activities of eliciting and representing variability in software artefacts, establishing and managing dependencies among different variabilities, and supporting the exploitation of the variabilities for building and evolving a family of software systems. Software product line (SPL) community has allocated huge amount of effort to develop various approaches to dealing with variability related challenges during the last two decade. Several dozens of VM approaches have been reported. However, there has been no systematic effort to study how the reported VM approaches have been evaluated.

Objective

The objectives of this research are to review the status of evaluation of reported VM approaches and to synthesize the available evidence about the effects of the reported approaches.

Method

We carried out a systematic literature review of the VM approaches in SPLE reported from 1990s until December 2007.

Results

We selected 97 papers according to our inclusion and exclusion criteria. The selected papers appeared in 56 publication venues. We found that only a small number of the reviewed approaches had been evaluated using rigorous scientific methods. A detailed investigation of the reviewed studies employing empirical research methods revealed significant quality deficiencies in various aspects of the used quality assessment criteria. The synthesis of the available evidence showed that all studies, except one, reported only positive effects.

Conclusion

The findings from this systematic review show that a large majority of the reported VM approaches have not been sufficiently evaluated using scientifically rigorous methods. The available evidence is sparse and the quality of the presented evidence is quite low. The findings highlight the areas in need of improvement, i.e., rigorous evaluation of VM approaches. However, the reported evidence is quite consistent across different studies. That means the proposed approaches may be very beneficial when they are applied properly in appropriate situations. Hence, it can be concluded that further investigations need to pay more attention to the contexts under which different approaches can be more beneficial.     

Survey of reliability and availability prediction methods from the viewpoint of software architecture

Many future software systems will be distributed across a network, extensively providing different kinds of services for their users. These systems must be highly reliable and provide services when required. Reliability and availability must be engineered into software from the onset of its development, and potential problems must be detected in the early stages, when it is easier and less expensive to implement modifications. The software architecture design phase is the first stage of software development in which it is possible to evaluate how well the quality requirements are being met. For this reason, a method is needed for analyzing software architecture with respect to reliability and availability. In this paper, we define a framework for comparing reliability and availability analysis methods from the viewpoint of software architecture. Our contribution is the comparison of the existing analysis methods and techniques that can be used for reliability and availability prediction at the architectural level. The objective is to discover which methods are suitable for the reliability and availability prediction of today’s complex systems, what are the shortcomings of the methods, and which research activities need to be conducted in order to overcome these identified shortcomings. The comparison reveals that none of the existing methods entirely fulfill the requirements that are defined in the framework. The comparison framework also defines the characteristics required of new reliability and availability analysis methods. Additionally, the framework is a valuable tool for selecting the best suitable method for architecture analysis. Furthermore, the framework can be extended and used for other evaluation methods as well.     

Systematic literature review of machine learning based software development effort estimation models

Context

Software development effort estimation (SDEE) is the process of predicting the effort required to develop a software system. In order to improve estimation accuracy, many researchers have proposed machine learning (ML) based SDEE models (ML models) since 1990s. However, there has been no attempt to analyze the empirical evidence on ML models in a systematic way.

Objective

This research aims to systematically analyze ML models from four aspects: type of ML technique, estimation accuracy, model comparison, and estimation context.

Method

We performed a systematic literature review of empirical studies on ML model published in the last two decades (1991-2010).

Results

We have identified 84 primary studies relevant to the objective of this research. After investigating these studies, we found that eight types of ML techniques have been employed in SDEE models. Overall speaking, the estimation accuracy of these ML models is close to the acceptable level and is better than that of non-ML models. Furthermore, different ML models have different strengths and weaknesses and thus favor different estimation contexts.

Conclusion

ML models are promising in the field of SDEE. However, the application of ML models in industry is still limited, so that more effort and incentives are needed to facilitate the application of ML models. To this end, based on the findings of this review, we provide recommendations for researchers as well as guidelines for practitioners.     

Supporting runtime software architecture: A bidirectional-transformation-based approach

Runtime software architectures (RSA) are architecture-level, dynamic representations of running software systems, which help monitor and adapt the systems at a high abstraction level. The key issue to support RSA is to maintain the causal connection between the architecture and the system, ensuring that the architecture represents the current system, and the modifications on the architecture cause proper system changes. The main challenge here is the abstraction gap between the architecture and the system. In this paper, we investigate the synchronization mechanism between architecture configurations and system states for maintaining the causal connections. We identify four required properties for such synchronization, and provide a generic solution satisfying these properties. Specifically, we utilize bidirectional transformation to bridge the abstraction gap between architecture and system, and design an algorithm based on it, which addresses issues such as conflicts between architecture and system changes, and exceptions of system manipulations. We provide a generative tool-set that helps developers implement this approach on a wide class of systems. We have successfully applied our approach on JOnAS JEE system to support it with C2-styled runtime software architecture, as well as some other cases between practical systems and typical architecture models.     

Potential and limitations of the ISBSG dataset in enhancing software engineering research: A mapping review

ContextThe International Software Benchmarking Standards Group (ISBSG) maintains a software development repository with over 6000 software projects. This dataset makes it possible to estimate a project’s size, effort, duration, and cost.ObjectiveThe aim of this study was to determine how and to what extent, ISBSG has been used by researchers from 2000, when the first papers were published, until June of 2012.MethodA systematic mapping review was used as the research method, which was applied to over 129 papers obtained after the filtering process.ResultsThe papers were published in 19 journals and 40 conferences. Thirty-five percent of the papers published between years 2000 and 2011 have received at least one citation in journals and only five papers have received six or more citations. Effort variable is the focus of 70.5% of the papers, 22.5% center their research in a variable different from effort and 7% do not consider any target variable. Additionally, in as many as 70.5% of papers, effort estimation is the research topic, followed by dataset properties (36.4%). The more frequent methods are Regression (61.2%), Machine Learning (35.7%), and Estimation by Analogy (22.5%). ISBSG is used as the only support in 55% of the papers while the remaining papers use complementary datasets. The ISBSG release 10 is used most frequently with 32 references. Finally, some benefits and drawbacks of the usage of ISBSG have been highlighted.ConclusionThis work presents a snapshot of the existing usage of ISBSG in software development research. ISBSG offers a wealth of information regarding practices from a wide range of organizations, applications, and development types, which constitutes its main potential. However, a data preparation process is required before any analysis. Lastly, the potential of ISBSG to develop new research is also outlined.     

Reconciling software development models: A quasi-systematic review

Purpose

The purpose of this paper is to characterize reconciliation among the plan-driven, agile, and free/open source software models of software development.

Design/methodology/approach

An automated quasi-systematic review identified 42 papers, which were then analyzed.

Findings

The main findings are: there exist distinct - organization, group and process - levels of reconciliation; few studies deal with reconciliation among the three models of development; a significant amount of work addresses reconciliation between plan-driven and agile development; several large organizations (such as Microsoft, Motorola, and Philips) are interested in trying to combine these models; and reconciliation among software development models is still an open issue, since it is an emerging area and research on most proposals is at an early stage.

Research limitations

Automated searches may not capture relevant papers in publications that are not indexed. Other data sources not amenable to execution of the protocol were not used. Data extraction was performed by only one researcher, which may increase the risk of threats to internal validity.

Implications

This characterization is important for practitioners wanting to be current with the state of research. This review will also assist the scientific community working with software development processes to build a common understanding of the challenges that must be faced, and to identify areas where research is lacking. Finally, the results will be useful to software industry that is calling for solutions in this area.

Originality/value

There is no other systematic review on this subject, and reconciliation among software development models is an emerging area. This study helps to identify and consolidate the work done so far and to guide future research. The conclusions are an important step towards expanding the body of knowledge in the field.