A systematic literature review on the industrial use of software process simulation

ContextSoftware process simulation modelling (SPSM) captures the dynamic behaviour and uncertainty in the software process. Existing literature has conflicting claims about its practical usefulness: SPSM is useful and has an industrial impact; SPSM is useful and has no industrial impact yet; SPSM is not useful and has little potential for industry.ObjectiveTo assess the conflicting standpoints on the usefulness of SPSM.MethodA systematic literature review was performed to identify, assess and aggregate empirical evidence on the usefulness of SPSM.ResultsIn the primary studies, to date, the persistent trend is that of proof-of-concept applications of software process simulation for various purposes (e.g. estimation, training, process improvement, etc.). They score poorly on the stated quality criteria. Also only a few studies report some initial evaluation of the simulation models for the intended purposes.ConclusionThere is a lack of conclusive evidence to substantiate the claimed usefulness of SPSM for any of the intended purposes. A few studies that report the cost of applying simulation do not support the claim that it is an inexpensive method. Furthermore, there is a paramount need for improvement in conducting and reporting simulation studies with an emphasis on evaluation against the intended purpose.     

Knowledge-based approaches in software documentation: A systematic literature review

ContextSoftware documents are core artifacts produced and consumed in documentation activity in the software lifecycle. Meanwhile, knowledge-based approaches have been extensively used in software development for decades, however, the software engineering community lacks a comprehensive understanding on how knowledge-based approaches are used in software documentation, especially documentation of software architecture design.ObjectiveThe objective of this work is to explore how knowledge-based approaches are employed in software documentation, their influences to the quality of software documentation, and the costs and benefits of using these approaches.MethodWe use a systematic literature review method to identify the primary studies on knowledge-based approaches in software documentation, following a pre-defined review protocol.ResultsSixty studies are finally selected, in which twelve quality attributes of software documents, four cost categories, and nine benefit categories of using knowledge-based approaches in software documentation are identified. Architecture understanding is the top benefit of using knowledge-based approaches in software documentation. The cost of retrieving information from documents is the major concern when using knowledge-based approaches in software documentation.ConclusionsThe findings of this review suggest several future research directions that are critical and promising but underexplored in current research and practice: (1) there is a need to use knowledge-based approaches to improve the quality attributes of software documents that receive less attention, especially credibility, conciseness, and unambiguity; (2) using knowledge-based approaches with the knowledge content in software documents which gets less attention in current applications of knowledge-based approaches in software documentation, to further improve the practice of software documentation activity; (3) putting more focus on the application of software documents using the knowledge-based approaches (knowledge reuse, retrieval, reasoning, and sharing) in order to make the most use of software documents; and (4) evaluating the costs and benefits of using knowledge-based approaches in software documentation qualitatively and quantitatively.     

Software process modeling languages: A systematic literature review

ContextOrganizations working in software development are aware that processes are very important assets as well as they are very conscious of the need to deploy well-defined processes with the goal of improving software product development and, particularly, quality. Software process modeling languages are an important support for describing and managing software processes in software-intensive organizations.ObjectiveThis paper seeks to identify what software process modeling languages have been defined in last decade, the relationships and dependencies among them and, starting from the current state, to define directions for future research.MethodA systematic literature review was developed. 1929 papers were retrieved by a manual search in 9 databases and 46 primary studies were finally included.ResultsSince 2000 more than 40 languages have been first reported, each of which with a concrete purpose. We show that different base technologies have been used to define software process modeling languages. We provide a scheme where each language is registered together with the year it was created, the base technology used to define it and whether it is considered a starting point for later languages. This scheme is used to illustrate the trend in software process modeling languages. Finally, we present directions for future research.ConclusionThis review presents the different software process modeling languages that have been developed in the last ten years, showing the relevant fact that model-based SPMLs (Software Process Modeling Languages) are being considered as a current trend. Each one of these languages has been designed with a particular motivation, to solve problems which had been detected. However, there are still several problems to face, which have become evident in this review. This let us provide researchers with some guidelines for future research on this topic.     

Impediments for software test automation: A systematic literature review

Automated software testing is a critical enabler for modern software development, where rapid feedback on the product quality is expected. To make the testing work well, it is of high importance that impediments related to test automation are prevented and removed quickly. An enabling factor for all types of improvement is to understand the nature of what is to be improved. We have performed a systematic literature review of reported impediments related to software test automation to contribute to this understanding. In this paper, we present the results from the systematic literature review: The list of identified publications, a categorization of identified impediments, and a qualitative discussion of the impediments proposing a socio‐technical system model of the use and implementation of test automation.     

A systematic review of systematic review process research in software engineering

ContextMany researchers adopting systematic reviews (SRs) have also published papers discussing problems with the SR methodology and suggestions for improving it. Since guidelines for SRs in software engineering (SE) were last updated in 2007, we believe it is time to investigate whether the guidelines need to be amended in the light of recent research.ObjectiveTo identify, evaluate and synthesize research published by software engineering researchers concerning their experiences of performing SRs and their proposals for improving the SR process.MethodWe undertook a systematic review of papers reporting experiences of undertaking SRs and/or discussing techniques that could be used to improve the SR process. Studies were classified with respect to the stage in the SR process they addressed, whether they related to education or problems faced by novices and whether they proposed the use of textual analysis tools.ResultsWe identified 68 papers reporting 63 unique studies published in SE conferences and journals between 2005 and mid-2012. The most common criticisms of SRs were that they take a long time, that SE digital libraries are not appropriate for broad literature searches and that assessing the quality of empirical studies of different types is difficult.ConclusionWe recommend removing advice to use structured questions to construct search strings and including advice to use a quasi-gold standard based on a limited manual search to assist the construction of search stings and evaluation of the search process. Textual analysis tools are likely to be useful for inclusion/exclusion decisions and search string construction but require more stringent evaluation. SE researchers would benefit from tools to manage the SR process but existing tools need independent validation. Quality assessment of studies using a variety of empirical methods remains a major problem.     

Process models for service-based applications: A systematic literature review

Context

Service-Oriented Computing (SOC) is a promising computing paradigm which facilitates the development of adaptive and loosely coupled service-based applications (SBAs). Many of the technical challenges pertaining to the development of SBAs have been addressed, however, there are still outstanding questions relating to the processes required to develop them.

Objective

The objective of this study is to systematically identify process models for developing service-based applications (SBAs) and review the processes within them. This will provide a useful starting point for any further research in the area. A secondary objective of the study is to identify process models which facilitate the adaptation of SBAs.

Method

In order to achieve this objective a systematic literature review (SLR) of the existing software engineering literature is conducted.

Results

During this research 722 studies were identified using a predefined search strategy, this number was narrowed down to 57 studies based on a set of strict inclusion and exclusion criteria. The results are reported both quantitatively in the form of a mapping study, as well as qualitatively in the form of a narrative summary of the key processes identified.

Conclusion

There are many process models reported for the development of SBAs varying in detail and maturity, this review has identified and categorised the processes within those process models. The review has also identified and evaluated process models which facilitate the adaptation of SBAs.     

Exploring principles of user-centered agile software development: A literature review

ContextIn the last decade, software development has been characterized by two major approaches: agile software development, which aims to achieve increased velocity and flexibility during the development process, and user-centered design, which places the goals and needs of the system’s end-users at the center of software development in order to deliver software with appropriate usability. Hybrid development models, referred to as user-centered agile software development (UCASD) in this article, propose to combine the merits of both approaches in order to design software that is both useful and usable.ObjectiveThis paper aims to capture the current state of the art in UCASD approaches and to derive generic principles from these approaches. More specifically, we investigate the following research question: Which principles constitute a user-centered agile software development approach?MethodWe conduct a systematic review of the literature on UCASD. Identified works are analyzed using a coding scheme that differentiates four levels of UCASD: the process, practices, people/social and technology dimensions. Through subsequent synthesis, we derive generic principles of UCASD.ResultsWe identified and analyzed 83 relevant publications. The analysis resulted in a comprehensive coding system and five principles for UCASD: (1) separate product discovery and product creation, (2) iterative and incremental design and development, (3) parallel interwoven creation tracks, (4) continuous stakeholder involvement, and (5) artifact-mediated communication.ConclusionOur paper contributes to the software development body of knowledge by (1) providing a broad overview of existing works in the area of UCASD, (2) deriving an analysis framework (in form a coding system) for works in this area, going beyond former classifications, and (3) identifying generic principles of UCASD and associating them with specific practices and processes.     

A systematic review of machine learning techniques for software fault prediction

BackgroundSoftware fault prediction is the process of developing models that can be used by the software practitioners in the early phases of software development life cycle for detecting faulty constructs such as modules or classes. There are various machine learning techniques used in the past for predicting faults.MethodIn this study we perform a systematic review of studies from January 1991 to October 2013 in the literature that use the machine learning techniques for software fault prediction. We assess the performance capability of the machine learning techniques in existing research for software fault prediction. We also compare the performance of the machine learning techniques with the statistical techniques and other machine learning techniques. Further the strengths and weaknesses of machine learning techniques are summarized.ResultsIn this paper we have identified 64 primary studies and seven categories of the machine learning techniques. The results prove the prediction capability of the machine learning techniques for classifying module/class as fault prone or not fault prone. The models using the machine learning techniques for estimating software fault proneness outperform the traditional statistical models.ConclusionBased on the results obtained from the systematic review, we conclude that the machine learning techniques have the ability for predicting software fault proneness and can be used by software practitioners and researchers. However, the application of the machine learning techniques in software fault prediction is still limited and more number of studies should be carried out in order to obtain well formed and generalizable results. We provide future guidelines to practitioners and researchers based on the results obtained in this work.     

Barriers in the selection of offshore software development outsourcing vendors: An exploratory study using a systematic literature review

Context

Software development outsourcing is a contract-based relationship between client and vendor organisations in which a client contracts out all or part of its software development activities to a vendor, who provides agreed services for remuneration.

Objective

The objective is to identify various barriers that have a negative impact on software outsourcing clients in the selection process of offshore software development outsourcing vendors.

Method

We have performed a systematic literature review (SLR) process for the identification of barriers. We have performed all the SLR steps such as the protocol development, initial selection, final selection, quality assessment, data extraction and data synthesis.

Results

We have identified barriers such as ‘language and cultural barriers’, ‘country instability’, ‘lack of project management’, ‘lack of protection for intellectual property rights’ and ‘lack of technical capability’ that generally have a negative impact on outsourcing clients. We have identified only one common frequently cited barrier in three types of organisations (i.e. small, medium and large) which is ‘language and cultural barriers’. We did not identify any common frequently cited barrier in three continents (Asia, North America and Europe) and in two decades (1990-1999 and 2000-mid 2008). The results also reveal the similarities and differences in the barriers identified through different study strategies.

Conclusions

Vendors should address frequently cited barriers such as ‘language and cultural barriers’, ‘country instability’, ‘lack of project management’, ‘lack of protection for intellectual property rights’ and ‘lack of technical capability’ in order to compete in the offshore outsourcing business.     

Industrial maintenance decision-making: A systematic literature review

The increasing competition among industries has leveraged the emergence of various tools and methods for maintenance decision-making support. This paper identifies in literature the application areas of industrial maintenance decision-making, the relationships between these areas and the ways in which authors integrate tools and methods. This information makes it possible to identify trends and deficiencies in this context, helping to centralize the efforts required for future work. This work follows a series of structured steps for a systematic literature review of papers related to the main topic available in online databases. The selected papers are subject to a content assessment and grouped according to the application areas. The direct comparison between these areas and the construction of a relational matrix provide a quantitative interpretation of the results and well-structured information. Additionally, this paper proposes a framework based on information from the literature, which summarizes the origin and flow of information used in the development of models, showing the relationship among application areas of decision making. The research undertaken identifies trends focused on joint production systems optimization and increasing the deployment of methods for autonomous equipment predictions.     

Analogy-based software development effort estimation: A systematic mapping and review

ContextAnalogy-based Software development Effort Estimation (ASEE) techniques have gained considerable attention from the software engineering community. However, existing systematic map and review studies on software development effort prediction have not investigated in depth several issues of ASEE techniques, to the exception of comparisons with other types of estimation techniques.ObjectiveThe objective of this research is twofold: (1) to classify ASEE studies which primary goal is to propose new or modified ASEE techniques according to five criteria: research approach, contribution type, techniques used in combination with ASEE methods, and ASEE steps, as well as identifying publication channels and trends and (2) to analyze these studies from five perspectives: estimation accuracy, accuracy comparison, estimation context, impact of the techniques used in combination with ASEE methods, and ASEE tools.MethodWe performed a systematic mapping of studies for which the primary goal is to develop or to improve ASEE techniques published in the period 1990–2012, and reviewed them based on an automated search of four electronic databases.ResultsIn total, we identified 65 studies published between 1990 and 2012, and classified them based on our predefined classification criteria. The mapping study revealed that most researchers focus on addressing problems related to the first step of an ASEE process, that is, feature and case subset selection. The results of our detailed analysis show that ASEE methods outperform the eight techniques with which they were compared, and tend to yield acceptable results especially when combining ASEE techniques with Fuzzy Logic (FL) or Genetic Algorithms (GA).ConclusionBased on the findings of this study, the use of other techniques such FL and GA in combination with an ASEE method is promising to generate more accurate estimates. However, the use of ASEE techniques by practitioners is still limited: developing more ASEE tools may facilitate the application of these techniques and then lead to increasing the use of ASEE techniques in industry.     

Web application testing: A systematic literature review

ContextThe web has had a significant impact on all aspects of our society. As our society relies more and more on the web, the dependability of web applications has become increasingly important. To make these applications more dependable, for the past decade researchers have proposed various techniques for testing web-based software applications. Our literature search for related studies retrieved 193 papers in the area of web application testing, which have appeared between 2000 and 2013.ObjectiveAs this research area matures and the number of related papers increases, it is important to systematically identify, analyze, and classify the publications and provide an overview of the trends and empirical evidence in this specialized field.MethodsWe systematically review the body of knowledge related to functional testing of web application through a systematic literature review (SLR) study. This SLR is a follow-up and complimentary study to a recent systematic mapping (SM) study that we conducted in this area. As part of this study, we pose three sets of research questions, define selection and exclusion criteria, and synthesize the empirical evidence in this area.ResultsOur pool of studies includes a set of 95 papers (from the 193 retrieved papers) published in the area of web application testing between 2000 and 2013. The data extracted during our SLR study is available through a publicly-accessible online repository. Among our results are the followings: (1) the list of test tools in this area and their capabilities, (2) the types of test models and fault models proposed in this domain, (3) the way the empirical studies in this area have been designed and reported, and (4) the state of empirical evidence and industrial relevance.ConclusionWe discuss the emerging trends in web application testing, and discuss the implications for researchers and practitioners in this area. The results of our SLR can help researchers to obtain an overview of existing web application testing approaches, fault models, tools, metrics and empirical evidence, and subsequently identify areas in the field that require more attention from the research community.     

Systematic literature review of machine learning based software development effort estimation models

Context

Software development effort estimation (SDEE) is the process of predicting the effort required to develop a software system. In order to improve estimation accuracy, many researchers have proposed machine learning (ML) based SDEE models (ML models) since 1990s. However, there has been no attempt to analyze the empirical evidence on ML models in a systematic way.

Objective

This research aims to systematically analyze ML models from four aspects: type of ML technique, estimation accuracy, model comparison, and estimation context.

Method

We performed a systematic literature review of empirical studies on ML model published in the last two decades (1991-2010).

Results

We have identified 84 primary studies relevant to the objective of this research. After investigating these studies, we found that eight types of ML techniques have been employed in SDEE models. Overall speaking, the estimation accuracy of these ML models is close to the acceptable level and is better than that of non-ML models. Furthermore, different ML models have different strengths and weaknesses and thus favor different estimation contexts.

Conclusion

ML models are promising in the field of SDEE. However, the application of ML models in industry is still limited, so that more effort and incentives are needed to facilitate the application of ML models. To this end, based on the findings of this review, we provide recommendations for researchers as well as guidelines for practitioners.     

A systematic literature review on the barriers faced by newcomers to open source software projects

ContextNumerous open source software projects are based on volunteers collaboration and require a continuous influx of newcomers for their continuity. Newcomers face barriers that can lead them to give up. These barriers hinder both developers willing to make a single contribution and those willing to become a project member.ObjectiveThis study aims to identify and classify the barriers that newcomers face when contributing to open source software projects.MethodWe conducted a systematic literature review of papers reporting empirical evidence regarding the barriers that newcomers face when contributing to open source software (OSS) projects. We retrieved 291 studies by querying 4 digital libraries. Twenty studies were identified as primary. We performed a backward snowballing approach, and searched for other papers published by the authors of the selected papers to identify potential studies. Then, we used a coding approach inspired by open coding and axial coding procedures from Grounded Theory to categorize the barriers reported by the selected studies.ResultsWe identified 20 studies providing empirical evidence of barriers faced by newcomers to OSS projects while making a contribution. From the analysis, we identified 15 different barriers, which we grouped into five categories: social interaction, newcomers’ previous knowledge, finding a way to start, documentation, and technical hurdles. We also classified the problems with regard to their origin: newcomers, community, or product.ConclusionThe results are useful to researchers and OSS practitioners willing to investigate or to implement tools to support newcomers. We mapped technical and non-technical barriers that hinder newcomers’ first contributions. The most evidenced barriers are related to socialization, appearing in 75% (15 out of 20) of the studies analyzed, with a high focus on interactions in mailing lists (receiving answers and socialization with other members). There is a lack of in-depth studies on technical issues, such as code issues. We also noticed that the majority of the studies relied on historical data gathered from software repositories and that there was a lack of experiments and qualitative studies in this area.     

A systematic literature review of studies on business process modeling quality

ContextBusiness process modeling is an essential part of understanding and redesigning the activities that a typical enterprise uses to achieve its business goals. The quality of a business process model has a significant impact on the development of any enterprise and IT support for that process.ObjectiveSince the insights on what constitutes modeling quality are constantly evolving, it is unclear whether research on business process modeling quality already covers all major aspects of modeling quality. Therefore, the objective of this research is to determine the state of the art on business process modeling quality: What aspects of process modeling quality have been addressed until now and which gaps remain to be covered?MethodWe performed a systematic literature review of peer reviewed articles as published between 2000 and August 2013 on business process modeling quality. To analyze the contributions of the papers we use the Formal Concept Analysis technique.ResultsWe found 72 studies addressing quality aspects of business process models. These studies were classified into different dimensions: addressed model quality type, research goal, research method, and type of research result. Our findings suggest that there is no generally accepted framework of model quality types. Most research focuses on empirical and pragmatic quality aspects, specifically with respect to improving the understandability or readability of models. Among the various research methods, experimentation is the most popular one. The results from published research most often take the form of intangible knowledge.ConclusionWe believe there is a lack of an encompassing and generally accepted definition of business process modeling quality. This evidences the need for the development of a broader quality framework capable of dealing with the different aspects of business process modeling quality. Different dimensions of business process quality and of the process of modeling still require further research.     

Factors influencing clients in the selection of offshore software outsourcing vendors: An exploratory study using a systematic literature review

Context

Offshore software development outsourcing is a modern business strategy for developing high quality software at low cost.

Objective

The objective of this research paper is to identify and analyse factors that are important in terms of the competitiveness of vendor organisations in attracting outsourcing projects.

Method

We performed a systematic literature review (SLR) by applying our customised search strings which were derived from our research questions. We performed all the SLR steps, such as the protocol development, initial selection, final selection, quality assessment, data extraction and data synthesis.

Results

We have identified factors such as cost-saving, skilled human resource, appropriate infrastructure, quality of product and services, efficient outsourcing relationships management, and an organisation's track record of successful projects which are generally considered important by the outsourcing clients. Our results indicate that appropriate infrastructure, cost-saving, and skilled human resource are common in three continents, namely Asia, North America and Europe. We identified appropriate infrastructure, cost-saving, and quality of products and services as being common in three types of organisations (small, medium and large). We have also identified four factors-appropriate infrastructure, cost-saving, quality of products and services, and skilled human resource as being common in the two decades (1990-1999 and 2000-mid 2008).

Conclusions

Cost-saving should not be considered as the driving factor in the selection process of software development outsourcing vendors. Vendors should rather address other factors in order to compete in the OSDO business, such as skilled human resource, appropriate infrastructure and quality of products and services.     

Scientific research ontology to support systematic review in software engineering

The term systematic review is used to refer to a specific methodology of research, developed in order to gather and evaluate the available evidence pertaining to a focused topic. It represents a secondary study that depends on primary study results to be accomplished. Several primary studies have been conducted in the field of Software Engineering in the last years, determining an increasing improvement in methodology. However, in most cases software is built with technologies and processes for which developers have insufficient evidence to confirm their suitability, limits, qualities, costs, and inherent risks. Conducting systematic reviews in Software Engineering consists in a major methodological tool to scientifically improve the validity of assertions that can be made in the field and, as a consequence, the reliability degree of the methods that are employed for developing software technologies and supporting software processes. This paper aims at discussing the significance of experimental studies, particularly systematic reviews, and their use in supporting software processes. A template designed to support systematic reviews in Software Engineering is presented, and the development of ontologies to describe knowledge regarding such experimental studies is also introduced.     

A systematic review of software architecture evolution research

Context

Software evolvability describes a software system’s ability to easily accommodate future changes. It is a fundamental characteristic for making strategic decisions, and increasing economic value of software. For long-lived systems, there is a need to address evolvability explicitly during the entire software lifecycle in order to prolong the productive lifetime of software systems. For this reason, many research studies have been proposed in this area both by researchers and industry practitioners. These studies comprise a spectrum of particular techniques and practices, covering various activities in software lifecycle. However, no systematic review has been conducted previously to provide an extensive overview of software architecture evolvability research.

Objective

In this work, we present such a systematic review of architecting for software evolvability. The objective of this review is to obtain an overview of the existing approaches in analyzing and improving software evolvability at architectural level, and investigate impacts on research and practice.

Method

The identification of the primary studies in this review was based on a pre-defined search strategy and a multi-step selection process.

Results

Based on research topics in these studies, we have identified five main categories of themes: (i) techniques supporting quality consideration during software architecture design, (ii) architectural quality evaluation, (iii) economic valuation, (iv) architectural knowledge management, and (v) modeling techniques. A comprehensive overview of these categories and related studies is presented.

Conclusion

The findings of this review also reveal suggestions for further research and practice, such as (i) it is necessary to establish a theoretical foundation for software evolution research due to the fact that the expertise in this area is still built on the basis of case studies instead of generalized knowledge; (ii) it is necessary to combine appropriate techniques to address the multifaceted perspectives of software evolvability due to the fact that each technique has its specific focus and context for which it is appropriate in the entire software lifecycle.