Usage and usefulness of technical software documentation: An industrial case study

ContextSoftware documentation is an integral part of any software development process. However, software practitioners are often concerned about the value, degree of usage and usefulness of documentation during development and maintenance.ObjectiveMotivated by the needs of NovAtel Inc. (NovAtel), a world-leading company developing software systems in support of global navigation satellite systems, and based on the results of a former systematic mapping study, we aimed at better understanding of the usage and the usefulness of various technical documents during software development and maintenance.MethodWe utilized the results of a former systematic mapping study and performed an industrial case study at NovAtel. From the joint definition of the analysis goals, the research method incorporates qualitative and quantitative analysis of 55 documents (design, test and process related) and 1630 of their revisions. In addition, we conducted a survey on the usage and usefulness of documents. A total of 25 staff members from the industrial partner, all having a medium to high level of experience, participated in the survey.ResultsIn the context of the case study, a number of findings were derived. They include that (1) technical documentation was consulted least frequently for maintenance purpose and most frequently as an information source for development, (2) source code was considered most frequently as the preferred information source during software maintenance, (3) there is no significant difference between the usage of various documentation types during both development and maintenance, and (4) initial hypotheses stating that up-to-date information, accuracy and preciseness have the highest impact on usefulness of technical documentation.ConclusionsIt is concluded that the usage of documentation differs for various purposes and it depends on the type of the information needs as well as the tasks to be completed (e.g., development and maintenance). The results have been confirmed to be helpful for the company under study, and the firm is currently implementing some of the recommendations given.     

Software Process Simulation Modeling: Systematic literature review

Changes and continuous progress in logistics and productive systems make the realization of improvements in decision making necessary. Simulation is a good support tool for this type of decisions because it allows reproducing processes virtually to study their behavior, to analyze the impact of possible changes or to compare different design alternatives without the high cost of scale experiments. Although process simulation is usually focused on industrial processes, over the last two decades, new proposals have emerged to bring simulation techniques into software engineering. This paper describes a Systematic Literature Review (SLR) which returned 8070 papers (published from 2013 to 2019) by a systematic search in 4 digital libraries. After conducting this SLR, 36 Software Process Simulation Modeling (SPSM) works were selected as primary studies and were documented following a specific characterization scheme. This scheme allows characterizing each proposal according to the paradigm used and its technology base as well as its future line of work. Our purpose is to identify trends and directions for future research on SPSM after identifying and studying which proposals in this topic have been defined and the relationships and dependencies between these proposals in the last five years. After finishing this review, it is possible to conclude that SPSM continues to be a topic that is very much addressed by the scientific community, but each contribution has been proposed with particular goals. This review also concludes that Agent-Based Simulation and System Dynamics paradigm is increasing and decreasing, respectively, its trend among SPSM proposals in the last five years. Regarding Discrete-Event Simulation paradigm, it seems that it is strengthening its position among research community in recent years to design new approaches.     

Using metrics in Agile and Lean Software Development – A systematic literature review of industrial studies

ContextSoftware industry has widely adopted Agile software development methods. Agile literature proposes a few key metrics but little is known of the actual metrics use in Agile teams.ObjectiveThe objective of this paper is to increase knowledge of the reasons for and effects of using metrics in industrial Agile development. We focus on the metrics that Agile teams use, rather than the ones used from outside by software engineering researchers. In addition, we analyse the influence of the used metrics.MethodThis paper presents a systematic literature review (SLR) on using metrics in industrial Agile software development. We identified 774 papers, which we reduced to 30 primary studies through our paper selection process.ResultsThe results indicate that the reasons for and the effects of using metrics are focused on the following areas: sprint planning, progress tracking, software quality measurement, fixing software process problems, and motivating people. Additionally, we show that although Agile teams use many metrics suggested in the Agile literature, they also use many custom metrics. Finally, the most influential metrics in the primary studies are Velocity and Effort estimate.ConclusionThe use of metrics in Agile software development is similar to Traditional software development. Projects and sprints need to be planned and tracked. Quality needs to be measured. Problems in the process need to be identified and fixed. Future work should focus on metrics that had high importance but low prevalence in our study, as they can offer the largest impact to the software industry.     

A systematic review of software architecture visualization techniques

ContextGiven the increased interest in using visualization techniques (VTs) to help communicate and understand software architecture (SA) of large scale complex systems, several VTs and tools have been reported to represent architectural elements (such as architecture design, architectural patterns, and architectural design decisions). However, there is no attempt to systematically review and classify the VTs and associated tools reported for SA, and how they have been assessed and applied.ObjectiveThis work aimed at systematically reviewing the literature on software architecture visualization to develop a classification of VTs in SA, analyze the level of reported evidence and the use of different VTs for representing SA in different application domains, and identify the gaps for future research in the area.MethodWe used systematic literature review (SLR) method of the evidence-based software engineering (EBSE) for reviewing the literature on VTs for SA. We used both manual and automatic search strategies for searching the relevant papers published between 1 February 1999 and 1 July 2011.ResultsWe selected 53 papers from the initially retrieved 23,056 articles for data extraction, analysis, and synthesis based on pre-defined inclusion and exclusion criteria. The results from the data analysis enabled us to classify the identified VTs into four types based on the usage popularity: graph-based, notation-based, matrix-based, and metaphor-based VTs. The VTs in SA are mostly used for architecture recovery and architectural evolution activities. We have also identified ten purposes of using VTs in SA. Our results also revealed that VTs in SA have been applied to a wide range of application domains, among which “graphics software” and “distributed system” have received the most attention.ConclusionSA visualization has gained significant importance in understanding and evolving software-intensive systems. However, only a few VTs have been employed in industrial practice. This review has enabled us to identify the following areas for further research and improvement: (i) it is necessary to perform more research on applying visualization techniques in architectural analysis, architectural synthesis, architectural implementation, and architecture reuse activities; (ii) it is essential to pay more attention to use more objective evaluation methods (e.g., controlled experiment) for providing more convincing evidence to support the promised benefits of using VTs in SA; (iii) it is important to conduct industrial surveys for investigating how software architecture practitioners actually employ VTs in architecting process and what are the issues that hinder and prevent them from adopting VTs in SA.     

Software evolution visualization: A systematic mapping study

BackgroundSoftware evolution is an important topic in software engineering. It generally deals with large amounts of data, as one must look at whole project histories as opposed to their current snapshot. Software visualization is the field of software engineering that aims to help people to understand software through the use of visual resources. It can be effectively used to analyze and understand the large amount of data produced during software evolution.ObjectiveThis study investigates Software Evolution Visualization (SEV) approaches, collecting evidence about how SEV research is structured, synthesizing current evidence on the goals of the proposed approaches and identifying key challenges for its use in practice.MethodsA mapping study was conducted to analyze how the SEV area is structured. Selected primary studies were classified and analyzed with respect to nine research questions.ResultsSEV has been used for many different purposes, especially for change comprehension, change prediction and contribution analysis. The analysis identified gaps in the studies with respect to their goals, strategies and approaches. It also pointed out to a widespread lack of empirical studies in the area.ConclusionResearchers have proposed many SEV approaches during the past years, but some have failed to clearly state their goals, tie them back to concrete problems, or formally validate their usefulness. The identified gaps indicate that there still are many opportunities to be explored in the area.     

Process models in the practice of distributed software development: A systematic review of the literature

ContextDistributed Software Development (DSD) has recently become an active research area. Although considerable research effort has been made in this area, as yet, no agreement has been reached as to an appropriate process model for DSD.PurposeThis paper is intended to identify and synthesize papers that describe process models for distributed software development in the context of overseas outsourcing, i.e. “offshoring”.MethodWe used a systematic review methodology to search seven digital libraries and one topic-specific conference.ResultsWe found 27 primary studies describing stage-related DSD process models. Only five of such studies looked into outsourcing to a subsidiary company (i.e. “internal offshoring”). Nineteen primary studies addressed the need for DSD process models. Eight primary studies and three literature surveys described stage-based DSD process models, but only three of such models were empirically evaluated.ConclusionWe need more research aimed at internal offshoring. Furthermore, proposed models need to be empirically validated.     

Knowledge management initiatives in software testing: A mapping study

ContextSoftware testing is a knowledge intensive process, and, thus, Knowledge Management (KM) principles and techniques should be applied to manage software testing knowledge.ObjectiveThis study conducts a survey on existing research on KM initiatives in software testing, in order to identify the state of the art in the area as well as the future research. Aspects such as purposes, types of knowledge, technologies and research type are investigated.MethodThe mapping study was performed by searching seven electronic databases. We considered studies published until December 2013. The initial resulting set was comprised of 562 studies. From this set, a total of 13 studies were selected. For these 13, we performed snowballing and direct search to publications of researchers and research groups that accomplished these studies.ResultsFrom the mapping study, we identified 15 studies addressing KM initiatives in software testing that have been reviewed in order to extract relevant information on a set of research questions.ConclusionsAlthough only a few studies were found that addressed KM initiatives in software testing, the mapping shows an increasing interest in the topic in the recent years. Reuse of test cases is the perspective that has received more attention. From the KM point of view, most of the studies discuss aspects related to providing automated support for managing testing knowledge by means of a KM system. Moreover, as a main conclusion, the results show that KM is pointed out as an important strategy for increasing test effectiveness, as well as for improving the selection and application of suited techniques, methods and test cases. On the other hand, inadequacy of existing KM systems appears as the most cited problem related to applying KM in software testing.     

A measurement framework for improving verification processes

This paper describes the application of quantitative techniques to software process improvement in an industrial environment. This investigation is based on theami approach which was developed during the ESPRIT II project 5494ami (Application of Metrics in Industry). The activity-based improvement paradigm behind ami (assess, analyse, metricate, improve) advocates measurement as an essential component of process improvement for understanding the process, monitoring the changes and evaluating the return on investment. This experience focuses on the improvement of reviews and inspections in a software development environment certified to ISO 9001. A measurement framework is put into place for understanding these verification processes and performing them efficiently. After a brief introduction of theami approach, the case study is described following the fourami activities. The first data analysis findings are emphasized as well as the role played by the management during the initiative.     

Understanding the characteristics of quality for software engineering processes: A Grounded Theory investigation

ContextSoftware engineering organizations routinely define and implement processes to support, guide and control project execution. An assumption underlying this process-centric approach to business improvement is that the quality of the process will influence the quality, cost and time-to-release of the software produced. A critical question thus arises of what constitutes quality for software engineering processes.ObjectiveTo identify criteria used by experienced practitioners to judge the quality of software engineering processes and to understand how knowledge of these criteria and their relationships may be useful for those undertaking software process improvement activities.MethodInterviews were conducted with 17 experienced software engineering practitioners from a range of geographies, roles and industry sectors. Published reports from 30 software process improvement case-studies were selected from multiple peer-reviewed software journals. A qualitative Grounded Theory-based methodology was employed to systematically analyze the collected data to synthesize a model of quality for software engineering processes.ResultsThe synthesized model suggests that practitioners perceive the overall quality of a software engineering process based on four quality attributes: suitability, usability, manageability and evolvability. Furthermore, these judgments are influenced by key properties related to the semantic content, structure, representation and enactment of that process. The model indicates that these attributes correspond to particular organizational perspectives and that these differing views may explain role-based conflicts in the judgement of process quality.ConclusionConsensus exists amongst practitioners about which characteristics of software engineering process quality most influence project outcomes. The model presented provides a terminological framework that can facilitate precise discussion of software engineering process issues and a set of criteria that can support activities for software process definition, evaluation and improvement. The potential exists for further development of this model to facilitate optimization of process properties to match organizational needs.     

Test case generation for the task tree type of architecture

ContextEmerging multicores and clusters of multicores that may operate in parallel have set a new challenge – development of massively parallel software composed of thousands of loosely coupled or even completely independent threads/processes, such as MapReduce and Java 3.0 workers, or Erlang processes, respectively. Testing and verification is a critical phase in the development of such software products.ObjectiveGenerating test cases based on operational profiles and certifying declared operational reliability figure of the given software product is a well-established process for the sequential type of software. This paper proposes an adaptation of that process for a class of massively parallel software – large-scale task trees.MethodThe proposed method uses statistical usage testing and operational reliability estimation based on operational profiles and novel test suite quality indicators, namely the percentage of different task trees and the percentage of different paths.ResultsAs an example, the proposed method is applied to operational reliability certification of a parallel software infrastructure named the TaskTreeExecutor. The paper proposes an algorithm for generating random task trees to enable that application. Test runs in the experiments involved hundreds and thousands of Win32/Linux threads thus demonstrating scalability of the proposed approach. For practitioners, the most useful result presented is the method for determining the number of task trees and the number of paths, which are needed to certify the given operational reliability of a software product. The practitioners may also use the proposed coverage metrics to measure the quality of automatically generated test suite.ConclusionThis paper provides a useful solution for the test case generation that enables the operational reliability certification process for a class of massively parallel software called the large-scale task trees. The usefulness of this solution was demonstrated by a case study – operational reliability certification of the real parallel software product.     

User-centred design does make a difference. The case of decision support systems in crop production

This paper lends weight to the argument that user-centred design is important in design and development of software systems by describing the case of decision support systems for crop production. Decision support systems (DSS) are increasingly being seen as useful mechanisms for the transfer of scientific knowledge and 'best practice'; particularly in the field of agriculture. Although many systems have been developed, few can be termed 'successful' (i.e. have a significant uptake by intended users, and used by them). This paper provides evidence to support the importance of a user-centred approach by showing that each issue considered to be relevant to the failure of DSS in crop production can be addressed by the appropriate inclusion of users in the design and development process. Positive evidence in the shape of successful systems that have employed UCD is also presented.     

Variability in quality attributes of service-based software systems: A systematic literature review

ContextVariability is the ability of a software artifact (e.g., a system, component) to be adapted for a specific context, in a preplanned manner. Variability not only affects functionality, but also quality attributes (e.g., security, performance). Service-based software systems consider variability in functionality implicitly by dynamic service composition. However, variability in quality attributes of service-based systems seems insufficiently addressed in current design practices.ObjectiveWe aim at (a) assessing methods for handling variability in quality attributes of service-based systems, (b) collecting evidence about current research that suggests implications for practice, and (c) identifying open problems and areas for improvement.MethodA systematic literature review with an automated search was conducted. The review included studies published between the year 2000 and 2011. We identified 46 relevant studies.ResultsCurrent methods focus on a few quality attributes, in particular performance and availability. Also, most methods use formal techniques. Furthermore, current studies do not provide enough evidence for practitioners to adopt proposed approaches. So far, variability in quality attributes has mainly been studied in laboratory settings rather than in industrial environments.ConclusionsThe product line domain as the domain that traditionally deals with variability has only little impact on handling variability in quality attributes. The lack of tool support, the lack of practical research and evidence for the applicability of approaches to handle variability are obstacles for practitioners to adopt methods. Therefore, we suggest studies in industry (e.g., surveys) to collect data on how practitioners handle variability of quality attributes in service-based systems. For example, results of our study help formulate hypotheses and questions for such surveys. Based on needs in practice, new approaches can be proposed.     

A tool supporting root cause analysis for synchronous retrospectives in distributed software teams

ContextRoot cause analysis (RCA) is a useful practice for software project retrospectives, and is typically carried out in synchronous collocated face-to-face meetings. Conducting RCA with distributed teams is challenging, as face-to-face meetings are infeasible. Lack of adequate real-time tool support exacerbates this problem. Furthermore, there are no empirical studies on using RCA in synchronous retrospectives of geographically distributed teams.ObjectiveThis paper presents a real-time cloud-based software tool (ARCA-tool) we developed to support RCA in distributed teams and its initial empirical evaluation. The feasibility of using RCA with distributed teams is also evaluated.MethodWe compared our tool with 35 existing RCA software tools. We conducted field studies of four distributed agile software teams at two international software product companies. The teams conducted RCA collaboratively in synchronous retrospective meetings by using the tool we developed. We collected the data using observations, interviews and questionnaires.ResultsComparison revealed that none of the existing 35 tools matched all the features of our ARCA-tool. The team members found ARCA-tool to be an essential part of their distributed retrospectives. They considered the software as efficient and very easy to learn and use. Additionally, the team members perceived RCA to be a vital part of the retrospectives. In contrast to the prior retrospective practices of the teams, the introduced RCA method was evaluated as efficient and easy to use.ConclusionRCA is a useful practice in synchronous distributed retrospectives. However, it requires software tool support for enabling real-time view and co-creation of a cause-effect diagram. ARCA-tool supports synchronous RCA, and includes support for logging problems and causes, problem prioritization, cause-effect diagramming, and logging of process improvement proposals. It enables conducting RCA in distributed retrospectives.     

Enhancing software artefact traceability recovery processes with link count information

ContextThe intensive human effort needed to manually manage traceability information has increased the interest in using semi-automated traceability recovery techniques. In particular, Information Retrieval (IR) techniques have been largely employed in the last ten years to partially automate the traceability recovery process.AimPrevious studies mainly focused on the analysis of the performances of IR-based traceability recovery methods and several enhancing strategies have been proposed to improve their accuracy. Very few papers investigate how developers (i) use IR-based traceability recovery tools and (ii) analyse the list of suggested links to validate correct links or discard false positives. We focus on this issue and suggest exploiting link count information in IR-based traceability recovery tools to improve the performances of the developers during a traceability recovery process.MethodTwo empirical studies have been conducted to evaluate the usefulness of link count information. The two studies involved 135 University students that had to perform (with and without link count information) traceability recovery tasks on two software project repositories. Then, we evaluated the quality of the recovered traceability links in terms of links correctly and erroneously traced by the students.ResultsThe results achieved indicate that the use of link count information significantly increases the number of correct links identified by the participants.ConclusionsThe results can be used to derive guidelines on how to effectively use traceability recovery approaches and tools proposed in the literature.     

A pattern-based model-driven approach for situational method engineering

ContextConstructing bespoke software development methodologies for specific project situations has become a crucial need, giving rise to Situational Method Engineering (SME). Compared with Software Engineering, SME has a long way to go yet; SME approaches are especially deficient as to support for modeling, portability, and automation. Model-Driven Development (MDD) has been effectively used for addressing these issues in Software Engineering, and is also considered a promising approach for resolving them in SME.ObjectiveThis paper aims to address the shortcomings of existing SME approaches by introducing a novel MDD approach, specifically intended for SME purposes, that uses a pattern-based approach for model transformation.MethodDeveloping a MDD approach for SME requires that a modeling framework, consisting of modeling levels, be defined for modeling software development methodologies. Transformation patterns should also be specified for converting the models from one level to the next. A process should then be defined for applying the framework and transformations patterns to real SME projects. The resulting MDD approach requires proper evaluation to demonstrate its applicability.ResultsA framework and a semi-automated process have been proposed that adapt pattern-based model transformation techniques for application to the methodology models used in SME. The transformation patterns have been implemented in the Medini-QVT model transformation tool, along with two supplementary method bases: one for mapping the situational factors of SME projects to requirements, and the other for mapping the requirements to method fragments. The method engineer can produce the methodology models by using the method bases and executing the transformation patterns via the tool.ConclusionThe validity of the proposed approach has been assessed based on special evaluation criteria, and also through application to a real-world project. Evaluation results indicate that the proposed approach addresses the deficiencies of existing approaches, and satisfies the practicality requirements of SME approaches.     

Gamification in software engineering – A systematic mapping

ContextGamification seeks for improvement of the user’s engagement, motivation, and performance when carrying out a certain task, by means of incorporating game mechanics and elements, thus making that task more attractive. Much research work has studied the application of gamification in software engineering for increasing the engagement and results of developers.ObjectiveThe objective of this paper is to carry out a systematic mapping of the field of gamification in software engineering in an attempt to characterize the state of the art of this field identifying gaps and opportunities for further research.MethodWe carried out a systematic mapping with a view to finding the primary studies in the existing literature, which were later classified and analyzed according to four criteria: the software process area addressed, the gamification elements used, the type of research method followed, and the type of forum in which they were published. A subjective evaluation of the studies was also carried out to evaluate them in terms of methodology, empirical evidence, integration with the organization, and replicability.ResultsAs a result of the systematic mapping we found 29 primary studies, published between January 2011 and June 2014. Most of them focus on software development, and to a lesser extent, requirements, project management, and other support areas. In the main, they consider very simple gamification mechanics such as points and badges, and few provide empirical evidence of the impact of gamification.ConclusionsExisting research in the field is quite preliminary, and more research effort analyzing the impact of gamification in SE would be needed. Future research work should look at other game mechanics in addition to the basic ones and should tackle software process areas that have not been fully studied, such as requirements, project management, maintenance, or testing. Most studies share a lack of methodological support that would make their proposals replicable in other settings. The integration of gamification with an organization’s existing tools is also an important challenge that needs to be taken up in this field.     

Validating a model-driven software architecture evaluation and improvement method: A family of experiments

ContextSoftware architectures should be evaluated during the early stages of software development in order to verify whether the non-functional requirements (NFRs) of the product can be fulfilled. This activity is even more crucial in software product line (SPL) development, since it is also necessary to identify whether the NFRs of a particular product can be achieved by exercising the variation mechanisms provided by the product line architecture or whether additional transformations are required. These issues have motivated us to propose QuaDAI, a method for the derivation, evaluation and improvement of software architectures in model-driven SPL development.ObjectiveWe present in this paper the results of a family of four experiments carried out to empirically validate the evaluation and improvement strategy of QuaDAI.MethodThe family of experiments was carried out by 92 participants: Computer Science Master’s and undergraduate students from Spain and Italy. The goal was to compare the effectiveness, efficiency, perceived ease of use, perceived usefulness and intention to use with regard to participants using the evaluation and improvement strategy of QuaDAI as opposed to the Architecture Tradeoff Analysis Method (ATAM).ResultsThe main result was that the participants produced their best results when applying QuaDAI, signifying that the participants obtained architectures with better values for the NFRs faster, and that they found the method easier to use, more useful and more likely to be used. The results of the meta-analysis carried out to aggregate the results obtained in the individual experiments also confirmed these results.ConclusionsThe results support the hypothesis that QuaDAI would achieve better results than ATAM in the experiments and that QuaDAI can be considered as a promising approach with which to perform architectural evaluations that occur after the product architecture derivation in model-driven SPL development processes when carried out by novice software evaluators.     

HCI and business practices in a collaborative method for augmented reality systems

ContextEvery interactive system is composed of a functional core and a user interface. However, the software engineering (SE) and human–computer interaction (HCI) communities do not share the same methods, models or tools. This usually induces a large work overhead when specialists from the two domains try to connect their applicative studies, especially when developing augmented reality systems that feature complex interaction cores.ObjectiveWe present in this paper the essential activities and concepts of a development method integrating the SE and HCI development practices, from the specifications down to the design, as well as their application on a case study.MethodThe efficiency of the method was tested in a qualitative study involving four pairs of SE and HCI experts in the design of an application for which an augmented reality interaction would provide better user performance than a classic interactive system. The effectivity of the method was evaluated in a qualitative study comparing the quality of three implementations of the same application fragment (based on the same analysis model), using software engineering metrics.ResultsThe first evaluation confirmed the ease of use of our method and the relevance of our tools for guiding the design process, but raised concerns on the handling of conflicting collaborative activities. The second evaluation gave indications that the structure of the analysis model facilitates the implementation of quality software (in terms of coupling, stability and complexity).ConclusionIt is concluded that our method enables design teams with different backgrounds in application development to collaborate for integrating augmented reality applications with information systems. Areas of improvement are also described.     

Investigating dependencies in software requirements for change propagation analysis

ContextThe dependencies between individual requirements have an important influence on software engineering activities e.g., project planning, architecture design, and change impact analysis. Although dozens of requirement dependency types were suggested in the literature from different points of interest, there still lacks an evaluation of the applicability of these dependency types in requirements engineering.ObjectiveUnderstanding the effect of these requirement dependencies to software engineering activities is useful but not trivial. In this study, we aimed to first investigate whether the existing dependency types are useful in practise, in particular for change propagation analysis, and then suggest improvements for dependency classification and definition.MethodWe conducted a case study that evaluated the usefulness and applicability of two well-known generic dependency models covering 25 dependency types. The case study was conducted in a real-world industry project with three participants who offered different perspectives.ResultsOur initial evaluation found that there exist a number of overlapping and/or ambiguous dependency types among the current models; five dependency types are particularly useful in change propagation analysis; and practitioners with different backgrounds possess various viewpoints on change propagation. To improve the state-of-the-art, a new dependency model is proposed to tackle the problems identified from the case study and the related literature. The new model classifies dependencies into intrinsic and additional dependencies on the top level, and suggests nine dependency types with precise definitions as its initial set.ConclusionsOur case study provides insights into requirement dependencies and their effects on change propagation analysis for both research and practise. The resulting new dependency model needs further evaluation and improvement.