Exploring a Bayesian and linear approach to requirements traceability

Context

For large software projects it is important to have some traceability between artefacts from different phases (e.g.requirements, designs, code), and between artefacts and the involved developers. However, if the capturing of traceability information during the project is felt as laborious to developers, they will often be sloppy in registering the relevant traceability links so that the information is incomplete. This makes automated tool-based collection of traceability links a tempting alternative, but this has the opposite challenge of generating too many potential trace relationships, not all of which are equally relevant.

Objective

This paper evaluates how to rank such auto-generated trace relationships.

Method

We present two approaches for such a ranking: a Bayesian technique and a linear inference technique. Both techniques depend on the interaction event trails left behind by collaborating developers while working within a development tool.

Results

The outcome of a preliminary study suggest the advantage of the linear approach, we also explore the challenges and potentials of the two techniques.

Conclusion

The advantage of the two techniques is that they can be used to provide traceability insights that are contextual and would have been much more difficult to capture manually. We also present some key lessons learnt during this research.     

Pattern-based framework for modularized software development and evolution robustness

Context

Software development is now facing much more challenges than ever before due to the intrinsic high complexity and the increasing demands of the quick-service-ready paradigm.

Objective

As the developers are now called for more quality software systems from the industries, there is insufficient guidance from the methodologies and standards of software engineering that can provide assistance to the rapid development of qualified business software.

Method

In this work, we discuss the advantages of the pattern-based software development. We verify the benefits using a pattern-based software framework called OS2F, and a corresponding system design architecture that is intended for the rapid development of web applications.

Results

The objective of the framework/architecture is that, through software patterns, developers should be able to separate the work of system development from the business rules so as to reduce the problems caused by a developer’s lack of business experiences.

Conclusion

Through a suitable pattern-based software framework, the quality of the product can thus be enhanced, software development time and cost decreased, and software evolution robustness improved.     

API2MoL: Automating the building of bridges between APIs and Model-Driven Engineering

Context

A software artefact typically makes its functionality available through a specialized Application Programming Interface (API) describing the set of services offered to client applications. In fact, building any software system usually involves managing a plethora of APIs, which complicates the development process. In Model-Driven Engineering (MDE), where models are the key elements of any software engineering activity, this API management should take place at the model level. Therefore, tools that facilitate the integration of APIs and MDE are clearly needed.

Objective

Our goal is to automate the implementation of API-MDE bridges for supporting both the creation of models from API objects and the generation of such API objects from models. In this sense, this paper presents the API2MoL approach, which provides a declarative rule-based language to easily write mapping definitions to link API specifications and the metamodel that represents them. These definitions are then executed to convert API objects into model elements or vice versa. The approach also allows both the metamodel and the mapping to be automatically obtained from the API specification (bootstrap process).

Method

After implementing the API2MoL engine, its correctness was validated using several APIs. Since APIs are normally large, we then developed a tool to implement the bootstrap process, which was also validated.

Results

We provide a toolkit (language and bootstrap tool) for the creation of bridges between APIs and MDE. The current implementation focuses on Java APIs, although its adaptation to other statically typed object-oriented languages is straightforward. The correctness, expressiveness and completeness of the approach have been validated with the Swing, SWT and JTwitter APIs.

Conclusion

API2MoL frees developers from having to manually implement the tasks of obtaining models from API objects and generating such objects from models. This helps to manage API models in MDE-based solutions.     

A systematic literature review of actionable alert identification techniques for automated static code analysis

Context

Automated static analysis (ASA) identifies potential source code anomalies early in the software development lifecycle that could lead to field failures. Excessive alert generation and a large proportion of unimportant or incorrect alerts (unactionable alerts) may cause developers to reject the use of ASA. Techniques that identify anomalies important enough for developers to fix (actionable alerts) may increase the usefulness of ASA in practice.

Objective

The goal of this work is to synthesize available research results to inform evidence-based selection of actionable alert identification techniques (AAIT).

Method

Relevant studies about AAITs were gathered via a systematic literature review.

Results

We selected 21 peer-reviewed studies of AAITs. The techniques use alert type selection; contextual information; data fusion; graph theory; machine learning; mathematical and statistical models; or dynamic detection to classify and prioritize actionable alerts. All of the AAITs are evaluated via an example with a variety of evaluation metrics.

Conclusion

The selected studies support (with varying strength), the premise that the effective use of ASA is improved by supplementing ASA with an AAIT. Seven of the 21 selected studies reported the precision of the proposed AAITs. The two studies with the highest precision built models using the subject program’s history. Precision measures how well a technique identifies true actionable alerts out of all predicted actionable alerts. Precision does not measure the number of actionable alerts missed by an AAIT or how well an AAIT identifies unactionable alerts. Inconsistent use of evaluation metrics, subject programs, and ASAs in the selected studies preclude meta-analysis and prevent the current results from informing evidence-based selection of an AAIT. We propose building on an actionable alert identification benchmark for comparison and evaluation of AAIT from literature on a standard set of subjects and utilizing a common set of evaluation metrics.     

MC Sandbox: Devising a tool for method-user-centered method configuration

Context

Method engineering approaches are often based on the assumption that method users are able to explicitly express their situational method requirements. Similar to systems requirements, method requirements are often vague and hard to explicate. In this paper we address the issue of involving method users early in method configuration. This is done through borrowing ideas from user-centered design and prototyping, and implementing them on the method engineering layer.

Objective

We design a computerized tool, MC Sandbox, to capture method requirements through the use of method-user-centered method configuration, hence bridging the gap between systems developers’ and method engineers’ understanding of and expectations on a situational method.

Method

The research method adopted can be characterized as multi-grounded action research. Our implementation of multi-grounded action research follows the traditional ‘canonical’ action research method, which has cycles of diagnosing, action planning, action taking, evaluating, and specifying learning. The research project comprised three such action research cycles where 10 action cases were performed.

Results

MC Sandbox has proven useful in eliciting and negotiating method requirements in a continuously ongoing dialog between the method users and the method engineers during configuration workshops. The results also show that the method engineer role rotated among the systems developers and that they were indeed committed to the negotiated methods during the systems development projects.

Conclusion

It is possible for method users to actively participate and construct suitable situational methods if they are provided with appropriate high-level modelling concepts, such as method components, configuration packages and configuration templates. This way, the project members’ understanding of the current development practice develops incrementally, both in terms of understanding the needs and available method support. In addition, both method requirements and commitments are made explicit, which are important aspects when working with method configuration from a collaboration point of view.     

Mutant generation for embedded systems using kernel-based software and hardware fault simulation

Context

Mutation testing is a fault-injection-based technique to help testers generate test cases for detecting specific and predetermined types of faults.

Objective

Before mutation testing can be effectively applied to embedded systems, traditional mutation testing needs to be modified. To inject a fault into an embedded system without causing any system failure or hardware damage is a challenging task as it requires some knowledge of the underlying layers such as the kernel and the corresponding hardware.

Method

We propose a set of mutation operators for embedded systems using kernel-based software and hardware fault simulation. These operators are designed for software developers so that they can use the mutation technique to test the entire system after the software is integrated with the kernel and hardware devices.

Results

A case study on a programmable logic controller for a digital reactor protection system in a nuclear power plant is conducted. Our results suggest that the proposed mutation operators are useful for fault-injection and this is evidenced by the fact that faults not injected by us were discovered in the subject software as a result of the case study.

Conclusion

We conclude that our mutation operators are useful for integration testing of an embedded system.     

Usability evaluation methods for the web: A systematic mapping study

Context

In recent years, many usability evaluation methods (UEMs) have been employed to evaluate Web applications. However, many of these applications still do not meet most customers’ usability expectations and many companies have folded as a result of not considering Web usability issues. No studies currently exist with regard to either the use of usability evaluation methods for the Web or the benefits they bring.

Objective

The objective of this paper is to summarize the current knowledge that is available as regards the usability evaluation methods (UEMs) that have been employed to evaluate Web applications over the last 14 years.

Method

A systematic mapping study was performed to assess the UEMs that have been used by researchers to evaluate Web applications and their relation to the Web development process. Systematic mapping studies are useful for categorizing and summarizing the existing information concerning a research question in an unbiased manner.

Results

The results show that around 39% of the papers reviewed reported the use of evaluation methods that had been specifically crafted for the Web. The results also show that the type of method most widely used was that of User Testing. The results identify several research gaps, such as the fact that around 90% of the studies applied evaluations during the implementation phase of the Web application development, which is the most costly phase in which to perform changes. A list of the UEMs that were found is also provided in order to guide novice usability practitioners.

Conclusions

From an initial set of 2703 papers, a total of 206 research papers were selected for the mapping study. The results obtained allowed us to reach conclusions concerning the state-of-the-art of UEMs for evaluating Web applications. This allowed us to identify several research gaps, which subsequently provided us with a framework in which new research activities can be more appropriately positioned, and from which useful information for novice usability practitioners can be extracted.     

Research synthesis in software engineering: A tertiary study

Context

Comparing and contrasting evidence from multiple studies is necessary to build knowledge and reach conclusions about the empirical support for a phenomenon. Therefore, research synthesis is at the center of the scientific enterprise in the software engineering discipline.

Objective

The objective of this article is to contribute to a better understanding of the challenges in synthesizing software engineering research and their implications for the progress of research and practice.

Method

A tertiary study of journal articles and full proceedings papers from the inception of evidence-based software engineering was performed to assess the types and methods of research synthesis in systematic reviews in software engineering.

Results

As many as half of the 49 reviews included in the study did not contain any synthesis. Of the studies that did contain synthesis, two thirds performed a narrative or a thematic synthesis. Only a few studies adequately demonstrated a robust, academic approach to research synthesis.

Conclusion

We concluded that, despite the focus on systematic reviews, there is limited attention paid to research synthesis in software engineering. This trend needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.     

The longitudinal, chronological case study research strategy: A definition, and an example from IBM Hursley Park

Context

There is surprisingly little empirical software engineering research (ESER) that has analysed and reported the rich, fine-grained behaviour of phenomena over time using qualitative and quantitative data. The ESER community also increasingly recognises the need to develop theories of software engineering phenomena e.g. theories of the actual behaviour of software projects at the level of the project and over time.

Objective

To examine the use of the longitudinal, chronological case study (LCCS) as a research strategy for investigating the rich, fine-grained behaviour of phenomena over time using qualitative and quantitative data.

Method

Review the methodological literature on longitudinal case study. Define the LCCS and demonstrate the development and application of the LCCS research strategy to the investigation of Project C, a software development project at IBM Hursley Park. Use the study to consider prospects for LCCSs, and to make progress on a theory of software project behaviour.

Results

LCCSs appear to provide insights that are hard to achieve using existing research strategies, such as the survey study. The LCCS strategy has basic requirements that data is time-indexed, relatively fine-grained and collected contemporaneous to the events to which the data refer. Preliminary progress is made on a theory of software project behaviour.

Conclusion

LCCS appears well suited to analysing and reporting rich, fine-grained behaviour of phenomena over time.     

Migration of information systems in the Italian industry: A state of the practice survey

Context

Software migration—and in particular migration towards the Web and towards distributed architectures—is a challenging and complex activity, and has been particularly relevant in recent years, due to the large number of migration projects the industry had to face off because of the increasing pervasiveness of the Web and of mobile devices.

Objective

This paper reports a survey aimed at identifying the state-of-the-practice of the Italian industry for what concerns the previous experiences in software migration projects—specifically concerning information systems—the adopted tools and the emerging needs and problems.

Method

The study has been carried out among 59 Italian Information Technology companies, and for each company a representative person had to answer an on-line questionnaire concerning migration experiences, pieces of technology involved in migration projects, adopted tools, and problems occurred during the project.

Results

Indicate that migration—especially towards the Web—is highly relevant for Italian IT companies, and that companies tend to increasingly adopt free and open source solutions rather than commercial ones. Results also indicate that the adoption of specific tools for migration is still very limited, either because of the lack of skills and knowledge, or due to the lack of mature and adequate options.

Conclusions

Findings from this survey suggest the need for further technology transfer between academia and industry for the purpose of favoring the adoption of software migration techniques and tools.     

Drivers of agile software development use: Dialectic interplay between benefits and hindrances

Context

Agile software development with its emphasis on producing working code through frequent releases, extensive client interactions and iterative development has emerged as an alternative to traditional plan-based software development methods. While a number of case studies have provided insights into the use and consequences of agile, few empirical studies have examined the factors that drive the adoption and use of agile.

Objective

We draw on intention-based theories and a dialectic perspective to identify factors driving the use of agile practices among adopters of this software development methodology.

Method

Data for the study was gathered through an anonymous online survey of software development professionals. We requested participation from members of a selected list of online discussion groups, and received 98 responses.

Results

Our analyses reveal that subjective norm and training play a significant role in influencing software developers’ use of agile processes and methods, while perceived benefits and perceived limitations are not primary drivers of agile use among adopters. Interestingly, perceived benefit emerges as a significant predictor of agile use only if adopters face hindrances to their agile practices.

Conclusion

We conclude that research in the adoption of software development innovations should examine the effects of both enabling and detracting factors and the interactions between them. Since training, subjective norm, and the interplay between perceived benefits and perceived hindrances appear to be key factors influencing the adoption of agile methods, researchers can focus on how to (a) perform training on agile methods more effectively, (b) facilitate the dialog between developers and managers about perceived benefits and hindrances, and (c) capitalize on subjective norm to publicize the benefits of agile methods within an organization. Further, when managing the transition to new software development methods, we recommend that practitioners adapt their strategies and tactics contingent on the extent of perceived hindrances to the change.     

Resolving unwanted couplings through interactive exploration of co-evolving software entities - An experience report

Context

Frequent changes to groups of software entities belonging to different parts of the system may indicate unwanted couplings between those parts. Visualizations of co-changing software entities have been proposed to help developers identify unwanted couplings. Identifying unwanted couplings, however, is only the first step towards an important goal of a software architect: to improve the decomposition of the software system. An in-depth analysis of co-changing entities is needed to understand the underlying reasons for co-changes, and also determine how to resolve the issues.

Objective

In this paper we discuss how interactive visualizations can support the process of analyzing the identified unwanted couplings.

Method

We applied a tool that interactively visualizes software evolution in 10 working sessions with architects and developers of a large embedded software system having a development history of more than a decade.

Results

The participants of the working sessions were overall very positive about their experiences with the interactive visualizations. In 70% of the cases investigated, a decision could be taken on how to resolve the unwanted couplings.

Conclusion

Our experience suggests that interactive visualization not only helps to identify unwanted couplings but it also helps experts to reason about and resolve them.     

Factors associated with quad bike loss of control events in agriculture

Objective

To determine personal and workplace factors associated with quad bike loss of control events (LCEs) on New Zealand farms.

Methods

Rural community databases were used to sample 130 farmers and farm employees (workers). Fieldwork and survey investigated for prevalence of LCEs; farm type; farm terrain; personal measures; and vehicle driving exposures.

Results

Seventy nine workers (61%) described a total of 200 LCEs. Increased driver height, increased body mass, non-flat farm terrain, increased driving speed and distance, and greater whole body vibration exposure were significantly associated with LCEs.

Conclusions

Taller and heavier drivers of quad bikes should be particularly vigilant for risk of an LCE. Vehicle speed, distance driven and choice of driving routes over difficult terrain are potentially modifiable factors which have behavioural components and should be considered as management strategies for reducing risk of on-farm quad bike LCEs.

Relevance to industry

Quad bike accidents are a considerable problem in agriculture. This research has identified a number of physical and driving factors that should be considered in the management strategies for reducing risk of on-farm quad bike accidents.     

Using mapping studies as the basis for further research - A participant-observer case study

Context

We are strong advocates of evidence-based software engineering (EBSE) in general and systematic literature reviews (SLRs) in particular. We believe it is essential that the SLR methodology is used constructively to support software engineering research.

Objective

This study aims to assess the value of mapping studies which are a form of SLR that aims to identify and categorise the available research on a broad software engineering topic.

Method

We used a multi-case, participant-observer case study using five examples of studies that were based on preceding mapping studies. We also validated our results by contacting two other researchers who had undertaken studies based on preceding mapping studies and by assessing review comments related to our follow-on studies.

Results

Our original case study identified 11 unique benefits that can accrue from basing research on a preceding mapping study of which only two were case specific. We also identified nine problems associated with using preceding mapping studies of which two were case specific. These results were consistent with the information obtained from the validation activities. We did not find an example of an independent research group making use of a mapping study produced by other researchers.

Conclusion

Mapping studies can save time and effort for researchers and provide baselines to assist new research efforts. However, they must be of high quality in terms of completeness and rigour if they are to be a reliable basis for follow-on research.     

Design guidelines for software processes knowledge repository development

Context

Staff turnover in organizations is an important issue that should be taken into account mainly for two reasons:

1.

Employees carry an organization’s knowledge in their heads and take it with them wherever they go

2.

Knowledge accessibility is limited to the amount of knowledge employees want to share

Objective

The aim of this work is to provide a set of guidelines to develop knowledge-based Process Asset Libraries (PAL) to store software engineering best practices, implemented as a wiki.

Method

Fieldwork was carried out in a 2-year training course in agile development. This was validated in two phases (with and without PAL), which were subdivided into two stages: Training and Project.

Results

The study demonstrates that, on the one hand, the learning process can be facilitated using PAL to transfer software process knowledge, and on the other hand, products were developed by junior software engineers with a greater degree of independence.

Conclusion

PAL, as a knowledge repository, helps software engineers to learn about development processes and improves the use of agile processes.     

An automated framework for software test oracle

Context

One of the important issues of software testing is to provide an automated test oracle. Test oracles are reliable sources of how the software under test must operate. In particular, they are used to evaluate the actual results that produced by the software. However, in order to generate an automated test oracle, oracle challenges need to be addressed. These challenges are output-domain generation, input domain to output domain mapping, and a comparator to decide on the accuracy of the actual outputs.

Objective

This paper proposes an automated test oracle framework to address all of these challenges.

Method

I/O Relationship Analysis is used to generate the output domain automatically and Multi-Networks Oracles based on artificial neural networks are introduced to handle the second challenge. The last challenge is addressed using an automated comparator that adjusts the oracle precision by defining the comparison tolerance. The proposed approach was evaluated using an industry strength case study, which was injected with some faults. The quality of the proposed oracle was measured by assessing its accuracy, precision, misclassification error and practicality. Mutation testing was considered to provide the evaluation framework by implementing two different versions of the case study: a Golden Version and a Mutated Version. Furthermore, a comparative study between the existing automated oracles and the proposed one is provided based on which challenges they can automate.

Results

Results indicate that the proposed approach automated the oracle generation process 97% in this experiment. Accuracy of the proposed oracle was up to 98.26%, and the oracle detected up to 97.7% of the injected faults.

Conclusion

Consequently, the results of the study highlight the practicality of the proposed oracle in addition to the automation it offers.     

An experimental assessment of module documentation-based testing

Context

Testing a module that has memory using the black-box approach has been found to be expensive and relatively ineffective. Instead, testing without knowledge of the specifications (white-box approach) may not be effective in showing whether a program has been properly implemented as stated in its specifications. We propose instead a grey-box approach called Module Documentation-based Testing or MD-Test, the heart of which is an automatic generation of the test oracle from the external and internal views of the module.

Objective

This paper presents an empirical analysis and comparison of MD-Test against three existing testing tools.

Method

The experiment was conducted using a mutation-testing approach, in two phases that assess the capability of MD-Test in general and its capability of evaluating test results in particular.

Results

The results of the general assessment indicate that MD-Test is more effective than the other three tools under comparison, where it is able to detect all faults. The second phase of the experiment, which is significant to this study, compares the capabilities of MD-Test and JUnit-black using the test evaluation results. Likewise, an analysis of the test evaluation results shows that MD-Test is more effective and efficient, where MD-Test is able to detect at least the same number of faults as, or is at par with, the black-box approach.

Conclusion

It is concluded that test evaluation using grey-box approach is more effective and efficient that the black-box approach when testing a module that has memory.     

Code churn estimation using organisational and code metrics: An experimental comparison

Context

Source code revision control systems contain vast amounts of data that can be exploited for various purposes. For example, the data can be used as a base for estimating future code maintenance effort in order to plan software maintenance activities. Previous work has extensively studied the use of metrics extracted from object-oriented source code to estimate future coding effort. In comparison, the use of other types of metrics for this purpose has received significantly less attention.

Objective

This paper applies machine learning techniques to unveil predictors of yearly cumulative code churn of software projects on the basis of metrics extracted from revision control systems.

Method

The study is based on a collection of object-oriented code metrics, XML code metrics, and organisational metrics. Several models are constructed with different subsets of these metrics. The predictive power of these models is analysed based on a dataset extracted from eight open-source projects.

Results

The study shows that a code churn estimation model built purely with organisational metrics is superior to one built purely with code metrics. However, a combined model provides the highest predictive power.

Conclusion

The results suggest that code metrics in general, and XML metrics in particular, are complementary to organisational metrics for the purpose of estimating code churn.     

The situational factors that affect the software development process: Towards a comprehensive reference framework

Context

An optimal software development process is regarded as being dependent on the situational characteristics of individual software development settings. Such characteristics include the nature of the application(s) under development, team size, requirements volatility and personnel experience. However, no comprehensive reference framework of the situational factors affecting the software development process is presently available.

Objective

The absence of such a comprehensive reference framework of the situational factors affecting the software development process is problematic not just because it inhibits our ability to optimise the software development process, but perhaps more importantly, because it potentially undermines our capacity to ascertain the key constraints and characteristics of a software development setting.

Method

To address this deficiency, we have consolidated a substantial body of related research into an initial reference framework of the situational factors affecting the software development process. To support the data consolidation, we have applied rigorous data coding techniques from Grounded Theory and we believe that the resulting framework represents an important contribution to the software engineering field of knowledge.

Results

The resulting reference framework of situational factors consists of eight classifications and 44 factors that inform the software process. We believe that the situational factor reference framework presented herein represents a sound initial reference framework for the key situational elements affecting the software process definition.

Conclusion

In addition to providing a useful reference listing for the research community and for committees engaged in the development of standards, the reference framework also provides support for practitioners who are challenged with defining and maintaining software development processes. Furthermore, this framework can be used to develop a profile of the situational characteristics of a software development setting, which in turn provides a sound foundation for software development process definition and optimisation.     

A framework for analysis and design of software reference architectures

Context

A software reference architecture is a generic architecture for a class of systems that is used as a foundation for the design of concrete architectures from this class. The generic nature of reference architectures leads to a less defined architecture design and application contexts, which makes the architecture goal definition and architecture design non-trivial steps, rooted in uncertainty.

Objective

The paper presents a structured and comprehensive study on the congruence between context, goals, and design of software reference architectures. It proposes a tool for the design of congruent reference architectures and for the analysis of the level of congruence of existing reference architectures.

Method

We define a framework for congruent reference architectures. The framework is based on state of the art results from literature and practice. We validate our framework and its quality as analytical tool by applying it for the analysis of 24 reference architectures. The conclusions from our analysis are compared to the opinions of experts on these reference architectures documented in literature and dedicated communication.

Results

Our framework consists of a multi-dimensional classification space and of five types of reference architectures that are formed by combining specific values from the multi-dimensional classification space. Reference architectures that can be classified in one of these types have better chances to become a success. The validation of our framework confirms its quality as a tool for the analysis of the congruence of software reference architectures.

Conclusion

This paper facilitates software architects and scientists in the inception, design, and application of congruent software reference architectures. The application of the tool improves the chance for success of a reference architecture.