An empirical research agenda for understanding formal methods productivity

ContextFormal methods, and particularly formal verification, is becoming more feasible to use in the engineering of large highly dependable software-based systems, but so far has had little rigorous empirical study. Its artefacts and activities are different to those of conventional software engineering, and the nature and drivers of productivity for formal methods are not yet understood.ObjectiveTo develop a research agenda for the empirical study of productivity in software projects using formal methods and in particular formal verification. To this end we aim to identify research questions about productivity in formal methods, and survey existing literature on these questions to establish face validity of these questions. And further we aim to identify metrics and data sources relevant to these questions.MethodWe define a space of GQM goals as an investigative framework, focusing on productivity from the perspective of managers of projects using formal methods. We then derive questions for these goals using Easterbrook et al.’s (2008) taxonomy of research questions. To establish face validity, we document the literature to date that reflects on these questions and then explore possible metrics related to these questions. Extensive use is made of literature concerning the L4.verified project completed within NICTA, as it is one of the few projects to achieve code-level formal verification for a large-scale industrially deployed software system.ResultsWe identify more than thirty research questions on the topic in need of investigation. These questions arise not just out of the new type of project context, but also because of the different artefacts and activities in formal methods projects. Prior literature supports the need for research on the questions in our catalogue, but as yet provides little evidence about them. Metrics are identified that would be needed to investigate the questions. Thus although it is obvious that at the highest level concepts such as size, effort, rework and so on are common to all software projects, in the case of formal methods, measurement at the micro level for these concepts will exhibit significant differences.ConclusionsEmpirical software engineering for formal methods is a large open research field. For the empirical software engineering community our paper provides a view into the entities and research questions in this domain. For the formal methods community we identify some of the benefits that empirical studies could bring to the effective management of large formal methods projects, and list some basic metrics and data sources that could support empirical studies. Understanding productivity is important in its own right for efficient software engineering practice, but can also support future research on cost-effectiveness of formal methods, and on the emerging field of Proof Engineering.     

Defining a Requirements Process Improvement Model

Both software organisations and the academic community are aware that the requirements phase of software development is in need of further support. We address this problem by creating a specialised Requirements Capability Maturity Model (R-CMM¹). The model focuses on the requirements engineering process as defined within the established Software Engineering Institute’s (SEI’s) software process improvement framework. Our empirical work with software practitioners is a primary motivation for creating this requirements engineering process improvement model. Although all organisations in our study were involved in software process improvement (SPI), they all showed a lack of control over many requirement engineering activities.This paper describes how the requirements engineering (RE) process is decomposed and prioritised in accordance with maturity goals set by the SEI’s Software Capability Maturity Model (SW CMM). Our R-CMM builds on the SEI’s framework by identifying and defining recommended RE sub-processes that meet maturity goals. This new focus will help practitioners to define their RE process with a view to setting realistic goals for improvement.Sarah Beecham is a research fellow in the Department of Maths and Computing in The Open University in the UK. She is currently working on the EPSRC funded CRESTES project () looking into modelling resource estimation for long-lived software. She has recently completed her PhD for a program of work entitled “A Requirements-based Software Process Maturity Model”. Current research interests are in estimation for software evolution and maintenance and in the general areas of software process improvement. Her particular research interests are in empirical methods in software engineering and requirements engineering.Tracy Hall leads the Systems & Software Research Group in the Department of Computer Science at the University of Hertfordshire. She specialises in the empirical investigation of technical and non-technical issues within software engineering. During the past ten years Tracy has successfully collaborated with many companies on a variety of research projects. She is very active in the Empirical Software Engineering community and is regularly invited to talk about empirical methods both in the UK and abroad. Tracy is an accomplished researcher having published over twenty high quality journal papers.Austen Rainer Austen Rainer is a senior lecturer at the University of Hertfordshire. He studied for his PhD at Bournemouth University, in conjunction with IBM Hursley Park. His current research interests include open source software development, longitudinal case study research, and the credibility of empirical evidence for researchers and software practitioners.     

Links between the personalities,views and attitudes of software engineers

Context:Successful software development and management depends not only on the technologies, methods and processes employed but also on the judgments and decisions of the humans involved. These, in turn, are affected by the basic views and attitudes of the individual engineers.Objective:The objective of this paper is to establish if these views and attitudes can be linked to the personalities of software engineers.Methods:We summarize the literature on personality and software engineering and then describe an empirical study on 47 professional engineers in ten different Swedish software development companies. The study evaluated the personalities of these engineers via the IPIP 50-item five-factor personality test and prompted them on their attitudes towards and basic views on their professional activities.Results:We present extensive statistical analyses of their responses to show that there are multiple, significant associations between personality factors and software engineering attitudes. The tested individuals are more homogeneous in personality than a larger sample of individuals from the general population.Conclusion:Taken together, the methodology and personality test we propose and the associated statistical analyses can help find and quantify relations between complex factors in software engineering projects in both research and practice.     

Automated COSMIC Function Point measurement using a requirements engineering ontology

ContextThere are two interrelated difficulties in requirements engineering processes. First, free-format modelling practices in requirements engineering activities may lead to low quality artefacts and productivity problems. Second, the COSMIC Function Point Method is not yet widespread in the software industry because applying measurement rules to imprecise and ambiguous textual requirements is difficult and requires additional human measurement effort. This challenge is common to all functional size measurement methods.ObjectiveIn this study, alternative solutions have been investigated to address these two difficulties. Information created during the requirements engineering process is formalized as an ontology that also becomes a convenient model for transforming requirements into COSMIC Function Point Method concepts.MethodA method is proposed to automatically measure the functional size of software by using the designed ontology. The proposed method has been implemented as a software application and verified with real projects conducted within the ICT department of a leading telecommunications provider in Turkey.ResultsWe demonstrated a novel method to measure the functional size of software in COSMIC FP automatically. It is based on a newly developed requirements engineering ontology. Our proposed method has several advantages over other methods explored in previous research.ConclusionManual and automated measurement results are in agreement, and the tool is promising for the company under study and for the industry at large.     

Are you biting off more than you can chew? A case study on causes and effects of overscoping in large-scale software engineering

ContextScope management is a core part of software release management and often a key factor in releasing successful software products to the market. In a market-driven case, when only a few requirements are known a priori, the risk of overscoping may increase.ObjectiveThis paper reports on findings from a case study aimed at understanding overscoping in large-scale, market-driven software development projects, and how agile requirements engineering practices may affect this situation.MethodBased on a hypothesis of which factors that may be involved in an overscoping situation, semi-structured interviews were performed with nine practitioners at a large, market-driven software company. The results from the interviews were validated by six (other) practitioners at the case company via a questionnaire.ResultsThe results provide a detailed picture of overscoping as a phenomenon including a number of causes, root causes and effects, and indicate that overscoping is mainly caused by operating in a fast-moving market-driven domain and how this ever-changing inflow of requirements is managed. Weak awareness of overall goals, in combination with low development involvement in early phases, may contribute to ‘biting off’ more than a project can ‘chew’. Furthermore, overscoping may lead to a number of potentially serious and expensive consequences, including quality issues, delays and failure to meet customer expectations. Finally, the study indicates that overscoping occurs also when applying agile requirements engineering practices, though the overload is more manageable and perceived to result in less wasted effort when applying a continuous scope prioritization, in combination with gradual requirements detailing and a close cooperation within cross-functional teams.ConclusionThe results provide an increased understanding of scoping as a complex and continuous activity, including an analysis of the causes, effects, and a discussion on possible impact of agile requirements engineering practices to the issue of overscoping. The results presented in this paper can be used to identify potential factors to address in order to achieve a more realistic project scope.     

浅谈软件项目的需求分析

当今世界各行各业对软件的依赖程度急剧增长,而在规定的时间和预算内开发出可靠并满足用户需求的软件系统,对于许多开发者来说都是一件非常困难的事情。由于软件项目中的需求变更而导致软件项目失败的案例越来越多,所以如何高质量完成需求分析已经被许多软件公司列入了重要的流程化管理中。需求分析的任务还不是确定系统怎样完成它的工作,而仅仅是确定系统必须完成哪些工作,也就是对目标系统提出完整、准确、清晰、具体的要求。     

A theory of distances in software engineering

ContextCoordinating a software project across distances is challenging. Even without geographical and time zone distances, other distances within a project can cause communication gaps. For example, organisational and cognitive distances between product owners and development-near roles such as developers and testers can lead to differences in understanding and interpretation of the business requirements. Applying good software development practices, known to enhance alignment and coordination within development projects, can alleviate these challenges.ObjectiveThe aim of our research is to identify and describe underlying factors which can explain why certain practices support aligning and coordinating software development projects.MethodWe have inductively generated a theory analysing empirical data consisting of 15 interviews from 5 different companies. The systematic and iterative analysis was based on an initial hypothesis that distances affect development, and on results from previous research.ResultsWe present a theory of distances that explains how practices improve the communication within a project by impacting distances between people, activities and artefacts. We also present a theoretical model of how specific alignment practices affect different types of distances.ConclusionsThe results provide a basis for further research and can be used by software organisations to improve on software practice.     

Integrating agile software development into stage-gate managed product development

Agile methods have evolved as a bottom-up approach to software development. However, as the software in embedded products is only one part of development projects, agile methods must coexist with project management models typically of the stage-gate type. This paper presents a qualitative case study of two large independent software system projects that have used eXtreme Programming (XP) for software development within contexts of stage-gate project management models. The study is comprised of open ended interviews with managers as well as practitioners, followed by a structured, fully traceable, qualitative analysis. We conclude that it is possible to integrate XP in a gate model context. Key issues for success are the interfaces towards the agile subproject and management attitudes towards the agile approach. Editor: Marvin Zelkowitz     

The state of the art in automated requirements elicitation

ContextIn large software development projects a huge number of unstructured text documents from various stakeholders becomes available and needs to be analyzed and transformed into structured requirements. This elicitation process is known to be time-consuming and error-prone when performed manually by a requirements engineer. Consequently, substantial research has been done to automate the process through a plethora of tools and technologies.ObjectiveThis paper aims to capture the current state of automated requirements elicitation and derive future research directions by identifying gaps in the existing body of knowledge and through relating existing works to each other. More specifically, we are investigating the following research question: What is the state of the art in research covering tool support for automated requirements elicitation from natural language documents?MethodA systematic review of the literature in automated requirements elicitation is performed. Identified works are categorized using an analysis framework comprising tool categories, technological concepts and evaluation approaches. Furthermore, the identified papers are related to each other through citation analysis to trace the development of the research field.ResultsWe identified, categorized and related 36 relevant publications. Summarizing the observations we made, we propose future research to (1) investigate alternative elicitation paradigms going beyond a pure automation approach (2) compare the effects of different types of knowledge on elicitation results (3) apply comparative evaluation methods and multi-dimensional evaluation measures and (4) strive for a closer integration of research activities across the sub-fields of automatic requirements elicitation.ConclusionThrough the results of our paper, we intend to contribute to the Requirements Engineering body of knowledge by (1) conceptualizing an analysis framework for works in the area of automated requirements elicitation, going beyond former classifications (2) providing an extensive overview and categorization of existing works in this area (3) formulating concise directions for future research.     

A multi-case study of agile requirements engineering and the use of test cases as requirements

ContextIt is an enigma that agile projects can succeed ‘without requirements’ when weak requirements engineering is a known cause for project failures. While agile development projects often manage well without extensive requirements test cases are commonly viewed as requirements and detailed requirements are documented as test cases.ObjectiveWe have investigated this agile practice of using test cases as requirements to understand how test cases can support the main requirements activities, and how this practice varies.MethodWe performed an iterative case study at three companies and collected data through 14 interviews and two focus groups.ResultsThe use of test cases as requirements poses both benefits and challenges when eliciting, validating, verifying, and managing requirements, and when used as a documented agreement. We have identified five variants of the test-cases-as-requirements practice, namely de facto, behaviour-driven, story-test driven, stand-alone strict and stand-alone manual for which the application of the practice varies concerning the time frame of requirements documentation, the requirements format, the extent to which the test cases are a machine executable specification and the use of tools which provide specific support for the practice of using test cases as requirements.ConclusionsThe findings provide empirical insight into how agile development projects manage and communicate requirements. The identified variants of the practice of using test cases as requirements can be used to perform in-depth investigations into agile requirements engineering. Practitioners can use the provided recommendations as a guide in designing and improving their agile requirements practices based on project characteristics such as number of stakeholders and rate of change.     

Naming the pain in requirements engineering: A design for a global family of surveys and first results from Germany

ContextFor many years, we have observed industry struggling in defining a high quality requirements engineering (RE) and researchers trying to understand industrial expectations and problems. Although we are investigating the discipline with a plethora of empirical studies, they still do not allow for empirical generalisations.ObjectiveTo lay an empirical and externally valid foundation about the state of the practice in RE, we aim at a series of open and reproducible surveys that allow us to steer future research in a problem-driven manner.MethodWe designed a globally distributed family of surveys in joint collaborations with different researchers and completed the first run in Germany. The instrument is based on a theory in the form of a set of hypotheses inferred from our experiences and available studies. We test each hypothesis in our theory and identify further candidates to extend the theory by correlation and Grounded Theory analysis.ResultsIn this article, we report on the design of the family of surveys, its underlying theory, and the full results obtained from Germany with participants from 58 companies. The results reveal, for example, a tendency to improve RE via internally defined qualitative methods rather than relying on normative approaches like CMMI. We also discovered various RE problems that are statistically significant in practice. For instance, we could corroborate communication flaws or moving targets as problems in practice. Our results are not yet fully representative but already give first insights into current practices and problems in RE, and they allow us to draw lessons learnt for future replications.ConclusionOur results obtained from this first run in Germany make us confident that the survey design and instrument are well-suited to be replicated and, thereby, to create a generalisable empirical basis of RE in practice.     

Facilitating construction of safety cases from formal models in Event-B

ContextCertification of safety–critical software systems requires submission of safety assurance documents, e.g., in the form of safety cases. A safety case is a justification argument used to show that a system is safe for a particular application in a particular environment. Different argumentation strategies (informal and formal) are applied to determine the evidence for a safety case. For critical software systems, application of formal methods is often highly recommended for their safety assurance.ObjectiveThe objective of this paper is to propose a methodology that combines two activities: formalisation of system safety requirements of critical software systems for their further verification as well as derivation of structured safety cases from the associated formal specifications.MethodWe propose a classification of system safety requirements in order to facilitate the mapping of informally defined requirements into a formal model. Moreover, we propose a set of argument patterns that aim at enabling the construction of (a part of) a safety case from a formal model in Event-B.ResultsThe results reveal that the proposed classification-based mapping of safety requirements into formal models facilitates requirements traceability. Moreover, the provided detailed guidelines on construction of safety cases aim to simplify the task of the argument pattern instantiation for different classes of system safety requirements. The proposed methodology is illustrated by numerous case studies.ConclusionFirstly, the proposed methodology allows us to map the given system safety requirements into elements of the formal model to be constructed, which is then used for verification of these requirements. Secondly, it guides the construction of a safety case, aiming to demonstrate that the safety requirements are indeed met. Consequently, the argumentation used in such a constructed safety case allows us to support it with formal proofs and model checking results used as the safety evidence.     

Detecting Web requirements conflicts and inconsistencies under a model-based perspective

Web requirements engineering is an essential phase in the software project life cycle for the project results. This phase covers different activities and tasks that in many situations, depending on the analyst's experience or intuition, help getting accurate specifications. One of these tasks is the conciliation of requirements in projects with different groups of users. This article presents an approach for the systematic conciliation of requirements in big projects dealing with a model-based approach. The article presents a possible implementation of the approach in the context of the NDT (Navigational Development Techniques) Methodology and shows the empirical evaluation in a real project by analysing the improvements obtained with our approach. The paper presents interesting results that demonstrate that we can get a reduction in the time required to find conflicts between requirements, which implies a reduction in the global development costs.     

Applying a selection method to choose Quality Attribute Techniques

ContextSoftware products have requirements on software quality attributes such as safety and performance. Development teams use various specific techniques to achieve these quality requirements. We call these “Quality Attribute Techniques” (QATs). QATs are used to identify, analyse and control potential product quality problems. Although QATs are widely used in practice, there is no systematic approach to represent, select, and integrate them in existing approaches to software process modelling and tailoring.ObjectiveThis research aims to provide a systematic approach to better select and integrate QATs into tailored software process models for projects that develop products with specific product quality requirements.MethodA selection method is developed to support the choice of appropriate techniques for any quality attribute, across the lifecycle. The selection method is based on three perspectives: (1) risk management; (2) process integration; and (3) cost/benefit using Analytic Hierarchy Process (AHP). An industry case study is used to validate the feasibility and effectiveness of applying the selection method.ResultsThe case study demonstrates that the selection method provides a more methodological and effective approach to choose QATs for projects that target a specific quality attribute, compared to the ad hoc selection performed by development teams.ConclusionThe proposed selection method can be used to systematically choose QATs for projects to target specific product qualities throughout the software development lifecycle.     

An extensive empirical study on C++ concurrency constructs

ContextNowadays concurrent programming is in large demand. The inherent support for concurrency is therefore increasingly important in programming languages. As for C++, an abundance of standard concurrency constructs have been supported since C++11. However, to date there is little work investigating how these constructs are actually used in developing real software.ObjectiveIn this paper, we perform an extensive empirical study to investigate the actual adoption of C++ concurrency constructs in open-source applications, with the goal to provide useful information for language designers and practitioners to improve the design and use of concurrency constructs.MethodWe first conduct a survey to understand the developers’ perception of concurrency constructs. Then, we analyze 492 open-source concurrent applications, comprising 131 million lines of C++ code, to collect the data of the practical adoption of concurrency constructs. Finally, we perform statistical analysis on the collected data and get the quantitative analysis results.ResultsUsing the experimental data, we uncover many interesting findings with respect to the usage of concurrency constructs, the management of synchronization, the relationship between standard concurrency constructs and third-party concurrency constructs, and the difference of applications in using concurrency constructs. For example, we find that: (1) thread-based constructs are significantly more often used to write concurrent programs than atomics-based constructs and task-based constructs; (2) lock-based constructs are significantly more often used to manage synchronization than lock-free constructs and blocking constructs; (3) developers in most projects do not move from third-party concurrency constructs to standard concurrency constructs; and (4) small-size applications introduce concurrency constructs more intensively and more quickly than medium-size applications and large-size applications.ConclusionsThis is the first extensive empirical study on C++ concurrency constructs. The results of this paper should be helpful for designing, teaching, and using C++ concurrency constructs.     

Towards a competency model for requirements analysts

Requirements engineering is a software development discipline, executed by requirements analysts (RAs), that includes requirements elicitation, analysis, specification and validation. Its successful outcome is very often essential to overall project success. However, there is a lack of systematically conducted empirical research on the competencies of RAs. This paper addressed this gap by conducting 64 interviews at eight major North American and European financial services companies. Our qualitative research design follows an interpretive approach and uses critical incident technique. We develop a competency model, which specifies 16 critical competencies, and integrates contextual and situational factors as well as results variables. ‘Consulting others’, ‘Testing assumptions and investigating’ and ‘Explaining concepts and opinions’ were the most frequently identified competencies. This indicates that for an effective analyst, close interaction and communication with customers is indeed crucial – but of equally importance is the critical questioning of the expressed needs. Surprisingly, applying specific tools and advanced techniques did not seem to play a significant role from the interviewees' perspective. This study contributes to theory as it is the first to elaborate a competency model for RAs. It also provides a foundation for the development of competency‐based training in companies and universities.     

Agent interaction for bioinformatics data management

As genome projects produce increasingly large quantities of sequence data, fast and reliable sequence analysis methods are required. Basic methods for comparing pairs of sequences or detecting patterns are well-developed, and now the key problem in analyzing this genomic data is how to integrate the software and primary databases in a flexible and robust way. The wide range of available programs conform to very different input, output, and processing requirements, typically with little consideration given to issues of integration. Key to addressing these issues appropriately is not to consider them as a result of the biological domain, but instead as an information processing problem that suggests nothing as much as an agentbased approach. In this paper, we introduce GeneWeaver, a multi-agent system for bioinformatics, and describe in detail the agent interactions which allow the integration and management of analysis methods and data. The system does not offer new methods but instead manages existing databases and analysis tools in an effective and flexible way, and facilitates easy and dynamic growth.     

Interpretative case studies on agile team productivity and management

ContextThe management of software development productivity is a key issue in software organizations, where the major drivers are lower cost and shorter time-to-market. Agile methods, including Extreme Programming and Scrum, have evolved as “light” approaches that simplify the software development process, potentially leading to increased team productivity. However, little empirical research has examined which factors do have an impact on productivity and in what way, when using agile methods.ObjectiveOur objective is to provide a better understanding of the factors and mediators that impact agile team productivity.MethodWe have conducted a multiple-case study for 6 months in three large Brazilian companies that have been using agile methods for over 2 years. We have focused on the main productivity factors perceived by team members through interviews, documentation from retrospectives, and non-participant observation.ResultsWe developed a novel conceptual framework, using thematic analysis to understand the possible mechanisms behind such productivity factors. Agile team management was found to be the most influential factor in achieving agile team productivity. At the intra-team level, the main productivity factors were team design (structure and work allocation) and member turnover. At the inter-team level, the main productivity factors were how well teams could be effectively coordinated by proper interfaces and other dependencies and avoiding delays in providing promised software to dependent teams.ConclusionTeams should be aware of the influence and magnitude of turnover, which has been shown negative for agile team productivity. Team design choices remain an important factor impacting team productivity, even more pronounced on agile teams that rely on teamwork and people factors. The intra-team coordination processes must be adjusted to enable productive work by considering priorities and pace between teams. Finally, the revised conceptual framework for agile team productivity supports further tests through confirmatory studies.     

Challenges of shared decision-making: A multiple case study of agile software development

ContextAgile software development changes the nature of collaboration, coordination, and communication in software projects.ObjectiveOur objective was to understand the challenges of shared decision-making in agile software development teams.MethodWe designed a multiple case study consisting of four projects in two software product companies that recently adopted Scrum. We collected data in semi-structured interviews, through participant observations, and from process artifacts.ResultsWe identified three main challenges to shared decision-making in agile software development: alignment of strategic product plans with iteration plans, allocation of development resources, and performing development and maintenance tasks in teams.ConclusionAgile software development requires alignment of decisions on the strategic, tactical, and operational levels in order to overcome these challenges. Agile development also requires a transition from specialized skills to redundancy of functions and from rational to naturalistic decision-making. This takes time; the case companies needed from one to two years to change from traditional, hierarchical decision-making to shared decision-making in software development projects.