Investigating the penalty reward calculus of software users and its impact on requirements prioritization

ContextThe current requirements engineering techniques for prioritization of software requirements implicitly assume that each user requirement will have an independent and symmetric impact on user satisfaction. For example, it is assumed that implementing a high priority user requirement will positively impact his satisfaction and not implementing a high priority user requirement will negatively impact his satisfaction. Further, the impacts of implementing multiple user requirements on his satisfaction are expected to be additive. But is this always the case?ObjectiveThis paper empirically examines whether the assumptions of symmetric and multiplicative impacts of user requirements on his satisfaction are valid. Further, the study assesses the relative efficacy of 5 methods of requirements prioritization in managing these effects as reflected by the user satisfaction with the prioritized requirement sets.MethodTo test for existence and mitigation of asymmetric effects an adaptation of the widely accepted PRCA (Penalty Reward Contrast Analysis) method was used for 5 requirements prioritization techniques. To test for existence and mitigation of multiplicative effects MHMR (Moderated Hierarchical Multiple Regression) a well-accepted technique for testing interaction effects was used.ResultsBoth asymmetric and multiplicative effects of software requirements on user satisfaction were observed for requirements prioritized using all 5 requirements prioritization methods raising questions about the efficacy of present day requirements prioritization techniques. Further, the results of the experiment led to proposing a new method for requirements prioritization for managing these effects.ConclusionThe study empirically demonstrates the complexities of prioritizing software requirements and calls for a new generation of methods to address them. Understanding and resolving these complexities will enable software providers to conserve resources by enabling them to parsimoniously selecting only those requirements for implementation in the software product that have maximum incremental impact on user satisfaction.     

A systematic literature review on requirement prioritization techniques and their empirical evaluation

[Context and Motivation] Many requirements prioritization approaches have been proposed, however not all of them have been investigated empirically in real-life settings. As a result, our knowledge of their applicability and actual use is incomplete. [Question/problem] A 2007 systematic review on requirements prioritization mapped out the landscape of proposed prioritization approaches and their prioritization criteria. To understand how this sub-field of requirements engineering has developed since 2007 and what evidence has been accumulated through empirical evaluations, we carried out a literature review that takes as input publications published between 2007 and 2019. [Principle ideas/results] We evaluated 102 papers that proposed and/or evaluated requirements prioritization methods. Our results show that the newly proposed requirements prioritization methods tend to use as basis fuzzy logic and machine learning algorithms. We also concluded that the Analytical Hierarchy Process is the most accurate and extensively used requirement prioritization method in industry. However, scalability is still its major limitation when requirements are large in number. We have found that machine learning has shown potential to deal with this limitation. Last, we found that experiments were the most used research method to evaluate the various aspects of the proposed prioritization approaches. [Contribution] This paper identified and evaluated requirements prioritization techniques proposed between 2007 and 2019, and derived some trends. Limitations of the proposals and implications for research and practice are identified as well.     

A systematic literature review of software requirements prioritization research

ContextDuring requirements engineering, prioritization is performed to grade or rank requirements in their order of importance and subsequent implementation releases. It is a major step taken in making crucial decisions so as to increase the economic value of a system.ObjectiveThe purpose of this study is to identify and analyze existing prioritization techniques in the context of the formulated research questions.MethodSearch terms with relevant keywords were used to identify primary studies that relate requirements prioritization classified under journal articles, conference papers, workshops, symposiums, book chapters and IEEE bulletins.Results73 Primary studies were selected from the search processes. Out of these studies; 13 were journal articles, 35 were conference papers and 8 were workshop papers. Furthermore, contributions from symposiums as well as IEEE bulletins were 2 each while the total number of book chapters amounted to 13.ConclusionPrioritization has been significantly discussed in the requirements engineering domain. However, it was generally discovered that, existing prioritization techniques suffer from a number of limitations which includes: lack of scalability, methods of dealing with rank updates during requirements evolution, coordination among stakeholders and requirements dependency issues. Also, the applicability of existing techniques in complex and real setting has not been reported yet.     

基于解释结构模型的需求最高优先级设定方法

对用户需求进行排序是软件项目开发的基础。针对当前需求优先级排序方法,将用户需求放在同一层次上进行比较设定优先级,而对需求层次性和对多个需求整体赋予优先级则考虑的不多。从需求的层次性出发,用解释结构模型对需求进行分层处理;然后对顶层需求的相关需求集进行交运算,整体设定需求的最高优先级;最后给出案例分析。结果表明,该方法能够突出需求层次性,直接生成最高优先级需求集,提高需求优先级设定的效率。     

Interactive requirements prioritization using a genetic algorithm

ContextThe order in which requirements are implemented affects the delivery of value to the end-user, but it also depends on technical constraints and resource availability. The outcome of requirements prioritization is a total ordering of requirements that best accommodates the various kinds of constraints and priorities. During requirements prioritization, some decisions on the relative importance of requirements or the feasibility of a given implementation order must necessarily resort to a human (e.g., the requirements analyst), possessing the involved knowledge.ObjectiveIn this paper, we propose an Interactive Genetic Algorithm (IGA) that includes incremental knowledge acquisition and combines it with the existing constraints, such as dependencies and priorities. We also assess the performance of the proposed algorithm.MethodThe validation of IGA was conducted on a real case study, by comparing the proposed algorithm with the state of the art, interactive prioritization technique Incomplete Analytic Hierarchy Process (IAHP).ResultsThe proposed method outperforms IAHP in terms of effectiveness, efficiency and robustness to decision maker errors.ConclusionIGA produces a good approximation of the reference requirements ranking, requiring an acceptable manual effort and tolerating a reasonable human error rate.     

Operational release planning in large-scale Scrum with multiple stakeholders – A longitudinal case study at F-Secure Corporation

ContextThe analysis and selection of requirements are important parts of any release planning process. Previous studies on release planning have focused on plan-driven optimization models. Unfortunately, solving the release planning problem mechanistically is difficult in an agile development context.ObjectiveWe describe how a release planning method was employed in two case projects in F-Secure, a large Finnish software company. We identify the benefits which the projects gained from the method, and analyze challenges in the cases and improvements made to the method during the case projects.MethodWe observed five release planning events and four retrospectives and we conducted surveys in the first two events. We conducted six post-project interviews. We conjoined the observation notes, survey results and interviews and analyzed them qualitatively and quantitatively.ResultsThe focal point of the method was release planning events where the whole project organization gathered to plan the next release. The planning was conducted by the development teams in close collaboration with each other and with the other stakeholders. We identified ten benefits which included improved communication, transparency, dependency management and decision making. We identified nine challenges which included the lacking preparation and prioritization of requirements, unrealistic schedules, insufficient architectural planning and lacking agile mindset. The biggest improvements to the method were the introduction of frequent status checks and a big visible planning status board.ConclusionThe release planning method ameliorated many difficult characteristics of the release planning problem but its efficiency was negatively affected by the performing organization that was in transition from a plan-driven to an agile development mindset. Even in this case the benefits clearly outweighed the challenges and the method enabled the early identification of the issues in the project.     

Are you biting off more than you can chew? A case study on causes and effects of overscoping in large-scale software engineering

ContextScope management is a core part of software release management and often a key factor in releasing successful software products to the market. In a market-driven case, when only a few requirements are known a priori, the risk of overscoping may increase.ObjectiveThis paper reports on findings from a case study aimed at understanding overscoping in large-scale, market-driven software development projects, and how agile requirements engineering practices may affect this situation.MethodBased on a hypothesis of which factors that may be involved in an overscoping situation, semi-structured interviews were performed with nine practitioners at a large, market-driven software company. The results from the interviews were validated by six (other) practitioners at the case company via a questionnaire.ResultsThe results provide a detailed picture of overscoping as a phenomenon including a number of causes, root causes and effects, and indicate that overscoping is mainly caused by operating in a fast-moving market-driven domain and how this ever-changing inflow of requirements is managed. Weak awareness of overall goals, in combination with low development involvement in early phases, may contribute to ‘biting off’ more than a project can ‘chew’. Furthermore, overscoping may lead to a number of potentially serious and expensive consequences, including quality issues, delays and failure to meet customer expectations. Finally, the study indicates that overscoping occurs also when applying agile requirements engineering practices, though the overload is more manageable and perceived to result in less wasted effort when applying a continuous scope prioritization, in combination with gradual requirements detailing and a close cooperation within cross-functional teams.ConclusionThe results provide an increased understanding of scoping as a complex and continuous activity, including an analysis of the causes, effects, and a discussion on possible impact of agile requirements engineering practices to the issue of overscoping. The results presented in this paper can be used to identify potential factors to address in order to achieve a more realistic project scope.     

RePizer: a framework for prioritization of software requirements

The standard software development life cycle heavily depends on requirements elicited from stakeholders. Based on those requirements, software development is planned and managed from its inception phase to closure. Due to time and resource constraints, it is imperative to identify the high-priority requirements that need to be considered first during the software development process. Moreover, existing prioritization frameworks lack a store of historical data useful for selecting the most suitable prioritization technique of any similar project domain. In this paper, we propose a framework for prioritization of software requirements, called RePizer, to be used in conjunction with a selected prioritization technique to rank software requirements based on defined criteria such as implementation cost. RePizer assists requirements engineers in a decision-making process by retrieving historical data from a requirements repository. RePizer also provides a panoramic view of the entire project to ensure the judicious use of software development resources. We compared the performance of RePizer in terms of expected accuracy and ease of use while separately adopting two different prioritization techniques, planning game (PG) and analytical hierarchy process (AHP). The results showed that RePizer performed better when used in conjunction with the PG technique.     

Improving test efficiency through system test prioritization

Software testing is an expensive process consuming at least 50% of the total development cost. Among the types of testing, system testing is the most expensive and complex. Companies are frequently faced with budgetary constraints, which may limit their ability to effectively complete testing efforts before delivering a software product. We build upon prior test case prioritization research and present a system-level approach to test case prioritization called Prioritization of Requirements for Test (PORT). PORT prioritizes system test cases based on four factors for each requirement: customer priority, implementation complexity, fault proneness, and requirements volatility. Test cases for requirements with higher priority based upon a weighted average of these factors are executed earlier in system test. An academic feasibility study and three post hoc industrial studies were conducted. Results indicate that PORT can be used to improve the rate of failure detection when compared with a random and operational profile-driven random approach. Furthermore, we investigated the contribution of the prioritization factors towards the improved rate of failure detection and found customer priority was the most significant contributor. Tool support is provided for the PORT scheme which allows for automatic collection of the four factor values and the resultant test case prioritization.     

Robustness indices and robust prioritization in QFD

The prioritization of engineering characteristics (ECs) provides an important basis for decision-making in QFD. However, the prioritization results in the conventional QFD may be misleading since it does not consider the uncertainty of input information. This paper develops two robustness indices and proposes the notion of robust prioritization that ensures the EC prioritization to be robust against the uncertainty. The robustness indices consider robustness from two perspectives, namely, the absolute ranking of ECs and the priority relationship among ECs. Based on the two indices, robust prioritization seeks to identify a set of ECs or a priority relationship among ECs in such a way that the result of robust prioritization is stable despite the uncertainty. Finally, the proposed robustness indices and robust prioritization are demonstrated in a case study conducted on the ADSL-based high-speed internet service.     

Tool-supported requirements prioritization: Comparing the AHP and CBRank methods

Requirements prioritization aims at identifying the most important requirements for a software system, a crucial step when planning for system releases and deciding which requirements to implement in each release. Several prioritization methods and supporting tools have been proposed so far. How to evaluate their properties, with the aim of supporting the selection of the most appropriate method for a specific project, is considered a relevant question.In this paper, we present an empirical study aiming at evaluating two state-of-the art tool-supported requirements prioritization methods, AHP and CBRank. We focus on three measures: the ease of use, the time-consumption and the accuracy. The experiment has been conducted with 23 experienced subjects on a set of 20 requirements from a real project. Results indicate that for the first two characteristics CBRank overcomes AHP, while for the accuracy AHP performs better than CBRank, even if the resulting ranks from the two methods are very similar. The majority of the users found CBRank the “overall best” method.     

An empirical study on the implementation and evaluation of a goal-driven software development risk management model

ContextBuilding a quality software product in the shortest possible time to satisfy the global market demand gives an enterprise a competitive advantage. However, uncertainties and risks exist at every stage of a software development project. These can have an extremely high influence on the success of the final software product. Early risk management practice is effective to manage such risks and contributes effectively towards the project success.ObjectiveDespite risk management approaches, a detailed guideline that explains where to integrate risk management activities into the project is still missing. Little effort has been directed towards the evaluation of the overall impact of a risk management method. We present a Goal-driven Software Development Risk Management Model (GSRM) and its explicit integration into the requirements engineering phase and an empirical investigation result of applying GSRM into a project.MethodWe combine the case study method with action research so that the results from the case study directly contribute to manage the studied project risks and to identify ways to improve the proposed methodology. The data is collected from multiple sources and analysed both in a qualitative and quantitative way.ResultsWhen risk factors are beyond the control of the project manager and project environment, it is difficult to control these risks. The project scope affects all the dimensions of risk. GSRM is a reasonable risk management method that can be employed in an industrial context. The study results have been compared against other study results in order to generalise findings and identify contextual factors.ConclusionA formal early stage risk management practice provides early warning related to the problems that exists in a project, and it contributes to the overall project success. It is not necessary to always consider budget and schedule constraints as top priority. There exist issues such as requirements, change management, and user satisfaction which can influence these constraints.     

Towards the prioritization of system test cases

During software development, companies are frequently faced with lack of time and resources, which limits their ability to effectively complete testing efforts. This paper presents a system‐level, value‐driven approach to test case prioritization called the Prioritization of Requirements for Test (PORT). PORT involves analysing and assigning value to each requirement using the following four factors: requirements volatility, customer priority, implementation complexity, and fault proneness. System test cases are prioritized such that the test cases for requirements with higher priority are executed earlier during system test. PORT was applied to four student team projects as well as an industrial case study. The results show that PORT improves the rate of detection of severe failures over random prioritization. Additionally, the results indicate that customer priority was the most important contributor towards improved rate of failure detection. Copyright © 2013 John Wiley & Sons, Ltd.     

A tool supporting root cause analysis for synchronous retrospectives in distributed software teams

ContextRoot cause analysis (RCA) is a useful practice for software project retrospectives, and is typically carried out in synchronous collocated face-to-face meetings. Conducting RCA with distributed teams is challenging, as face-to-face meetings are infeasible. Lack of adequate real-time tool support exacerbates this problem. Furthermore, there are no empirical studies on using RCA in synchronous retrospectives of geographically distributed teams.ObjectiveThis paper presents a real-time cloud-based software tool (ARCA-tool) we developed to support RCA in distributed teams and its initial empirical evaluation. The feasibility of using RCA with distributed teams is also evaluated.MethodWe compared our tool with 35 existing RCA software tools. We conducted field studies of four distributed agile software teams at two international software product companies. The teams conducted RCA collaboratively in synchronous retrospective meetings by using the tool we developed. We collected the data using observations, interviews and questionnaires.ResultsComparison revealed that none of the existing 35 tools matched all the features of our ARCA-tool. The team members found ARCA-tool to be an essential part of their distributed retrospectives. They considered the software as efficient and very easy to learn and use. Additionally, the team members perceived RCA to be a vital part of the retrospectives. In contrast to the prior retrospective practices of the teams, the introduced RCA method was evaluated as efficient and easy to use.ConclusionRCA is a useful practice in synchronous distributed retrospectives. However, it requires software tool support for enabling real-time view and co-creation of a cause-effect diagram. ARCA-tool supports synchronous RCA, and includes support for logging problems and causes, problem prioritization, cause-effect diagramming, and logging of process improvement proposals. It enables conducting RCA in distributed retrospectives.     

Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize is an open question because the studies have all focused on a single procedural language, C, and a few specific types of test suites. In particular, Java and the JUnit testing framework are being used extensively to build software systems in practice, and the effectiveness of prioritization techniques on Java systems tested under JUnit has not been investigated. We have therefore designed and performed a controlled experiment examining whether test case prioritization can be effective on Java programs tested under JUnit, and comparing the results to those achieved in earlier studies. Our analyses show that test case prioritization can significantly improve the rate of fault detection of JUnit test suites, but also reveal differences with respect to previous studies that can be related to the language and testing paradigm. To investigate the practical implications of these results, we present a set of cost-benefits models for test case prioritization, and show how the effectiveness differences observed can result in savings in practice, but vary substantially with the cost factors associated with particular testing processes.     

Challenges of shared decision-making: A multiple case study of agile software development

ContextAgile software development changes the nature of collaboration, coordination, and communication in software projects.ObjectiveOur objective was to understand the challenges of shared decision-making in agile software development teams.MethodWe designed a multiple case study consisting of four projects in two software product companies that recently adopted Scrum. We collected data in semi-structured interviews, through participant observations, and from process artifacts.ResultsWe identified three main challenges to shared decision-making in agile software development: alignment of strategic product plans with iteration plans, allocation of development resources, and performing development and maintenance tasks in teams.ConclusionAgile software development requires alignment of decisions on the strategic, tactical, and operational levels in order to overcome these challenges. Agile development also requires a transition from specialized skills to redundancy of functions and from rational to naturalistic decision-making. This takes time; the case companies needed from one to two years to change from traditional, hierarchical decision-making to shared decision-making in software development projects.     

Decision-making in management information systems research: the utility of policy capturing methodology

Abstract

Although decision-making represents a fundamental issue in management information systems (MIS), obtaining accurate assessments of the factors affecting employees' decisions may be difficult using traditional methods such as ratings and rankings. Policy capturing, a little-used method in MIS, represents a potentially important alternative to more traditional methods. After demonstrating that policy capturing has been underutilized in MIS, the paper illustrates the use of policy capturing in two decision-making contexts—computer training and software selection. These two studies contrast policy capturing results with more traditional methods, and draw implications for research.     

Past and future of software architectural decisions – A systematic mapping study

ContextThe software architecture of a system is the result of a set of architectural decisions. The topic of architectural decisions in software engineering has received significant attention in recent years. However, no systematic overview exists on the state of research on architectural decisions.ObjectiveThe goal of this study is to provide a systematic overview of the state of research on architectural decisions. Such an overview helps researchers reflect on previous research and plan future research. Furthermore, such an overview helps practitioners understand the state of research, and how research results can help practitioners in their architectural decision-making.MethodWe conducted a systematic mapping study, covering studies published between January 2002 and January 2012. We defined six research questions. We queried six reference databases and obtained an initial result set of 28,895 papers. We followed a search and filtering process that resulted in 144 relevant papers.ResultsAfter classifying the 144 relevant papers for each research question, we found that current research focuses on documenting architectural decisions. We found that only several studies describe architectural decisions from the industry. We identified potential future research topics: domain-specific architectural decisions (such as mobile), achieving specific quality attributes (such as reliability or scalability), uncertainty in decision-making, and group architectural decisions. Regarding empirical evaluations of the papers, around half of the papers use systematic empirical evaluation approaches (such as surveys, or case studies). Still, few papers on architectural decisions use experiments.ConclusionOur study confirms the increasing interest in the topic of architectural decisions. This study helps the community reflect on the past ten years of research on architectural decisions. Researchers are offered a number of promising future research directions, while practitioners learn what existing papers offer.     

Similarity testing for access control

ContextAccess control is among the most important security mechanisms, and XACML is the de facto standard for specifying, storing and deploying access control policies. Since it is critical that enforced policies are correct, policy testing must be performed in an effective way to identify potential security flaws and bugs. In practice, exhaustive testing is impossible due to budget constraints. Therefore the tests need to be prioritized so that resources are focused on their most relevant subset.ObjectiveThis paper tackles the issue of access control test prioritization. It proposes a new approach for access control test prioritization that relies on similarity.MethodThe approach has been applied to several policies and the results have been compared to random prioritization (as a baseline). To assess the different prioritization criteria, we use mutation analysis and compute the mutation scores reached by each criterion. This helps assessing the rate of fault detection.ResultsThe empirical results indicate that our proposed approach is effective and its rate of fault detection is higher than that of random prioritization.ConclusionWe conclude that prioritization of access control test cases can be usefully based on similarity criteria.     

On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localization

ContextTesting and debugging consume a significant portion of software development effort. Both processes are usually conducted independently despite their close relationship with each other. Test adequacy is vital for developers to assure that sufficient testing effort has been made, while finding all the faults in a program as soon as possible is equally important. A tight integration between testing and debugging activities is essential.ObjectiveThe paper aims at finding whether three factors, namely, the adequacy criterion to gauge a test suite, the size of a prioritized test suite, and the percentage of such a test suite used in fault localization, have significant impacts on integrating test case prioritization techniques with statistical fault localization techniques.MethodWe conduct a controlled experiment to investigate the effectiveness of applying adequate test suites to locate faults in a benchmark suite of seven Siemens programs and four real-life UNIX utility programs using three adequacy criteria, 16 test case prioritization techniques, and four statistical fault localization techniques. We measure the proportion of code needed to be examined in order to locate a fault as the effectiveness of statistical fault localization techniques. We also investigate the integration of test case prioritization and statistical fault localization with postmortem analysis.ResultThe main result shows that on average, it is more effective for a statistical fault localization technique to utilize the execution results of a MC/DC-adequate test suite than those of a branch-adequate test suite, and is in turn more effective to utilize the execution results of a branch-adequate test suite than those of a statement-adequate test suite. On the other hand, we find that none of the fault localization techniques studied can be sufficiently effective in suggesting fault-relevant statements that can fit easily into one debug window of a typical IDE.ConclusionWe find that the adequacy criterion and the percentage of a prioritized test suite utilized are major factors affecting the effectiveness of statistical fault localization techniques. In our experiment, the adoption of a stronger adequacy criterion can lead to more effective integration of testing and debugging.