Equality in cumulative voting: A systematic review with an improvement proposal

ContextPrioritization is an essential part of requirements engineering, software release planning and many other software engineering disciplines. Cumulative Voting (CV) is known as a relatively simple method for prioritizing requirements on a ratio scale. Historically, CV has been applied in decision-making in government elections, corporate governance, and forestry. However, CV prioritization results are of a special type of data—compositional data.ObjectivesThe purpose of this study is to aid decision-making by collecting knowledge on the empirical use of CV and develop a method for detecting prioritization items with equal priority.MethodsWe present a systematic literature review of CV and CV analysis methods. The review is based on searching electronic databases and snowball sampling of the found primary studies. Relevant studies are selected based on titles, abstracts, and full text inspection. Additionally, we propose Equality of Cumulative Votes (ECVs)—a CV result analysis method that identifies prioritization items with equal priority.ResultsCV has been used in not only requirements prioritization and release planning but also in e.g. software process improvement, change impact analysis and model driven software development. The review presents a collection of state of the practice studies and CV result analysis methods. In the end, ECV was applied to 27 prioritization cases from 14 studies and identified nine groups of equal items in three studies.ConclusionsWe believe that the analysis of the collected studies and the CV result analysis methods can help in the adoption of CV prioritization method. The evaluation of ECV indicates that it is able to detect prioritization items with equal priority and thus provide the practitioner with a more fine-grained analysis.     

利用佳点集遗传算法的白盒测试用例优先排序

在软件演化过程中,测试用例优先排序作为一种高效实用的回归测试技术,对于提高缺陷的早期检测速率和降低测试成本有重要意义。针对传统遗传算法在白盒测试用例优先排序中收敛速度慢和稳定性差的问题,采用佳点集遗传算法求解白盒测试用例优先排序问题。算法根据程序实体覆盖矩阵对个体进行编码,以程序实体覆盖平均百分比作为适应度函数,采用随机抽样选择算子和佳点集交叉算子产生新一代种群。实验选择6个典型的基准开源项目,以语句、分支和方法作为程序实体,实验结果表明佳点集遗传算法收敛速度快、稳定性好,为回归测试提供了一个有效的测试用例优先排序方法,有助于尽早发现软件缺陷,降低测试成本。     

Prioritized aggregation operators based on the priority degrees in multicriteria decision-making

We consider the multicriteria decision-making (MCDM) problems where there exists a prioritization relationship over the criteria. We introduce the concept of the priority degree. Then we give three kinds of prioritized aggregation operators based on the priority degrees: the prioritized averaging operator with the priority degrees, the prioritized scoring operator with the priority degrees, and the prioritized ordered weighted averaging operator with the priority degrees. Some desired properties of these prioritized aggregation operators are also investigated. The priority degree plays an important role in the prioritized MCDM problems. We also investigate how to select a proper priority degree according to the giving decision information. By using an illustrative example, we show that the prioritized aggregation operators based on the priority degrees provide the decision-makers more choices and they are more flexible in the process of decision-making.     

Consolidating SWOT analysis with nonhomogeneous uncertain preference information

SWOT analysis is an important support tool for decision-making, and is commonly used to systematically analyze organizations’ internal and external environments. However, one of its deficiencies is in the measurement and evaluation of prioritization of the factors and strategies. This paper is aimed to present a novel quantified SWOT analytical method based multiple criteria group decision-making (MCGDM) concept, in which the priorities of SWOT factors and groups are derived by multiple decision makers (DMs) with nonhomogeneous uncertain preference information (NUPI), such as interval multiplicative preference relations, interval fuzzy preference relations, and uncertain linguistic preference relations. In this method, the SWOT analysis provides a basic frame within which to perform analyses of decision situations, in turn, MCGDM methods assist in carrying out SWOT more analytically and in elaborating the results of the analyses so that SWOT factors and groups can be prioritized with respect to the entire SWOT. The uniform and aggregation of the NUPI and the derivation of priorities for SWOT groups and factors are investigated in detail. Finally, an example is to validate the procedure of the proposed method.     

Analytic hierarchy prioritization process in the AHP application development: A prioritization operator selection approach

In the analytic hierarchy process, prioritization of the reciprocal matrix is a core issue to influence the final decision choice. Various prioritization methods have been proposed, but none of prioritization methods performs better than others in every inconsistent case. To address the prioritation operator selection problem, this paper proposes the analytic hierarchy prioritization process, which is an objective hierarchy model (without using subjective pairwise comparisons) to approximate the real priority vectors with selection of the most appropriate prioritization operator among the various prioritization candidates, for a reciprocal matrix, and on the basis of a list of measurement criteria. Nine important prioritization operators and seven measurement criteria are illustrated in AHPP. Two previous applications are revised and illustrate the validity and usability of the proposed model. The results show that the most appropriate prioritization operator is dependent of the content of the reciprocal matrix and AHPP is an appropriate method to address the prioritization problem to make better decisions.     

Test case prioritization: a systematic mapping study

Test case prioritization techniques, which are used to improve the cost-effectiveness of regression testing, order test cases in such a way that those cases that are expected to outperform others in detecting software faults are run earlier in the testing phase. The objective of this study is to examine what kind of techniques have been widely used in papers on this subject, determine which aspects of test case prioritization have been studied, provide a basis for the improvement of test case prioritization research, and evaluate the current trends of this research area. We searched for papers in the following five electronic databases: IEEE Explorer, ACM Digital Library, Science Direct, Springer, and Wiley. Initially, the search string retrieved 202 studies, but upon further examination of titles and abstracts, 120 papers were identified as related to test case prioritization. There exists a large variety of prioritization techniques in the literature, with coverage-based prioritization techniques (i.e., prioritization in terms of the number of statements, basic blocks, or methods test cases cover) dominating the field. The proportion of papers on model-based techniques is on the rise, yet the growth rate is still slow. The proportion of papers that use datasets from industrial projects is found to be 64 %, while those that utilize public datasets for validation are only 38 %. On the basis of this study, the following recommendations are provided for researchers: (1) Give preference to public datasets rather than proprietary datasets; (2) develop more model-based prioritization methods; (3) conduct more studies on the comparison of prioritization methods; (4) always evaluate the effectiveness of the proposed technique with well-known evaluation metrics and compare the performance with the existing methods; (5) publish surveys and systematic review papers on test case prioritization; and (6) use datasets from industrial projects that represent real industrial problems.     

基于历史信息的自适应测试用例优先级技术

在软件迭代开发的过程中,测试用例优先级技术因能有效地提高回归测试的效率,降低时间开销和人力成本,受到研究者的广泛关注,许多优化方法相继被提出。但是目前的研究多倾向于以需求和覆盖率作为排序准则,并且是一种静态排序。为此,提出一种基于历史信息的测试用例优先级技术,并在测试用例的执行过程中动态自适应地调整测试用例的优先级,以尽可能早地发现缺陷,达到预期的检错目标。在课题组开发的项目中运用该方法,验证了该方法的有效性。     

Ordinal Priority Approach (OPA) in Multiple Attribute Decision-Making

The current study aims to present a new method called Ordinal Priority Approach (OPA) in Multiple Attribute Decision-Making (MADM). This method can be used in individual or group decision-making (GDM). In the case of GDM, through this method, we first determine the experts and their priorities. The priority of experts may be determined based on their experience and/or knowledge. After prioritization of the experts, the attributes are prioritized by each expert. Meanwhile, each expert ranks the alternatives based on each attribute, and the sub-attributes if any. Ultimately, by solving the presented linear programming model of this method, the weights of the attributes, alternatives, experts, and sub-attributes would be obtained simultaneously. A significant advantage of the proposed method is that it does not make use of pairwise comparison matrix, decision-making matrix (no need for numerical input), normalization methods, averaging methods for aggregating the opinions of experts (in GDM) and linguistic variables. Another advantage of this method is the possibility for experts to only comment on the attributes and alternatives for which they have sufficient knowledge and experience. The validity of the proposed model has been evaluated using several group and individual instances. Finally, the proposed method has been compared with other methods such as AHP, BWM, TOPSIS, VIKOR, PROMETHEE and QUALIFLEX. Based on comparisons among the weights and ranks using Spearman and Pearson correlation coefficients, the proposed method has an applicable performance compared with other methods.     

Multiattribute group decision-making based on Pythagorean fuzzy Einstein prioritized aggregation operators

Pythagorean fuzzy set (PFS) is a powerful tool to deal with the imprecision and vagueness. Many aggregation operators have been proposed by many researchers based on PFSs. But the existing methods are under the hypothesis that the decision-makers (DMs) and the attributes are at the same priority level. However, in real group decision-making problems, the attribute and DMs may have different priority level. Therefore, in this paper, we introduce multiattribute group decision-making (MAGDM) based on PFSs where there exists a prioritization relationship over the attributes and DMs. First we develop Pythagorean fuzzy Einstein prioritized weighted average operator and Pythagorean fuzzy Einstein prioritized weighted geometric operator. We study some of its desirable properties such as idempotency, boundary, and monotonicity in detail. Moreover we propose a MAGDM approach based on the developed operators under Pythagorean fuzzy environment. Finally, an illustrative example is provided to illustrate the practicality of the proposed approach.     

基于函数调用路径的回归测试用例选择排序方法研究

针对在回归测试过程中,因为不断修复软件中存在的缺陷所造成的测试工作量大、测试效率低等问题,论文将测试用例选择与优先级排序技术相结合,以面向函数调用的路径覆盖生成方法为基础,提出了一种面向函数调用路径（Functions Calling Path, FCP）的测试用例选择与排序方法。首先根据函数调用关系图,对程序中被修改函数与其他函数的关联性进行分析,从初始测试用例集中选择测试用例,形成回归测试用例集;然后对这些测试用例进行优先级排序,并动态地调整优先级排序结果;最后,对优先级排序结果进行再次选择,确定最小的回归测试用例集。实验结果表明,测试用例选择与排序方法对优化回归测试用例是有效的,大大减少了回归测试用例数量,降低了回归测试成本。     

On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques

Regression testing is an important activity in the software life cycle, but it can also be very expensive. To reduce the cost of regression testing, software testers may prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. One potential goal of test case prioritization techniques is to increase a test suite's rate of fault detection (how quickly, in a run of its test cases, that test suite can detect faults). Previous work has shown that prioritization can improve a test suite's rate of fault detection, but the assessment of prioritization techniques has been limited primarily to hand-seeded faults, largely due to the belief that such faults are more realistic than automatically generated (mutation) faults. A recent empirical study, however, suggests that mutation faults can be representative of real faults and that the use of hand-seeded faults can be problematic for the validity of empirical results focusing on fault detection. We have therefore designed and performed two controlled experiments assessing the ability of prioritization techniques to improve the rate of fault detection of test case prioritization techniques, measured relative to mutation faults. Our results show that prioritization can be effective relative to the faults considered, and they expose ways in which that effectiveness can vary with characteristics of faults and test suites. More importantly, a comparison of our results with those collected using hand-seeded faults reveals several implications for researchers performing empirical studies of test case prioritization techniques in particular and testing techniques in general     

基于基尼系数-交叉效率的多属性决策方法

在数据包络分析(DEA)方法的基础上,提出一种基于基尼系数-交叉效率的多属性决策方法,用于解决具有多个投入、产出指标的多属性决策问题.首先,借鉴基尼系数的优化准则构建基尼系数-交叉评价策略模型,从而得到相对唯一的DEA交叉效率矩阵;然后,应用基尼准则计算各个效率值所包含的信息纯度,并借之实现交叉效率矩阵的集结;最后,根据集结结果对决策单元进行排序和择优.所提决策方法不仅能够克服传统DEA交叉效率方法的交叉评价策略选择难的问题,而且能够保证决策过程的客观性和公平性.同时,所提方法还对交叉评价所得的决策信息进行提纯,为科学合理地进行决策提供更多的有效信息.通过对中国各地区的医疗资源配置效率进行实证,验证了所提出方法的有效性和实用性.     

Combining different prioritization methods in the analytic hierarchy process synthesis

A multicriteria approach for combining prioritization methods within the analytic hierarchy process (AHP) is proposed. The leading assumption is that for each particular decision problem and related hierarchy, AHP must not necessarily employ only one prioritization method (e.g. eigenvector method). If more available methods are used to identify the best estimates of local priorities for each comparison matrix in the hierarchy, then the estimate of final alternatives’ priorities should also be the best possible, which is in natural concordance with an additive compensatory structure of the AHP synthesis. The most popular methods for deriving priorities from comparison matrices are identified as candidates (alternatives) to participate in AHP synthesis: additive normalization, eigenvector, weighted least-squares, logarithmic least-squares, logarithmic goal programming and fuzzy preference programming. Which method will be used depends on the result of multicriteria evaluation of their priority vectors’ performance with regard to suggested deviation and rank reversal measures. Two hierarchies with matrices of size 3–6 are used to illustrate an approach.     

RePizer: a framework for prioritization of software requirements

The standard software development life cycle heavily depends on requirements elicited from stakeholders. Based on those requirements, software development is planned and managed from its inception phase to closure. Due to time and resource constraints, it is imperative to identify the high-priority requirements that need to be considered first during the software development process. Moreover, existing prioritization frameworks lack a store of historical data useful for selecting the most suitable prioritization technique of any similar project domain. In this paper, we propose a framework for prioritization of software requirements, called RePizer, to be used in conjunction with a selected prioritization technique to rank software requirements based on defined criteria such as implementation cost. RePizer assists requirements engineers in a decision-making process by retrieving historical data from a requirements repository. RePizer also provides a panoramic view of the entire project to ensure the judicious use of software development resources. We compared the performance of RePizer in terms of expected accuracy and ease of use while separately adopting two different prioritization techniques, planning game (PG) and analytical hierarchy process (AHP). The results showed that RePizer performed better when used in conjunction with the PG technique.     

基于蚁群算法的测试用例优先排序

测试用例优先排序技术通过优化测试用例的执行次序来提高软件测试的效率,是增强型软件测试和回归测试的重要研究课题。针对基于需求的测试用例优先排序问题,提出了一种基于蚁群算法的求解方法,采用不同的测试用例间距离及用例序列评价策略,给出了该方法的2种不同实现方式。首先,针对黑盒测试特点,设计了基于需求的一般性测试用例序列评价指标;其次,提出测试用例吸引度概念,基于测试用例吸引度定义了测试用例间的距离;然后,给出了信息素更新策略、最优解集更新策略、局部最优解突变策略等主要设计策略,分别实现了该方法基于距离和基于指标的2种实现方式。实验结果表明,该方法具有很好的全局寻优能力,整体效果上优于粒子群算法、遗传算法和随机测试。     

Improving the efficiency of functional verification based on test prioritization

Functional verification has become the key bottleneck that delays time-to-market during the embedded system design process. And simulation-based verification is the mainstream practice in functional verification due to its flexibility and scalability. In practice, the success of the simulation-based verification highly depends on the quality of functional tests in use which is usually evaluated by coverage metrics. Since test prioritization can provide a way to simulate the more important tests which can improve the coverage metrics evidently earlier, we propose a test prioritization approach based on the clustering algorithm to obtain a high coverage level earlier in the simulation process. The k-means algorithm, which is one of the most popular clustering algorithms and usually used for the test prioritization, has some shortcomings which have an effect on the effectiveness of test prioritization. Thus we propose three enhanced k-means algorithms to overcome these shortcomings and improve the effectiveness of the test prioritization. Then the functional tests in the simulation environment can be ordered with the test prioritization based on the enhanced k-means algorithms. Finally, the more important tests, which can improve the coverage metrics evidently, can be selected and simulated early within the limited simulation time. Experimental results show that the enhanced k-means algorithms are more accurate and efficient than the standard k-means algorithm for the test prioritization, especially the third enhanced k-means algorithm. In comparison with simulating all the tests randomly, the more important tests, which are selected with the test prioritization based on the third enhanced k-means algorithm, achieve almost the same coverage metrics in a shorter time, which achieves a 90% simulation time saving.     

Enhancing decision-making flexibility by introducing a new last aggregation evaluating approach based on multi-criteria group decision making and Pythagorean fuzzy sets

Uncertainty is an important factor in any decision-making process. Different tools and approaches have been introduced to handle the uncertain environment of group decision making. One of the latest tools in dealing with uncertainty is Pythagorean fuzzy sets (PFSs). These sets extend the concept of intuitionistic fuzzy sets. To show the advantages of these new sets, this paper offers a novel last aggregation group decision-making process for weighting and evaluating. The methodology employs a new approach in computing the weight of decision makers. Moreover, the concept of entropy is applied to address the fuzziness of weights of evaluation criteria in the process. The method develops a new index in ranking the alternatives. Finally, the proposed method is last aggregation, which means it will be more precise in situations with high variations in decision makers’ judgments. To show the applicability of the method, an example from the literature is adopted and solved for internet companies.     

Improving test efficiency through system test prioritization

Software testing is an expensive process consuming at least 50% of the total development cost. Among the types of testing, system testing is the most expensive and complex. Companies are frequently faced with budgetary constraints, which may limit their ability to effectively complete testing efforts before delivering a software product. We build upon prior test case prioritization research and present a system-level approach to test case prioritization called Prioritization of Requirements for Test (PORT). PORT prioritizes system test cases based on four factors for each requirement: customer priority, implementation complexity, fault proneness, and requirements volatility. Test cases for requirements with higher priority based upon a weighted average of these factors are executed earlier in system test. An academic feasibility study and three post hoc industrial studies were conducted. Results indicate that PORT can be used to improve the rate of failure detection when compared with a random and operational profile-driven random approach. Furthermore, we investigated the contribution of the prioritization factors towards the improved rate of failure detection and found customer priority was the most significant contributor. Tool support is provided for the PORT scheme which allows for automatic collection of the four factor values and the resultant test case prioritization.     

测试用例排序的研究

回归测试是一个成本很高的测试过程。为了减少回归测试的成本,可以使用测试用例排序技术。测试用例排序是指按照事先确定的目标重新安排测试用例集中测试用例的执行次序,使得具有高优先级的测试用例比低优先级的测试用例在测试过程中更早执行。本文描述了测试用例排序问题;给出了两个一般测试用例排序算法,即总计排序算法和 附加排序算法;根据不同的覆盖准则（如语句、分支和定义-使用等）,可以从这两个一般算法得到对应的排序算法;最后,讨论了测试用例排序算法的有效性。     

Differential prioritization in feature selection and classifier aggregation for multiclass microarray datasets

The high dimensionality of microarray datasets endows the task of multiclass tissue classification with various difficulties—the main challenge being the selection of features deemed relevant and non-redundant to form the predictor set for classifier training. The necessity of varying the emphases on relevance and redundancy, through the use of the degree of differential prioritization (DDP) during the search for the predictor set is also of no small importance. Furthermore, there are several types of decomposition technique for the feature selection (FS) problem—all-classes-at-once, one-vs.-all (OVA) or pairwise (PW). Also, in multiclass problems, there is the need to consider the type of classifier aggregation used—whether non-aggregated (a single machine), or aggregated (OVA or PW). From here, first we propose a systematic approach to combining the distinct problems of FS and classification. Then, using eight well-known multiclass microarray datasets, we empirically demonstrate the effectiveness of the DDP in various combinations of FS decomposition types and classifier aggregation methods. Aided by the variable DDP, feature selection leads to classification performance which is better than that of rank-based or equal-priorities scoring methods and accuracies higher than previously reported for benchmark datasets with large number of classes. Finally, based on several criteria, we make general recommendations on the optimal choice of the combination of FS decomposition type and classifier aggregation method for multiclass microarray datasets.