Evaluating and selecting software packages: A review

Evaluating and selecting software packages that meet an organization’s requirements is a difficult software engineering process. Selection of a wrong software package can turn out to be costly and adversely affect business processes. The aim of this paper is to provide a basis to improve the process of evaluation and selection of the software packages. This paper reports a systematic review of papers published in journals and conference proceedings. The review investigates methodologies for selecting software packages, software evaluation techniques, software evaluation criteria, and systems that support decision makers in evaluating software packages. The key findings of the review are: (1) analytic hierarchy process has been widely used for evaluation of the software packages, (2) there is lack of a common list of generic software evaluation criteria and its meaning, and (3) there is need to develop a framework comprising of software selection methodology, evaluation technique, evaluation criteria, and system to assist decision makers in software selection.     

Comparing the relative importance of evaluation criteria in proprietary and open‐source enterprise application software selection – a conjoint study of ERP and Office systems

Until recently, organizations willing to acquire application systems have had no choice but to adopt proprietary software. With the advent of open‐source software (OSS), a new model for developing and distributing software has entered the stage. OSS has evolved from a generally horizontal infrastructure towards more highly visible applications in vertical domains, giving information systems (IS) managers more degrees of freedom in their selection of enterprise application software (EAS). Although a large body of research exists on the relative importance of evaluation criteria for proprietary EAS, the role of OSS in the EAS evaluation process has received little attention so far. To address this research gap, this study represents the first empirical investigation to compare the relative importance of evaluation criteria in proprietary and open‐source EAS selection. Through an online survey, we evaluated the responses of IS managers of 358 organizations to a conjoint study spawning 8592 trade‐off pair comparisons and 3580 purchase evaluations on proprietary and open‐source enterprise resource planning (ERP) and Office software packages. The results show that the relative importance of evaluation criteria significantly varies between proprietary and open‐source ERP systems. Implementation factors such as ease of implementation and support are much more crucial in the evaluation of open‐source than of proprietary ERP systems, which is generally due to IS managers' risk mitigation behaviour. Interestingly, there are no major differences in the ranking of evaluation criteria between proprietary and open‐source Office systems. We conclude our paper with a detailed discussion of our findings and their implications for researchers, companies, EAS vendors and open‐source communities.     

On recommendation problems beyond points of interest

Recommendation systems aim to recommend items or packages of items that are likely to be of interest to users. Previous work on recommendation systems has mostly focused on recommending points of interest (POI), to identify and suggest top-k items or packages that meet selection criteria and satisfy compatibility constraints on items in a package, where the (packages of) items are ranked by their usefulness to the users. As opposed to prior work, this paper investigates two issues beyond POI recommendation that are also important to recommendation systems. When there exist no sufficiently many POI that can be recommended, we propose (1) query relaxation recommendation to help users revise their selection criteria, or (2) adjustment recommendation to guide recommendation systems to modify their item collections, such that the users׳ requirements can be satisfied.We study two related problems, to decide (1) whether the query expressing the selection criteria can be relaxed to a limited extent, and (2) whether we can update a bounded number of items, such that the users can get desired recommendations. We establish the upper and lower bounds of these problems, all matching, for both combined and data complexity, when selection criteria and compatibility constraints are expressed in a variety of query languages, for both item recommendation and package recommendation. To understand where the complexity comes from, we also study the impact of variable sizes of packages, compatibility constraints and selection criteria on the analyses of these problems. Our results indicate that in most cases the complexity bounds of query relaxation and adjustment recommendation are comparable to their counterparts of the basic recommendation problem for testing whether a given set of (resp. packages of) items makes top-k items (resp. packages). In other words, extending recommendation systems with the query relaxation and adjustment recommendation functionalities typically does not incur extra overhead.     

Relative importance of evaluation criteria for enterprise systems: a conjoint study

Abstract.  While a large body of research exists on the development and implementation of software, organizations are increasingly acquiring enterprise software packages [e.g. enterprise resource planning (ERP) systems] instead of custom developing their own software applications. To be competitive in the marketplace, software package development firms must manage the three-pronged trade-off between cost, quality and functionality. Surprisingly, prior research has made little attempt to investigate the characteristics of packaged software that influence management information system (MIS) managers' likelihood of recommending purchase. As a result, both the criteria by which MIS managers evaluate prospective packaged systems and the attributes that lead to commercially competitive ERP software products are poorly understood. This paper examines this understudied issue through a conjoint study. We focus on ERP systems, which are among the largest and most complex packaged systems that are purchased by organizations. In a conjoint study, 1008 evaluation decisions based on hypothetical ERP software package profiles were completed by managers in 126 organizations. The study represents the first empirical investigation of the relative importance that managers ascribe to various factors that are believed to be important in evaluating packaged software. The results provide important insights for both organizations that acquire such systems and those that develop them. The results show that functionality, reliability, cost, ease of use and ease of customization are judged to be important criteria, while ease of implementation and vendor reputation were not found to be significant. Functionality and reliability were found to be the most heavily weighted factors. We conclude the paper with a detailed discussion of the results and their implications for software acquisition and development practice.     

Framework for evaluation and selection of the software packages: A hybrid knowledge based system approach

Evaluation and selection of the software packages is complicated and time consuming decision making process. Selection of inappropriate software package can turn out to be costly and adversely affects business processes and functioning of the organization. In this paper we describe (i) generic methodology for software selection, (ii) software evaluation criteria, and (iii) hybrid knowledge based system (HKBS) approach to assist decision makers in evaluation and selection of the software packages. The proposed HKBS approach employs an integrated rule based and case based reasoning techniques. Rule based reasoning is used to capture user needs of the software package and formulate a problem case. Case based reasoning is used to retrieve and compare candidate software packages with the user needs of the package. This paper also evaluates and compares HKBS approach with the widely used existing software evaluation techniques such as analytic hierarchy process (AHP) and weighted scoring method (WSM).     

A hierarchical framework for evaluating simulation software

In simulation software selection problems, packages are evaluated either on their own merits or in comparison with other packages. In either method, a comprehensive list of criteria for evaluation of simulation software is essential for proper selection. Although various simulation software evaluation checklists do exist, there are differences in the lists provided and the terminologies used. This paper presents a hierarchical framework for simulation software evaluation consisting of seven main groups and several subgroups. An explanation for each criterion is provided and an analysis of the usability of the proposed framework is further discussed.     

A methodology for the automation of medium or small manufacturing companies

Many products are produced by discrete manufacturing plants in lots of 100 items or fewer. Currently, most small and medium-sized manufacturing companies are far from computer-integrated automation. New computer-integrated manufacturing systems with their associated hardwares require tremendous initial investments that are beyond the means of most small and medium-sized companies. In this paper we investigate how these companies can upgrade and automate their existing facilities at a minimal and viable cost. In Section 2 we discuss main references (books and trade journals) that contain detailed information on software packages for automation. In Section 3 we discuss the criteria for selecting software packages. In Section 4 we discuss how to use the obtained information to create a data base of different alternatives. In Section 5 we outline different areas of manufacturing. For illustrations, we discuss manufacturing planning, intelligent systems, and computer areas. In Section 6 we discuss a new method for the selection of the best alternative using the data base. The method is a simple interactive paired comparison method based on a computer package developed on a personal computer.     

Integration strategy for distributed intelligent systems

In this article, we describe a new framework for designing real-time intelligent control systems. An integrated intelligent system is a large knowledge integration environment that consists of both symbolic reasoning systems (expert systems) and numerical computation packages. These modular software programs are controlled by a meta-system which manages the selection, operation, and communication of these programs. This new architecture can serve as a universal configuration to develop high-performance intelligent systems for many complicated application domains in the real-time manufacturing process. As an example, an intelligent optimal control is utilized to illustrate the integrated intelligent control system.     

Modeling the growth of complex software function dependency networks

Software engineering efforts can potentially benefit much from a good understanding of the structures of existing software systems and the processes governing their development. Towards that end, we study software systems by means of the complex network analysis framework. We model a software package as a network, with nodes representing the functions in the package and edges representing the dependencies among the functions. Our empirical analysis of five widely-adopted open-source software packages reveals a set of interesting features of such networks, which cannot be adequately reproduced by existing complex network models. We then set out to develop a new network growth model, explicitly imitating generally-advocated software development principals, such as divide-and-conquer, modularization, high intra-module cohesion, and low inter-module coupling. Results of our analytical derivations and numeric studies show that our model can more closely reproduce the particular features exhibited by real-world software packages, thus hopefully better explaining the phenomena of concern.     

Universal software packages to model the distributed-parameter systems

Consideration was given to the universal software packages for modeling objects and distributed-parameter systems obeying the partial differential equations. The packages may serve as important tools for industrial automation because the majority of industrial objects have distributed parameters. The review presented the results of comparing a number of commercial packages against a set of user-defined criteria. Its final part was devoted to some freeware packages. Conclusions were drawn, and recommendations for using the considered packages were given.     

Modelling Off-the-Shelf Information Systems Requirements: An Ontological Approach

Requirements for choosing off-the-shelf information systems (OISR) differ from requirements for development of new information systems in that they do not necessarily provide complete specifications, thus allowing flexibility in matching an existing IS to the stated needs. We present a framework for OISR conceptual models that consists of four essential elements: business processes, business rules, information objects and required system services. We formalise the definitions of these concepts based on an ontological model. The ontology-based OISR model provides a framework to evaluate modelling languages on how appropriate they are for OISR requirements specifications. The evaluation framework is applied to the Object-Process Methodology, and its results are compared with a similar evaluation of ARIS. This comparison demonstrates the effectiveness of the ontological framework for evaluating modelling tools on how well they can guide selection, implementation and integration of purchased software packages.     

A fuzzy VIKOR method for supplier selection based on entropy measure for objective weighting

Recently, resolving the problem of evaluation and ranking the potential suppliers has become as a key strategic factor for business firms. With the development of intelligent and automated information systems in the information era, the need for more efficient decision making methods is growing. The VIKOR method was developed to solve multiple criteria decision making (MCDM) problems with conflicting and non-commensurable criteria assuming that compromising is acceptable to resolve conflicts. On the other side objective weights based on Shannon entropy concept could be used to regulate subjective weights assigned by decision makers or even taking into account the end-users’ opinions. In this paper, we treat supplier selection as a group multiple criteria decision making (GMCDM) problem and obtain decision makers’ opinions in the form of linguistic terms. Then, these linguistic terms are converted to trapezoidal fuzzy numbers. We extended the VIKOR method with a mechanism to extract and deploy objective weights based on Shannon entropy concept. The final result is obtained through next steps based on factors R, S and Q. A numerical example is proposed to illustrate an application of the proposed method.     

An ERP software selection process with using artificial neural network based on analytic network process approach

An enterprise resource planning (ERP) software selection is known to be multi attribute decision making (MADM) problem. This problem has been modeled according with analytic network process (ANP) method due to fact that it considers criteria and sub criteria relations and interrelations in selecting the software.Opinions of many experts are obtained while building ANP model for the selection ERP then opinions are reduced to one single value by methods like geometric means so as to get desired results. To use ANP model for the selection of ERP for a new organization, a new group of expert’s opinions are needed. In this case the same problem will be in counter. In the proposed model, when ANP and ANN models are setup, an ERP software selection can be made easily by the opinions of one single expert. In that case calculation of geometric mean of answers that obtained from many experts will be unnecessary. Additionally the effect of subjective opinion of one single decision maker will be avoided. In terms of difficulty, ANP has some difficulties due to eigenvalue and their limit value calculation.An ANN model has been designed and trained with using ANP results in order to calculate ERP software priority. The artificial neural network (ANN) model is trained by results obtained from ANP. It seems that there is no any major difficulty in order to predict software priorities with trained ANN model. By this results ANN model has been come suitable for using in the selection of ERP for another new decision.     

Empirical performance evaluation of graphics recognition systems

Presents a methodology for evaluating graphics recognition systems operating on images that contain straight lines, circles, circular arcs, and text blocks. It enables an empirical comparison of vectorization software packages and uses practical performance evaluation methods that can be applied to complete vectorization systems. The methodology includes a set of matching criteria for pairs of graphical entities, a set of performance evaluation metrics, and a benchmark for the evaluation of graphics recognition systems. The benchmark was tested on three systems. The results are reported and analyzed in the paper     

面向操作系统版本构建的软件包依赖关系分析

基于Linux的开源操作系统发行版本通常都是由一系列相互关联的软件包构建而成,由于软件包数量庞大、互相之间依赖和冲突等关系复杂,操作系统版本构建也变得复杂,定制裁剪的粒度和精度不足,可能存在很多冗余软件包。目前,操作系统版本的构建主要还是根据工程经验进行组织,特别是软件包的选择缺少理论分析和指导。给出基于依赖关系的操作系统版本构建的基本模型,并结合Ubuntu开源发行版本的版本构建过程和软件仓库数据,对软件包的出入度、优先级和分类等特征进行统计分析和模型验证,从依赖关系角度总结出操作系统版本构建的主要原则,为后续提升版本自动化定制裁剪的精度以及软件发行演化和故障分析提供指导和重要参考。     

Evaluation methodology for performance and system economics for robotic devices

This paper presents a mathematical model for the selection and evaluation of robots, which consists of critical factors such as maximum return on investment, objective factors such as payback value to give a more complete cost perspective, and subjective factors including hardware/software, vendor performance, and internal adaptation. This evaluation procedure can be used for the selection of robots for any robot implementation, such as CAD/CAM, CIM and military applications.     

An integrated fuzzy MCDM based approach for robot selection considering objective and subjective criteria

Robots with vastly different capabilities and specifications are available for a wide range of applications. Selection of a robot for a specific application has become more complicated due to increase in the complexity, advanced features and facilities that are continuously being incorporated into the robots by different manufacturers. The aim of this paper is to present an integrated approach for the optimal selection of robots by considering both objective and subjective criteria. The approach utilizes Fuzzy Delphi Method (FDM), Fuzzy Analytical Hierarchical Process (FAHP), Fuzzy modified TOPSIS or Fuzzy VIKOR and Brown–Gibson model for robot selection. FDM is used to select the list of important objective and subjective criteria based on the decision makers’ opinion. Fuzzy AHP method is then used to find out the weight of each criterion (both objective and subjective). Fuzzy modified TOPSIS or Fuzzy VIKOR method is then used to rank the alternatives based on objective and subjective factors. The rankings obtained are used to calculate the robot selection index based on Brown–Gibson model. The proposed methodology is illustrated with a case study related to selection of robot for teaching purpose. It is found that the highest ranked alternative based on Fuzzy VIKOR is closest to the ideal solution.     

Package Coupling Measurement in Object-Oriented Software

The grouping of correlated classes into a package helps in better organization of modern object-oriented software. The quality of such packages needs to be measured so as to estimate their utilization. In this paper, new package coupling metrics are proposed, which also take into consideration the hierarchical structure of packages and direction of connections among package elements. The proposed measures have been validated theoretically as well as empirically using 18 packages taken from two open source software systems. The results obtained from this study show strong correlation between package coupling and understandability of the package which suggests that proposed metrics could be further used to represent other external software quality factors.