Information systems evaluation: navigating through the problem domain

Information systems (IS) make it possible to improve organizational efficiency and effectiveness, which can provide competitive advantage. There is, however, a great deal of difficulty reported in the normative literature when it comes to the evaluation of investments in IS, with companies often finding themselves unable to assess the full implications of their IS infrastructure. Although many of the savings resulting from IS are considered suitable for inclusion within traditional accountancy frameworks, it is the intangible and non-financial benefits, together with indirect project costs that complicate the justification process. In exploring this phenomenon, the paper reviews the normative literature in the area of IS evaluation, and then proposes a set of conjectures. These were tested within a case study to analyze the investment justification process of a manufacturing IS investment. The idiosyncrasies of the case study and problems experienced during its attempts to evaluate, implement, and realize the holistic implications of the IS investment are presented and critically analyzed. The paper concludes by identifying lessons learnt and thus, proposes a number of empirical findings for consideration by decision-makers during the investment evaluation process.     

Team performance and information system implementation

Although the Earned Value Method (EVM), a multi-dimensional project control system, has been widely used since 1960, there has been little research focused specifically on its application in IS implementation. The management of IS projects—including the accurate forecasting of project duration—is complicated by the strong impact of the learning curve on the effectiveness of EVM. This study makes new contributions by advancing formulas for the accurate prediction of project duration, and by developing a decision support model in which the learning curve is fully integrated with EVM, and performance changes due to learning are isolated from other factors affecting project duration. The study makes three contributions to the understanding and use of EVM in IS implementation: (1) It provides an extended version of EVM, (2) it provides an illustration of the application of EVM to a real project situation based on empirical data from an ERP implementation project, and (3) it provides a theoretical basis for empirical studies of IS project control techniques. The decision support model can be used to determine both the learning curve coefficient and the project completion date during the early stages of a project, and it thus offers a significant practical contribution to the management of IS projects.

The role of IT in organizational innovation – A systematic literature review

Information technology (IT) is broadly recognized as an important element in organizational innovation, however there has been relatively little integration of Information Systems (IS) research on the role of IT in organizational innovation. Such integration is particularly important in view of recent calls to examine the evolving ontological nature of IT, wherein IT artifacts are now shaping physical reality. We systematically examine innovation-related literature published in the leading IS journals in the period between 2009 and 2020 and we identify and summarize the core theoretical discourses in this domain. We find that extant literature largely examines the role of IT in innovation at a high level of abstraction, focusing on IT investments and IT capabilities, and consequently precluding granular insights on who, what, when, where, how, and why in relation to organizational innovation. To address this limitation and to provide a structuring lens for future research, we develop the organizational innovation system framework which posits that actors, IT artifacts, actions and organizational context are the key elements in organizational innovation systems that require more granular examination to yield deeper insights on how information technology contributes to organizational innovation. We re-examine the literature through the organizational innovation system lens and we identify gaps in extant research. We also outline potential directions for expanding the scope of future research.     

A new framework for managing IT-enabled business change

Abstract. Although information systems and technology (IS/IT) investments have always caused varying degrees of business change, the main purpose of many of today's IS/IT implementations is to change the business and/or organization in some significant way. However, most organizations' approaches to managing IS/IT developments have changed little in the last 15–20 years and are heavily dependent on methodologies of IS/IT development and associated project management principles. The philosophy being adopted by many organizations today is that there are few IS/IT projects (some infrastructure investments only) that do not cause business changes and therefore they are really 'business projects involving IS/IT'. This exploratory research set out to devise a new framework for 'IT and change' management, which is more appropriate to today's realities than traditional approaches. The initial framework was derived from 'first principles', then adapted and further developed by applying and testing it in a range of real projects in organizations. This paper summarizes the results of that work.     

Towards an integrated measurement of IS project performance: The project performance scorecard

Determining the contribution of information system (IS) projects is a difficult endeavour. This research presents a framework for evaluating and measuring IS project performance that seek to address this problem while moving away from the reliance on the traditional assessment method (time, cost, specifications). When IS projects fail or succeed is difficult to determine what conditions enabled the outcome. However, if we are to develop a higher level of IS project management competence we need frameworks that can help in assessing IS project outcomes while critically reflecting and learning from IS project management practices. The Project Performance Scorecard (PPS) recommended here is one framework that can help to advance our understanding IS project management and evaluation approaches. This framework integrates concepts from existing models of IS success, the Balanced Scorecard and project management practices. A case study approach is used to illustrate the use of the PPS, and offers insights into how it can be useful in assessing the performance of IS projects.

Appropriate computer-mediated communication: An Australian indigenous information system case study

This article discusses ways to operationalise the concept of culturally appropriate computer-mediated communication, utilising information systems (IS) development methodologies and adopting a postmodern and postcolonial perspective. By way of illustration, it describes progress on the participative development of the Ieramugadu Cultural Information System. This project is designed to develop and evaluate innovative procedures for elicitation, analysis, storage and communication of indigenous cultural heritage information. It is investigating culturally appropriate IS design techniques, multimedia approaches and ways to ensure protection of secret/sacred information. Ethical issues are foregrounded.     

Internet-based information systems use in organizations: an information studies perspective

Abstract. This case study investigates various ways in which different internet-based information systems (IS) are used by organizational participants. Borrowing theoretical insights on information behaviour accumulated over 50 years of information studies research, a conceptual framework is presented to help understand and assess the social and organizational impacts of internet-based IS. The framework describes the use of internet-based IS as a dynamic cycle of information needs–seeking–use activity situated in the context of a firm's information environment. Research questions pertain to the process of how individuals in organizations seek and use information from internet-based IS to satisfy information needs. In terms of information needs, this involves understanding the problem situations that lead participants to use internet-based IS, as well as the characteristics of those problems beyond subject matter. With respect to information seeking, this involves analysing how information from internet-based systems is displayed and formatted to signal their potential usefulness. In terms of information use, this involves how information obtained from internet-based systems is used in practice to resolve or redefine problems. Both quantitative and qualitative research methods are used. Data collection involves web tracking to identify significant episodes of internet-based IS activity, as well as one-on-one interviews to explore the context behind these episodes. Results suggest that it is possible and valuable to identify scenarios of internet-based IS use dominant in an organizational work setting. Doing so can help to identify ways to improve the situated use of internet-based IS that ameliorate the information needs–seeking–use cycle in firms.     

RefModPM: A Conceptual Reference Model for Project Management Information Systems

Project management information systems have changed considerably in the last decade. Today they go far beyond traditional scheduling as well as resource management. These systems have become comprehensive information systems (IS) that support the entire life cycle of projects, project programmes and project portfolios. In this context, project-oriented organizations face a new challenge: the design, implementation and operation of such information systems has become increasingly complex, since numerous processes have to be considered, diverse stakeholder interests taken into account and corresponding software systems selected. The reference model for project management IS (RefMod^PM) presented here addresses these challenges and aims to support corresponding implementation projects. This article introduces RefMod^PM by discussing its construction process and evaluation, as well as providing an overview of its architecture and presenting one part of the reference model in detail.     

Factors influencing effective use of big data: A research framework

Information systems (IS) research has explored “effective use” in a variety of contexts. However, it is yet to specifically consider it in the context of the unique characteristics of big data. Yet, organizations have a high appetite for big data, and there is growing evidence that investments in big data solutions do not always lead to the derivation of intended value. Accordingly, there is a need for rigorous academic guidance on what factors enable effective use of big data. With this paper, we aim to guide IS researchers such that the expansion of the body of knowledge on the effective use of big data can proceed in a structured and systematic manner and can subsequently lead to empirically driven guidance for organizations. Namely, with this paper, we cast a wide net to understand and consolidate from literature the potential factors that can influence the effective use of big data, so they may be further studied. To do so, we first conduct a systematic literature review. Our review identifies 41 factors, which we categorize into 7 themes, namely data quality; data privacy and security and governance; perceived organizational benefit; process management; people aspects; systems, tools, and technologies; and organizational aspects. To explore the existence of these themes in practice, we then analyze 45 published case studies that document insights into how specific companies use big data successfully. Finally, we propose a framework for the study of effective use of big data as a basis for future research. Our contributions aim to guide researchers in establishing the relevance and relationships within the identified themes and factors and are a step toward developing a deeper understanding of effective use of big data.     

Analysing information systems evaluation: another look at an old problem

Information systems (IS) evaluation is a thorny problem. In this paper we re-examine the area in light of recent developments in the field. Our examination begins with an example of a common contemporary IS assessment problem, viz. evaluating outsourcing. The example highlights the organizational and political issues that make evaluation fraught with difficulties. The paper argues that IS evaluation is a ‘necessary evil’ but the context in which IS are developed and used has become much more demanding and complex. A conceptual framework, first proposed in an earlier paper as a way to classify the literature, is presented and brought up to date. The framework is then used to re-examine the outsourcing example, demonstrating the usefulness of the framework.     

The Use of a Formalised Risk Model in NHS Information System Development

Information systems (IS) and technology are used extensively throughout the National Health Service (NHS), and the 1998 national information strategy, ‘Information for Health’, sets out how the NHS will be developing and implementing IS to support patient care within the next decade. This new IS initiative is set against a mixed record of success of IS projects in the NHS, with a number of high-profile failures. This paper highlights the need to consider the ‘organisational issues’ involved in systems implementation to avoid failures. It goes on to advocate the use of a process-oriented and organisation studies-based model for risk analysis and management for use in NHS IS projects. Two famous NHS case studies are used to validate the model. It is concluded that there is a real need in the NHS for tools to better control the inherent risks involved in IS development and implementation. Ultimately, the success of IS projects in the NHS is crucial if they want to best utilise clinical and patient information, with the overall aim of improving the efficiency and standard of the nation’s health care.     

Process: Learning,Working and Innovation

Worldwide the environment in which IT initiatives are deployed is changing. Increasingly, IT investments are being made to produce market outcomes as well as internal operating efficiencies. Learning about the linkage between the IT investments and business value is a critical challenge in the service sector for industries such as financial services, transportation and retailing where IT is endemic to the products and services delivered to customers [1]. Using a case study in a large North American insurance company and recent theory [2] as our guide, we identify ways in which Information Systems (IS) organizations can become more innovative by using learning as the bridge between working and innovation.     

信息系统项目管理研究

信息系统在组织中的地位和作用至关重要,但是信息系统建设的失败率还是很高,通过对信息系统项目的特点、信息系统项目成功与失败的标准、信息系统项目管理的发展阶段、信息系统项目管理的内容、信息系统项目管理的过程、信息系统项目管理的工具和技巧等的分析与讨论,以期帮助全面理解信息系统项目管理,指导信息系统项目管理和信息系统建设。     

Visualising a knowledge mapping of information systems investment evaluation

Information systems (IS) facilitate organisations to increase responsiveness and reduce the costs of their supply chain. This paper seeks to make a contribution through exploring and visualising knowledge mapping from the perspective of IS investment evaluation. The evaluation of IS is regarded as a challenging and complex process, which becomes even more difficult with the increased complexity of IS. The intricacy of IS evaluation, however, is due to numerous interrelated factors (e.g. costs, benefits and risks) that have human or organisational dimensions. With this in mind, there appears to be an increasing need to assess investment decision-making processes, to better understand the often far-reaching implications associated with technology adoption and interrelated knowledge components (KC). Through the identification and extrapolation of key learning issues from the literature and empirical findings, organisations can better improve their business processes and thereby their effectiveness and efficiency, while preventing others from making costly oversights that may not necessarily be only financial. In seeking to enlighten the often obscure evaluation of IS investments, this paper attempts to inductively emphasise the dissemination of knowledge and learning through the application of a fuzzy Expert System (ES) based knowledge mapping technique (i.e. Fuzzy Cognitive Map [FCM]). The rationale for exploring knowledge and IS investment evaluation is that a knowledge map will materialise for others to exploit during their specific technology evaluation. This is realised through conceptualising the explicit and tacit investment drivers. Among the several findings drawn from this research, the key resulting knowledge mapping through FCM demonstrated the complex, multifaceted and emergent behaviour of causal relationships within the knowledge area. The principal relationships and knowledge within IS investment evaluation are illustrated as being determined by a blend of managerial and user perspectives.     

Issues in information systems planning

Numerous difficulties face information systems managers as they develop IS plans. Three nominal group technique sessions used IS practitioners from different levels of management to identify specific difficulties. The findings indicate a wide variety of problems faced by these managers. Top IS managers were concerned most with learning the objectives of top general management. Middle IS managers were concerned most with adopting a methodology for planning. Operating IS managers were concerned most with understanding the perceived needs of the user. These results suggest the need for more sophisticated management techniques (or the better use of existing techniques) and the need for research to address the development, utilization and evaluation of such techniques.     

Developing a frame of reference for <Emphasis Type="Italic">ex-ante</Emphasis> IT/IS investment evaluation

Investment appraisal techniques are an integral part of many traditional capital budgeting processes. However, the adoption of Information Systems (IS) and the development of resulting infrastructures are being increasingly viewed on the basis of consumption. Consequently, decision-makers are now moving away from the confines of rigid capital budgeting processes, which have traditionally compared IS with non-IS-related investments. With this in mind, the authors seek to dissect investment appraisal from the broader capital budgeting process to allow a deeper understanding of the mechanics involved with IS justification. This analysis presents conflicting perspectives surrounding the scope and sensitivity of traditional appraisal methods. In contributing to this debate, the authors present taxonomies of IS benefit types and associated natures, and discuss the resulting implications of using traditional appraisal techniques during the IS planning and decision-making process. A frame of reference that can be used to navigate through the variety of appraisal methods available to decision-makers is presented and discussed. Taxonomies of appraisal techniques that are classified by their respective characteristics are also presented. Perspectives surrounding the degree of involvement that financial appraisal should play during decision making and the limitations surrounding investment appraisal techniques are identified.     

Virtual organizational learning in open source software development projects

We studied virtual organizational learning in open source software (OSS) development projects. Specifically, our research focused on learning effects of OSS projects and the factors that affect the learning process. The number and percentage of resolved bugs and bug resolution time of 118 SourceForge.net OSS projects were used to measure the learning effects. Projects were characterized by project type, number and experience of developers, number of bugs, and bug resolution time. Our results provided evidence of virtual organizational learning in OSS development projects and support for several factors as determinants of performance. Team size was a significant predictor, with mid-sized project teams functioning best. Teams of three to seven developers exhibited the highest efficiency over time and teams of eight to 15 produced the lowest mean time for bug resolution. Increasing the percentage of bugs assigned to specific developers or boosting developer participation in other OSS projects also improved performance. Furthermore, project type introduced variability in project team performance.     

The linkage between reporting quality and performance in IS projects

Recent research has suggested that flawed status reporting is a serious concern in IS projects. However, the linkage between reporting quality and project performance has not been empirically confirmed. Our investigation consisted of two complementary studies, both of which employed previously validated measures of reporting quality and project outcomes. The first considered the perceptions of 210 IS project members of the quality of status reports they submitted to their project managers. The second considered the perceptions of 485 IS project managers of the quality of project reports they received from project members reporting to them. Both showed that the perceived quality of project reporting was less than perfect and was significantly associated with project task and psychological outcomes. Moreover, the second study results suggested that reporting quality was also related to organizational outcomes. We offer recommendations for project managers.