Information systems effectiveness: research designs for causal inference

Abstract. This paper examines the capacity of the research designs of 37 empirical studies of information systems (IS) effectiveness to provide a basis for the development of theories about behaviour related to IS effectiveness. The power of each study to support causal inference was evaluated in terms of (a) its handling of the time dimension, (b) its ability to weigh differences and (c) its resistance to internal validity threats that pose alternative explanations for its reported findings. Of the reviewed studies, 29.7% could account for the time dimension, 32.4% employed a comparison group and 16.2% were not susceptible to any internal validity threats. Only 13.5% of the studies combined an accounting for the time dimension with the use of a comparison group. Of these, however, only 5.4% were also invulnerable to internal validity threats. The research designs of nearly 95% of these published studies were deficient in supporting causal inference. In those studies, suggestions that one variable was causally related to another variable could not be substantiated. Encouragement for the future capacity of IS effectiveness research to support causal inference was found in a trend towards the use of quasiexperimental designs. Recommendations are made regarding ways to increase the inferential capacity of research designs employed in the study of IS effectiveness.