Some soft measures for performance analysis: The “core” dimensions of expert system quality

The post-implementation evaluation of new technologies such as artificial intelligence, multimedia or parallel computing systems which are in operation is a much neglected problem. There is a lack of methods, techniques and tools for the practitioner to analyze the performance such systems. In earlier work, we have presented a framework and methodology that described a socio-technical approach for evaluating expert systems. In short, we determined the features and characteristics of expert systems that are most critical for their “implementation success”. This was the result of empirical evidence from a first-ever field study of expert systems in production. In this paper, specifically tailored to an audience of generalists, we retrospectively put our work into the framework of performance measurement and analysis. The end-result of our investigations is that we have definitional clarity as to how we may alternately evaluate such decision support systems.