Measuring users' performance with interactive information systems

Abstract The purpose of this study was to investigate the discrete forms of knowledge required for successful use of interactive information systems (IIS). Using an existing framework, an instrument was created to measure the achievement of three discrete forms of knowledge among novice users of an CD-ROM encyclopedia. The instrument included tasks designed to measure users' knowledge and skills in managing and controlling the system, navigating between information nodes and selecting appropriate search strategies given information related problems. The study examined hierarchical knowledge and compared the development of these forms of knowledge and skills among novices using two different IIS. The results found significant differences in the user performances between two systems in tasks that measured navigation skills and an understanding of the information organisation. It is suggested that these differences were due to differences in the interface design of the two systems and suggestions are made for selecting and implementing IIS systems in school and educational settings.