Metrics for evaluating human information interaction systems

Society today has a wealth of information available due to information technology. The challenge facing researchers working in information access is how to help users easily locate the information needed. Evaluation methodologies and metrics are important tools to assess progress in human information interaction (HII). To properly evaluate these systems, evaluations need to consider the performance of the various components, the usability of the system, and the impact of the system on the end user. Current usability metrics are adequate for evaluating the efficiency, effectiveness, and user satisfaction of such systems. Performance measures for new intelligent technologies will have to be developed. Regardless of how well the systems are and how usable the systems are, it is critical that impact measures are developed. For HII systems to be useful, we need to assess how well information analysts work with the systems. This evaluation needs to go beyond technical performance metrics and usability metrics. What are the metrics for evaluating utility? This paper describes research efforts focused on developing metrics for the intelligence community that measure the impact of new software to facilitate information interaction.