User studies of knowledge acquisition tools: methodology and lessons learned

Knowledge acquisition research concerned with the development of knowledge acquisition tools is in need of a methodological approach to evaluation. This paper describes experimental methodology to conduct studies and experiments of users modifying knowledge bases with knowledge acquisition tools. The paper also reports on the lessons learned from several experiments that have been performed using this methodology. The hope is that it will help others design user evaluations of knowledge acquisition tools. Ideas are discussed for improving the current methodology and some open issues that remain.