Towards validation and refinement of rule-based systems

A methodology for the validation of rule-based expert systems is presented as a multi-step process that has three central themes: (1) creation of a useful set of test inputs that cover the domain; (2) a methodology that evaluates the system's responses to the test inputs and compares it to the responses of a panel of human experts; and (3) use of the validation results for system refinement. The presented methodology is performed in a loop. The starting point is a rule base and the loop ends up in a (hopefully) better rule base. The first three steps of this process have been published as separate issues in earlier papers by the authors. Here, these issues are just outlined and references are provided for detailed information. This paper gives an overview of the entire process and describes the relation between its steps. The last step, system refinement, is the new scientific issue here and thus, presented in more detail. Here, the rules are modified according to the results of evaluating the test cases. The base of this rule base reconstruction is both a 'rule-associated validity' and the existence of a 'better rated' human solution.