Performance evaluation of rule‐based expert systems: An example from medical billing domain

Most of the research in the area of performance evaluation of rule‐based expert systems (RBESs) is focused on verification and validation issues. Many researchers discuss usability, usefulness, portability, and response time for the evolution of RBES. The final goal of all such studies is to construct a system with optimal, accurate knowledge base. Arguably, a system with best knowledge base is actually worthless if it is never utilized in the real world. We have proposed “benefit” as a measure of evaluation and suggested some guidelines for performance evaluation of RBESs. The proposed measure has been demonstrated for performance evaluation of an RBES applied in the medical billing domain. Results showed that the system has saved hundreds of working hours during the evaluation period of 3 months. Moreover, other associated measures have also been considered. Associated measures in the medical billing domain are “claim rejection rate”—reduced by 54%—and “claim aging,” which has decreased from 34 to 28 days due to the RBES. Guidelines proposed by this research can be applied for the evaluation of expert systems implemented in other application domains, including in the first place business decision support systems.