Accuracy statistics for judging soft classification

In the literature one can find different accuracy measures that are built from the error matrix. However, standard accuracy assessment, which is based on the error matrix, is incomplete when dealing with fuzzy sets or when errors do not have the same importance. In this paper, we propose an extension of the error concept for soft (or crisp) classification that will be able to extend standard accuracy measures (e.g., overall, producer's, user's or Kappa statistic) that can be used in any framework: errors with different importance, soft classifier and crisp reference data (expert) or with a fuzzy expert. In particular, a weighted measure is built that takes into account the preferences of the decision maker in order to differentiate some errors that must not be considered equal.