Quantifying uncertainty in multicriteria concept selection methods

Decision support for concept selection in engineering design ranges from simple pairwise comparison techniques to methods that consider multiple criteria. The Analytic Hierarchy Process, or AHP, is an example of a multicriteria selection tool with wide-spread industry application. It is recognized by responsible practitioners that AHP, like other decision support methods, is best used not as an optimization tool, but as a means of clarifying individual or group attitudes; the numerical rankings that are its output are not definitive. This paper offers a means to quantify how differently two alternatives must be ranked by AHP to instill confidence that one is truly better than the other, a question that is in practice always answered using intuition. The quantification of uncertainty in AHP relies on the extension of concepts from statistical hypothesis testing. The procedure is not stochastic in the same way that physical measurement is, so probability distributions are created over relevant parameters. The quantified uncertainty depends, as in all statistical analysis, upon the assumed distributions. The uncertainty in AHP is quantified from two distinct points of view. The first makes the assumption that AHP is structurally correct but subject to measurement “error” in the pairwise comparisons, while the second quantifies the uncertainties introduced by AHP’s failure to consider different level of compensation in trade-offs among criteria.