Notions of thoroughness, efficiency, and validity: Are they valid in HCI practice?

In this paper I argue that while the notions of thoroughness, efficiency, and validity of problems identified in usability tests are mandatory for researchers seeking to establish the effectiveness of a given testing procedure, especially the notions of thoroughness and efficiency are irrelevant in HCI practice. In research devoted to validating a given test methodology, it is imperative that all usability problems be identified in the product or application used as a test bed. In HCI practice, however, the objective is to identify as many usability problems as possible with limited resources and within a limited time frame, to define and implement solutions to these early in the development process. It is impossible to know whether all usability problems have been identified in a particular test or type of evaluation unless testing is repeated until it reaches an asymptote, a point at which no new problems emerge in a test. Asymptotic testing is not, and should not be, done in practice; it is as unfeasible as it is irrelevant in a work context. In the absence of a complete usability problem set, the notions of thoroughness and efficiency are meaningless and also impossible to calculate. While validity can be assessed for individual problems in practical usability tests, it cannot yield information about the effectiveness of the testing method per se because the problem set is unlikely to be complete. An example is provided to support my argument.

Relevance to industry

The point of a usability test is to identify as many usability problems as possible, with limited resources. The resulting problems hopefully include the most severe ones, but not the entire problem set. While HCI practitioners should know about thoroughness, efficiency, and validity of the test methods they elect to employ, they should not attempt to assess these from their test findings. Neither thoroughness nor efficiency can be determined from an incomplete problem set, and the notion of validity is tied to the severity of individual usability problems, not to the testing method as such.