An Improved Usability Measure Based on Novice and Expert Performance

The novice–expert ratio method (NEM) pinpoints user interface design problems by identifying the steps in a task that have a high ratio of novice to expert completion time. This study tested the construct validity of NEM's ratio measure against common alternatives. Data were collected from 337 participants who separately performed 10 word-completion tasks on a cellular phone interface. The logarithm, ratio, Cohen's d, and Hedges's ? measures had similar construct validity, but Hedges's ? provided the most accurate measure of effect size. All these measures correlated more strongly with self-reported interface usability and interface knowledge when applied to the number of actions required to complete a task than when applied to task completion time. A weighted average of both measures had the highest correlation. The relatively high correlation between self-reported interface usability and a weighted Hedges's ? measure as compared to the correlations found in the literature indicates the usefulness of the weighted Hedges's ? measure in identifying usability problems.