Measurement for evaluating the learnability and resilience of methods of cognitive work

Some experiments on human–computer interaction are aimed at evaluating hypotheses concerning cognitive work. Other experiments are intended to evaluate the software tools that shape the cognitive work. In both cases, effective experimentation is premised on the control and factorial analysis of sources of variability. This entails programmes of experimentation. However, sociotechnical systems are generally a ‘moving target’ in terms of the pace of change. The objective of this study was to create a general approach to experimental design and the measurement of cognitive work that can satisfy the requirements for experimentation and yet can also provide a ‘fast track’ to the evaluation of software-supported cognitive work. A measure called i-bar is presented, which is the inverse of the mid-range. The statistic is derived from data on trials-to-criterion in tasks that require practice and learning. This single measure is interpreted as a conjoint measurement scale, permitting: (a) evaluation of sensitivity of the principal performance measure (which is used to set the metric for trials to criterion); (b) evaluation of the learnability of the work method (i.e. the goodness of the software tool); (c) evaluation of the resilience of the work method. It is shown that it is possible to mathematically model such order statistics and derive methods for estimating likelihoods. This involves novel ways of thinking about statistical analysis for discrete non-Gaussian distributions. The idea and method presented herein should be applicable to the study of the effects of any training or intervention, including software interventions designed to improve legacy work methods and interventions that involve creating entirely new cognitive work systems.