Evaluation of decision-tree rating scales for mental workload estimation

Recent studies suggest that a decision-tree rating scale called the Modified Cooper-Harper (MCH) rating scale is a globally sensitive indicator of change in mental loading (Wierwille and Casali 1983 a). The present study was directed at developing refinements in the scale and at obtaining additional background information. The MCH scale and five design variations of the scale were studied in two independent aircraft-simulator experiments. Aspects studied were the decision-tree structure, the number of categories, the decision sequence and the effects of computer implementation. Results using the rating scales indicate that the MCH scale and its computerized version are generally more consistent than the others. Attendant questionnaire results indicate that pilots base their ratings on the same factors that researchers believe are the important elements of the multidimensional construct of workload.     

Validity and reliability of rating scales for seated pressure discomfort

To investigate seated pressure discomfort requires a valid and reliable technique to measure discomfort. The aims of this study were to test the validity and reliability of several rating scales and select the best for investigation of pressure discomfort. Six scales were tested: a category partitioning scale, the Borg CR-10 scale, the Corlett discomfort scale, an 8-point ordinal scale, a modified intensity and discomfort scale, and a 21-point ratio scale. Twelve subjects took part in the repeated measures test in two sessions, one week apart. A test seating device generated interface pressure from underneath a foam cushion. Four levels of stimulus, 60, 85, 120, and 165 mmHg, were presented to the seated mid-thigh region. Perceived pressure intensity, discomfort level due to the pressure, and overall discomfort were reported using each of the six scales. Reliability was examined by test-retest correlation, relative rating change and coefficient of variations, and validity examined by absoluteness of rating and the functional consistency. Generally, subjects were capable of reporting their sensation of pressure intensity and discomfort by using the rating scale technique. However, the accuracy of rating strongly depended upon the properties of the scale. The category partitioning scale was found to be highly reliable and most valid for rating pressure intensity and perceived discomfort. This scale was also preferred by subjects when compared with the other five scales. Properties of all the six scales were defined and summarised.     

Computer-based rating method for evaluating multiple visual stimuli on multiple scales

In this research, two joint evaluation rating methods (focus-on-attribute and drag-and-drop) and a separate evaluation rating method (focus-on-stimulus) are proposed for rating multiple visual stimuli with respect to multiple scales. All three interactive methods incorporate a real-time adjusting mechanism, allowing respondents to interactively adjust their ratings. Using the traditional paper-and-pencil method as the basis for comparison, the performance of these computer-based rating methods and the proposed real-time adjusting mechanism were investigated. First, it was found that the computer-based methods not only produced results equivalent to those produced by the paper-and-pencil method, but also improved the data quality by preventing the occurrence of missing values and allowing respondents to adjust their scores. Second, the two joint evaluation methods performed better than the separate evaluation method in terms of usage experience, test–retest reliability, and the likelihood of simulating the daily shopping experience, in which multiple consumer products are often compared simultaneously. Of the two joint evaluation methods, the drag-and-drop method received better subjective evaluations than the focus-on-attribute method. Third, the proposed real-time adjusting mechanism was found to obtain better subjective satisfaction in rating. In summary, for measuring multiple visual stimuli with multiple scales, the computer-based joint evaluation rating methods using a real-time adjusting mechanism were found to be better choices among the measurement tools investigated.     

Subjective rating scales: science or art?

Subjective rating scales are widely used in almost every aspect of ergonomics research and practice for the assessment of workload, fatigue, usability, annoyance and comfort, and lesser known qualities such as urgency and presence, but are they truly scientific? This paper raises some of the key issues as a basis for debate. First, it is argued that all empirical observations, including those conventionally labelled as ‘objective’, are unavoidably subjective. Shared meaning between observers, or intersubjectivity, is the key criterion of scientific probity. The practical steps that can be taken to increase intersubjective agreement are discussed and the well-known sources of error and bias in human judgement reviewed. The role of conscious experience as a mechanism for appraising the environment and guiding behaviour has important implications for the interpretation of subjective reports. The view that psychometric measures do not conform to the requirements of truly ‘scientific’ measurement is discussed. Human judgement of subjective attributes is essentially ordinal and, unlike physical measures, can be matched to interval scales only with difficulty, but ordinal measures can be used successfully both to develop and test substantive theories using multivariate statistical techniques. Constructs such as fatigue are best understood as latent or inferred variables defined by a set of manifest or directly observed indicator variables. Both construct validity and predictive validity are viewed from this perspective and this helps to clarify several problems including the dissociation between measures of different aspects of a given construct, the question of whether physical (e.g. physiological) measures should be preferred to subjective measures and whether a single measure of constructs which are essentially multidimensional having both subjective and physical components is desirable. Finally, the fitness of subjective ratings to different purposes within the broad field of ergonomics research is discussed. For testing of competing hypotheses concerning the mechanisms underlying human performance, precise quantitative predictions are rarely needed. The same is frequently true of comparative evaluation of competing designs. In setting design standards, however, something approaching the level of measurement required for precise quantitative prediction is required, but this is difficult to achieve in practice. Although it may be possible to establish standards within restricted contexts, general standards for broadly conceived constructs such as workload are impractical owing to the requirement for representative sampling of tasks, work environments and personnel.     

Subjective rating scales in ergonomics: a reply

Comments on the target paper are acknowledged. Whilst there is still some concern that subjective rating scales are scientifically suspect, the general view is that verbal reports, including ratings, constitute objective data, which can be of considerable value in ergonomics research and practice. The main anxiety attached to their use is to ensure acceptable levels of reliability and validity. Quantification in the strict sense can be achieved by some measures but is by no means essential for all scientific and practical purposes. The value of ordinal and qualitative data obtained by subjective judgements should not be underestimated, especially in predicting future performance. Many commonly used constructs such as fatigue, stress, mental workload, usability, etc. are complex and multidimensional, often combining both ‘subjective’ and ‘objective’ measures. The validity of individual dimensions and complex constructs lies principally in their relationships with other variables of interest in the context of the specific investigation. The question of design standards based partly or wholly on such scales should therefore be treated with some caution.     

How scales influence user rating behaviour in recommender systems

Many websites allow users to rate items and share their ratings with others, for social or personalisation purposes. In recommender systems in particular, personalised suggestions are generated by predicting ratings for items that users are unaware of, based on the ratings users provided for other items. Explicit user ratings are collected by means of graphical widgets referred to as ‘rating scales’. Each system or website normally uses a specific rating scale, in many cases differing from scales used by other systems in their granularity, visual metaphor, numbering or availability of a neutral position. While many works in the field of survey design reported on the effects of rating scales on user ratings, these, however, are normally regarded as neutral tools when it comes to recommender systems. In this paper, we challenge this view and provide new empirical information about the impact of rating scales on user ratings, presenting the results of three new studies carried out in different domains. Based on these results, we demonstrate that a static mathematical mapping is not the best method to compare ratings coming from scales with different features, and suggest when it is possible to use linear functions instead.     

Examples of the use of rating scales in ergonomics research

The majority of ergonomics investigations using rating scales employ either an analogue or a category type of scale. Both have their limitations. This paper discusses a method of combining the two scales to produce a valuable assessment tool for the ergonomist. Examples of its use in both laboratory and field situations are discussed, as are its limitations.     

The effect of scale manipulations on validity: targetting frequency rating scales for anticipated performance levels

A common problem with standard five-point frequency rating scales is their inability to differentiate between objects within a relatively narrow band of the rating dimension. Two alternatives for increasing a scale's ability to reflect existing differences are: increasing the number of positions on the rating scale, or packing the rating scale with quantifiers from a particular portion of the frequency dimension. In this study, three types of rating scale - a standard five-point balanced scale, a longer nine-point balanced scale, and a five-point packed scale - were used to rate two videotaped samples of behaviour, one displaying performance levels from 10% to 100% and the other from 70% to 90%. Each subject's ratings were correlated with true performance levels in each sample of behaviour as a measure of validity. Results showed that for ratings of the wide range of behaviour all three types of scales provided average correlations between the ratings and actual frequencies of the event which exceeded 0.90, with the longer nine-point scale yielding a significantly higher mean correlation than the other two scales. For ratings of the narrow performance range, the nine-point scale provided the highest correlation with the actual frequencies, followed by the packed scale and the standard five-point balanced scale. All differences were significant. Findings suggest that increasing the number of scale positions can significantly increase the validity of ratings obtained. Also, though to a lesser degree, validity of ratings may be enhanced in a shorter scale by using quantifiers from the portion of the frequency continuum where performance are anticipated to lie.     

New heuristics for one-dimensional bin-packing

Several new heuristics for solving the one-dimensional bin packing problem are presented. Some of these are based on the minimal bin slack (MBS) heuristic of Gupta and Ho. A different algorithm is one based on the variable neighbourhood search metaheuristic. The most effective algorithm turned out to be one based on running one of the former to provide an initial solution for the latter. When tested on 1370 benchmark test problem instances from two sources, this last hybrid algorithm proved capable of achieving the optimal solution for 1329, and could find for 4 instances solutions better than the best known. This is remarkable performance when set against other methods, both heuristic and optimum seeking.Scope and purposePacking items into boxes or bins is a task that occurs frequently in distribution and production. A large variety of different packing problems can be distinguished, depending on the size and shape of the items, as well as on the form and capacity of the bins (H. Dyckhoff and U. Finke, Cutting and Packing in Production and Distribution: a Typology and Bibliography, Springer, Berlin, 1992). Similar problems occur in minimising material wastage while cutting pieces into particular smaller ones and in the scheduling of identical processors in order to minimise total completion time. This work addresses the basic packing problem, known as the one-dimensional bin packing problem, where it is required to pack a number of items into the smallest possible number of bins of pre-specified equal capacity. Even though this problem is simple to state, it is NP hard, i.e., it is unlikely that there exists an algorithm that could solve every instance of it in polynomial time. Solution of more general realistic packing problems is probably contingent upon the availability of effective and computationally efficient solution procedures for the basic problem. In this work we present several heuristics capable of doing that. Extensive computational testing attests to the power of these heuristics, as well as to their computational efficiency.     

The effect of set size on the interpretation of quantifiers used in rating scales

Two studies are reported which investigated how people interpret quantifiers of amount such as are commonly used in questionnaires and rating scales. The results indicated that the interpretation of certain quantifiers and rating scales. The results indicated that the interpretation of certain quantifiers varied depending on the context in which they occurred. Low-magnitude quantifiers (e g, 'few', 'several') seemed to signify a much greater proportion when they described small set sizes than when they described relatively large ones. This means that it will be virtually impossible to find quantifiers for use in rating scales which achieve the desirable property of interval scaling. Despite this, some quantifiers are clearly more consistent in their interpretation and more appropriate to use than others, and recommendations are made as to the best ones to use in different situations.     

Network-based heuristics for constraint-satisfaction problems

Many AI tasks can be formulated as constraint-satisfaction problems (CSP), i.e., the assignment of values to variables subject to a set of constraints. While some CSPs are hard, those that are easy can often be mapped into sparse networks of constraints which, in the extreme case, are trees. This paper identifies classes of problems that lend themselves to easy solutions, and develops algorithms that solve these problems optimally. The paper then presents a method of generating heuristic advice to guide the order of value assignments based on both the sparseness found in the constraint network and the simplicity of tree-structured CSPs. The advice is generated by simplifying the pending subproblems into trees, counting the number of consistent solutions in each simplified subproblem, and comparing these counts to decide among the choices pending in the original problem.     

Faster heuristics for graph burning

Applied Intelligence - Graph burning is a process of information spreading through the network by an agent in discrete steps. The problem is to find an optimal sequence of nodes that have to be...     

Optimization models for determining nitric acid equilibria in supercritical water

Optimization models are developed to determine equilibrium constants for dissociation, redox and disproportionation reactions involving nitric and nitrous acid, NO₂, NO, N₂O, NO⁻₃, and oxygen in supercritical water at temperatures of 380 and 400°C and at pressures of 276–414 bar. A constrained nonlinear programming (NLP) model of moderate size is developed to estimate equilibrium constants and extinction coefficients. This model is optimized using an implementation of the generalized reduced gradient (GRG) algorithm. Although it was only possible to measure the absorbance of two of the twelve species by UV-Vis spectroscopy, six equilibrium constants and two extinction coefficients could be optimized. Significant improvements of these values are obtained with a Jackknife Statistical implementation. Linear extrapolation of the log K vs density plots to zero density results in values which are very close to independent gas-phase values.     

Iterative-improvement-based declustering heuristics for multi-disk databases

Data declustering is an important issue for reducing query response times in multi-disk database systems. In this paper, we propose a declustering method that utilizes the available information on query distribution, data distribution, data-item sizes, and disk capacity constraints. The proposed method exploits the natural correspondence between a data set with a given query distribution and a hypergraph. We define an objective function that exactly represents the aggregate parallel query-response time for the declustering problem and adapt the iterative-improvement-based heuristics successfully used in hypergraph partitioning to this objective function. We propose a two-phase algorithm that first obtains an initial K-way declustering by recursively bipartitioning the data set, then applies multi-way refinement on this declustering. We provide effective gain models and efficient implementation schemes for both phases. The experimental results on a wide range of realistic data sets show that the proposed method provides a significant performance improvement compared with the state-of-the-art declustering strategy based on similarity-graph partitioning.