Evaluation of decision-tree rating scales for mental workload estimation

Recent studies suggest that a decision-tree rating scale called the Modified Cooper-Harper (MCH) rating scale is a globally sensitive indicator of change in mental loading (Wierwille and Casali 1983 a). The present study was directed at developing refinements in the scale and at obtaining additional background information. The MCH scale and five design variations of the scale were studied in two independent aircraft-simulator experiments. Aspects studied were the decision-tree structure, the number of categories, the decision sequence and the effects of computer implementation. Results using the rating scales indicate that the MCH scale and its computerized version are generally more consistent than the others. Attendant questionnaire results indicate that pilots base their ratings on the same factors that researchers believe are the important elements of the multidimensional construct of workload.     

Validity and reliability of rating scales for seated pressure discomfort

To investigate seated pressure discomfort requires a valid and reliable technique to measure discomfort. The aims of this study were to test the validity and reliability of several rating scales and select the best for investigation of pressure discomfort. Six scales were tested: a category partitioning scale, the Borg CR-10 scale, the Corlett discomfort scale, an 8-point ordinal scale, a modified intensity and discomfort scale, and a 21-point ratio scale. Twelve subjects took part in the repeated measures test in two sessions, one week apart. A test seating device generated interface pressure from underneath a foam cushion. Four levels of stimulus, 60, 85, 120, and 165 mmHg, were presented to the seated mid-thigh region. Perceived pressure intensity, discomfort level due to the pressure, and overall discomfort were reported using each of the six scales. Reliability was examined by test-retest correlation, relative rating change and coefficient of variations, and validity examined by absoluteness of rating and the functional consistency. Generally, subjects were capable of reporting their sensation of pressure intensity and discomfort by using the rating scale technique. However, the accuracy of rating strongly depended upon the properties of the scale. The category partitioning scale was found to be highly reliable and most valid for rating pressure intensity and perceived discomfort. This scale was also preferred by subjects when compared with the other five scales. Properties of all the six scales were defined and summarised.     

Computer-based rating method for evaluating multiple visual stimuli on multiple scales

In this research, two joint evaluation rating methods (focus-on-attribute and drag-and-drop) and a separate evaluation rating method (focus-on-stimulus) are proposed for rating multiple visual stimuli with respect to multiple scales. All three interactive methods incorporate a real-time adjusting mechanism, allowing respondents to interactively adjust their ratings. Using the traditional paper-and-pencil method as the basis for comparison, the performance of these computer-based rating methods and the proposed real-time adjusting mechanism were investigated. First, it was found that the computer-based methods not only produced results equivalent to those produced by the paper-and-pencil method, but also improved the data quality by preventing the occurrence of missing values and allowing respondents to adjust their scores. Second, the two joint evaluation methods performed better than the separate evaluation method in terms of usage experience, test–retest reliability, and the likelihood of simulating the daily shopping experience, in which multiple consumer products are often compared simultaneously. Of the two joint evaluation methods, the drag-and-drop method received better subjective evaluations than the focus-on-attribute method. Third, the proposed real-time adjusting mechanism was found to obtain better subjective satisfaction in rating. In summary, for measuring multiple visual stimuli with multiple scales, the computer-based joint evaluation rating methods using a real-time adjusting mechanism were found to be better choices among the measurement tools investigated.     

Subjective rating scales in ergonomics: a reply

Comments on the target paper are acknowledged. Whilst there is still some concern that subjective rating scales are scientifically suspect, the general view is that verbal reports, including ratings, constitute objective data, which can be of considerable value in ergonomics research and practice. The main anxiety attached to their use is to ensure acceptable levels of reliability and validity. Quantification in the strict sense can be achieved by some measures but is by no means essential for all scientific and practical purposes. The value of ordinal and qualitative data obtained by subjective judgements should not be underestimated, especially in predicting future performance. Many commonly used constructs such as fatigue, stress, mental workload, usability, etc. are complex and multidimensional, often combining both ‘subjective’ and ‘objective’ measures. The validity of individual dimensions and complex constructs lies principally in their relationships with other variables of interest in the context of the specific investigation. The question of design standards based partly or wholly on such scales should therefore be treated with some caution.     

How scales influence user rating behaviour in recommender systems

Many websites allow users to rate items and share their ratings with others, for social or personalisation purposes. In recommender systems in particular, personalised suggestions are generated by predicting ratings for items that users are unaware of, based on the ratings users provided for other items. Explicit user ratings are collected by means of graphical widgets referred to as ‘rating scales’. Each system or website normally uses a specific rating scale, in many cases differing from scales used by other systems in their granularity, visual metaphor, numbering or availability of a neutral position. While many works in the field of survey design reported on the effects of rating scales on user ratings, these, however, are normally regarded as neutral tools when it comes to recommender systems. In this paper, we challenge this view and provide new empirical information about the impact of rating scales on user ratings, presenting the results of three new studies carried out in different domains. Based on these results, we demonstrate that a static mathematical mapping is not the best method to compare ratings coming from scales with different features, and suggest when it is possible to use linear functions instead.     

Examples of the use of rating scales in ergonomics research

The majority of ergonomics investigations using rating scales employ either an analogue or a category type of scale. Both have their limitations. This paper discusses a method of combining the two scales to produce a valuable assessment tool for the ergonomist. Examples of its use in both laboratory and field situations are discussed, as are its limitations.     

The effect of scale manipulations on validity: targetting frequency rating scales for anticipated performance levels

A common problem with standard five-point frequency rating scales is their inability to differentiate between objects within a relatively narrow band of the rating dimension. Two alternatives for increasing a scale's ability to reflect existing differences are: increasing the number of positions on the rating scale, or packing the rating scale with quantifiers from a particular portion of the frequency dimension. In this study, three types of rating scale - a standard five-point balanced scale, a longer nine-point balanced scale, and a five-point packed scale - were used to rate two videotaped samples of behaviour, one displaying performance levels from 10% to 100% and the other from 70% to 90%. Each subject's ratings were correlated with true performance levels in each sample of behaviour as a measure of validity. Results showed that for ratings of the wide range of behaviour all three types of scales provided average correlations between the ratings and actual frequencies of the event which exceeded 0.90, with the longer nine-point scale yielding a significantly higher mean correlation than the other two scales. For ratings of the narrow performance range, the nine-point scale provided the highest correlation with the actual frequencies, followed by the packed scale and the standard five-point balanced scale. All differences were significant. Findings suggest that increasing the number of scale positions can significantly increase the validity of ratings obtained. Also, though to a lesser degree, validity of ratings may be enhanced in a shorter scale by using quantifiers from the portion of the frequency continuum where performance are anticipated to lie.     

The effect of set size on the interpretation of quantifiers used in rating scales

Two studies are reported which investigated how people interpret quantifiers of amount such as are commonly used in questionnaires and rating scales. The results indicated that the interpretation of certain quantifiers and rating scales. The results indicated that the interpretation of certain quantifiers varied depending on the context in which they occurred. Low-magnitude quantifiers (e g, 'few', 'several') seemed to signify a much greater proportion when they described small set sizes than when they described relatively large ones. This means that it will be virtually impossible to find quantifiers for use in rating scales which achieve the desirable property of interval scaling. Despite this, some quantifiers are clearly more consistent in their interpretation and more appropriate to use than others, and recommendations are made as to the best ones to use in different situations.     

Faster heuristics for graph burning

Applied Intelligence - Graph burning is a process of information spreading through the network by an agent in discrete steps. The problem is to find an optimal sequence of nodes that have to be...     

Iterative-improvement-based declustering heuristics for multi-disk databases

Data declustering is an important issue for reducing query response times in multi-disk database systems. In this paper, we propose a declustering method that utilizes the available information on query distribution, data distribution, data-item sizes, and disk capacity constraints. The proposed method exploits the natural correspondence between a data set with a given query distribution and a hypergraph. We define an objective function that exactly represents the aggregate parallel query-response time for the declustering problem and adapt the iterative-improvement-based heuristics successfully used in hypergraph partitioning to this objective function. We propose a two-phase algorithm that first obtains an initial K-way declustering by recursively bipartitioning the data set, then applies multi-way refinement on this declustering. We provide effective gain models and efficient implementation schemes for both phases. The experimental results on a wide range of realistic data sets show that the proposed method provides a significant performance improvement compared with the state-of-the-art declustering strategy based on similarity-graph partitioning.     

New heuristics for planning operating rooms

We tackle the operating room planning problem of the Plastic Surgery and Major Burns Specialty of the University Hospital “Virgen del Rocio” in Seville (Spain). The decision problem is to assign an intervention date and an operating room to a set of surgeries on the waiting list, minimizing access time for patients with diverse clinical priority values. This problem has been previously addressed in the literature considering different objective functions. The clinical priority depends on the surgery priority and the number of days spent on the waiting list. We propose a set of 83 heuristics (81 constructive heuristics, a composite heuristic, and a meta-heuristic) based on a new solution encoding, and we compare these methods against existing heuristics from the literature for solving operating room planning problems. The heuristics are adapted to the problem under consideration (i.e. considering all constraints and the new objective function), being re-implemented using the information provided by the authors. In total, after a calibration procedure, we compare 17 heuristics. The computational experiments show that our proposed meta-heuristic is the best for the problem under consideration. Finally, the proposed heuristics are tested using data from the Plastic Surgery and Major Burns Specialty. The results show significant improvements on several key performance indicators (number of scheduled surgeries, quality of surgical plan, resources utilization, etc.) when comparing with the actual results obtained by the specialty in the current practice. The aforementioned hospital is currently implementing the heuristic methods.     

On greedy heuristics for steiner minimum trees

We disprove a conjecture of Shor and Smith on a greedy heuristic for the Steiner minimum tree by showing that the length ratio between the Steiner minimum tree and the greedy tree constructed by their method for the same set of points can be arbitrarily close to3/2. We also propose a new conjecture.Supported in part by the National Science Foundation under Grant CCR-9208913.     

Set-structured and cost-sharing heuristics for classical planning

Problem abstractions based on (either completely or partially) ignoring delete effects of the actions provide the basis for some seminal classical planning heuristics. However, the palette of the conceptual tools exploited by these heuristics remains rather limited. We study a framework for approximating the optimal cost solutions for problems with no delete effects that bridges between certain works on heuristic-search classical planning and on probabilistic reasoning. Our analysis results in developing a novel heuristic function that combines “informed” set-structured cost estimates and “conservative” action cost sharing. Our empirical comparative evaluation provides a clear evidence for the attractiveness of this heuristic estimate. In addition, we examine a (suggested before in the context of probabilistic reasoning) admissible heuristic based on a stronger variant of action cost sharing. We show that what is good for “typical” problems of probabilistic reasoning turns out not to be so for “typical” problems of classical planning, and provide a formal account for that difference.     

Efficient heuristics for all-port multicast in wormhole-routedhypercubes

This paper presents several recursive heuristic methods for multicasting in all-port dimension-ordered wormhole-routed hypercubes. The methods described are stepwise contention-free and are primarily designed to reduce the number of communication steps required to complete the multicast operation. Comparisons show that the number of steps can be significantly reduced compared to depth contention-free solutions previously described. These methods are also shown to be source-controlled depth contention-free and can be considered a generalization of the broadcast method described previously, which is the most efficient method known     

Some heuristics for the consensus ranking problem

In this paper, we propose a consensus ranking function and several heuristics to solve the consensus ranking problem. Empirical testing results of the solution strategies are presented to indicate their solution efficiency.