Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Examined the effects of knowledge of a ratee's prior performance on evaluations of present performance. Subjects received knowledge of either good or poor prior performances and then viewed and rated a videotape depicting average performance. In Study 1, some subjects received knowledge of the ratee's prior performance by directly viewing videotapes of good or poor ratee behavior, whereas others only reviewed written performance ratings completed by those subjects who had actually viewed the ratee. A contrast effect occurred when knowledge of prior performance was obtained by observing ratee behavior, but an assimilation effect occurred when knowledge of prior performance was obtained by reviewing performance ratings. In Study 2, subjects viewed videotapes of good or poor performances prior to viewing an average performance by the same ratee. However, the separate ratee performances were observed over a more realistic time interval than that used in Study 1 (3 weeks vs. 1 h). No significant contrast effects were observed. In Study 3, subjects reviewed written ratings of prior performances before viewing an average videotape. Subjects who reviewed extremely good (or poor) prior performance ratings provided more extreme ratings of the "average" performance than did subjects who reviewed less extreme ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bias in Behaviorally Anchored Rating Scales: Global or scale-specific?

Murphy and Constans (1987) showed that behavioral anchors can be a source of bias in ratings but did not determine whether bias in ratings of one dimension would spread to rating of other performance dimensions. In two separate studies, we manipulated behavioral anchors on one of the scales developed by Murphy and Constans (1987) and examined the effect of bias in these scales on ratings of dimensions that varied in their conceptual and evaluative similarity to the dimension that was manipulated. Neither study showed evidence that bias in ratings of one dimension spreads to ratings of other performance dimensions. These results suggest that rating bias of the sort reported by Murphy and Constans will not have a substantial effect on performance ratings in organizations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive categorization and dimensional schemata: A process approach to the study of halo in performance ratings.

Compared 2 models of the cognitive processes underlying performance ratings: a traditional model outlined by W. C. Borman (see record 1980-26801-001), and a cognitive categorization model proposed by J. M. Feldman (see record 1981-24524-001). To distinguish these 2 models, 120 college students were shown 1 of 2 videotapes of a college lecturer in which 3 of 5 dimensions of performance were manipulated to be opposite to that of the lecturer's overall performance. Ratings were made either immediately after viewing the videotape or 2 days later. Results indicate that the traditional model was appropriate for describing the rating process in both the immediate and the delayed rating conditions. However, a large halo effect was also found that was consistent with the categorization model despite conditions designed to minimize the likelihood of halo. Additional effects of cognitive categorization included a tendency to make errors in later recall of lecturing incidents consistent with Ss' general impression. (48 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The consequences of halo "error" in performance ratings: A field study of the moderating effect of halo on test validation results.

Performance ratings of 294 clerical workers in a validation study of clerical ability tests indicated that halo, measured as the standard deviation across dimensions, consistently moderated the relationships between dimension ratings and scores on valid tests. Greater halo resulted in higher validity coefficients, and also was related to higher performance ratings. In an additional analysis, statistically controlling for the effect of the overall rating on dimension ratings resulted in poorer validation results, with dimension ratings rarely adding additional variance to that of overall ratings. The results of this study contradict the traditionally held view of halo as a rating "error," and are consistent with recent laboratory studies that have found accuracy and halo positively related. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater selective attention ability and appraisal effectiveness: The effect of a cognitive style on the accuracy of differentiation among ratees.

Examined the effect of the level of selective-attention ability of 359 undergraduate raters on the accuracy of ratings in 2 studies. A group-administered measure of field dependence–independence was used to divide Ss into high- or low-selectivity levels. In Study 1, 288 psychology undergraduates rated 4 vignettes of instructors by either direct inspection or from memory. In Study 2, 71 management undergraduates rated the vignettes. Results show that memory manipulation significantly influenced rating accuracy. In both studies, Ss high in selectivity provided significantly more accurate appraisals than did Ss low in selectivity. It is suggested that cognitive operations underly the effects found in these studies. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Retrieval processes underlying confidence in comprehension judgments.

People's ratings of confidence in ability to answer questions about previously studied expository passages are virtually uncorrelated with test performance. This correlation would be attenuated if confidence ratings were influenced by momentary accessibility (e.g., latency to initial access of probed information) and if test performance were influenced by other aspects of retrieval (e.g., total recall over an interval). To see if momentary accessibility influences confidence ratings, Experiment 1 manipulated accessibility by using self- versus other-generated title probes. Both confidence and accessibility measures varied with generation condition. Experiment 2 provided three findings. First it assessed whether accessibility is the only common factor underlying confidence and domain familiarity ratings by observing whether confidence and familiarity covary wth accessibility partialed out. Marginal covariation implied either some common factor other than accessibility, imperfect measurement of accessibility, or both. Second, negative correlations of rating latencies with rating magnitudes implied that access latency, momentary access rate, or both, affect the ratings. Third, calibration of comprehension was poor, but nonzero. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Affect and appraisal accuracy: Liking as an integral dimension in evaluating performance.

Tested the possibility that liking may influence rating accuracy by operating as an integral dimension in 66 undergraduate raters. Ss evaluated vignettes of 4 professors; liking was manipulated with 40 trait terms (e.g., amusing, greedy, bashful) that engendered different liking levels, but had little implication for professor performance. Results indicate a significant effect on rating accuracy, suggesting that liking is an integral dimension that is difficult to separate from performance dimensions. Results support the potential importance of affect in appraisal. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The category effect with rating scales: Number of categories, number of stimuli, and method of presentation.

Four experiments were conducted, using approximately 500 undergraduates, to compare category ratings of sets of stimuli with different skewing (i.e., positive and negative) under different conditions. Results indicate that squares received higher category ratings when the smaller sizes were presented more frequently than the larger sizes. This shift in the rating scale was greater when there were either fewer categories (the category effect) or more stimuli. Similar shifts were obtained whether the stimuli were presented successively for judgment or simultaneously. The category effect also occurred when Ss were not told how many categories to use until after the contextual stimuli had been presented. It is suggested that a simple range-frequency model describes most of the shifts in scale by variations in a single weighting parameter. However, these shifts are predicted by an elaborated model in which the number of representations of any stimulus in working memory is limited by a principle of consistent assignment of each stimulus to a single category. It is concluded that this elaborated model correctly predicts the disappearance of the category effect when contexts are manipulated by varying the spacing of stimulus values rather than by varying their relative frequencies. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The systematic distortion hypothesis, halo, and accuracy: An individual-level analysis.

In several social perception studies investigators have concluded that raters' semantic conceptual similarity schemata serve to guide and constrain dimensional covariance in the rating judgment process. This effect has been hypothesized to be most likely when ratings are memory based and raters lack relevant job or ratee information. Recent research that has explored the effects of conceptual similarity schemata on performance ratings and halo error has provided some limited support for this systematic distortion hypothesis (SDH). However, these studies are limited because researchers have examined this phenomenon using group-level analyses, whereas the theory references individual-level judgment processes. The present study investigated the phenomena at the individual level. The effects of varying levels of rater job knowledge (high, medium, and low) and familiarity with ratees (high and low) were examined for conceptual similarity–rating and rating–true-score covariation relations, for measures of halo, and for rating accuracy components. Results provided support for the SDH, but indicated a boundary condition for its operation and revealed some surprising findings for individual-level rater halo. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interdependence of successive absolute judgments.

The aim was "to determine whether successive absolute judgments of a cognitive nature are interdependent, and if so, to evaluate a method for controlling this bias." The design involved rating populations of countries, with one group of Ss rating a "test" country following rating of a sparsely populated one and a second group following the rating of a populous country. This type of design was used several times in longer lists of words. Results of 8 test items indicated that "ratings tend to be biased in the direction of the previous rating… bias increases as the number of response categories increases." When the Ss were instructed to rate extreme stimuli first, no bias effect was found. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues.

Considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. 93 college students (mean age 22 yrs) trained using either FOR or control procedures, observed 3 managers on videotape, and rated the managers on 3 performance dimensions. Results supported the hypothesis that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An investigation of the validity of expert true score estimates in appraisal research.

Researchers of appraisal rating accuracy have defined ratee true scores as the mean ratings given by experts provided with enhanced opportunities to observe performance. The external validity of accuracy research depends on the relevance of these expert estimates to true performance. In our study, we provided expert raters with enhanced opportunities to view videotapes of five ratees under conditions of high true dimension intercorrelation or low true intercorrelation. The accuracy of expert ratings was compared with the accuracy of nonexpert ratings of the same tapes viewed under more typical rating conditions. Subjects' ratings were compared with ratee true scores defined in terms of objective worker output. Results indicated that experts were more accurate than nonexperts, regardless of the true dimension intercorrelations. Accuracy indices computed by using objective true scores were highly correlated with indices computed by using the mean expert ratings as true score estimates. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Attitudes, self-monitoring, and appraisal behaviors.

This study examined the influence of attitudes and self-monitoring on leniency (elevation accuracy) of performance ratings and personnel decisions. In addition, moderating effects of self-monitoring on the relationship between attitudes and accuracy of ratings and decisions were investigated. Attitudes and self-monitoring tendency of 210 managers–professionals were measured, and ratings provided and decisions made by them were used to test 3 sets of hypotheses. Moderated regression and follow-up split-group analyses indicated that self-monitoring moderated the relationship between attitudes toward accurate appraisal and rating accuracy. Self-monitoring significantly influenced rating and decision accuracy such that accuracy declined with increasing level of self-monitoring. Results highlight the influence of rater's personality on appraisal behaviors. Implications of results and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of the purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance.

Previous research has suggested that ratings used to make administrative decisions are lenient when compared with ratings obtained for research purposes only. The present study examined the effects of the purpose of rating on multivariate measures of accuracy in observing teacher behavior as well as measures of accuracy in evaluating teaching performance. 45 undergraduates viewed and evaluated videotaped lectures under conditions in which they were informed that their ratings would be used for research only or for making important decisions about those being rated. The purpose of rating did not affect measures of accuracy in rating the frequency with which a number of critical behaviors occurred on each tape. The purpose of rating also did not affect multivariate measures of performance rating accuracy. Purpose did, however, affect the relationship between accuracy in observing teacher behavior and accuracy in evaluating teaching performance. It is suggested that purpose affects the way in which raters process behavioral information without necessarily affecting the general level of rating. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Convergence of stranger ratings of personality and intelligence with self-ratings, partner ratings, and measured intelligence.

Several studies have shown above-chance agreement of self-reports on extraversion and conscientiousness with ratings by strangers, indicating that ratings by strangers might be quite accurate. Because self-reports are a less-than-ideal criterion to evaluate the accuracy of stranger ratings, however, the present study compared them also with ratings by acquaintances and with targets' performance on an intelligence test. Ratings of extraversion, conscientiousness, and intelligence by strangers having been exposed to a videotape of targets were significantly related to self-reports of these traits as well as to ratings by acquaintances. Moreover, ratings of intelligence by strangers were related to targets' measured intelligence, provided that judges had been exposed to a sound film of the targets. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of subsequent performance on evaluations of previous performance.

Tested the hypothesis that subsequent performance levels would bias the recall and evaluations of a ratee's previous level of performance with 183 undergraduates, who rated 3 videotaped lectures in either immediate or delayed rating conditions. The 1st videotape depicted an average level of performance and was followed by either 2 good lectures or 2 poor lectures. A significant performance level?×?time of rating interaction was found, in which memory-based ratings were biased in the direction of subsequent performance (i.e., when there was a delay between observation and rating, Ss who had seen an average lecture followed by good lectures rated that average lecture more favorably than did Ss who had seen that same lecture followed by poor lectures). It is suggested that raters are biased in favor of recalling behaviors that are consistent with their general impression of a ratee and that subsequent performance may systematically alter the rater's recall of the ratee's previous behavior. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)