Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy.

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are performance appraisal ratings from different rating sources comparable?

The purpose of this study was to test whether a multisource performance appraisal instrument exhibited measurement invariance across different groups of raters. Multiple-groups confirmatory factor analysis as well as item response theory (IRT) techniques were used to test for invariance of the rating instrument across self, peer, supervisor, and subordinate raters. The results of the confirmatory factor analysis indicated that the rating instrument was invariant across these rater groups. The IRT analysis yielded some evidence of differential item and test functioning, but it was limited to the effects of just 3 items and was trivial in magnitude. Taken together, the results suggest that the rating instrument could be regarded as invariant across the rater groups, thus supporting the practice of directly comparing their ratings. Implications for research and practice are discussed, as well as for understanding the meaning of between-source rating discrepancies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training.

A total of 52 supervisory personnel were trained under one of three performance-appraisal training programs: rater error (response set) training, observation training, or decision-making training. Halo, leniency, range restriction, and accuracy measures were collected before and after training from the three training groups and a no-training control group. The results suggested that although the traditional rater error training, best characterized as inappropriate response set training, reduced the classic rater errors (or statistical effects), it also detrimentally affected rating accuracy. However, observation and decision-making training caused performance rating accuracy to increase after training, but did little to reduce classic rater effects. The need for a reconceptualization of rater training content and measurement focus was discussed in terms of the uncertain relation between statistical rating effects and accuracy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Role of the rater in performance appraisal.

Notes that ratings for performance appraisal are frequently made by supervisors. In the present study, judgments of effectiveness for 153 hospital nurses were obtained from the nurse herself and her peers in addition to her supervisor, using the same rating form. Factor analysis indicated that each rating source could be clearly identified and characterized. The data reaffirm the notion that interrater disagreement may reflect systematic rater bias as well as meaningful differences in the ways in which judgments are made. Implications for comprehensive appraisals are suggested. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The systematic distortion hypothesis, halo, and accuracy: An individual-level analysis.

In several social perception studies investigators have concluded that raters' semantic conceptual similarity schemata serve to guide and constrain dimensional covariance in the rating judgment process. This effect has been hypothesized to be most likely when ratings are memory based and raters lack relevant job or ratee information. Recent research that has explored the effects of conceptual similarity schemata on performance ratings and halo error has provided some limited support for this systematic distortion hypothesis (SDH). However, these studies are limited because researchers have examined this phenomenon using group-level analyses, whereas the theory references individual-level judgment processes. The present study investigated the phenomena at the individual level. The effects of varying levels of rater job knowledge (high, medium, and low) and familiarity with ratees (high and low) were examined for conceptual similarity–rating and rating–true-score covariation relations, for measures of halo, and for rating accuracy components. Results provided support for the SDH, but indicated a boundary condition for its operation and revealed some surprising findings for individual-level rater halo. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The frame of reference approach as a solution to an assessment center dilemma.

This article describes how the frame of reference (FOR) approach to rater training for performance appraisal purposes (H. J. Bernardin, 1979; H. J. Bernardin & M. R. Buckley, 1981) was applied to traditional assessment center ratings and rater training. The method by which an FOR was established for the assessment center ratings is presented, including (a) definitions of dimensions of performance, (b) definitions of qualitative levels of performance within each dimension, and (c) specific behavioral examples of levels of performance on an item-by-item basis within dimensions. The resulting FOR was used to structure the training and certification of raters with the expectation of minimizing sources of rater unreliability. Implications for assessment center reliability, validity, and employee perceptions are also discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of rater training programs: Error training and accuracy training.

108 undergraduates were randomly assigned to 1 of 4 experimental groups to rate videotaped performances of several managers talking with a problem subordinate. The research employed a single-factor experimental design in which rater error training (RET), rater accuracy training (RAT), rating error and accuracy training (RET/RAT), and no training were compared for 2 rating errors (halo and leniency) and accuracy of performance evaluations. Differences in program effectiveness for various performance dimensions were also assessed. Results show that RAT yielded the most accurate ratings and no-training the least accurate ratings. The presence of error training (RET or RET/RAT) was associated with reduced halo, but the presence of accuracy training (RAT or RET/RAT) was associated with less leniency. Dimensions?×?Training interactions revealed that training was not uniformly effective across the rating dimensions. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of ratee race effects in performance ratings.

Conducted a meta-analysis of how the race of the ratee affects performance ratings by examining 74 studies with a total sample of 17,159 ratees for White raters and 14 studies with 2,428 ratees for Black raters. The 5 moderators examined were the study setting, rater training, type of rating, rating purpose, and the racial composition of the work group. Results show that the corrected mean correlations between ratee race and ratings for White and Black raters were .183 and –.220, with 95% confidence intervals that excluded zero for both rater groups. Substantial moderating effects were found for study setting and for the saliency of Blacks in the sample. Race effects were more likely in field settings when Blacks composed a small percentage of the work force. Both Black and White raters gave significantly higher ratings to members of their own race. It is suggested that future research should focus on understanding the process underlying race effects. References for the studies included are appended. (47 ref) (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Reducing the performance-cue bias in work behavior ratings: Can groups help?

The authors examined whether the performance-cue bias can be reduced by relying on groups as raters. Study participants (N = 333) were provided with feedback regarding the performance of a workgroup and, after observing the group, assigned to an individual or group rater condition to complete a behavioral rating instrument. Results revealed that when provided with positive (vs. negative) feedback, individuals attributed more effective and fewer ineffective behaviors to the workgroup; however, group ratings were unaffected by the feedback. In addition, feedback biased the decision criteria and false alarm rates of individuals but not of groups. Discussion of when groups may attenuate versus amplify bias in performance appraisal judgments emphasizes 2 key elements--bias magnitude and task perception. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Job knowledge, ratee familiarity, conceptual similarity and halo error: An exploration.

The results of numerous social perception studies have led researchers to conclude that raters' implicit cognitive schemata regarding trait and behavior covariance may play a crucial role in the rating judgment process. W. H. Cooper (see PA, Vol 66:9176 and 9262) proposed one such cognitive schema, semantic conceptual similarity, as a key source of halo error in job performance ratings but was unable to reproduce the results of previous social perception research. The present study, with 186 undergraduates, employed baseball players as target ratees to examine the effects of job and ratee knowledge on the relations of raters' conceptual similarity schemata with rating and true score covariance. The results are consistent with the systematic distortion hypothesis presented by R. A. Shweder (see record 1976-07240-001). The association between conceptual similarity and rating covariance was significantly greater when Ss lacked sufficient job and/or ratee knowledge. Moreover, the degree of halo was also significantly greater when Ss lacked relevant job and ratee knowledge. The advantages of using objective measures of actual performance as true score estimates in the study of rater cognitive processes are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rating format on goal-setting dimensions: A field experiment.

Atotal of 10 managers and their 62 subordinates participated in a field experiment that examined the effects of performance appraisal feedback deriving from two rating formats—behavioral observation scales (BOS) and graphic rating scales—on three goal-setting dimensions. Multivariate analyses of variance and a priori t tests revealed, as hypothesized, that the BOS format yielded significantly higher levels of goal clarity, goal acceptance, and goal commitment, the three goal-setting dimensions. Results are discussed in terms of systemic characteristics, that is, a "built-in" focus on specific behaviors, rather than a translation of generic performance dimensions into specific work behaviors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Performance appraisal decisions as a function of rater training and purpose of the appraisal.

130 undergraduates rated 33 paragraphs describing the performance of supermarket checkers for one of the following purposes: merit raise, development, or retention. The paragraphs were assembled using previously scaled behavioral anchors describing 5 dimensions of performance. The authors conclude that (a) purpose of the rating was a more important variable in explaining the overall variability in ratings than was rater training; (b) training raters to evaluate for some purposes led to more accurate evaluations than training for other purposes; and (c) rater strategy varied with purpose of the rating (i.e., identical dimensions were weighed, combined, and integrated differently as a function of purpose). (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Delay and distortion: Tacit influences on performance appraisal effectiveness.

A laboratory study of performance appraisal delay and distortion (a) investigated the effects of rater gender, performance level, and feedback requirement, and (b) defined feedback delay as the separate components of appraisal delay and feedback session scheduling delay. Subjects were 64 university business students who had at least 6 months of supervisory experience. Female raters delayed longer when conducting performance appraisals and when scheduling feedback sessions with subordinates, and more positively distorted ratings than did male raters. Moderately low performers were evaluated and scheduled for feedback sessions later, and were evaluated with more positive distortion than were moderately high performers. When required to provide feedback, raters delayed longer appraising performance and distorted ratings more positively. Significant interactions suggest that when feedback is required, female raters may delay appraisals, delay scheduling feedback sessions, and more positively distort their ratings of performance, particularly when rating low performers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Nonperformance influences on performance evaluations: A laboratory phenomenon?

Reviews research that investigated the effects of nonperformance factors (i.e., gender and race) on a variety of organizational criteria, including performance evaluations. It is argued that previous findings are attributable to a research design that bears little resemblance to the performance appraisal process in real organizational contexts. 134 Black and 417 White male candidates for a police-department promotion were rated on a battery of attitude and behavior measures by 3 of the 14 Black and 18 White interviewers to examine the effects of 2 nonperformance factors (ratee and rater race) and an index of ratee past performance on performance ratings. Results of a higher-order MANOVA showed significant effects of ratee race, past performance, rater race, and a Ratee?×?Rater interaction. All of these sources of variance combined, however, accounted for no more than 4% of the total variance in performance ratings. Reasons for the low relationship between past performance and oral interview performance, which involve dissimilarity between rating dimensions and interview demand characteristics, are discussed. Thus, the applicability of results from past laboratory studies to performance evaluation in real organizational environments is questioned. (45 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Magnitude and moderators of bias in observer ratings: A meta-analysis.

Bias in observer ratings compromises generalizability of measurement, typically resulting in attenuation of observed associations between variables. This quantitative review of 79 generalizability studies including raters as a facet examines bias in observer ratings in published psychological research and identifies properties of rating systems likely to place them at risk for problems with rater bias. For the rating systems studied, an average of 37% of score variance was attributable to 2 types of rater bias: (a) raters' differential interpretations of the rating scale and (b) their differential evaluations of the same targets. Ratings of explicit attributes (e.g., frequency counts) contained negligible bias variance, whereas ratings of attributes requiring rater inference contained substantial bias variance. Rater training ameliorated but did not solve the problem of bias in inferential rating scales. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individualism-collectivism.

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)