The frame of reference approach as a solution to an assessment center dilemma.

This article describes how the frame of reference (FOR) approach to rater training for performance appraisal purposes (H. J. Bernardin, 1979; H. J. Bernardin & M. R. Buckley, 1981) was applied to traditional assessment center ratings and rater training. The method by which an FOR was established for the assessment center ratings is presented, including (a) definitions of dimensions of performance, (b) definitions of qualitative levels of performance within each dimension, and (c) specific behavioral examples of levels of performance on an item-by-item basis within dimensions. The resulting FOR was used to structure the training and certification of raters with the expectation of minimizing sources of rater unreliability. Implications for assessment center reliability, validity, and employee perceptions are also discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of rater training programs: Error training and accuracy training.

108 undergraduates were randomly assigned to 1 of 4 experimental groups to rate videotaped performances of several managers talking with a problem subordinate. The research employed a single-factor experimental design in which rater error training (RET), rater accuracy training (RAT), rating error and accuracy training (RET/RAT), and no training were compared for 2 rating errors (halo and leniency) and accuracy of performance evaluations. Differences in program effectiveness for various performance dimensions were also assessed. Results show that RAT yielded the most accurate ratings and no-training the least accurate ratings. The presence of error training (RET or RET/RAT) was associated with reduced halo, but the presence of accuracy training (RAT or RET/RAT) was associated with less leniency. Dimensions?×?Training interactions revealed that training was not uniformly effective across the rating dimensions. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The impact of training on the accuracy of Direct Behavior Ratings (DBR).

To date, extant research has not established how rater training affects the accuracy of data yielded from Direct Behavior Rating (DBR) methods. The purpose of the current study was to examine whether providing users of DBR methods with a training session that utilized practice and performance feedback would increase rating accuracy. It was hypothesized that exposure to direct training procedures would result in greater accuracy than exposure to a brief familiarization training session. Results were consistent with initial hypotheses in that ratings conducted by trained participants were more accurate than those conducted by the untrained participants. Implications for future practice and research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy.

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Improving the accuracy of performance evaluations: Comparison of three methods of performance appraiser training.

A total of 52 supervisory personnel were trained under one of three performance-appraisal training programs: rater error (response set) training, observation training, or decision-making training. Halo, leniency, range restriction, and accuracy measures were collected before and after training from the three training groups and a no-training control group. The results suggested that although the traditional rater error training, best characterized as inappropriate response set training, reduced the classic rater errors (or statistical effects), it also detrimentally affected rating accuracy. However, observation and decision-making training caused performance rating accuracy to increase after training, but did little to reduce classic rater effects. The need for a reconceptualization of rater training content and measurement focus was discussed in terms of the uncertain relation between statistical rating effects and accuracy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive complexity and appraisal effectiveness: Back to the drawing board?

Tested C. E. Schneier's (see record 1978-11450-001) cognitive compatibility theory. In Exps I and II, 100 undergraduates rated college instructors and professor vignettes, respectively. Results show that rater cognitive complexity was unrelated to rating accuracy, halo error, acceptability of rating format, or confidence in ratings. In Exp III, 31 police sergeants rated patrol officers, and the results show that halo error and acceptability of formats were unrelated to cognitive complexity. In Exp IV, 95 undergraduates' ratings of managerial performance and instructor effectiveness showed no support for the cognitive compatibility theory. However, the data showed that raters' ability to generate dimensions was significantly related to halo error in instructors' ratings. Implications for cognitive compatibility theory and future research with the method of generating performance dimensions are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Delay and distortion: Tacit influences on performance appraisal effectiveness.

A laboratory study of performance appraisal delay and distortion (a) investigated the effects of rater gender, performance level, and feedback requirement, and (b) defined feedback delay as the separate components of appraisal delay and feedback session scheduling delay. Subjects were 64 university business students who had at least 6 months of supervisory experience. Female raters delayed longer when conducting performance appraisals and when scheduling feedback sessions with subordinates, and more positively distorted ratings than did male raters. Moderately low performers were evaluated and scheduled for feedback sessions later, and were evaluated with more positive distortion than were moderately high performers. When required to provide feedback, raters delayed longer appraising performance and distorted ratings more positively. Significant interactions suggest that when feedback is required, female raters may delay appraisals, delay scheduling feedback sessions, and more positively distort their ratings of performance, particularly when rating low performers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of ratee race effects in performance ratings.

Conducted a meta-analysis of how the race of the ratee affects performance ratings by examining 74 studies with a total sample of 17,159 ratees for White raters and 14 studies with 2,428 ratees for Black raters. The 5 moderators examined were the study setting, rater training, type of rating, rating purpose, and the racial composition of the work group. Results show that the corrected mean correlations between ratee race and ratings for White and Black raters were .183 and –.220, with 95% confidence intervals that excluded zero for both rater groups. Substantial moderating effects were found for study setting and for the saliency of Blacks in the sample. Race effects were more likely in field settings when Blacks composed a small percentage of the work force. Both Black and White raters gave significantly higher ratings to members of their own race. It is suggested that future research should focus on understanding the process underlying race effects. References for the studies included are appended. (47 ref) (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Role of the rater in performance appraisal.

Notes that ratings for performance appraisal are frequently made by supervisors. In the present study, judgments of effectiveness for 153 hospital nurses were obtained from the nurse herself and her peers in addition to her supervisor, using the same rating form. Factor analysis indicated that each rating source could be clearly identified and characterized. The data reaffirm the notion that interrater disagreement may reflect systematic rater bias as well as meaningful differences in the ways in which judgments are made. Implications for comprehensive appraisals are suggested. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A method for investigating sources of rater bias.

Few studies in counseling and psychotherapy have investigated rater bias. The purpose of this study was to outline a method for studying rater bias. We studied three potential sources of rater bias: (a) characteristics of the rater, client, and therapist; (b) the similarity of characteristics between rater and therapist or client; and (c) perceived similarity between rater and therapist or client. We used a new rater-bias measure. The data for the study were ratings on the Collaborative Study Psychotherapy Rating Scale for 826 sessions of psychotherapy in the Treatment of Depression Collaborative Research Program. High interrater reliability was found for all scales of the measure. We found evidence of rater bias only on the facilitative conditions scale. Rater bias was not found for the other scales, perhaps because of the extensive development of the measure, careful selection of the raters, lengthy rater training, and continued contact with raters throughout the rating period. The rater-bias measure may be useful to other researchers as a means of testing the reactivity of their measures to rater bias. Finally, the method for investigating rater bias can be used by other researchers to evaluate rater bias. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding the latent structure of job performance ratings.

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n?=?2,350 and n?=?2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated, that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of the relationship between true halo and halo error in performance ratings.

The purpose of this study was to test competing theories regarding the relationship between true halo (actual dimensional correlations) and halo rater error (effects of raters' general impressions on specific ratee qualities) at both the individual and group level of analysis. Consistent with the prevailing general impression model of halo rater error, results at both the individual and group level analyses indicated a null (vs. positive or negative) true halo-halo rater error relationship. Results support the ideas that (a) the influence of raters' general impressions is homogeneous across rating dimensions despite wide variability in levels of true halo; (b) in assigning ratings, raters rely both on recalled observations of actual ratee behaviors and on general impressions of ratees in assigning dimensional ratings; and (c) these 2 processes occur independently of one another. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding frame-of-reference training: The impact of training on the recall of performance information.

Extends research on the cognitive mechanisms underlying frame-of-reference (FOR) rater training by examining the impact of FOR training on the recall of performance information. It was hypothesized that the shared performance schema fostered by FOR training would serve as the basis for information processing, resulting in better recall for behavioral performance information as well as more accurate ratings of individual ratees. 174 FOR-trained Ss produced more accurate performance ratings, as measured by L. Cronbach's (1955) differential accuracy and differential elevation components, than did 142 control-trained Ss. FOR-trained Ss also recalled more behaviors, representing more performance dimensions, and exhibited less evaluative clustering and a larger relationship between memory and judgment. No differences were found between control and FOR Ss on measures of recognition accuracy. Implications for the evaluative judgment process are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individualism-collectivism.

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Multitrait–multimethod–multirater analysis of performance ratings for law enforcement personnel.

144 deputy sheriffs were rated on 9 job performance dimensions with 2 rating scales by 2 raters. Results indicate that the rating scales (the Multiple Item Appraisal Form and the Global Dimension Appraisal Form) developed in this study were able to minimize the major problems often associated with performance ratings (i.e., leniency error, restriction of range, and low reliability). A multitrait/multimethod analysis indicated that the rating scales possessed high convergent and discriminant validity. A multitrait/multirater analysis indicated that although the interrater agreement and the degree of rated discrimination on different traits by different raters were good, there was a substantial rater bias, or strong halo effect. This halo effect in the ratings, however, may really be a legitimate general factor rather than an error. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are performance appraisal ratings from different rating sources comparable?

The purpose of this study was to test whether a multisource performance appraisal instrument exhibited measurement invariance across different groups of raters. Multiple-groups confirmatory factor analysis as well as item response theory (IRT) techniques were used to test for invariance of the rating instrument across self, peer, supervisor, and subordinate raters. The results of the confirmatory factor analysis indicated that the rating instrument was invariant across these rater groups. The IRT analysis yielded some evidence of differential item and test functioning, but it was limited to the effects of just 3 items and was trivial in magnitude. Taken together, the results suggest that the rating instrument could be regarded as invariant across the rater groups, thus supporting the practice of directly comparing their ratings. Implications for research and practice are discussed, as well as for understanding the meaning of between-source rating discrepancies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A test of the context dependency of three causal models of halo rater error.

C. E. Lance et al (see record 1994-17452-001) tested 3 different causal models of halo rater error (general impression [GI], salient dimension [SD], and inadequate discrimination [ID] models) and found that the GI model better accounted for observed halo rating error than did the SD or ID models. It was also suggested that the type of halo rater error that occurs might vary as a function of rating context. The purpose of this study was to determine whether rating contexts could be manipulated that favored the operation of each of these 3 halo-error models. Results indicate, however, that GI halo error occurred in spite of experimental conditions designed specifically to induce other forms of halo rater error. This suggests that halo rater error is a unitary phenomenon that should be defined as the influence of a rater's general impression on ratings of specific ratee qualities. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Accuracy of performance ratings as affected by rater training and perceived purpose of rating.

Investigated the effects of perceived purpose for rating and training type on the following dependent variables: accuracy, leniency/severity, and illusory halo. The purpose factor comprised 3 levels: a hiring purpose, a feedback purpose, and a research-only purpose. The training factor comprised 4 levels: rater error (RE) training, frame-of-reference (FOR) training, the combination of both methods, and no training. With both factors crossed, 164 undergraduates were randomly assigned to 1 of 12 conditions and viewed videotapes of lectures given by bogus graduate assistants. Heterogeneity of variance made it necessary to apply a conservative analytical strategy. Training significantly affected 2 measures of accuracy and halo such that a training condition that contained an FOR component did better than RE or no training. The conservativeness of the conservative analytic strategy made effects for the purpose factor on correlation accuracy, leniency/severity, and halo only tentative; it dissipated the 1 interaction effect of the 2 factors on distance accuracy. Discussion centers on (a) comparison of the results with those of S. Zedeck and W. Cascio (see record 1983-09102-001), (b) potential reasons for the heteroscedasticity, and (c) implications for the development of student evaluations of university instructors. (32 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)