Effects of rater goals on rating patterns: Evidence from an experimental field study.

The goal-based perspective of performance appraisals suggests that raters who pursue different goals give different performance ratings. Yet previous studies have not provided strong empirical evidence that there are different impacts of different goals on mean rating and discriminability, nor have they provided evidence of a goal-rating causality. The authors extend the literature by manipulating rater goals in the context of peer evaluations of graded group projects with a sample of 104 undergraduate students. They find that (a) pursuing a harmony goal increased mean rating and decreased discriminability, and (b) pursuing a fairness goal increased mean rating and decreased discriminability when the group projects had not ended and increased mean rating but did not change discriminability when the group projects had ended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Delay and distortion: Tacit influences on performance appraisal effectiveness.

A laboratory study of performance appraisal delay and distortion (a) investigated the effects of rater gender, performance level, and feedback requirement, and (b) defined feedback delay as the separate components of appraisal delay and feedback session scheduling delay. Subjects were 64 university business students who had at least 6 months of supervisory experience. Female raters delayed longer when conducting performance appraisals and when scheduling feedback sessions with subordinates, and more positively distorted ratings than did male raters. Moderately low performers were evaluated and scheduled for feedback sessions later, and were evaluated with more positive distortion than were moderately high performers. When required to provide feedback, raters delayed longer appraising performance and distorted ratings more positively. Significant interactions suggest that when feedback is required, female raters may delay appraisals, delay scheduling feedback sessions, and more positively distort their ratings of performance, particularly when rating low performers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating frame-of-reference rater training effectiveness using performance schema accuracy.

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey & McIntyre, 1987; Woehr, 1994). The present study tested the hypothesis that, following training, frame-of-reference–trained raters would possess schemas of performance that are more similar to a referent schema, as compared with control-trained raters. Schema accuracy was also hypothesized to be positively related to rating accuracy. Results supported these hypotheses. Implications for frame-of-reference training research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are performance appraisal ratings from different rating sources comparable?

The purpose of this study was to test whether a multisource performance appraisal instrument exhibited measurement invariance across different groups of raters. Multiple-groups confirmatory factor analysis as well as item response theory (IRT) techniques were used to test for invariance of the rating instrument across self, peer, supervisor, and subordinate raters. The results of the confirmatory factor analysis indicated that the rating instrument was invariant across these rater groups. The IRT analysis yielded some evidence of differential item and test functioning, but it was limited to the effects of just 3 items and was trivial in magnitude. Taken together, the results suggest that the rating instrument could be regarded as invariant across the rater groups, thus supporting the practice of directly comparing their ratings. Implications for research and practice are discussed, as well as for understanding the meaning of between-source rating discrepancies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Raters Who Pursue Different Goals Give Different Ratings.

J. N. Cleveland and K. R. Murphy (1992) suggested that phenomena such as rater errors and interrater disagreements could be understood in terms of differences in the goals pursued by various raters. We measured 19 rating goals of students at the beginning of a semester, grouped them into scales, and correlated these with teacher evaluations collected at the end of the semester. We found significant multiple correlations, both within classes and in an analysis of the pooled sample (adjusting for instructor mean differences, incremental R2 = .08). Measures of rating goals obtained after raters had observed a significant proportion of ratee performance accounted for variance (incremental R2 = .07) not accounted for by measures of goals obtained at the beginning of the semester. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Influence of performance evaluation rating segmentation on motivation and fairness perceptions.

This study investigated the impact of rating segmentation (i.e., the number of alternative appraisal categories available for rating employee performance) on motivation and perceptions of fairness. Participants were 305 student volunteers who performed a catalog search task, A 2 X 4 factorial design with 2 performance rating systems--low segmentation (3 categories) and moderate segmentation (5 categories)--and 4 performance levels was used. Overall, the results indicated that the 5-category system resulted in higher self-efficacy regarding participants' ability to reach the next higher rating category, higher goals for rating improvement, and higher rating improvements than the 3-category system. The effects of rating system and performance rating on rating improvement were partially mediated by self-efficacy and personal goals. The rating system and the performance rating affected perceptions of distributive and procedural justice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Motivations to work and organizational performance.

Measures of 3 types of motivation to work were related to 2 criteria of job performance, both of which reflect the degree to which the organization has rewarded individual behaviors. In the white-collar sample (N = 1047), composed largely of technical personnel, low performers were motivated primarily by the social environment of the job and, to a lesser extent, by the opportunity of gaining recognition through advancement; few significant relationships were found between intrinsic self-actualizing motivations and job performance. In the blue-collar sample (N = 421), no significant relationships were found between any of the motivational measures and job performance. With advancing age and tenure, work became more meaningful for high performers but less meaningful for low performers, although the importance of the social environment increased for both high and low performers. (17 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Transfer of management training from alternative perspectives.

One hundred seven management training evaluations were meta-analyzed to compare effect sizes for the transfer of managerial training derived from different rating sources (self, superior, peer, and subordinate) and broken down by both study- and training-related variables. For studies as a whole, and interpersonal management skills training studies in particular, transfer effects based on trainees' self-ratings, and to a lesser extent ratings from their superiors, were largest and most varied across studies. In contrast, transfer effects based on peer ratings, and particularly subordinate ratings, were substantially smaller and more homogeneous. This pattern was consistent across different sources of studies, features of evaluation design, and within a subset of 14 studies that each included all 4 rating sources. Across most rating sources, transfer of training was greatest for studies conducted in nonmilitary settings, when raters were likely to have known whether the manager being rated had attended training, when criteria were targeted to training content, when training content was derived from an analysis of tasks and skill requirements, and when training included opportunities for practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Is There a General Factor in Ratings of Job Performance? A Meta-Analytic Framework for Disentangling Substantive and Error Influences.

A database integrating 90 years of empirical studies reporting intercorrelations among rated job performance dimensions was used to test the hypothesis of a general factor in job performance. After controlling for halo error and 3 other sources of measurement error, there remained a general factor in job performance ratings at the construct level accounting for 60% of total variance. Construct-level correlations among rated dimensions of job performance were substantially inflated by halo for both supervisory (33%) and peer (63%) intrarater correlations. These findings have important implications for the measurement of job performance and for theories of job performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multitrait–multimethod–multirater analysis of performance ratings for law enforcement personnel.

144 deputy sheriffs were rated on 9 job performance dimensions with 2 rating scales by 2 raters. Results indicate that the rating scales (the Multiple Item Appraisal Form and the Global Dimension Appraisal Form) developed in this study were able to minimize the major problems often associated with performance ratings (i.e., leniency error, restriction of range, and low reliability). A multitrait/multimethod analysis indicated that the rating scales possessed high convergent and discriminant validity. A multitrait/multirater analysis indicated that although the interrater agreement and the degree of rated discrimination on different traits by different raters were good, there was a substantial rater bias, or strong halo effect. This halo effect in the ratings, however, may really be a legitimate general factor rather than an error. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessing Measurement Equivalence Across Rating Sources: A Multitrait-Multirater Approach.

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding the latent structure of job performance ratings.

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n?=?2,350 and n?=?2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated, that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The five-factor model of personality and managerial performance: Validity gains through the use of 360 degree performance ratings.

This study investigated the usefulness of the five-factor model (FFM) of personality in predicting two aspects of managerial performance (task vs. contextual) assessed by utilizing the 360 degree performance rating system. The authors speculated that one reason for the low validity of the FFM might be the failure of single-source (e.g., supervisor) ratings to comprehensively capture the construct of managerial performance. The operational validity of personality was found to increase substantially (50%–74%) across all of the FFM personality traits when both peer and subordinate ratings were added to supervisor ratings according to the multitrait–multimethod approach. Furthermore, the authors responded to the recent calls to validate tests via a multivariate (e.g., multitrait–multimethod) approach by decomposing overall managerial performance into task and contextual performance criteria and by using multiple rating perspectives (sources). Overall, this study contributes to the evidence that personality may be even more useful in predicting managerial performance if the performance criteria are less deficient. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater accountability on the accuracy and the favorability of performance ratings.

The authors tested the effects of holding raters accountable for their performance ratings on the accuracy and the favorability of those ratings. Undergraduate research participants (N?=?247) completed an inbasket exercise and observed a videotaped simulation during 2 sessions over a 2-wk period. The simulation presented performance information on 4 simulated subordinates portrayed through videotaped vignettes. True performance scores were manipulated by varying the proportion of positive and negative performance vignettes presented for each subordinate. Participants who were made to feel accountable by having to justify their ratings to the experimenter in writing rated their simulated subordinates more accurately. In another experimental condition, accountable raters who were told their subordinates' previous performance ratings were too low rated their subordinates more favorably than did raters in the same experimental condition who were not accountable. (PsycINFO Database Record (c) 2010 APA, all rights reserved)