Cognitive complexity and appraisal effectiveness: Back to the drawing board?

Tested C. E. Schneier's (see record 1978-11450-001) cognitive compatibility theory. In Exps I and II, 100 undergraduates rated college instructors and professor vignettes, respectively. Results show that rater cognitive complexity was unrelated to rating accuracy, halo error, acceptability of rating format, or confidence in ratings. In Exp III, 31 police sergeants rated patrol officers, and the results show that halo error and acceptability of formats were unrelated to cognitive complexity. In Exp IV, 95 undergraduates' ratings of managerial performance and instructor effectiveness showed no support for the cognitive compatibility theory. However, the data showed that raters' ability to generate dimensions was significantly related to halo error in instructors' ratings. Implications for cognitive compatibility theory and future research with the method of generating performance dimensions are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Influence of performance evaluation rating segmentation on motivation and fairness perceptions.

This study investigated the impact of rating segmentation (i.e., the number of alternative appraisal categories available for rating employee performance) on motivation and perceptions of fairness. Participants were 305 student volunteers who performed a catalog search task, A 2 X 4 factorial design with 2 performance rating systems--low segmentation (3 categories) and moderate segmentation (5 categories)--and 4 performance levels was used. Overall, the results indicated that the 5-category system resulted in higher self-efficacy regarding participants' ability to reach the next higher rating category, higher goals for rating improvement, and higher rating improvements than the 3-category system. The effects of rating system and performance rating on rating improvement were partially mediated by self-efficacy and personal goals. The rating system and the performance rating affected perceptions of distributive and procedural justice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Attitudes, self-monitoring, and appraisal behaviors.

This study examined the influence of attitudes and self-monitoring on leniency (elevation accuracy) of performance ratings and personnel decisions. In addition, moderating effects of self-monitoring on the relationship between attitudes and accuracy of ratings and decisions were investigated. Attitudes and self-monitoring tendency of 210 managers–professionals were measured, and ratings provided and decisions made by them were used to test 3 sets of hypotheses. Moderated regression and follow-up split-group analyses indicated that self-monitoring moderated the relationship between attitudes toward accurate appraisal and rating accuracy. Self-monitoring significantly influenced rating and decision accuracy such that accuracy declined with increasing level of self-monitoring. Results highlight the influence of rater's personality on appraisal behaviors. Implications of results and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Between-Individual Comparisons in Performance Evaluation: A Perspective From Prospect Theory.

This article examines how between-individual comparisons influence performance evaluations in rating tasks. The authors demonstrated a systematic change in the perceived difference across ratees as a result of changing the way performance information is expressed. Study 1 found that perceived performance difference between 2 individuals was greater when their objective performance levels were presented with small numbers (e.g., absence rates of 2% vs. 5%) than when they were presented with large numbers (e.g., attendance rates of 98% vs. 95%). Extending this finding to situations involving trade-offs between multiple performance attributes across ratees, Study 2 showed that the relative preference for 1 ratee over another actually reversed when the presentation format of the performance information changed. The authors draw upon prospect theory (D. Kahneman & A. Tversky, 1979; A. Tversky & D. Kahneman, 1981) to offer a theoretical framework describing the between-individual comparison aspect of performance evaluation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy.

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct revalidation of a forced-choice rating form.

A forced-choice rating form was revalidated by using a type of construct validation based on the hypothesis that a manager's effectiveness is reflected in the performance level of his subordinates. 11 plant managers were ranked on overall effectiveness by 3 independent judges, and the relationship between these rankings and the average performance report scores of 142 first-line supervisors in the respective plants was determined by analysis of variance and correlation techniques. Results showed a significant overall relationship between plant-manager rankings and production-supervisor scores on the forced-choice form (p = .005) and significant correlations on 2 of the 6 subscales, with the highest relationship apparent in the Human Relations area (p = .025). The findings support the hypothesis of a relationship between management effectiveness and subordinate performance, and provide evidence to indicate continued validity of the rating instrument. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of frame-of-reference training and information configuration on memory organization and rating accuracy.

It was hypothesized that encoding conditions would substitute for, or neutralize, the effects of frame-of-reference (FOR) training on rating accuracy by encouraging or impeding the person organization of behavior in memory. Undergraduates (N?=?121) were trained with FOR or control procedures, observed videotaped manager performance in a blocked or a mixed order, rated the managers on 3 performance dimensions, and free-recalled target performance vignettes. FOR training and blocked information improved rating accuracy and led to person-based recall; however, person organization was uncorrelated with accuracy. Results are discussed in terms of R. S. Wyer and T. K. Srull's (1989) model of person memory and judgment from which it is proposed that memory organization for behaviors may be unnecessary for rating accuracy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Affect and appraisal accuracy: Liking as an integral dimension in evaluating performance.

Tested the possibility that liking may influence rating accuracy by operating as an integral dimension in 66 undergraduate raters. Ss evaluated vignettes of 4 professors; liking was manipulated with 40 trait terms (e.g., amusing, greedy, bashful) that engendered different liking levels, but had little implication for professor performance. Results indicate a significant effect on rating accuracy, suggesting that liking is an integral dimension that is difficult to separate from performance dimensions. Results support the potential importance of affect in appraisal. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of a criterion of research productivity.

"A forced-choice rating scale designed to determine the extent of a person's productive research behavior was developed at a Midwestern physical science foundation. Of the two experimental scales developed the better form showed an interrator reliability coefficient of .62 and a validity of .60. When the ratings of two raters were averaged the validity of the scale increased to .74." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effect of peer rating treatments on group morale.

During a training program at a Canadian Forces military indoctrination centre, six intact sub-groups consisting of a total of 166 male Ss were randomly assigned to a control and two experimental groups. Each of the two experimental groups was administered a different form of peer rating treatment. Using a morale survey instrument whose relevance, reliability, and validity were empirically established, pre-treatment and post-treatment measures were derived for the three groups. For the nomination approach peer rating treatment a positive effect on group morale was found (P = .05). No significant effect on group morale was demonstrated for the absolute approach peer rating treatment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of category prototypes on performance-rating accuracy.

Two studies examined whether the accessibility of performance prototypes influences performance appraisals. Pilot studies revealed students used performance prototypes when rating instructor performance. Study 1 manipulated the accessibility of these prototypes and the time delay of performance ratings. Results showed no effect of the prime on rating error and accuracy; however, discrimination accuracy decreased over time and recognition bias became more conservative. Study 2 manipulated prototype accessibility and type of rating stimuli (videotape vs. vignette). Rating accuracy and recall were higher for vignette than videotape stimuli, and only those participants exposed to the vignette exhibited priming effects. Results supported transfer-appropriate processing and implied that cognitive primes may have a stronger effect on performance ratings based on "paper-people" than videotaped stimuli. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater selective attention ability and appraisal effectiveness: The effect of a cognitive style on the accuracy of differentiation among ratees.

Examined the effect of the level of selective-attention ability of 359 undergraduate raters on the accuracy of ratings in 2 studies. A group-administered measure of field dependence–independence was used to divide Ss into high- or low-selectivity levels. In Study 1, 288 psychology undergraduates rated 4 vignettes of instructors by either direct inspection or from memory. In Study 2, 71 management undergraduates rated the vignettes. Results show that memory manipulation significantly influenced rating accuracy. In both studies, Ss high in selectivity provided significantly more accurate appraisals than did Ss low in selectivity. It is suggested that cognitive operations underly the effects found in these studies. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Four semantic rating scales compared.

In the construction of semantic rating scales, what format works best? "The questionnaire used in this investigation consisted of 24 eight-step rating scales… . All the scales were bipolar, and were defined by words or phrases at the scale poles, in the manner of the Semantic Differential… . The ratings were made by approximately 400 housewives… interviewed in their own homes by 11 professional interviewers." Rating and interviewer reports were the criteria. Methods were compared. "Whatever method is chosen, the choice should be made with a thoughtful evaluation of the influences these details of format and administration may have upon results." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The consequences of halo "error" in performance ratings: A field study of the moderating effect of halo on test validation results.

Performance ratings of 294 clerical workers in a validation study of clerical ability tests indicated that halo, measured as the standard deviation across dimensions, consistently moderated the relationships between dimension ratings and scores on valid tests. Greater halo resulted in higher validity coefficients, and also was related to higher performance ratings. In an additional analysis, statistically controlling for the effect of the overall rating on dimension ratings resulted in poorer validation results, with dimension ratings rarely adding additional variance to that of overall ratings. The results of this study contradict the traditionally held view of halo as a rating "error," and are consistent with recent laboratory studies that have found accuracy and halo positively related. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater goals on rating patterns: Evidence from an experimental field study.

The goal-based perspective of performance appraisals suggests that raters who pursue different goals give different performance ratings. Yet previous studies have not provided strong empirical evidence that there are different impacts of different goals on mean rating and discriminability, nor have they provided evidence of a goal-rating causality. The authors extend the literature by manipulating rater goals in the context of peer evaluations of graded group projects with a sample of 104 undergraduate students. They find that (a) pursuing a harmony goal increased mean rating and decreased discriminability, and (b) pursuing a fairness goal increased mean rating and decreased discriminability when the group projects had not ended and increased mean rating but did not change discriminability when the group projects had ended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating frame-of-reference rater training effectiveness using performance schema accuracy.

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey & McIntyre, 1987; Woehr, 1994). The present study tested the hypothesis that, following training, frame-of-reference–trained raters would possess schemas of performance that are more similar to a referent schema, as compared with control-trained raters. Schema accuracy was also hypothesized to be positively related to rating accuracy. Results supported these hypotheses. Implications for frame-of-reference training research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)