A comparison of behavioral expectation scales and graphic rating scales.

Compared ratings derived from behavioral expectation scales developed by 147 personnel management students with ratings based on graphic rating scales. The ratees were 4 college professors, and the raters were the 183 students in their classes. The behaviorally anchored scales resulted in less halo error, or alternatively, more independence in ratings of different dimensions of performance. The behaviorally anchored scales did not correct for leniency in ratings. These results were observed both among raters who participated in developing the behavioral expectation scales and among similar raters who did not take part in this process. The factor structures of the 2 rating formats were essentially equivalent in "cleanness." Neither solution was judged superior to the other. However, the behavioral expectation scale format possessed greater discriminant validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of the purpose of rating on accuracy in observing teacher behavior and evaluating teaching performance.

Previous research has suggested that ratings used to make administrative decisions are lenient when compared with ratings obtained for research purposes only. The present study examined the effects of the purpose of rating on multivariate measures of accuracy in observing teacher behavior as well as measures of accuracy in evaluating teaching performance. 45 undergraduates viewed and evaluated videotaped lectures under conditions in which they were informed that their ratings would be used for research only or for making important decisions about those being rated. The purpose of rating did not affect measures of accuracy in rating the frequency with which a number of critical behaviors occurred on each tape. The purpose of rating also did not affect multivariate measures of performance rating accuracy. Purpose did, however, affect the relationship between accuracy in observing teacher behavior and accuracy in evaluating teaching performance. It is suggested that purpose affects the way in which raters process behavioral information without necessarily affecting the general level of rating. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of performance feedback from behaviorally anchored rating scales.

Compared a behaviorally anchored rating scale (BARS) developed according to the procedure of H. Bernardin et al (see record 1976-08614-001) with a summated rating scale. Using both scale formats (designed to evaluate teaching effectiveness), 859 undergraduates rated 32 instructors during spring classes, and 314 undergraduates rated 19 instructors during summer classes. Students rated instructors halfway through the course and at its end. Instructors received feedback from the 1st rating period on either the BARS (including written behavioral observations) or on the summated rating format. Analyses of covariance revealed no format effect on the performance ratings obtained during the 2nd rating period in the spring experiment. BARS feedback, however, produced greater behavioral change among instructors than did feedback from the alternative format used during the summer experiment. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A view of what can happen when behavioral expectation scales are developed in one setting and used in another.

Pairs of supervisors in 2 large hospitals rated 24 groups of their subordinate nurses, using P. C. Smith and L. M. Kendall's Behavioral Expectation Scale (BES) and a simpler, numerically anchored format. A counter-balanced research design was used to explore the differences in the operational effectiveness of the 2 formats. Results indicate that in terms of interrater reliability and confidence in ratings, the BES was slightly superior. However, when the simpler scale was used there was significantly less leniency effect and raters were better able to discriminate among different ratees in terms of performance. It is concluded that when a behavioral expectation scale is transplanted from one setting to another, the effectiveness of the scaled-expectations format may suffer because the raters do not participate in scale development and/or certain anchors are inappropriate for the new situation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive complexity and appraisal effectiveness: Back to the drawing board?

Tested C. E. Schneier's (see record 1978-11450-001) cognitive compatibility theory. In Exps I and II, 100 undergraduates rated college instructors and professor vignettes, respectively. Results show that rater cognitive complexity was unrelated to rating accuracy, halo error, acceptability of rating format, or confidence in ratings. In Exp III, 31 police sergeants rated patrol officers, and the results show that halo error and acceptability of formats were unrelated to cognitive complexity. In Exp IV, 95 undergraduates' ratings of managerial performance and instructor effectiveness showed no support for the cognitive compatibility theory. However, the data showed that raters' ability to generate dimensions was significantly related to halo error in instructors' ratings. Implications for cognitive compatibility theory and future research with the method of generating performance dimensions are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An investigation of the validity of expert true score estimates in appraisal research.

Researchers of appraisal rating accuracy have defined ratee true scores as the mean ratings given by experts provided with enhanced opportunities to observe performance. The external validity of accuracy research depends on the relevance of these expert estimates to true performance. In our study, we provided expert raters with enhanced opportunities to view videotapes of five ratees under conditions of high true dimension intercorrelation or low true intercorrelation. The accuracy of expert ratings was compared with the accuracy of nonexpert ratings of the same tapes viewed under more typical rating conditions. Subjects' ratings were compared with ratee true scores defined in terms of objective worker output. Results indicated that experts were more accurate than nonexperts, regardless of the true dimension intercorrelations. Accuracy indices computed by using objective true scores were highly correlated with indices computed by using the mean expert ratings as true score estimates. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings, judgment decomposition, and their impact on interrater agreement and rating accuracy.

One strategy suggested for improving the accuracy of the complex evaluative judgments involved in performance evaluation is to decompose them into a series of simpler judgments. Another is to collect observations in a distributional rating scheme in which raters estimate the frequencies of different classes of behavior and performance is assessed in terms of the relative frequencies of effective and ineffective behaviors. Distributional ratings were compared to Likert-type ratings of videotaped lectures at 3 levels of dimensional decomposition; ratings were evaluated in terms of interrater agreement and rating accuracy. Decomposition led to increased agreement and accuracy, but the use of distributional ratings did not. The practical implications of the results are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating frame-of-reference rater training effectiveness using performance schema accuracy.

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey & McIntyre, 1987; Woehr, 1994). The present study tested the hypothesis that, following training, frame-of-reference–trained raters would possess schemas of performance that are more similar to a referent schema, as compared with control-trained raters. Schema accuracy was also hypothesized to be positively related to rating accuracy. Results supported these hypotheses. Implications for frame-of-reference training research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of ratee race effects in performance ratings.

Conducted a meta-analysis of how the race of the ratee affects performance ratings by examining 74 studies with a total sample of 17,159 ratees for White raters and 14 studies with 2,428 ratees for Black raters. The 5 moderators examined were the study setting, rater training, type of rating, rating purpose, and the racial composition of the work group. Results show that the corrected mean correlations between ratee race and ratings for White and Black raters were .183 and –.220, with 95% confidence intervals that excluded zero for both rater groups. Substantial moderating effects were found for study setting and for the saliency of Blacks in the sample. Race effects were more likely in field settings when Blacks composed a small percentage of the work force. Both Black and White raters gave significantly higher ratings to members of their own race. It is suggested that future research should focus on understanding the process underlying race effects. References for the studies included are appended. (47 ref) (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Comparison of training methods for improving the psychometric quality of experimental and administrative performance ratings.

Studied 52 nursing supervisors to examine the effects of a lecture on rating errors, discussion about errors, and participation in scale construction on both experimental and subsequent administrative ratings. On experimental ratings, scale construction reduced halo and variability errors, lecture reduced variability errors, and discussion increased variability errors. These results held true only for raters who began making ratings within 1 wk after training, before administration of questionnaires designed to measure rater motivation and knowledge. (27 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater accountability on the accuracy and the favorability of performance ratings.

The authors tested the effects of holding raters accountable for their performance ratings on the accuracy and the favorability of those ratings. Undergraduate research participants (N?=?247) completed an inbasket exercise and observed a videotaped simulation during 2 sessions over a 2-wk period. The simulation presented performance information on 4 simulated subordinates portrayed through videotaped vignettes. True performance scores were manipulated by varying the proportion of positive and negative performance vignettes presented for each subordinate. Participants who were made to feel accountable by having to justify their ratings to the experimenter in writing rated their simulated subordinates more accurately. In another experimental condition, accountable raters who were told their subordinates' previous performance ratings were too low rated their subordinates more favorably than did raters in the same experimental condition who were not accountable. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A clarification of some issues regarding the development and use of behaviorally anchored ratings scales (BARS).

Attempts to clarify some issues regarding the development and use of behaviorally anchored rating scales (BARS). The P. C. Smith and L. M. Kendall (1963) format is distinguished from other approaches to BARS, and research comparing their relative effectiveness is reviewed. Their format is discussed as a method that is designed to enhance future observations and to foster a common frame of reference in observer raters. Emphasis is also placed on BARS as an observation-rating system that provides data for the assessment of estimates of accuracy for individual raters. Responses are made to criticisms dealing with the rating process of BARS, the relative effectiveness of BARS vs summated scales, and the role and characteristics of the behavioral anchors. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)