Effectiveness of performance feedback from behaviorally anchored rating scales.

Compared a behaviorally anchored rating scale (BARS) developed according to the procedure of H. Bernardin et al (see record 1976-08614-001) with a summated rating scale. Using both scale formats (designed to evaluate teaching effectiveness), 859 undergraduates rated 32 instructors during spring classes, and 314 undergraduates rated 19 instructors during summer classes. Students rated instructors halfway through the course and at its end. Instructors received feedback from the 1st rating period on either the BARS (including written behavioral observations) or on the summated rating format. Analyses of covariance revealed no format effect on the performance ratings obtained during the 2nd rating period in the spring experiment. BARS feedback, however, produced greater behavioral change among instructors than did feedback from the alternative format used during the summer experiment. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behaviorally anchored bias: A replication and extension of Murphy and Constans.

A recent study by K. R. Murphy and J. I. Constans (see record 1988-06380-001) found that behaviorally anchored rating scales (BARS) containing observed but unrepresentative behaviors biased ratings in the direction of those behaviors. This study extends the work of Murphy and Constans by adding a retention interval (immediate rating vs 1-week delay) as an independent variable. The principal finding of Murphy and Constans, that BARS induced a novel form of rating bias, was replicated. This bias was found in both the immediate and delayed rating conditions. Implications and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of behavioral expectation scales and graphic rating scales.

Compared ratings derived from behavioral expectation scales developed by 147 personnel management students with ratings based on graphic rating scales. The ratees were 4 college professors, and the raters were the 183 students in their classes. The behaviorally anchored scales resulted in less halo error, or alternatively, more independence in ratings of different dimensions of performance. The behaviorally anchored scales did not correct for leniency in ratings. These results were observed both among raters who participated in developing the behavioral expectation scales and among similar raters who did not take part in this process. The factor structures of the 2 rating formats were essentially equivalent in "cleanness." Neither solution was judged superior to the other. However, the behavioral expectation scale format possessed greater discriminant validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effect of appraisal instrument on managerial perceptions of fairness and satisfaction with appraisals from their peers.

Examined whether the type of appraisal instrument (behavioral observation scales [BOS], behaviorally anchored rating scales [BARS], trait scales, or using no formal appraisal instrument) affected satisfaction with a peer appraisal and perceptions of fairness. 91 managers, while working in teams on a simulated task, provided one another with feedback. Satisfaction with peer appraisals was higher when BOS, BARS, or no formal instrument (control) was used to give feedback than was the case with a trait scale. Procedural justice was perceived as higher when either BOS or no instrument was used to give feedback than when the feedback was based on a trait scale. Trait scales were not perceived to be an acceptable instrument by peers for assessing their performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Time urgency: Conceptual and construct development.

Although time urgency is seen as unidimensional in traditional measures of the Type A behavior pattern, F. J. Landy, H. Rastegary, J. Thayer, and C. Colvin (see record 1992-03955-001) concluded that it is a multidimensional construct. They developed behaviorally anchored rating scales (BARS) that correspond to 7 dimensions of time urgency. This multitrait-multimethod study used 4 raters' evaluations of 183 students at 2 times to examine the construct validity of the BARS measures. This study also investigated environmental and individual difference variables that were linked to time urgency. Confirmatory factor analyses failed to support a hypothesized 7-dimension, 4-method solution but did show good fit indices for a 5-dimension, 4-method model. Potential directions for future research on time urgency are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Job Behavior of Construction Project Managers: Determinants and Assessment

The paper reports the findings of an investigation into the psychosocial aspects of the work of construction project managers (PMs) in the United Kingdom. A theoretical model, which identified the different variables that may influence the behavior and effectiveness of PMs was developed. This study has used an adapted form of the behaviorally anchored rating scales (BARS) to measure a PM's effectiveness. The technique yielded five behavioral dimensions by which a PM's role may be described, and it was used in a survey involving 62 PMs. Analysis of the data suggests that a PM's effectiveness was influenced by situational and personal variables. The results strongly suggest that by understanding the complexity of such relationships, construction firms can influence the direction and behavior of PMs. The results also suggest that the behavior of PMs and project performance varied independently.     

Relations among criteria of police performance.

Job performance measures consisting of 35 objective indices and ratings on 8 behaviorally anchored rating scales (BARS) were available for 795 nonminority (mean age, 29.8 yrs) and 147 minority (mean age, 28.2 yrs) police officers. Eight of the 35 objective measures, plus age and job tenure, were used as predictors of the sum of the 8 BARS. Identical predictor sets validly forecast supervisory ratings in both minority and nonminority groups whether or not age and tenure were included. Unit weights were inferior to regression weights in both groups. It is concluded that supervisory ratings are linearly predictable from objective performance indices for both minority and nonminority subordinates, a finding that comports with civil rights legislation and recent US Supreme Court decisions. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behaviorally anchored scales for assessing behavioral science teaching.

Attempted to (a) identify critical behavioral dimensions associated with teaching behavioral science courses at the university level, (b) develop a set of behaviorally anchored rating scales (BARS) to assess these dimensions, and (c) make a psychometric comparison of the BARS with the Student Instructional Report (SIR), a measure of student perceptions of teacher performance. Based on data obtained from 381 graduate and undergraduate students, BARS had less skewness, lower intercorrelation of ratings on various behavioral dimensions, and lower variability in ratings than the SIR. Content validity and latent factor structure of the new instrument are also reported. (French summary) (18 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Pseudotaxonicity in MAMBAC and MAXCOV analyses of rating-scale data: Turning continua into classes by manipulating observer's expectations.

Taxometric procedures such as mean above minus below a cut and maximum covariance can determine whether a trait is distributed as a discrete latent class. These methods have been used to infer taxonic structure in several personality and psychopathology constructs, often from analyses of rating scale data. This is problematic given (a) well established biases in ratings, (b) the human tendency to think categorically, and (c) implicit typological models of personality and psychopathology among expert raters. Using an experimental method in which the cognitive sets of raters were manipulated as dimensional versus categorical, it is demonstrated that pseudotaxonicity can be created readily with rating scale measures. This suggests that researchers avoid an exclusive reliance on rating scales when conducting taxometrics investigations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multitrait–multimethod–multirater analysis of performance ratings for law enforcement personnel.

144 deputy sheriffs were rated on 9 job performance dimensions with 2 rating scales by 2 raters. Results indicate that the rating scales (the Multiple Item Appraisal Form and the Global Dimension Appraisal Form) developed in this study were able to minimize the major problems often associated with performance ratings (i.e., leniency error, restriction of range, and low reliability). A multitrait/multimethod analysis indicated that the rating scales possessed high convergent and discriminant validity. A multitrait/multirater analysis indicated that although the interrater agreement and the degree of rated discrimination on different traits by different raters were good, there was a substantial rater bias, or strong halo effect. This halo effect in the ratings, however, may really be a legitimate general factor rather than an error. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evidence incompatible with a cognitive compatibility theory of rating behavior.

Assessed the cognitive complexity of 96 undergraduates with the group version of the Role Construct Repertory (REP) Test, a factor analysis of REP test data, and a sorting task. Performance ratings for 3 of the Ss' instructors were obtained with behaviorally anchored rating scales, mixed standard rating scales, graphic rating scales, and simple "alternate" 3-point rating scales. No differences in leniency, halo, or range restriction emerged either as a function of raters' cognitive complexity or a Cognitive Complexity?×?Scale Format interaction. Raters' confidence in their ratings was not associated with either cognitive complexity or rating scale format. It is concluded that researchers of performance ratings should exercise restraint before confidently conferring moderator variable status on a cognitive complexity construct. (25 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Raters Who Pursue Different Goals Give Different Ratings.

J. N. Cleveland and K. R. Murphy (1992) suggested that phenomena such as rater errors and interrater disagreements could be understood in terms of differences in the goals pursued by various raters. We measured 19 rating goals of students at the beginning of a semester, grouped them into scales, and correlated these with teacher evaluations collected at the end of the semester. We found significant multiple correlations, both within classes and in an analysis of the pooled sample (adjusting for instructor mean differences, incremental R2 = .08). Measures of rating goals obtained after raters had observed a significant proportion of ratee performance accounted for variance (incremental R2 = .07) not accounted for by measures of goals obtained at the beginning of the semester. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A view of what can happen when behavioral expectation scales are developed in one setting and used in another.

Pairs of supervisors in 2 large hospitals rated 24 groups of their subordinate nurses, using P. C. Smith and L. M. Kendall's Behavioral Expectation Scale (BES) and a simpler, numerically anchored format. A counter-balanced research design was used to explore the differences in the operational effectiveness of the 2 formats. Results indicate that in terms of interrater reliability and confidence in ratings, the BES was slightly superior. However, when the simpler scale was used there was significantly less leniency effect and raters were better able to discriminate among different ratees in terms of performance. It is concluded that when a behavioral expectation scale is transplanted from one setting to another, the effectiveness of the scaled-expectations format may suffer because the raters do not participate in scale development and/or certain anchors are inappropriate for the new situation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating frame-of-reference rater training effectiveness using performance schema accuracy.

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey & McIntyre, 1987; Woehr, 1994). The present study tested the hypothesis that, following training, frame-of-reference–trained raters would possess schemas of performance that are more similar to a referent schema, as compared with control-trained raters. Schema accuracy was also hypothesized to be positively related to rating accuracy. Results supported these hypotheses. Implications for frame-of-reference training research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)