Effects of raters' stress on the dispersion and favorability of performance ratings.

We tested effects of raters' stress on the favorability and dispersion of performance ratings. In all, 120 undergraduates completed either a stressful or an unstressful inbasket exercise, either before or after they saw a videotaped portrayal of a manager's job performance. Then they rated the manager on several performance dimensions. Ratings provided by participants who completed the stressful inbasket showed less dispersion across performance dimensions, but no differences on favorability. Whether participants were stressed before or after viewing the performance videotape made no significant difference in the favorability or dispersion of their ratings. This suggests that stressful experiences affect at least the retrieval stage of information processing and might affect the input stage as well. There was a significant interaction with individual differences in Type A pattern. In the unstressful condition, Type A scores were positively correlated with dispersion (r?=?.32, p?r?=?–.15), but not statistically significant. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An examination of the relationship between the accuracy and favorability of ratings.

Administered a scale to 68 senior marketing students, who rated the prospecting ability of a life insurance agent on 7 videotaped telephone approaches. 3 randomly selected subgroups were given different background information about the agent. It was found that induced mental sets varied significantly in terms of favorability. A differential accuracy phenomenon (dap) was the only significant source of variance. However, neither the dap nor the overall accuracy of ratings was related to the favorability of s's general impression of the ratee. Findings make suspect the current practice of operationalizing leniency error by means of the average level of favorability of global ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater training and diary-keeping on psychometric error in ratings.

Compared the psychometric properties of ratings on behavioral expectation scales (BES) across 4 groups totalling 156 undergraduate raters. Groups differed with respect to amount of prior training (1 hr or more), the nature of psychometric errors, and the extent of exposure to scales (read scales and recorded observed critical incidents, discussed general scale dimensions, or no exposure to scales). Three Ss from each group rated 1 of 13 instructors during the last week of a 10-wk term. Significantly less leniency error and halo effect, plus higher interrater reliability, were found for the group that had received the hour of training and full exposure to the BES. Ss who had received only training had significantly less halo error than those that had received no training. The need for rater training prior to observation and the use of BES as a context for observation are discussed. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating frame-of-reference rater training effectiveness using performance schema accuracy.

Frame-of-reference training has been shown to be an effective intervention for improving the accuracy of performance ratings (e.g., Woehr & Huffcutt, 1994). Despite evidence in support of the effectiveness of frame-of-reference training, few studies have empirically addressed the ultimate goal of such training, which is to teach raters to share a common conceptualization of performance (Athey & McIntyre, 1987; Woehr, 1994). The present study tested the hypothesis that, following training, frame-of-reference–trained raters would possess schemas of performance that are more similar to a referent schema, as compared with control-trained raters. Schema accuracy was also hypothesized to be positively related to rating accuracy. Results supported these hypotheses. Implications for frame-of-reference training research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater training on leniency and halo errors in student ratings of instructors.

Investigated the effects (over time) of a comprehensive vs an abbreviated rater training session on relative levels of leniency error and halo effect. 80 undergraduates (20 per group) rated all of their nonlaboratory instructors over 1, 2, or 3 rating periods using either behavioral expectation scales or summated rating scales. Tests on psychometric error were also administered at these times. Results indicate that the psychometric quality of ratings was superior for the group receiving the comprehensive training, and both training groups were superior to the control groups at the 1st measurement period. No differences were found between any groups in later comparisons. A consistent relationship was found between scores on the tests of psychometric error and error as measured on the ratings. Results are discussed in terms of the diminishing effect of rater training over rating periods, the relationship of internal and external criteria of training effects, the practical significance of differences between groups, and the importance of rating context on rating quality. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of comparative performance information on the accuracy of self-ratings and agreement between self- and supervisor ratings.

The manner in which social comparison performance information affects the accuracy of self-ratings and the agreement between self- and supervisor ratings was investigated in a laboratory experiment using 163 undergraduates. Ss proofread a series of articles for 30 min and then made self-ratings of work performance. Half of the subjects received social comparison information prior to making self-ratings. Correlations between self- and supervisor evaluations and between self-evaluations and objective performance indicators were significantly larger when self-raters were presented with the same comparative performance information that was available to supervisors. Implications of the findings for future research on self- and supervisor evaluations are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Accuracy of performance ratings as affected by rater training and perceived purpose of rating.

Investigated the effects of perceived purpose for rating and training type on the following dependent variables: accuracy, leniency/severity, and illusory halo. The purpose factor comprised 3 levels: a hiring purpose, a feedback purpose, and a research-only purpose. The training factor comprised 4 levels: rater error (RE) training, frame-of-reference (FOR) training, the combination of both methods, and no training. With both factors crossed, 164 undergraduates were randomly assigned to 1 of 12 conditions and viewed videotapes of lectures given by bogus graduate assistants. Heterogeneity of variance made it necessary to apply a conservative analytical strategy. Training significantly affected 2 measures of accuracy and halo such that a training condition that contained an FOR component did better than RE or no training. The conservativeness of the conservative analytic strategy made effects for the purpose factor on correlation accuracy, leniency/severity, and halo only tentative; it dissipated the 1 interaction effect of the 2 factors on distance accuracy. Discussion centers on (a) comparison of the results with those of S. Zedeck and W. Cascio (see record 1983-09102-001), (b) potential reasons for the heteroscedasticity, and (c) implications for the development of student evaluations of university instructors. (32 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of frame-of-reference training on rater accuracy under alternative time delays.

Investigated the effects of frame-of-reference (FOR) training on various indexes of distance and correlational accuracy under alternative time delays. 150 Ss were assigned randomly to either FOR- or control- (i.e., minimal) training conditions, with 1 of 3 time delays: (1) no delay between training, observation, and rating; (2) ratings performed 2 days following training and ratee observations; or (3) ratee observations and ratings completed 2 days following training. Hypotheses were proposed predicting specific relationships between accuracy, recall memory, and learning, depending on the delay period. Overall, results support the categorization perspective on FOR-training effectiveness; however, different results were obtained depending on the type of accuracy index and time delay. The implications of these findings are discussed in terms of how they relate to the conceptual distinction between distance and correlational accuracy and to the role of on-line, memory-based, and inference-memory-based processing in the ratings of FOR trained raters. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives.

We use levels-of-processing theory and social facilitation theory to explain the effect of training format and group size on distance and correlation accuracy, leniency-severity, halo, retention of training and pretraining information, and subject arousal. The training factor included frame-of-reference (FOR) training, information only (INFO) training, and no training (NOT). Group size was n?=?1, n?=?6, and n?=?12, respectively. A total of 108 subjects, randomly assigned to one of nine Training?×?Group Size conditions, viewed and rated videotaped lectures. Results indicated that FOR training effected improved retention of training information, improved distance accuracy, and less halo over INFO training or NOT (p     

Longitudinal study of the effects of rater training on psychometric error in ratings.

66 supervisory engineers were randomly assigned to (a) an intense training group, (b) a discussion group, or (c) a nontrained comparison group. The intense training and discussion Ss received 14 hrs of rater training designed to minimize halo and leniency error and to use the organization's behavioral expectation scale for engineers more effectively. A longitudinal research design was used to study the halo and leniency errors 6 mo before training (TB), 6 mo after training (T?), and 12 mo after training (T??). Using covariance analysis, ANOVA with repeated measures, and planned comparisons, findings indicate that the intense training module (which included a 6-hr videotape block) was superior to the discussion and comparison groups in reducing halo and leniency error. However, a noticeable dissipation of training effect on these psychometric errors was identified when the T? and T?? data were examined. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Role of the rater in performance appraisal.

Notes that ratings for performance appraisal are frequently made by supervisors. In the present study, judgments of effectiveness for 153 hospital nurses were obtained from the nurse herself and her peers in addition to her supervisor, using the same rating form. Factor analysis indicated that each rating source could be clearly identified and characterized. The data reaffirm the notion that interrater disagreement may reflect systematic rater bias as well as meaningful differences in the ways in which judgments are made. Implications for comprehensive appraisals are suggested. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of ratee task performance and interpersonal factors on supervisor and peer performance ratings.

The study examines the effects of a wide array of rater–ratee relationship and ratee-characteristic variables on supervisor and peer job-performance ratings. Interpersonal ratings, job performance ratings, and ratee scores on ability, job knowledge, and technical proficiency were available for 493 to 631 first-tour US Army soldiers. Results of supervisor and peer ratings-path models showed ratee ability, knowledge, and proficiency accounted for 13% of the variance in supervisor performance ratings and 7% for the peer ratings. Among the interpersonal variables, ratee dependability had the strongest effect for both models. Ratee friendliness and likability had little effect on the performance ratings. Inclusion of the interpersonal factors increased the variance accounted for in the ratings to 28% and 19%, respectively. Discussion focuses on the relative contribution of ratee technical and contextual performance to raters' judgments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Perceived similarity and accuracy of peer ratings.

The hypothesis that raters will be more accurate in rating peers perceived to be similar to themselves, suggested by Mumford (1983) and derived from social comparison theory, was examined. Subjects were 681 Israeli entrants to a military training program. Shortly after course inception, subjects were asked to review the performance of squad members and to forecast their final grade. Subjects also judged peers' similarity to self overall (general similarity), in course achievement (foreground similarity), and in military experience (background similarity). Analysis revealed that accuracy was markedly lower in the evaluation of dissimilar others. Although the same patterns of results was observed for all forms of similarity, stronger effects resulted when similarity was measured in terms of general and foreground characteristics. Implications for future theory and research as well as for the practical application of peer assessment are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Systematic distortions in memory-based behavior ratings and performance evaluations: Consequences for rating accuracy.

To test the hypothesis that memory-based ratings should be less accurate than ratings collected under conditions that minimize demands on memory, data were obtained from 82 undergraduates who had rated 4 videotapes of graduate student lecturers. Half of the tapes were rated immediately after they were viewed; Ss returned the following day and rated the remaining tapes from memory. Memory-based behavior ratings and performance evaluations showed higher intercorrelations (more halo) than did ratings that were collected immediately after viewing the ratee's performance. However, ratings were systematically more accurate in the delayed-rating condition than in the immediate-rating condition. It is argued that (1) under certain conditions, raters may depend on their general impressions of ratees rather than on their memory for specific details; and (2) these schematic evaluations may preserve a greater proportion of valid information, as compared with irrelevant detail, than is available immediately after observing ratee behavior. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some differential effects of race of rater and ratee on early peer ratings of combat aptitude.

Peer ratings of combat potential made by 669 Caucasian and Negro army recruits of the squad members were examined. The hypothesis that ratees would receive higher ratings from members of their own race than from members of another race was supported (p     

Feedback motivation and reactions to personality interpretations that differ in favorability and accuracy.

Personality feedback based on participants' actual test scores was systematically varied in terms of accuracy and favorability and presented to subjects for the purpose of comparing reactions to the various types of feedback information. Responses to the self-relevant information were related to individual differences in the recipient's general level of desire for personal feedback. For most subjects, ratings of feedback accuracy showed stronger endorsement for positive over negative feedback, and for accurate over inaccurate feedback. Raters who exhibited high levels of desire for feedback, however, did not discriminate between favorable and unfavorable accurate feedback, or between accurate and inaccurate favorable feedback. The results are discussed in terms of (a) the reliability of personal endorsements for results derived from psychological testing devices and (b) potential for personal adjustment indicated by higher levels of desire for personal feedback. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of task performance, helping, voice, and organizational loyalty on performance appraisal ratings.

Despite the fact that several studies have investigated the relationship between organizational citizenship behavior and performance appraisal ratings, the vast majority of these studies have been cross-sectional, correlational investigations conducted in organizational settings that do not allow researchers to establish the causal nature of this relationship. To address this lack of knowledge regarding causality, the authors conducted 2 studies designed to investigate the effects of task performance, helping behavior, voice, and organizational loyalty on performance appraisal evaluations. Findings demonstrated that each of these forms of behavior has significant effects on performance evaluation decisions and suggest that additional attention should be directed at both voice and organizational loyalty as important forms of citizenship behavior aimed at the organization. (PsycINFO Database Record (c) 2010 APA, all rights reserved)