Variability across sources of performance ratings.

Examined the comparability of peer ratings, supervisor ratings, and self-ratings of counseling performance early in training. Earlier studies of the relationship of performance ratings from different sources have indicated some comparability across source of rating, particularly late in the training process. The results of the present study with 30 21–50 yr old graduate students enrolled in a prepracticum counseling methods course indicate considerable variability across sources of ratings used early in training. Implications and practical training strategies are discussed. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater bias in performance ratings: Superior, self-, and peer ratings.

Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

More sources of variance in accurate empathy ratings: A response.

Three criticisms of my original data are addressed. First, Cicchetti and Ryan confound a methodological and a statistical issue when criticizing Beutler, Johnson, Neville, and Workman for failure to investigate the role of rater memory in the inflation of reliability ratings. Second, data are presented to further suggest that even when ratings are based on the number of segments rather than the number of patients, statistical nonindependience does not result in inflation. Third, the burden of demonstrating that the usual procedure for assessing reliability is not justified rests with the critic, and there are no current data that adequately demonstrate the inappropriateness of the procedure. Hence, it is logical to suggest that Truax and Carkhuff's procedure is justified until proven otherwise. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further evidence for the validity of assessment center dimensions: A meta-analysis of the incremental criterion-related validity of dimension ratings.

This study investigates the incremental variance in job performance explained by assessment center (AC) dimensions over and above personality and cognitive ability. The authors extend previous research by using meta-analysis to examine the relationships between AC dimensions, personality, cognitive ability, and job performance. The results indicate that the 7 summary AC dimensions postulated by W. Arthur, Jr., E. A. Day, T. L. McNelly, & P. S. Edens (2003) are distinguishable from popular individual difference constructs and explain a sizeable proportion of variance in job performance beyond cognitive ability and personality. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behavior-based versus trait-oriented performance ratings: An empirical study.

23 US Navy officer raters evaluated the performance of subordinate officers using (a) behaviorally anchored scales, (b) scales containing the same dimensions and definitions but without behavioral anchors, and (c) a series of scales involving trait-oriented dimensions, also without anchors. Comparisons of the formats' psychometric properties indicated that the behaviorally anchored scale format was somewhat superior to the other 2 on 3 of 4 dependent variables (involving estimates of leniency, halo, interrater agreement, and degree of differentiation among ratees). However, the magnitudes of the differences due to formats were small, in no case exceeding more than 5% of the variance on the dependent variable. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating dynamic performance: The influence of salient Gestalt characteristics on performance ratings.

It is well recognized that performance changes over time. However, the effect of these changes on overall assessments of performance is largely unknown. In a laboratory experiment, we examined the influence of salient Gestalt characteristics of a dynamic performance profile on supervisory ratings. We manipulated performance trend (flat, linear-improving, linear-deteriorating, U-shaped, and ∩-shaped), performance variation (small, large), and performance mean (negative, zero, positive) within subjects and display format (graphic, tabular) between subjects. Participants received and evaluated information about the weekly performance of different employees over a simulated 26-week period. Results showed strong main effects on performance ratings of both performance mean and performance trend, as well as interactions with display format. Theoretical and practical implications of the results are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Trait validity of airline stewardess performance ratings.

Correlated performance ratings obtained at various decision points in the career of 121 airline stewardesses with ability and motive scores on the Project TALENT test battery to illustrate a method of assessing trait evaluations of employees or potential employees. Although the trait validity was low, training and on-the-job performance ratings of stewardesses did reflect knowledge of etiquette and typical high school behavior characterized by sociability. The method also illuminated aspects of the interviewer ratings and termination data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sociometric ratings as predictors of academic performance.

An experiment to determine the degree of accuracy with which sociometric ratings can be used to predict academic performance. A 14-item sociometric questionnaire was administered to 86 students before any academic grades were available. 2 rating measures, one emphasizing future job performance and the other dealing with personal adjustment, were found to be significantly related to academic performance (r = .40, p     

Adaptability measures as predictors of performance ratings.

How well do measures of temperamental and motivational characteristics in training predict success of performance of officers as indicated by Officer Effectiveness Ratings? S were selected from 666 cadets at one base and 69 at another. Variables used were 3 global indicies of adaptability and 10 variables representing the common factors. "(a) measures of adaptability to training and Air Force Life are more highly related to later officer performance than are measures of aptitude or ability; (b) assessments of a man's functioning involving personal judgments of peers, superiors, and experts are predictive of later performance as an Air Force officer." (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Cognitive categorization and quality of performance ratings.

The effects of cognitive categorization of raters on accuracy, leniency, and halo of performance evaluations were investigated in a field setting. One hundered seventy-four subordinates evaluated the performance of their managers on three performance dimensions. Managers were categorized as congruent or incongruent based on subordinates' perceptions of the extent to which the manager's behavior met the subordinates' expectations. The results indicated that the quality of ratings assigned by subordinates was related to the cognitive categories used. As hypothesized, ratings of managers who were categorized as congruent were found to be more accurate and also to contain more leniency and halo tendency than the ratings of managers who were categorized as incongruent. Implications of these findings for performance-appraisal research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some sources of variance in "accurate empathy" ratings.

Recorded the initial therapy interviews of 8 psychotherapists with 54 inpatients. Interviews were rated on the accurate empathy scale by 2 raters. Estimates of the consistency of therapists from 1 segment to another within the same session and from one patient to the next suggest that accurate empathy may not be a stable quality of the therapist, as is usually assumed, but instead may reflect a dyadic or relationship variable. Thus, the usual method of assessing the interrater reliability of the scale using the number of patient-therapist combinations rather than the number of therapists alone may be defensible in spite of recent criticisms. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further investigation of common knowledge effects on job analysis ratings.

Previous research (Smith & Hakel, 1979) raised the possibility that the Position Analysis Questionnaire (PAQ) only captures common knowledge, or stereotypes, about jobs. Cornelius, DeNisi, and Blencoe (1984) presented data to refute this, but found that the number of PAQ items rated does not apply (DNA) was related to the agreement between naive raters and expert raters. The present study used data from 87 analysts and 24 jobs. Naive ratings were those obtained from analysts who had not studied the job, whereas expert ratings were those obtained from raters who had observed the job, interviewed incumbents, and written task statements describing the job. These ratings were then compared to target-score profiles obtained from PAQ services. Results confirmed earlier hypotheses that large numbers of DNA items artifactually inflate correlational estimates of agreement between expert and naive raters. In addition to this artifact, results also supported the view that the PAQ is less appropriate as a job analysis tool for some types of jobs. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding the latent structure of job performance ratings.

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n?=?2,350 and n?=?2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated, that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Examined the effects of knowledge of a ratee's prior performance on evaluations of present performance. Subjects received knowledge of either good or poor prior performances and then viewed and rated a videotape depicting average performance. In Study 1, some subjects received knowledge of the ratee's prior performance by directly viewing videotapes of good or poor ratee behavior, whereas others only reviewed written performance ratings completed by those subjects who had actually viewed the ratee. A contrast effect occurred when knowledge of prior performance was obtained by observing ratee behavior, but an assimilation effect occurred when knowledge of prior performance was obtained by reviewing performance ratings. In Study 2, subjects viewed videotapes of good or poor performances prior to viewing an average performance by the same ratee. However, the separate ratee performances were observed over a more realistic time interval than that used in Study 1 (3 weeks vs. 1 h). No significant contrast effects were observed. In Study 3, subjects reviewed written ratings of prior performances before viewing an average videotape. Subjects who reviewed extremely good (or poor) prior performance ratings provided more extreme ratings of the "average" performance than did subjects who reviewed less extreme ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further evidence for mixed emotions.

Emotion theorists have long debated whether valence, which ranges from pleasant to unpleasant states, is an irreducible aspect of the experience of emotion or whether positivity and negativity are separable in experience. If valence is irreducible, it follows that people cannot feel happy and sad at the same time. Conversely, if positivity and negativity are separable, people may be able to experience such mixed emotions. The authors tested several alternative interpretations for prior evidence that happiness and sadness can co-occur in bittersweet situations (i.e., those containing both pleasant and unpleasant aspects). One possibility is that subjects who reported mixed emotions merely vacillated between happiness and sadness. The authors tested this hypothesis in Studies 1–3 by asking subjects to complete online continuous measures of happiness and sadness. Subjects reported more simultaneously mixed emotions during a bittersweet film clip than during a control clip. Another possibility is that subjects in earlier studies reported mixed emotions only because they were explicitly asked whether they felt happy and sad. The authors tested this hypothesis in Studies 4–6 with open-ended measures of emotion. Subjects were more likely to report mixed emotions after the bittersweet clip than the control clip. Both patterns occurred even when subjects were told that they were not expected to report mixed emotions (Studies 2 and 5) and among subjects who did not previously believe that people could simultaneously feel happy and sad (Studies 3 and 6). These results provide further evidence that positivity and negativity are separable in experience. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The psychological meaning of acquiescence set: Further evidence.

"This paper provides further evidence on the hypothesis that 'negative' items—those stating authoritarian beliefs or attitudes—are more valid than 'positive' items for measuring authoritarianism." (PsycINFO Database Record (c) 2010 APA, all rights reserved)