Understanding frame-of-reference training: The impact of training on the recall of performance information.

Extends research on the cognitive mechanisms underlying frame-of-reference (FOR) rater training by examining the impact of FOR training on the recall of performance information. It was hypothesized that the shared performance schema fostered by FOR training would serve as the basis for information processing, resulting in better recall for behavioral performance information as well as more accurate ratings of individual ratees. 174 FOR-trained Ss produced more accurate performance ratings, as measured by L. Cronbach's (1955) differential accuracy and differential elevation components, than did 142 control-trained Ss. FOR-trained Ss also recalled more behaviors, representing more performance dimensions, and exhibited less evaluative clustering and a larger relationship between memory and judgment. No differences were found between control and FOR Ss on measures of recognition accuracy. Implications for the evaluative judgment process are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers.

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (1) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (2) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (3) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Performance appraisal decisions as a function of rater training and purpose of the appraisal.

130 undergraduates rated 33 paragraphs describing the performance of supermarket checkers for one of the following purposes: merit raise, development, or retention. The paragraphs were assembled using previously scaled behavioral anchors describing 5 dimensions of performance. The authors conclude that (a) purpose of the rating was a more important variable in explaining the overall variability in ratings than was rater training; (b) training raters to evaluate for some purposes led to more accurate evaluations than training for other purposes; and (c) rater strategy varied with purpose of the rating (i.e., identical dimensions were weighed, combined, and integrated differently as a function of purpose). (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy.

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of rater training programs: Error training and accuracy training.

108 undergraduates were randomly assigned to 1 of 4 experimental groups to rate videotaped performances of several managers talking with a problem subordinate. The research employed a single-factor experimental design in which rater error training (RET), rater accuracy training (RAT), rating error and accuracy training (RET/RAT), and no training were compared for 2 rating errors (halo and leniency) and accuracy of performance evaluations. Differences in program effectiveness for various performance dimensions were also assessed. Results show that RAT yielded the most accurate ratings and no-training the least accurate ratings. The presence of error training (RET or RET/RAT) was associated with reduced halo, but the presence of accuracy training (RAT or RET/RAT) was associated with less leniency. Dimensions?×?Training interactions revealed that training was not uniformly effective across the rating dimensions. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Number of assessment center dimensions as a determinant of assessor accuracy.

Undergraduates (N?=?131) were trained as assessors, who evaluated the performance of confederates in an assessment center simulation on 3, 6, or 9 dimensions. Number of dimensions significantly affected some assessment center judgments but not others. Ss who rated a small number of dimensions classified behaviors more accurately and made more accurate ratings than did Ss who rated a large number of dimensions. Number of dimensions did not affect the accuracy of assessors' observations nor the discriminant validity of their dimension ratings. Given these results and the findings of others (e.g., J. R. Hinrichs and S. Haanpera; see record 1978-20114-001), developers of assessment centers should limit the cognitive demands placed on assessors by, for example, minimizing the number of dimensions assessors are required to process. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of assessment center ratings for the prediction of performance ratings and school climate of school administrators.

Describes the validation of an assessment center used to select school administrators on the basis of ratings of 153 school administrators who had participated in the center between 1976 and 1981. Behaviorally anchored rating scales were obtained from supervisors, teachers, and support staff; and measures of 7 school climate dimensions were provided by teachers, students, and support staff. Results show a significant relationship between an overall assessment center placement recommendation and supervisory, teacher, and support staff ratings on most performance dimensions. Climate measures were significantly related to few assessment-center ratings regardless of the source of the climate measure. It is suggested that the assessment center proved valid for a sample from widely distant geographic areas and school districts of differing sizes and levels. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of predictor and criterion judgments of managerial performance using the multitrait-multimethod approach.

Used the multitrait-multimethod matrix technique to examine the predictive validity of ratings of management potential derived from an industrial assessment center program. Ss were 71 professional, technical, and lower level managerial personnel. Psychologists' and managers' ratings on 13 assessment dimensions were correlated with supervisors' ratings of current job performance on the same dimensions. Ratings obtained of on-the-job performance were lower in quality than the predictor ratings. The median reliability of the criterion ratings was .52 compared to median reliabilities of .85 and .89 for the psychologists' and managers' ratings, respectively. Supervisors failed to differentiate the various dimensions. Results are compared with findings of other studies that used this technique to determine the sources of unreliability in the criterion. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An exercise design approach to understanding assessment center dimension and exercise constructs.

Arobust finding in assessment center research is that ratings cluster according to exercises and not dimensions. Seeking further understanding of exercise effects, the authors proposed 2 exercise-based factors (exercise form and exercise content) as sources of variance in assessment center ratings. Exercise designs were manipulated so that 2 levels of form (leaderless group discussions and role-play exercises) were crossed with 2 levels of content (cooperative and competitive task designs). Analysis of variance (ANOVA) and confirmatory factor analysis procedures applied to multitrait–multimethod ratings of 89 high school student assessees revealed that most of the variance in the ratings was explained by exercises and not dimensions. Exercise form accounted for 16% of method variance, whereas exercise content had near zero effect. The form effect was primarily due to higher correlations of dimension ratings across role-play exercises. Practical implications and future directions for exercise design research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Constructs and assessment center dimensions: Some troubling empirical findings.

Maintains that exercise design alone serves as weak evidence for construct validity due to the complexity of the typical assessment center rating and evaluation process. Evidence that assessment ratings accurately reflect the complex traits they purport to measure has not been provided to date. To explore this issue, the interrelationships among dimensional ratings between and within exercises in 3 assessment centers were examined. 559 candidates for upper-level management from a multinational firm, a civil service commission, and a retailer were assessed on managerial skills (including communication and interpersonal skills). Postexercise ratings of assessment dimensions were correlated and factor analyzed. The resulting factor pattern for all 3 organizations represented exercises rather than dimensions. In 2 of the organizations, the mean correlation among across-exercise ratings of individual dimensions was near zero. These findings suggest that assessment ratings do not measure the intended constructs. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Common comparison standards: An approach to improving agreement between self and supervisory performance ratings.

This study examined differential comparison standards (i.e., comparative bases for performance evaluation) and their effects on agreement between supervisory raters and self raters within the context of a performance appraisal system. The purpose of the research was to examine differential comparison standards as an underlying mechanism in the traditionally poor correlation between self and supervisor performance ratings. Supervisor and subordinate rater dyads (N ?=?106 dyads) evaluated job performance across 3 dimensions, using 5 different comparison standards (ambiguous, internal, absolute, relative, and multiple). Results support the hypotheses, indicating that more explicit and objective comparison standards produced higher levels of interrater agreement. The implications of these findings in terms of comparison standards being adopted in current research and future performance appraisal systems are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of the relationship between true halo and halo error in performance ratings.

The purpose of this study was to test competing theories regarding the relationship between true halo (actual dimensional correlations) and halo rater error (effects of raters' general impressions on specific ratee qualities) at both the individual and group level of analysis. Consistent with the prevailing general impression model of halo rater error, results at both the individual and group level analyses indicated a null (vs. positive or negative) true halo-halo rater error relationship. Results support the ideas that (a) the influence of raters' general impressions is homogeneous across rating dimensions despite wide variability in levels of true halo; (b) in assigning ratings, raters rely both on recalled observations of actual ratee behaviors and on general impressions of ratees in assigning dimensional ratings; and (c) these 2 processes occur independently of one another. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of behavioral assessment for predicting military recruiter performance.

Determined the validity of an assessment center designed to help select US Army recruiters. 16 experienced recruiters assessed 57 soldiers entering the Army's recruiter school, and assessment ratings were correlated with subsequent performance in training. A composite of assessment ratings yielded corrected validities of near .50. In contrast, first-impression evaluations, ratings based on a structured interview, and scores on a test developed to predict success in military recruiting correlated near zero with performance in training. Results confirm that valid assessment does not require behavioral scientists as assessors, and analyses suggest that statistical composites of assessment ratings on individual exercises may be slightly more valid than clinical consensus judgments made after discussion of assessment performance. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of race and sex effects on performance ratings.

The effects of rater source, rater and ratee race, rater and ratee sex, and job type were investigated on ratings collected for 8,642 first-term Army enlisted personnel. Ratings were made on 10 behaviorally based dimensions developed for evaluating all first-term soldiers. Results of between-Ss analyses similar to those conducted in past research revealed significant main effects and interactions for sex, race, rater source, and job type, but the variance accounted for by these effects was minimal. Repeated measures analyses were also performed, with each ratee evaluated by one Black and one White rater for the race effects analyses and one female and one male rater for the sex effects analyses. These analyses, which unconfounded rater bias and actual performance differences, yielded results similar to those obtained with the between-Ss design. Implications of the findings are discussed. ? (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A reexamination of the relation between halo error and accuracy.

The traditional assumption has been that halo error is negatively related to accuracy of ratings. W. H. Cooper (1981) evaluated this assumption by examining correlation coefficients between measures of accuracy and halo error from five earlier studies of performance and trait ratings. Because the correlation coefficients were typically positive, Cooper concluded that a "paradoxical" positive relation exists between halo error and accuracy. However, there is no paradox; some of these positive correlation coefficients were between halo error and inaccuracy, whereas others were based on analyses that did not take into consideration negative halo errors. When analyses that correct these problems were performed on two sets of data (R. Tallarigo, 1986, n?=?107; R. J. Vance, K. W. Kuhnert, & J. L. Farr, 1978, n?=?112), all significant (p?negative. The use of halo error measures, the possibility of negative halo errors, and implications of the results for rater training are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues.

Considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. 93 college students (mean age 22 yrs) trained using either FOR or control procedures, observed 3 managers on videotape, and rated the managers on 3 performance dimensions. Results supported the hypothesis that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding the latent structure of job performance ratings.

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n?=?2,350 and n?=?2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated, that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical comparison of the relative effects of rater response biases on three rating scale formats.

Compared Smith-Kendall type behaviorally anchored scales for derived performance dimensions (Format 1), scales for the same dimensions but without the behavioral anchors (Format 2), and scales for dimensions selected on an a priori basis (Format 3) on the basis of susceptibility to rater response biases. Raters were 30 graduate students and ratees were 3 associate professors whom the raters had had in succession during their 1st year of graduate study. Leniency error and composite halo error were present in all ratings; there was no evidence of relative or absolute halo errors in any ratings. There was some evidence that the use of scales for derived dimensions reduced leniency error and increased the amount of variance attributable to ratee differences. The scale reliabilities of the 3 formats were also determined. A discussion of the feasibility of obtaining relatively independent scales for several job performance dimensions is included. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Dimensions that make a difference: Examining the impact of in-role and extrarole behaviors on supervisory ratings.

Levels of in-role and extrarole performance were experimentally manipulated in a 3?×?2 within-Ss design. Dimensions capturing both in-role and extrarole behaviors (i.e., citizenship behaviors directed toward other individuals) strongly influenced various measures of rater search strategies as well as the ratings given by 116 supervisors evaluating secretarial performance. Furthermore, as predicted by D. W. Organ (1990), dimension ratings assigned to ratees with high levels of extrarole behavior displayed significantly more halo than ratings given to ratees with neutral levels of extrarole behavior. Tests for the effects of level of extrarole behavior on 2 measures of rater accuracy produced contradictory results. Implications are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)