Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings.

Investigated the effects of a short training session designed to reduce halo error in performance ratings. 90 low and middle managers rated 1 of 6 hypothetical first-line supervisors on 6 performance dimensions according to behavior displayed in a prepared vignette. Ratings were taken prior to and following the 5-min training session, with rater-ratee combinations counterbalanced. The vignettes were developed to contain previously scaled behavior examples, thus enabling the calculation of "true" criterion scores for each dimension. Comparisons between these "true" criterion and the performance ratings revealed that the training session significantly reduced halo, while leaving validity of the ratings generally unaffected. Performance ratings completed after training possessed lower reliability, although raters provided somewhat more accurate performance profiles. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Conceptual similarity as a source of illusory halo in job performance ratings.

Tested whether a possible source of difficulty in materially reducing illusory halo in job performance ratings is raters' beliefs that rating categories are conceptually similar and hence covary, thereby inflating observed correlation matrices. 11 graduate business administration students evaluated the conceptual similarities among job dimensions within 3 jobs. The previously observed interdimension correlation matrices were successfully predicted by Ss' mean conceptual similarity scores. When the observed correlation matrix obtained by W. C. Borman (see record 1980-26801-001) was compared with the normative true score matrix, the conceptual similarity scores were found to be inferior predictors of the observed correlation matrix compared with the superior predictive ability of the normative true score matrix. It is suggested that conceptual similarities among job dimensions represent one potentially recalcitrant source of illusory halo in performance ratings, particularly when ratings are based on encoded observations that have decayed in memory. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A reexamination of the relation between halo error and accuracy.

The traditional assumption has been that halo error is negatively related to accuracy of ratings. W. H. Cooper (1981) evaluated this assumption by examining correlation coefficients between measures of accuracy and halo error from five earlier studies of performance and trait ratings. Because the correlation coefficients were typically positive, Cooper concluded that a "paradoxical" positive relation exists between halo error and accuracy. However, there is no paradox; some of these positive correlation coefficients were between halo error and inaccuracy, whereas others were based on analyses that did not take into consideration negative halo errors. When analyses that correct these problems were performed on two sets of data (R. Tallarigo, 1986, n?=?107; R. J. Vance, K. W. Kuhnert, & J. L. Farr, 1978, n?=?112), all significant (p?negative. The use of halo error measures, the possibility of negative halo errors, and implications of the results for rater training are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of race and sex effects on performance ratings.

The effects of rater source, rater and ratee race, rater and ratee sex, and job type were investigated on ratings collected for 8,642 first-term Army enlisted personnel. Ratings were made on 10 behaviorally based dimensions developed for evaluating all first-term soldiers. Results of between-Ss analyses similar to those conducted in past research revealed significant main effects and interactions for sex, race, rater source, and job type, but the variance accounted for by these effects was minimal. Repeated measures analyses were also performed, with each ratee evaluated by one Black and one White rater for the race effects analyses and one female and one male rater for the sex effects analyses. These analyses, which unconfounded rater bias and actual performance differences, yielded results similar to those obtained with the between-Ss design. Implications of the findings are discussed. ? (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some reflections on general performance dimensions and halo rating error.

Criticizes an article by F. Landy et al (see record 1981-00274-001), which assumed that a general factor present in the intercorrelations of ratings of performance was generated by halo rating errors. A number of decision rules were used to generate an analytic procedure for extracting halo errors, correcting ratings, and interpreting factors present in the corrected correlations. A number of the decision rules, as well as the initial assumption, seem to be rules of thumb or to depend on implicit theories of ratings and not on empirical data. Reanalysis of the rating data to allow both general 2nd- and 1st-order factors to be expressed in terms of item loadings recovered the structure present in the correlation of the original ratings as well as the psychological meanings of the 1st-order factors. General factors in rating data resemble general factors in measures of human ability. It is argued that removing general factors as if they were halo rather than true score may eliminate more of the variance from rating data than is justifiable. (8 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The future of partial correlation as a means to reduce halo in performance ratings.

Examines the practice of statistically controlling for halo effects in performance ratings. It is argued that there are 2 major problems with the partial correlation approach to removing halo from ratings data. First, the correct use of the technique depends on the validity of specific causal assumptions regarding the rating process that have not received empirical evaluation to date. The 2nd problem concerns analytic procedures employed in previous tests of the partialing approach. Reanalysis of data reported by F. Landy et al (see record 1981-00274-001) indicates that previous conclusions regarding the effectiveness of partialing may have been artifacts of the way the data were analyzed. It is felt that criticisms of the partial correlation approach to halo reduction are sufficient to suggest suspension of its use in any nonresearch context until necessary additional research is performed. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive categorization and dimensional schemata: A process approach to the study of halo in performance ratings.

Compared 2 models of the cognitive processes underlying performance ratings: a traditional model outlined by W. C. Borman (see record 1980-26801-001), and a cognitive categorization model proposed by J. M. Feldman (see record 1981-24524-001). To distinguish these 2 models, 120 college students were shown 1 of 2 videotapes of a college lecturer in which 3 of 5 dimensions of performance were manipulated to be opposite to that of the lecturer's overall performance. Ratings were made either immediately after viewing the videotape or 2 days later. Results indicate that the traditional model was appropriate for describing the rating process in both the immediate and the delayed rating conditions. However, a large halo effect was also found that was consistent with the categorization model despite conditions designed to minimize the likelihood of halo. Additional effects of cognitive categorization included a tendency to make errors in later recall of lecturing incidents consistent with Ss' general impression. (48 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reputation, halo, and ratings of counseling programs.

Tested the hypothesis that previous ratings of programs in psychology reflect both an experimental psychology and general institutional halo bias. A questionnaire similar to one used in an earlier study of graduate programs by H. D. Roose and C. J. Andersen (1970) was used to survey the responses of 598 professionals in the field of counseling psychology. Respondents were furnished with a listing of 70 doctoral programs in counseling psychology and other closely related programs and were asked to rate each of the programs. It was found that applied programs in counseling psychology received ratings that differed from overall ratings of psychology in general. Programs ranked as strong, good, and adequate are listed. Ratings were related to institutional halo, program age, rater knowledge of program, geographic location, and approved status by the American Psychological Association. Implications for program evaluation are discussed, and users of reputational ratings are cautioned about the need for supplemental information. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An examination of the relationship between the accuracy and favorability of ratings.

Administered a scale to 68 senior marketing students, who rated the prospecting ability of a life insurance agent on 7 videotaped telephone approaches. 3 randomly selected subgroups were given different background information about the agent. It was found that induced mental sets varied significantly in terms of favorability. A differential accuracy phenomenon (dap) was the only significant source of variance. However, neither the dap nor the overall accuracy of ratings was related to the favorability of s's general impression of the ratee. Findings make suspect the current practice of operationalizing leniency error by means of the average level of favorability of global ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The consequences of halo "error" in performance ratings: A field study of the moderating effect of halo on test validation results.

Performance ratings of 294 clerical workers in a validation study of clerical ability tests indicated that halo, measured as the standard deviation across dimensions, consistently moderated the relationships between dimension ratings and scores on valid tests. Greater halo resulted in higher validity coefficients, and also was related to higher performance ratings. In an additional analysis, statistically controlling for the effect of the overall rating on dimension ratings resulted in poorer validation results, with dimension ratings rarely adding additional variance to that of overall ratings. The results of this study contradict the traditionally held view of halo as a rating "error," and are consistent with recent laboratory studies that have found accuracy and halo positively related. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Discrepancy between self- and observer ratings of performance in social phobics.

28 Ss meeting Diagnostic and Statistical Manual of Mental Disorders-III-Revised (DSM-III-R) criteria for social phobia and without comorbid affective disorder and 33 nonclinical controls were asked to present a brief, impromptu speech to a small audience. Speakers themselves, as well as members of the audience, rated each speaker on a public speaking questionnaire that included both specific items (e.g., voice shook) and global items (e.g., appeared confident). For global items, no significant difference was indicated between the 2 groups on observers' ratings of public speaking performance. However, social phobics rated their own performance worse than did nonclinical controls, and there was a significantly greater discrepancy between self and other ratings for social phobics than controls. Fear of negative evaluation was the only significant predictor of the self–other discrepancy on global items. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measurement error masks bipolarity in affect ratings.

For years, affect researchers have debated about the true dimensionality of mood. Some have argued that positive and negative moods are largely independent and can be experienced simultaneously. Others claim that mood is bipolar, that joy and sorrow represent opposite ends of a single dimension. The 3 studies presented in this article suggest that the evidence that purportedly shows the independence of seemingly opposite mood states, that is, low correlations between positive and negative moods, may be the result of failures to consider biases due to random and nonrandom response error. When these sources of error are taken into account using multiple methods of mood assessment, a largely bipolar structure for affect emerges. The data herein speak to the importance of a multimethod approach to the measurement of mood. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the statistical correction of halo error.

Attempts to eliminate halo error from rating scales by statistical correction have assumed halo to be a systematic error associated with a ratee–rater pair that adds performance-irrelevant variance to ratings. Furthermore, overall performance ratings have been assumed to reflect this bias. Consideration of the source of halo error, however, raises the possibility that the cognitive processes resulting in halo also mediate expectations of and interactions with employees, indirectly influencing true performance and ability via instruction, feedback, and reinforcement. If so, it would not be possible to correct for halo error using overall performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater training on leniency and halo errors in student ratings of instructors.

Investigated the effects (over time) of a comprehensive vs an abbreviated rater training session on relative levels of leniency error and halo effect. 80 undergraduates (20 per group) rated all of their nonlaboratory instructors over 1, 2, or 3 rating periods using either behavioral expectation scales or summated rating scales. Tests on psychometric error were also administered at these times. Results indicate that the psychometric quality of ratings was superior for the group receiving the comprehensive training, and both training groups were superior to the control groups at the 1st measurement period. No differences were found between any groups in later comparisons. A consistent relationship was found between scores on the tests of psychometric error and error as measured on the ratings. Results are discussed in terms of the diminishing effect of rater training over rating periods, the relationship of internal and external criteria of training effects, the practical significance of differences between groups, and the importance of rating context on rating quality. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analyzing the relationship between grades and job performance.

Employers and academics have differing views on the value of grades for predicting job performance. Employers often believe grades are useful predictors, and they make hiring decisions that are based on them. Many academics believe that grades have little predictive validity. Past meta-analyses of the grades–performance relationship have suffered either from small sample sizes or the inability to correct observed correlations for research artifacts. This study demonstrated the observed correlation between grades and job performance was .16. Correction for research artifacts increased the correlation to the .30s. Several factors were found to moderate the relationship. The most powerful factors were the year of research publication and the time between graduation and performance measurement. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Nature and consequences of halo error: A critical analysis.

The definition of halo error that dominated researchers' thinking for most of this century implied that (1) halo error was common, (2) it was a rater error, with true and illusory components, (3) it led to inflated correlations among rating dimensions and was due to the influence of a general evaluation on specific judgments, and (4) it had negative consequences and should be avoided or removed. Research is reviewed showing that all of the major elements of this conception of halo are either wrong or problematic. Because of unresolved confounds of true and illusory halo and the often unclear consequences of halo errors, the authors suggest a moratorium on the use of halo indices as dependent measures in applied research. They suggest specific directions for further research on halo that take into account the context in which judgments are formed and ratings are obtained and that more clearly distinguish between actual halo errors and the apparent halo effect. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Control of halo error: A multiple regression approach.

Comments on an article by F. J. Landy et al (see record 1981-00274-001) that suggests a method for excluding halo variance in rating scales. It is argued that this approach may result in excluding true variance. The present article presents a conceptualization of the halo effect in terms of a suppressor variable. Accordingly, a multiple regression approach for the treatment of halo variance is suggested. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Initial decisions and subsequent performance ratings.

Examined how information initially encountered for one decision was later used for evaluating ratee performance. 101 undergraduates viewed videotaped performances of 4 male carpenters performing 4 tasks. Half of the Ss were instructed to view the performance with the intent of rating the deservedness of each worker for outside contracting work; the other half were asked to designate the one best worker for this work. Two days later, 70 of the 101 Ss were asked to rate (from memory) each worker's performance. Findings indicate that raters initially designating one worker for a positive outcome rated all workers higher than raters making initial deservedness ratings. This elevation in ratings, which occurred for both subsequent overall evaluations and task ratings, may have reflected both the tendency to inflate ratings given the individual receiving the initial treatment and leniency toward the other ratees for whom representations of ability may not have been well established or remembered. Overall delayed performance ratings were influenced by these initial contextual factors. Results suggest that the pattern in which information is presented to raters and the nature of previous decisions may affect memory-based performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater training and diary-keeping on psychometric error in ratings.

Compared the psychometric properties of ratings on behavioral expectation scales (BES) across 4 groups totalling 156 undergraduate raters. Groups differed with respect to amount of prior training (1 hr or more), the nature of psychometric errors, and the extent of exposure to scales (read scales and recorded observed critical incidents, discussed general scale dimensions, or no exposure to scales). Three Ss from each group rated 1 of 13 instructors during the last week of a 10-wk term. Significantly less leniency error and halo effect, plus higher interrater reliability, were found for the group that had received the hour of training and full exposure to the BES. Ss who had received only training had significantly less halo error than those that had received no training. The need for rater training prior to observation and the use of BES as a context for observation are discussed. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)