The systematic distortion hypothesis, halo, and accuracy: An individual-level analysis.

In several social perception studies investigators have concluded that raters' semantic conceptual similarity schemata serve to guide and constrain dimensional covariance in the rating judgment process. This effect has been hypothesized to be most likely when ratings are memory based and raters lack relevant job or ratee information. Recent research that has explored the effects of conceptual similarity schemata on performance ratings and halo error has provided some limited support for this systematic distortion hypothesis (SDH). However, these studies are limited because researchers have examined this phenomenon using group-level analyses, whereas the theory references individual-level judgment processes. The present study investigated the phenomena at the individual level. The effects of varying levels of rater job knowledge (high, medium, and low) and familiarity with ratees (high and low) were examined for conceptual similarity–rating and rating–true-score covariation relations, for measures of halo, and for rating accuracy components. Results provided support for the SDH, but indicated a boundary condition for its operation and revealed some surprising findings for individual-level rater halo. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Conceptual similarity as a source of illusory halo in job performance ratings.

Tested whether a possible source of difficulty in materially reducing illusory halo in job performance ratings is raters' beliefs that rating categories are conceptually similar and hence covary, thereby inflating observed correlation matrices. 11 graduate business administration students evaluated the conceptual similarities among job dimensions within 3 jobs. The previously observed interdimension correlation matrices were successfully predicted by Ss' mean conceptual similarity scores. When the observed correlation matrix obtained by W. C. Borman (see record 1980-26801-001) was compared with the normative true score matrix, the conceptual similarity scores were found to be inferior predictors of the observed correlation matrix compared with the superior predictive ability of the normative true score matrix. It is suggested that conceptual similarities among job dimensions represent one potentially recalcitrant source of illusory halo in performance ratings, particularly when ratings are based on encoded observations that have decayed in memory. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of the relationship between true halo and halo error in performance ratings.

The purpose of this study was to test competing theories regarding the relationship between true halo (actual dimensional correlations) and halo rater error (effects of raters' general impressions on specific ratee qualities) at both the individual and group level of analysis. Consistent with the prevailing general impression model of halo rater error, results at both the individual and group level analyses indicated a null (vs. positive or negative) true halo-halo rater error relationship. Results support the ideas that (a) the influence of raters' general impressions is homogeneous across rating dimensions despite wide variability in levels of true halo; (b) in assigning ratings, raters rely both on recalled observations of actual ratee behaviors and on general impressions of ratees in assigning dimensional ratings; and (c) these 2 processes occur independently of one another. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive complexity and appraisal effectiveness: Back to the drawing board?

Tested C. E. Schneier's (see record 1978-11450-001) cognitive compatibility theory. In Exps I and II, 100 undergraduates rated college instructors and professor vignettes, respectively. Results show that rater cognitive complexity was unrelated to rating accuracy, halo error, acceptability of rating format, or confidence in ratings. In Exp III, 31 police sergeants rated patrol officers, and the results show that halo error and acceptability of formats were unrelated to cognitive complexity. In Exp IV, 95 undergraduates' ratings of managerial performance and instructor effectiveness showed no support for the cognitive compatibility theory. However, the data showed that raters' ability to generate dimensions was significantly related to halo error in instructors' ratings. Implications for cognitive compatibility theory and future research with the method of generating performance dimensions are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the statistical correction of halo error.

Attempts to eliminate halo error from rating scales by statistical correction have assumed halo to be a systematic error associated with a ratee–rater pair that adds performance-irrelevant variance to ratings. Furthermore, overall performance ratings have been assumed to reflect this bias. Consideration of the source of halo error, however, raises the possibility that the cognitive processes resulting in halo also mediate expectations of and interactions with employees, indirectly influencing true performance and ability via instruction, feedback, and reinforcement. If so, it would not be possible to correct for halo error using overall performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of ratee task performance and interpersonal factors on supervisor and peer performance ratings.

The study examines the effects of a wide array of rater–ratee relationship and ratee-characteristic variables on supervisor and peer job-performance ratings. Interpersonal ratings, job performance ratings, and ratee scores on ability, job knowledge, and technical proficiency were available for 493 to 631 first-tour US Army soldiers. Results of supervisor and peer ratings-path models showed ratee ability, knowledge, and proficiency accounted for 13% of the variance in supervisor performance ratings and 7% for the peer ratings. Among the interpersonal variables, ratee dependability had the strongest effect for both models. Ratee friendliness and likability had little effect on the performance ratings. Inclusion of the interpersonal factors increased the variance accounted for in the ratings to 28% and 19%, respectively. Discussion focuses on the relative contribution of ratee technical and contextual performance to raters' judgments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationship between job knowledge and the reliability of conceptual similarity schemata.

Several investigators have argued that raters often rely on the conceptual similarity among performance dimension labels to guide the pattern of their performance ratings. Recent studies have used individual-level conceptual similarity (COS) judgments to investigate this systematic distortion hypothesis and related performance rating issues. In this article the results from 4 studies are reported in which 171 subjects completed COS judgments on 2 occasions. In 3 separate studies the reliability of COS schemata was found to be positively related to the rater's relevant job knowledge. In a 4th study it was found that changes in COS schemata over a 9-week interval may result from COS unreliability as much as from any meaningful reconceptualization of COS structure. Implications for performance rating research are reviewed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A test of the context dependency of three causal models of halo rater error.

C. E. Lance et al (see record 1994-17452-001) tested 3 different causal models of halo rater error (general impression [GI], salient dimension [SD], and inadequate discrimination [ID] models) and found that the GI model better accounted for observed halo rating error than did the SD or ID models. It was also suggested that the type of halo rater error that occurs might vary as a function of rating context. The purpose of this study was to determine whether rating contexts could be manipulated that favored the operation of each of these 3 halo-error models. Results indicate, however, that GI halo error occurred in spite of experimental conditions designed specifically to induce other forms of halo rater error. This suggests that halo rater error is a unitary phenomenon that should be defined as the influence of a rater's general impression on ratings of specific ratee qualities. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Is halo error a property of the rater, ratees, or the specific behaviors observed?

The stability of halo errors when the ratees, the specific behavioral episodes observed, or both varied was studied. In a laboratory study, halo errors were highly unstable when either the ratees or the ratee behaviors varied (average stability coefficients were .20 and .18 when ratee behavior or both ratees and their behavior varied, respectively), but halo errors were moderately stable when the ratees and the specific performance segments viewed were kept constant. In a field study using actual teacher ratings in which the ratee, the ratee's role, or the semester in which ratings were obtained was varied, very low stability coefficients were again found. The results suggest that halo error is not a stable characteristic of the rater or the ratees but rather is partly a characteristic of the unique rating situation. Practical and theoretical implications are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Halo and performance appraisal research: A critical examination.

The different conceptual and operational definitions of halo are reviewed, and problems when using halo as a dependent measure in performance rating research and practice are pointed out. Four major points are emphasized: (1) There is no agreed on conceptual definition of halo; (2) the different conceptual definitions of halo are not systematically related to different operational definitions (i.e., measures) of halo; (3) halo measures may be poor indexes of rating quality in that different halo measures are not strongly interrelated and halo measures are not related to measures of rating validity or accuracy; and (4) although halo may be a poor measure of rating quality, it may or may not be an important measure of the rating process. The utility of assessing halo to determine the psychometric quality of rating data is questioned. Halo may be more appropriately used as a measure to study cognitive processing, rather than as a measure of performance rating outcome. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Patterns of parental childrearing and subsequent vulnerability to cognitive distu bance.

Research with normals and schizophrenics has indicated the importance of maternal childrearing control and nurturance. The 1st experiment tested whether patterns of maternal control and nurturance would relate to the son's subsequent cognitive performance under conditions involving social censure. Ss were 63 college males. Ss rating their mothers as highly controlling-low nurturant (rejecting pattern) were poorer in conceptual performance than Ss whose mothers were rated as low controlling-highly nurturant (accepting pattern). The same results were obtained when 74 males were divided into paternal childrearing pattern groups and their conceptual performance compared. It was proposed that social censure elicits responses which interfere with effective cognitive performance in Ss whose rejecting childrearing histories have mediated low self-esteem. (1 p. ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evidence incompatible with a cognitive compatibility theory of rating behavior.

Assessed the cognitive complexity of 96 undergraduates with the group version of the Role Construct Repertory (REP) Test, a factor analysis of REP test data, and a sorting task. Performance ratings for 3 of the Ss' instructors were obtained with behaviorally anchored rating scales, mixed standard rating scales, graphic rating scales, and simple "alternate" 3-point rating scales. No differences in leniency, halo, or range restriction emerged either as a function of raters' cognitive complexity or a Cognitive Complexity?×?Scale Format interaction. Raters' confidence in their ratings was not associated with either cognitive complexity or rating scale format. It is concluded that researchers of performance ratings should exercise restraint before confidently conferring moderator variable status on a cognitive complexity construct. (25 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating leniency and halo in multisource feedback ratings: Testing cultural assumptions of power distance and individualism-collectivism.

This study extends multisource feedback research by assessing the effects of rater source and raters' cultural value orientations on rating bias (leniency and halo). Using a motivational perspective of performance appraisal, the authors posit that subordinate raters followed by peers will exhibit more rating bias than superiors. More important, given that multisource feedback systems were premised on low power distance and individualistic cultural assumptions, the authors expect raters' power distance and individualism-collectivism orientations to moderate the effects of rater source on rating bias. Hierarchical linear modeling on data collected from 1,447 superiors, peers, and subordinates who provided developmental feedback to 172 military officers show that (a) subordinates exhibit the most rating leniency, followed by peers and superiors; (b) subordinates demonstrate more halo than superiors and peers, whereas superiors and peers do not differ; (c) the effects of power distance on leniency and halo are strongest for subordinates than for peers and superiors; (d) the effects of collectivism on leniency were stronger for subordinates and peers than for superiors; effects on halo were stronger for subordinates than superiors, but these effects did not differ for subordinates and peers. The present findings highlight the role of raters' cultural values in multisource feedback ratings. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

An investigation of the validity of expert true score estimates in appraisal research.

Researchers of appraisal rating accuracy have defined ratee true scores as the mean ratings given by experts provided with enhanced opportunities to observe performance. The external validity of accuracy research depends on the relevance of these expert estimates to true performance. In our study, we provided expert raters with enhanced opportunities to view videotapes of five ratees under conditions of high true dimension intercorrelation or low true intercorrelation. The accuracy of expert ratings was compared with the accuracy of nonexpert ratings of the same tapes viewed under more typical rating conditions. Subjects' ratings were compared with ratee true scores defined in terms of objective worker output. Results indicated that experts were more accurate than nonexperts, regardless of the true dimension intercorrelations. Accuracy indices computed by using objective true scores were highly correlated with indices computed by using the mean expert ratings as true score estimates. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Inhibitory effect of schematic processing on perceptual encoding.

Hypothesized that although schemata facilitate organized conceptual processing, and hence recall, they simultaneously inhibit perceptual encoding. This inhibitory effect should emerge because schemata allow perceivers to rely on prior knowledge in place of incoming information and because schemata facilitate selective attention. Exps 1 and 2 demonstrated that Ss encode less relevant perceptual information when they are provided with or are able to induce a schema. Exp 3 demonstrated that Ss encode less relevant perceptual information when they are self-schematic in a domain, even though they have better recall for that information. Exp 4 demonstrated that Ss encode less irrelevant perceptual information when they are provided with a schema. Thus, results show that although schemata facilitate recall, they simultaneously inhibit perceptual encoding. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues.

Considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. 93 college students (mean age 22 yrs) trained using either FOR or control procedures, observed 3 managers on videotape, and rated the managers on 3 performance dimensions. Results supported the hypothesis that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behaviorally anchored rating scales: Effects of education and job experience of raters and ratees.

Studied the effects of 2 levels of rater and ratee experience and education, as well as their possible interaction, on behaviorally anchored rating scales. A total of 370 male police personnel participated, of whom 71 were sergeants and 299 were police officers. Eight dependent variables, each a 9-point behaviorally anchored rating scale describing 1 dimension of police officer performance, were subjected to fixed-effects, unweighted-means analyses of variance. Results indicate that raters' experience and raters' education accounted for most of the statistically significant effects. Likewise Raters' Experience?×?Education and Raters' Education?×?Ratees' Education interactions were statistically significant. All significant effects were weak, however, as indicated by overlaps of 82–92% between distributions, and eta-squares for all significant F ratios of .01–.03. Hence, neither rater nor ratee characteristics exerted any practically significant effects on observed behaviorally anchored ratings. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Schematic and situational determinants of depressed and nondepressed students' interpretation of feedback.

We examined whether depressed persons' social skill deficits contribute to their negative cognitions and whether this contribution is independent of their negative schemata. Depressed (n?=?60) and nondepressed (n?=?60) Ss engaged in group discussions. We assessed Ss' social competence schemata with a questionnaire and Ss' actual level of social competence in the discussion through objective ratings made by codiscussants and outside observers. We found that independently of their negative schemata, depressed Ss' social skill deficits explained a significant portion of the variance in their more negative interpretation of feedback (relative to nondepressed Ss'). This suggests that real deficits in depressed persons' performance compound the effects of their negative schemata and further contribute to their negative cognitions. We also further explored findings by B. M. Dykman et al (see record 1989-18948-001) and P.M. Lewinsohn et al (see record 1980-12088-001). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of frame-of-reference training on rater accuracy under alternative time delays.

Investigated the effects of frame-of-reference (FOR) training on various indexes of distance and correlational accuracy under alternative time delays. 150 Ss were assigned randomly to either FOR- or control- (i.e., minimal) training conditions, with 1 of 3 time delays: (1) no delay between training, observation, and rating; (2) ratings performed 2 days following training and ratee observations; or (3) ratee observations and ratings completed 2 days following training. Hypotheses were proposed predicting specific relationships between accuracy, recall memory, and learning, depending on the delay period. Overall, results support the categorization perspective on FOR-training effectiveness; however, different results were obtained depending on the type of accuracy index and time delay. The implications of these findings are discussed in terms of how they relate to the conceptual distinction between distance and correlational accuracy and to the role of on-line, memory-based, and inference-memory-based processing in the ratings of FOR trained raters. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some reflections on general performance dimensions and halo rating error.

Criticizes an article by F. Landy et al (see record 1981-00274-001), which assumed that a general factor present in the intercorrelations of ratings of performance was generated by halo rating errors. A number of decision rules were used to generate an analytic procedure for extracting halo errors, correcting ratings, and interpreting factors present in the corrected correlations. A number of the decision rules, as well as the initial assumption, seem to be rules of thumb or to depend on implicit theories of ratings and not on empirical data. Reanalysis of the rating data to allow both general 2nd- and 1st-order factors to be expressed in terms of item loadings recovered the structure present in the correlation of the original ratings as well as the psychological meanings of the 1st-order factors. General factors in rating data resemble general factors in measures of human ability. It is argued that removing general factors as if they were halo rather than true score may eliminate more of the variance from rating data than is justifiable. (8 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)