Rater reliability and the heterogeneity of scale anchors.

"Raters (N=200) used 5-category rating scales, which varied in the heterogeneity of the verbal anchors defining the end categories, to rate for preference value two lists of ten foods. The food lists differed in the homogeneity of food stimuli on each list. Measures of individual rater reliability and rater bias were computed and analyzed as to the effect of scale and list differences. Reliability was significantly smaller for the more homogeneous list and increased linearly as a function of the heterogeneity of the end anchors. Rater bias was unaffected by either scale or list differences." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bias in psychotherapist ratings of client transference and insight.

The psychology literature at large considers rater bias to be a substantial source of error in observer ratings. Yet, it is typically ignored by psychotherapy researchers using participant (psychotherapist/client) ratings. In particular, interrater variability, or differences between raters' overall tendency to rate others favorably or unfavorably, has been a largely ignored source of error in studies that use psychotherapists and/or clients as raters. Ignoring rater bias can have serious consequences for statistical power and for interpretation of research findings. Rater bias may be a particular problem in psychotherapy research, as psychotherapists are often asked to rate subjective variables that require much rater inference. Consequently, we examined the extent to which rater bias is a factor in psychotherapist ratings of client transference and insight, by comparing psychotherapist variance from these ratings to psychotherapist variance in ratings of client-perceived emotional intelligence, using Hierarchical Linear Modeling. Results suggest that bias may be a substantial source of error in psychotherapist process and relationship ratings, accounting for, on average, 38% of the total variance in scores, and 30% after accounting for perceived emotional intelligence. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Modeling Error Variance in Job Specification Ratings: The Influence of Rater, Job, and Organization-Level Factors.

The authors modeled sources of error variance in job specification ratings collected from 3 levels of raters across 5 organizations (N=381). Variance components models were used to estimate the variance in ratings attributable to true score (variance between knowledge, skills, abilities, and other characteristics [KSAOs]) and error (KSAO-by-rater and residual variance). Subsequent models partitioned error variance into components related to the organization, position level, and demographic characteristics of the raters. Analyses revealed that the differential ordering of KSAOs by raters was not a function of these characteristics but rather was due to unexplained rating differences among the raters. The implications of these results for job specification and validity transportability are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The predictability of ratings as a function of interrater agreement.

"The hypothesis tested was that high agreement among the ratings assigned the same men by different raters does not necessarily imply predictable ratings." 3 superior officers rated 100 submariners on personal adjustment and technical competence. Each rating group was divided into 4 samples for interrater agreement. Correlations were computed with 3 predictor variables. The results support the hypothesis given. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some differential effects of race of rater and ratee on early peer ratings of combat aptitude.

Peer ratings of combat potential made by 669 Caucasian and Negro army recruits of the squad members were examined. The hypothesis that ratees would receive higher ratings from members of their own race than from members of another race was supported (p     

Source-Monitoring Training: Toward Reducing Rater Expectancy Effects in Behavioral Measurement.

The authors developed a source-monitoring procedure to reduce the biasing effects of rater expectations on behavioral measurement. Study participants (N = 224) were given positive or negative information regarding the performance of a group and, after observing the group, were assigned to a source-monitoring or control condition. Raters in the source-monitoring condition were instructed to report only behaviors that evoked detailed memories (remember judgments) and to avoid reporting behaviors based on feelings of familiarity (know judgments). Results revealed that controlling raters' response strategy reduced (and often eliminated) the biasing effects of performance expectations. These findings advance our understanding of the performance-cue bias and offer a potentially useful technique for decreasing rater bias. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Erratum to Cascio and Valenzi.

Reports an error in the original article by Wayne F. Cascio and Enzo R. Valenzi (Journal of Applied Psychology, 1978, Vol. 63, No. 1, pp. 22-28). In the last sentence of the Results section of the article, the values are incorrect. The corrected values for line 33 of page 26 are provided. (The following abstract of this article originally appeared in record 1979-24955-001.) Job performance measures consisting of 35 objective indices and ratings on 8 behaviorally anchored rating scales (BARS) were available for 795 nonminority (mean age, 29.8 yrs) and 147 minority (mean age, 28.2 yrs) police officers. Eight of the 35 objective measures, plus age and job tenure, were used as predictors of the sum of the 8 BARS. Identical predictor sets validly forecast supervisory ratings in both minority and nonminority groups whether or not age and tenure were included. Unit weights were inferior to regression weights in both groups. It is concluded that supervisory ratings are linearly predictable from objective performance indices for both minority and nonminority subordinates, a finding that comports with civil rights legislation and recent US Supreme Court decisions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison between ipsative and normative ratings of personality.

This is a report of results of 2 studies in which ipsative ratings, treated normatively, were correlated with corresponding normative ratings in a test of the functional equivalence of the 2 forms of measurement. "Both analyses showed an almost complete equivalence between the 2 methods. A possible reason for this relationship was discussed and some of the advantages of the ipsative approach presented." The opinion that a number of current rating and assessment procedures bear revision in light of these findings was offered. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Graphic ratings and attitude measurement: A comparison of research tactics.

This paper suggests that under certain conditions attitudes may be measured as validly and as reliably with a single "attitude report question" as with a multi-item attitude scale. 84 physicans responded to 8 Guttman scales, and also rated their attitudes on 8 comparable graphic rating scales. These data were factor analyzed, and the communalities used as an estimate of the minimal scale reliability. In general, the graphic rating scales proved as reliable as the Guttman scales. An examination of the interscale correlations showed that similar conclusions would be drawn from either technique. It is suggested that a single graphic rating may usefully substitute for a multi-item attitude scale when the attitude continuum is unidimensional. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater bias in performance ratings: Superior, self-, and peer ratings.

Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Contrast and convergence effects in ratings of foods.

"A set of assumptions was made that led to the hypothesis that preference ratings for poor quality food will be lower when preceded by a good quality food than when preceded by another poor quality item (contrast effects). It was also hypothesized that preference for a good quality food will be higher when preceded by another good quality item than when preceded by a poor quality product (convergence effects). The other predictions were that preference will increase with successive presentations of the same quality item, provided no opposite quality intervenes. The predictions concerning preference for the poor quality foods were clearly confirmed, but those involving the good quality foods were not substantiated." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evidence of the construct validity of developmental ratings of managerial performance.

The construct validity of developmental ratings of managerial performance was assessed by using 2 data sets, each based on a different 360° rating instrument. Specifically, the authors investigated the nature of the constructs measured by developmental ratings, the structural relationships among those constructs, and the generalizability of results across 4 rater perspectives (boss, peer, subordinate, and self). A structure with 4 lower order factors (Technical Skills, Administrative Skills, Human Skills, and Citizenship Behaviors) and 2 higher order factors (Task Performance and Contextual Performance) was tested against competing models. Results consistently supported the lower order constructs, but the higher order structure was problematic, indicating that the structure of ratings is not yet well understood. Multisample analyses indicated few practically significant differences in factor structures across perspectives. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Intrajudge consistency in ratings of heterogeneous persons.

"The existence and relative power of a unitary attitude toward diverse persons, implied in statements of Kelly (1955) and Sullivan (1956), were investigated by utilizing Cronbach's (1955) operational definition of 'implicit personality theory.' Although the results supported the concept of a general response set, the fragility of intrajudge consistency relative to stimulus factors points to a construct difficult to validate. No differences in consistency were found to be related to sex… The findings suggest less theoretical emphasis upon [judge]… bias and greater attention to response variability and 'reality' as evinced by interjudge agreement concerning stimulus attributes." From Psyc Abstracts 36:04:4HE05G. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Media ratings for violence and sex: Implications for policymakers and parents.

This article reviews research on the implementation of media-rating systems, parents' use and evaluation of them, and the impact of ratings on children. Although half or more of parents report using media-rating systems, understanding of various components of the systems is low, particularly for television ratings. A meta-analysis of national polls shows that parents overwhelmingly prefer that ratings specify content, rather than giving age recommendations. A second meta-analysis, of experiments testing the effects of ratings on children's interest in programs, shows that ratings indicating restricted or controversial content have a deterrent effect for children under age 8 but that, by age 11 and especially for boys, the ratings show a small enticement effect. This effect occurs for both age-based and content-based ratings. Implications for policymakers and parents are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Convergent job expectations and ratings of industrial foremen.

A significant positive correlation was hypothesized between evaluations of foremen made by superiors and subordinates and the degree to which foremen share and accurately predict superior-subordinate expectations regarding the foreman's job behavior. The Ss, (8 superiors, 32 foremen, and 377 subordinates) responded to a questionnaire composed of "consideration" and "structure" items yielding "expectation-convergence scores" which were correlated with evaluation measures. In a department providing close face-to-face interaction, a significant proportion of the variance in superior evaluations was accounted for by the hypothesis of expectation convergence. Implications for common personnel practices are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Self and parent ratings of executive functioning after adolescent traumatic brain injury.

Objective: To examine the level of agreement between adolescents with traumatic brain injury (TBI) and their parents in standardized ratings of executive functioning, and to determine correlates of discrepancies between those ratings. Participants: Ninety-eight 11- to 16-year-old adolescents with TBI and their parents, and 97 neuropsychologically healthy controls. Method: Five-year consecutive series of rehabilitation referrals for TBI. Measures: Behavior Rating Inventory of Executive Function–Self Report (BRIEF–SR) and Behavior Rating Inventory of Executive Function (BRIEF) parent report versions. Results: Self and parent ratings were moderately positively correlated in both the TBI group and the control group, but parents generally identified more executive dysfunction than did the adolescents. Parent-adolescent discrepancies were statistically significantly greater in the TBI group than in the control group on the Metacognitive index but not the Behavioral Regulation index. The degree of the former discrepancy was predicted by duration of coma in the TBI group. Conclusions: Adolescents with more severe TBI may underestimate their own degree of executive dysfunction in daily life, particularly aspects of metacognitive abilities, possibly, in part, because of an organic-based lack of deficit awareness. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Published student ratings of instruction: Revealing and reconciling the views of students and faculty.

The authors surveyed students (N?=?396) and faculty (N?=?156) at a 2-year college to determine their views toward publishing student ratings of instruction. Students favored published ratings of instruction, whereas faculty did not. Students cited many advantages of published ratings and rated the likelihood of potential benefits as high relative to faculty. In contrast, faculty cited numerous disadvantages of published ratings and rated the likelihood of potential costs as high relative to students. The authors discuss reasons for the contrasting views of students and faculty and offer suggestions for reconciling them. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Supervisory ratings and attitudes.

Do supervisors tend to be biased in their ranking and rating of workers? 42 janitors were ranked and rated by 4 supervisors and rated by the division chief. The highest and lowest were compared by 10 factor scores. "Supervisors of our hospital maintenance men valued most the abilities least prevalent in the men (leadership and executive ability) and valued least the attributes most abundant in these men (social adjustment and personal charm). (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The criterion problem and creativity in film: Psychometric characteristics of various measures.

Creativity in movies is a topic of growing interest in both the psychological and the marketing literature. Much research has been invested into determining the impact of cinematic quality on film success and finding successful predictors of cinematic creativity. For these reasons, research into the way creativity is measured in film is of considerable importance. This study examines a variety of measures of cinematic quality (movie ratings from a variety of sources) and determines the degree of agreement among different types of measures, the predictive value of these measures, and the effect of the timing of these measures on their predictive value. Results indicate that there is a high degree of agreement among types of movie ratings, that reviews through release day tend to be marginally higher than those that appear later, and that reviews are more highly correlated with later box office success (gross) than with early box office success. A surprising result of this study was that the number of ratings a movie received was a slightly better predictor of box office success than the actual movie ratings. Possible explanations for and implications of these results are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Personality development in emerging adulthood: Integrating evidence from self-ratings and spouse ratings.

The authors examined self-ratings and spouse ratings in a young adult newlywed sample across a 2-year interval. Rank-order stability correlations were consistently high and did not differ across the 2 types of ratings. As expected, self-ratings showed significant increases in conscientiousness and agreeableness- and declines in neuroticism/negative affectivity- over time. Spouse ratings yielded a very different pattern, however, showing significant decreases in conscientiousness, agreeableness, extraversion, and openness across the study interval. Spouse ratings also showed evidence of a "honeymoon effect," such that they tended to be more positive than self-ratings at Time 1. This effect had dissipated by the 2nd assessment; in fact, the spouse ratings tended to be more negative at Time 2. Analyses of individual-level change revealed little convergence between self- and spouse-rated change, using both raw change scores and reliable change index scores. Finally, correlational and regression analyses indicated that changes in spouse ratings were significantly associated with changes in marital satisfaction; in contrast, changes in self-ratings essentially were unrelated to marital satisfaction. These results highlight the value of collecting multimethod data in studies of adult personality development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)