Are performance appraisal ratings from different rating sources comparable?

The purpose of this study was to test whether a multisource performance appraisal instrument exhibited measurement invariance across different groups of raters. Multiple-groups confirmatory factor analysis as well as item response theory (IRT) techniques were used to test for invariance of the rating instrument across self, peer, supervisor, and subordinate raters. The results of the confirmatory factor analysis indicated that the rating instrument was invariant across these rater groups. The IRT analysis yielded some evidence of differential item and test functioning, but it was limited to the effects of just 3 items and was trivial in magnitude. Taken together, the results suggest that the rating instrument could be regarded as invariant across the rater groups, thus supporting the practice of directly comparing their ratings. Implications for research and practice are discussed, as well as for understanding the meaning of between-source rating discrepancies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Organization of information in memory and the performance appraisal process: Evidence from the field.

No studies dealing with cognitive processes in performance appraisal have been conducted in field settings, raising questions about the usefulness of this research for practice. The field experiments described here, conducted in 2 organizations, were designed to evaluate interventions that laboratory research has suggested enable raters to better organize performance information in memory: structured diary keeping and structured recall. After these interventions, raters had more positive reactions to the appraisal process, were better able to recall performance information, and produced ratings that were less elevated and better able to discriminate between and within ratees. The implications of these results for practice and for cognitive research in performance appraisal are discussed, along with the limitations of these studies and the problems with criteria for evaluating ratings in the field. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Enhanced co-orientation in the perception of friends: A social relations analysis.

Newcomb's (1953) idea of co-orientation (interdependence between two persons' attitudes or perceptions) is used as a framework within which interpersonal perception between friends and acquaintances is examined. The principal question is whether co-orientation effects are stronger for friendship dyads than for acquaintance dyads. More specifically, the study examines the degree to which consensus, assimilation, self–other agreement, and assumed similarity differ. The social relations model is used to analyze a data set that included 16 living groups with 119 friend dyads and 1,668 acquaintance dyads. Results indicate that co-orientation effects are more pronounced in friendship dyads. The increment in co-orientation effects is largely due to similarities in the unique or idiosyncratic perceptions that people have of friendship pairs as well as the unique agreement about others that friends have with one another. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Organization of information used for performance appraisals: Role of diary-keeping.

We designed a laboratory study (N?=?63) to test a series of hypotheses concerning diary-keeping derived from a social-cognitive view of the performance appraisal process. Raters who were given unorganized performance information were expected to store it in memory in patterns consistent with those used to organize diaries. Results supported this hypothesis. Moreover, raters preferred to organize diaries according to ratees, and diaries organized in this way resulted in more accurate recall and performance ratings. This effect held even though raters consulted diaries when making ratings. These results provide support for a cognitive role for diaries, such that they allow raters to impose organization on unorganized information. We discuss implications for research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Delay and distortion: Tacit influences on performance appraisal effectiveness.

A laboratory study of performance appraisal delay and distortion (a) investigated the effects of rater gender, performance level, and feedback requirement, and (b) defined feedback delay as the separate components of appraisal delay and feedback session scheduling delay. Subjects were 64 university business students who had at least 6 months of supervisory experience. Female raters delayed longer when conducting performance appraisals and when scheduling feedback sessions with subordinates, and more positively distorted ratings than did male raters. Moderately low performers were evaluated and scheduled for feedback sessions later, and were evaluated with more positive distortion than were moderately high performers. When required to provide feedback, raters delayed longer appraising performance and distorted ratings more positively. Significant interactions suggest that when feedback is required, female raters may delay appraisals, delay scheduling feedback sessions, and more positively distort their ratings of performance, particularly when rating low performers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Biomechanical analysis of nonreamed tibial intramedullary nailing after simulated transverse fracture and fibulectomy

PURPOSE: To examine differences between early adolescent girls' and their mothers' perceptions of girls' depressive symptoms. METHODS: 313 daughter-mother dyads completed the Children's Depression Inventory. RESULTS: Low to modest agreement was found for most symptoms, although higher agreement was found for symptoms relating to school performance. The hypothesis that girls would report more ideational symptoms and mothers more behavioral symptoms of depression was tested; girls generally reported more ideational and behavioral symptoms when differences occurred. However, several specific ideational symptoms (feeling like crying; feeling sad; guilt; worrying) tended to be more frequently endorsed by girls and had particularly poor daughter-mother agreement. Examining third variables associated with daughter-mother agreement, girls scoring high on social desirability tended to have smaller daughter-minus-mother difference scores for ideational, but not for behavioral items; therefore, social desirability may be associated with girls underreporting ideational symptoms. CONCLUSIONS: Mothers appear to be reliable raters of symptoms related to school functioning, but may be less aware of certain covert depressive symptoms in their early adolescent daughters.     

Cross-informant agreement for ratings for social skill and problem behavior ratings: An investigation of the Social Skills Improvement System—Rating Scales.

One of the most consistent findings in rating scale research with children and adolescents is the modest agreement among different informants’ ratings. The present study systematically explored patterns of agreement among teachers, parents/caregivers, and students in domains of social skills and problem behaviors using the Social Skills Improvement System—Rating Scales (SSIS–RS; F. M. Gresham & S. N. Elliott, 2008). Two subsamples from the normative sample of the SSIS–RS were used. The first sample of participants consisted of 168 students who had all 3 informants (parent, teacher, and self) complete the SSIS–RS scales, which was necessary to assess agreement across different raters. The second sample consisted of 164 students who had raters in a similar or same role (father–mother, teacher–teacher). The results replicated an extensive literature showing that cross-informant agreements for social skills and problem behaviors are weak to moderate. The current study invoked multitrait–multimethod logic to interpret the correlations among raters derived from different informants and showed that the convergent validity coefficients were consistently stronger than the discriminant validity correlations. Implications for assessment practices and future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Category-based and feature-based processes in performance appraisal: Integrating visual and computerized sources of performance data.

Recent developments in computerized performance monitoring (CPM) raise issues about how supervisors use monitored information. On the basis of theories of person perception that distinguish between category-based and feature-based information-processing strategies, the authors examined how raters integrate performance information from 2 sources: visual and computerized. Ss received positive or negative computerized data and viewed positive or negative episodes of a secretary's performance. Visual data were most influential in determining raters' processing strategies, suggesting that raters used category-based strategies when viewing positive performance and increased their use of feature-based strategies when viewing negative performance, regardless of the evaluative implications of the computerized data. Implications for performance appraisal and CPM systems are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Consensus in personality judgments: Moderating effects of target-rater acquaintanceship and behavior observability.

Conducted a peer rating study (N?=?111) to determine the effects of (a) level of acquaintanceship between rater and target and (b) degree of public observability of rated personality traits on peers' perceptions of target personality characteristics. As hypothesized, I found the agreement between peer ratings and target self-ratings to vary linearly and directly with acquaintanceship. In addition, acquaintanceship interacted with observability such that the public visibility of the behavior domain being judged was an important determinant of agreement for low to moderately acquainted peer dyads but not for highly acquainted dyads. Contrary to expectations, however, trait observability did not show a main effect with regard to self–peer agreement. The basis of the study is described with reference to the lens model of inferential behavior, and implications of the results are discussed with reference to past and future attempts at evaluating consensus and accuracy in person perception. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effects of purpose of appraisal and individual differences in stereotypes of women on sex differences in performance ratings: A laboratory and field study.

We investigated the effects of purpose of appraisal and individual differences in stereotypes of women on the evaluations of male and female ratees in two studies. In Study 1, 52 female and 51 male undergraduate students with traditional or nontraditional stereotypes of women evaluated written vignettes of either male or female ratees. Raters were informed that their evaluations would be used for scale development (experimental purposes) or for merit pay and promotion decisions (administrative decisions). Analyses revealed that female ratees were evaluated less accurately by raters with traditional stereotypes of women than by raters with nontraditional stereotypes of women. Such differences occurred, however, only when appraisals were made for administrative decisions. In Study 2, 810 raters with traditional or nontraditional stereotypes of women evaluated the teaching effectiveness of eight male and six female professors. Analyses indicated that women were evaluated more favorably by raters with nontraditional stereotypes of women than by raters with traditional stereotypes of women. Implications of the findings for sex differences in appraisal, future research, and organizational effectiveness are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater training and rating accuracy: Training for dimensional accuracy versus training for ratee differentiation.

Undergraduate Ss possessing normative or idiosyncratic rating standards were given frame-of-reference training, rater-error training, training that controlled for structural similarities between frame-of-reference training and rater-error training, or null control training. Hypothesized pretest differences that normative raters are more accurate than idiosyncratic raters were not found. However, when data were collapsed across rating aptitude, different trainings were found to improve different measures of accuracy. Frame-of-reference trainees were most accurate on stereotype accuracy and differential accuracy, rater-error trainees were most accurate on elevation, and all groups improved on differential elevation. Results are discussed in relation to the role of rater aptitude in frame-of-reference training and the future of rater-training programs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further investigation of common knowledge effects on job analysis ratings.

Previous research (Smith & Hakel, 1979) raised the possibility that the Position Analysis Questionnaire (PAQ) only captures common knowledge, or stereotypes, about jobs. Cornelius, DeNisi, and Blencoe (1984) presented data to refute this, but found that the number of PAQ items rated does not apply (DNA) was related to the agreement between naive raters and expert raters. The present study used data from 87 analysts and 24 jobs. Naive ratings were those obtained from analysts who had not studied the job, whereas expert ratings were those obtained from raters who had observed the job, interviewed incumbents, and written task statements describing the job. These ratings were then compared to target-score profiles obtained from PAQ services. Results confirmed earlier hypotheses that large numbers of DNA items artifactually inflate correlational estimates of agreement between expert and naive raters. In addition to this artifact, results also supported the view that the PAQ is less appropriate as a job analysis tool for some types of jobs. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Personal and interpersonal predictors of supervisors' avoidance of evaluating subordinates.

Supervisors' decisions about whether to formally evaluate their subordinates for each of 2 consecutive performance appraisal periods were found to be related to a linear combination of the following variables: subordinate's time under current supervisor, subordinate's job experience, subordinate–supervisor trust, supervisor's initiating structure behaviors, and subordinate's confidence in the operation of the performance appraisal system. A follow-up analysis, in which the criterion was the consistency with which subordinates were rated in both performance appraisal periods, suggested that relations among the predictors, relative to the criterion, may be quite complex. This complexity may reflect the decision strategy used by raters to combine information on the focal predictors when deciding whether to evaluate a particular subordinate. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Leader–member agreement: A vertical dyad linkage approach.

The hypothesis derived from the vertical dyad linkage model of leadership development and investigated in this study is that agreement between a leader and a member regarding the meaning of certain mutually experienced events and situations will vary as a positive function of the quality of their dyadic exchanges. By employing dyadic measurement procedures, 109 university managerial dyads were assessed at 3 separate times with 3-mo intervals between assessments. Two different measures of dyadic agreement were used. Results show moderately high agreement within high- and intermediate-quality dyads and low agreement within low-quality dyads. Results were replicated on an independent sample of 41 managerial dyads. Implications are discussed. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Perceived similarity and accuracy of peer ratings.

The hypothesis that raters will be more accurate in rating peers perceived to be similar to themselves, suggested by Mumford (1983) and derived from social comparison theory, was examined. Subjects were 681 Israeli entrants to a military training program. Shortly after course inception, subjects were asked to review the performance of squad members and to forecast their final grade. Subjects also judged peers' similarity to self overall (general similarity), in course achievement (foreground similarity), and in military experience (background similarity). Analysis revealed that accuracy was markedly lower in the evaluation of dissimilar others. Although the same patterns of results was observed for all forms of similarity, stronger effects resulted when similarity was measured in terms of general and foreground characteristics. Implications for future theory and research as well as for the practical application of peer assessment are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Counselor technical activity in cases with improving working alliances and continuing-poor working alliances.

Counselor technical activity in counseling dyads was examined with working alliances that were client-rated as improving (8 dyads) or as continuing poor (7 dyads). Ss were 15 volunteer clients paired with 15 counselor trainees for 4 counseling sessions. Clients rated the strength of the working alliance for each session, and raters scored counselor technical activity on the dimensions of supportive vs challenging, distant vs involved, permissive vs controlling, thematically oriented vs concretely oriented, and here-and-now oriented vs there-and-then oriented. Counselors were relatively more challenging, thematically focused, and here-and-now oriented in the improving dyads than in the continuing-poor alliance dyads. The implications of these findings for theoretical formulations and counseling practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding why they don’t see eye to eye: An examination of leader–member exchange (LMX) agreement.

Although it is an explicitly dyadic approach to leadership, some leader–member exchange (LMX) research has been characterized by relatively low levels of agreement between leader and member judgments of the relationship. Using a combination of meta-analytic methods and primary data collection, the authors sought to explore several theoretically and methodologically meaningful factors that might account for lower levels of agreement. On the basis of data from 64 independent samples (N = 10,884 dyads), the authors found that overall agreement was moderate in nature (ρ = .37). In addition, they found that longer relationship tenure, affectively oriented relationship dimensions, and ad hoc sampling techniques showed the highest levels of agreement. Empirical results from 98 matched dyads revealed that the extent of LMX agreement increases as the length of relationship tenure and intensity of dyadic interaction increases. Implications for LMX theory and future empirical research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)