The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Role of early supervisory experience in supervisor performance.

Explores the role of early supervisory experience and cognitive ability in 1st-line supervisor performance. Similar to the F. L. Schmidt et al (see record 1986-31441-001) study of nonsupervisors, this research tested structural models hypothesizing relationships among supervisory experience, cognitive ability, supervisory knowledge and proficiency, and performance ratings, using a sample of 570 2nd-tour soldiers. The Schmidt et al model with an additional ability?→?experience path provided the best fit. The significant ability?→?experience path was interpreted as indicating that demonstrated ability contributes to soldiers being given the opportunity to obtain supervisory experience. Experience had a greater impact on supervisor proficiency than on supervisor knowledge. Ability had a greater impact on supervisor knowledge than on proficiency. Discussion focuses on the personal characteristics that might be involved in being assigned supervisory responsibilities. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Three approaches to the investigation of subgroup bias in performance measurement: Review, results, and conclusions.

Three methods of assessing subgroup bias in performance measurement commonly found in the literature are identified. After a review of these approaches, findings are reported from analyses of data collected in the US Army's Project A (J. P. Campbell, 1987). Correlations between nonrating performance measures and supervisor ratings were generally not moderated by race, but correlations between nonrating indicators of negative performance and ratings assigned by peers were. In addition, significant interactions between rater and ratee race on performance ratings were not eliminated when variance in the nonrating measures was removed from the ratings provided by Black and White raters. Conclusions about the magnitude and nature of bias in supervisor and peer ratings are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Examination of race and sex effects on performance ratings.

The effects of rater source, rater and ratee race, rater and ratee sex, and job type were investigated on ratings collected for 8,642 first-term Army enlisted personnel. Ratings were made on 10 behaviorally based dimensions developed for evaluating all first-term soldiers. Results of between-Ss analyses similar to those conducted in past research revealed significant main effects and interactions for sex, race, rater source, and job type, but the variance accounted for by these effects was minimal. Repeated measures analyses were also performed, with each ratee evaluated by one Black and one White rater for the race effects analyses and one female and one male rater for the sex effects analyses. These analyses, which unconfounded rater bias and actual performance differences, yielded results similar to those obtained with the between-Ss design. Implications of the findings are discussed. ? (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Examined the effects of knowledge of a ratee's prior performance on evaluations of present performance. Subjects received knowledge of either good or poor prior performances and then viewed and rated a videotape depicting average performance. In Study 1, some subjects received knowledge of the ratee's prior performance by directly viewing videotapes of good or poor ratee behavior, whereas others only reviewed written performance ratings completed by those subjects who had actually viewed the ratee. A contrast effect occurred when knowledge of prior performance was obtained by observing ratee behavior, but an assimilation effect occurred when knowledge of prior performance was obtained by reviewing performance ratings. In Study 2, subjects viewed videotapes of good or poor performances prior to viewing an average performance by the same ratee. However, the separate ratee performances were observed over a more realistic time interval than that used in Study 1 (3 weeks vs. 1 h). No significant contrast effects were observed. In Study 3, subjects reviewed written ratings of prior performances before viewing an average videotape. Subjects who reviewed extremely good (or poor) prior performance ratings provided more extreme ratings of the "average" performance than did subjects who reviewed less extreme ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding the latent structure of job performance ratings.

This study quantified the effects of 5 factors postulated to influence performance ratings: the ratee's general level of performance, the ratee's performance on a specific dimension, the rater's idiosyncratic rating tendencies, the rater's organizational perspective, and random measurement error. Two large data sets, consisting of managers (n?=?2,350 and n?=?2,142) who received developmental ratings on 3 performance dimensions from 7 raters (2 bosses, 2 peers, 2 subordinates, and self) were used. Results indicated, that idiosyncratic rater effects (62% and 53%) accounted for over half of the rating variance in both data sets. The combined effects of general and dimensional ratee performance (21% and 25%) were less than half the size of the idiosyncratic rater effects. Small perspective-related effects were found in boss and subordinate ratings but not in peer ratings. Average random error effects in the 2 data sets were 11% and 18%. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Nonperformance influences on performance evaluations: A laboratory phenomenon?

Reviews research that investigated the effects of nonperformance factors (i.e., gender and race) on a variety of organizational criteria, including performance evaluations. It is argued that previous findings are attributable to a research design that bears little resemblance to the performance appraisal process in real organizational contexts. 134 Black and 417 White male candidates for a police-department promotion were rated on a battery of attitude and behavior measures by 3 of the 14 Black and 18 White interviewers to examine the effects of 2 nonperformance factors (ratee and rater race) and an index of ratee past performance on performance ratings. Results of a higher-order MANOVA showed significant effects of ratee race, past performance, rater race, and a Ratee?×?Rater interaction. All of these sources of variance combined, however, accounted for no more than 4% of the total variance in performance ratings. Reasons for the low relationship between past performance and oral interview performance, which involve dissimilarity between rating dimensions and interview demand characteristics, are discussed. Thus, the applicability of results from past laboratory studies to performance evaluation in real organizational environments is questioned. (45 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interpersonal facilitation and job dedication as separate facets of contextual performance.

This study attempts to refine the construct of contextual performance by dividing it into 2 narrower constructs, interpersonal facilitation and job dedication. Supervisors rated 975 U.S. Air Force mechanics on at least 1 of 4 aspects of job performance (different supervisors rated each aspect of performance), and 515 of these mechanics also completed self-report individual difference measures. Correlations between performance ratings and individual difference variables support distinguishing task performance from interpersonal facilitation but not from job dedication. Thus this study suggests the need to redefine task performance to include motivational elements of job dedication. Then task performance would include task proficiency and motivation to perform one's own tasks effectively, and contextual performance would include interpersonal skills, the motivation to maintain good working relationships and help others perform their tasks. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance.

Based on data from 4 independent studies reported by R. Vineberg and E. N. Taylor (1972) with a total sample size of 1,474, path analysis was used to examine the causal impact of job experience on job knowledge, performance capability as measured by job sample tests, and supervisory ratings of job performance. Findings support the conclusion that (1) when mean job experience is 2–3 yrs, there is substantial variance in job experience and (2) when the jobs are of an intermediate complexity level, job experience has a substantial direct impact on job knowledge and a smaller direct impact on performance capabilities as assessed by job sample measures. Job experience also has a substantial indirect effect on work sample performance through its effect on job knowledge, which, in turn, was found to be the strongest determinant of work sample performance. The pattern and magnitude of causal effects of general mental ability were similar to those of job experience. The effect of job knowledge on supervisory ratings was several times stronger than the effect of job sample performance, confirming the findings of J. E. Hunter (1983). When job experience was held constant, the direct impact of ability on the acquisition of job knowledge increased substantially, and this, in turn, increased the indirect effect of ability on job sample performance. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Variability across sources of performance ratings: Further evidence.

33 1st-yr graduate students (aged 21–55 yrs) completed a simulated counseling interview. Self-, peer, and supervisor ratings were obtained for the simulated interviews using 7 measures of counseling performance. A multiple analysis of variance (MANOVA) indicated statistically significant differences among self-, peer, and supervisor ratings. Analyses also showed significant differences on 4 of the 7 dependent measures. Implications for training are discussed. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Psychological, family, and peer predictors of academic outcomes in first- through fifth-grade children.

This study tested a model that posited that 3 diverse sets of academic outcomes (memory, verbal, and nonverbal aptitudes [N?=?521]; passing proficiency tests in reading and mathematics [N?=?122]; and end-of-year ratings of verbal and mathematical skills by teachers [N?=?159]) for 1st- through 5th-grade children were uniquely determined by psychological (verbal and visual-motor "school readiness" skills at kindergarten, cognitive self-control, and academic self-competence), family (behavioral involvement of an adult family member in the child's schooling), and peer (the average academic performance of members of one's peer group) factors. Verbal readiness skills were uniquely linked to 5 of the 7 academic outcomes. An outcome-specific view of what sets of factors are linked to academic performance was favored in that, of the 5 predictors, only school readiness accounted for unique portions of the variance in aptitudes; only school readiness and cognitive self-control accounted for unique portions of the variance in whether proficiency tests were passed; and only school readiness, cognitive self-control, and the academic performance of one's peer group accounted for unique portions of the variance in end-of-year ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Can a manager have a life and a career? International and multisource perspectives on work-life balance and career advancement potential.

The present study was the first cross-national examination of whether managers who were perceived to be high in work-life balance were expected to be more or less likely to advance in their careers than were less balanced, more work-focused managers. Using self ratings, peer ratings, and supervisor ratings of 9,627 managers in 33 countries, the authors examined within-source and multisource relationships with multilevel analyses. The authors generally found that managers who were rated higher in work-life balance were rated higher in career advancement potential than were managers who were rated lower in work-life balance. However, national gender egalitarianism, measured with Project GLOBE scores, moderated relationships based on supervisor and self ratings, with stronger positive relationships in low egalitarian cultures. The authors also found 3-way interactions of work-life balance ratings, ratee gender, and gender egalitarianism in multisource analyses in which self balance ratings predicted supervisor and peer ratings of advancement potential. Work-life balance ratings were positively related to advancement potential ratings for women in high egalitarian cultures and men in low gender egalitarian cultures, but relationships were nonsignificant for men in high egalitarian cultures and women in low egalitarian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Role of ability and prior knowledge in complex training performance.

A causal model of the role of general cognitive ability and prior job knowledge in subsequent job-knowledge acquisition and work-sample performance during training was developed. Participants were 3,428 US Air Force officers in pilot training. The measures of ability and prior job knowledge came from the Air Force Officer Qualifying Test. The measures of job knowledge acquired during training were derived from classroom grades. Work-sample measures came from check flight ratings. The causal model showed that ability directly influenced the acquisition of job knowledge. General cognitive ability influenced work samples through job knowledge. Prior job knowledge had almost no influence on subsequent job knowledge but directly influenced the early work sample. Early training job knowledge influenced subsequent job knowledge and work-sample performance. Finally, early work-sample performance strongly influenced subsequent work-sample performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correlates of perceived fairness and accuracy of performance evaluation.

Employee perceptions of the fairness and accuracy of a performance evaluation system were examined by means of a questionnaire administered to all exempt managerial and professional employees of a large manufacturing organization. Primary (N?=?355) and hold-out (N?=?356) samples were identified, and a forward stepwise multiple regression analysis was performed on the primary sample. Cross-validation indicated that a 5-variable linear composite accounted for 29% of the variance in the dependent variable. Frequency of evaluation, identification of goals to eliminate weaknesses, and supervisor knowledge of a subordinate's level of performance and job duties were significantly related to perceptions of fairness and accuracy of performance evaluation. (3 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behaviorally anchored rating scales: Effects of education and job experience of raters and ratees.

Studied the effects of 2 levels of rater and ratee experience and education, as well as their possible interaction, on behaviorally anchored rating scales. A total of 370 male police personnel participated, of whom 71 were sergeants and 299 were police officers. Eight dependent variables, each a 9-point behaviorally anchored rating scale describing 1 dimension of police officer performance, were subjected to fixed-effects, unweighted-means analyses of variance. Results indicate that raters' experience and raters' education accounted for most of the statistically significant effects. Likewise Raters' Experience?×?Education and Raters' Education?×?Ratees' Education interactions were statistically significant. All significant effects were weak, however, as indicated by overlaps of 82–92% between distributions, and eta-squares for all significant F ratios of .01–.03. Hence, neither rater nor ratee characteristics exerted any practically significant effects on observed behaviorally anchored ratings. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe.

The interrater reliabilities of ratings of 9,975 ratees from 79 organizations were examined as a function of length of exposure to the ratee. It was found that there was a strong, nonlinear relationship between months of exposure and interrater reliability. The correlation between a logarithmic transformation of months of experience and reliability was .73 for one type of ratings and .65 for another type. The relationship was strongest during the first 12 months on the job. Changes in reliability were accounted for mostly by changes in criterion variance. Asymptotic levels of reliability were only about .60, even with 10–20 yrs of experience. Implications for estimating reliabilities in individual and meta-analytic studies and for performance appraisal were presented, and possible explanations of the reliability–variance relationship were advanced. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Parameters in the prediction of police officer performance.

Examined the relationship between police officer's performance on psychological assessment instruments and on-the-job performance. 65 prospective police officers (mean age slightly over 27 yrs) who were eventually selected for work in either an inner-city police department, a major university police department, or a community college police department were evaluated initially using a battery of psychological tests (the MMPI, the Eysenck Personality Inventory, the FIRO-B, the Shipley-Institute of Living Scale for Measuring Intellectual Impairment, the SCL-90, and the Bender-Gestalt). Ss were seen for reevaluation every 2 yrs. Performance criteria were derived from a 22-item supervisor's rating scale, measures of technical ability, and 7 performance criteria. Information from personnel records was also obtained. Analysis showed that community-college-based Ss earned higher ratings for both interpersonal and technical ability, and differences as a function of personality patterns emerged that cut across departments. The 5 female Ss had lower supervisory ratings of interpersonal ability. Positive attributes were influenced by Ss' departmental assignment, with inner-city Ss seeking more schooling and receiving more commendations than Ss in the other groups. Ss' in-service behavior was associated with patterns and elevation of their MMPI profiles and interpersonal needs. Overall results show a high amount of association between psychological variables and criterion behaviors. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of applicant sample on ratings of valid information in an employment setting.

Studied the effect of the previous job applicant upon ratings of test results for the following applicant. 2 informational aspects of the previous job applicant were varied: overall level of information favorability, and order of information within the previous job applicant. Interviewers were life insurance agency managers who through classroom instruction had been trained in the interpretation of test results. Results indicate that 55% of the criterion variance was accounted for by the test results while the favorability of the previous job applicant accounted for 4%. Thus, although the quality of the previous job applicant had a statistically significant effect on the evaluation of valid test results, its effect on the employment decision was negligible. By implication, the results also points to the necessity of building models on additional criteria than statistical significance. (25 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Job knowledge, ratee familiarity, conceptual similarity and halo error: An exploration.

The results of numerous social perception studies have led researchers to conclude that raters' implicit cognitive schemata regarding trait and behavior covariance may play a crucial role in the rating judgment process. W. H. Cooper (see PA, Vol 66:9176 and 9262) proposed one such cognitive schema, semantic conceptual similarity, as a key source of halo error in job performance ratings but was unable to reproduce the results of previous social perception research. The present study, with 186 undergraduates, employed baseball players as target ratees to examine the effects of job and ratee knowledge on the relations of raters' conceptual similarity schemata with rating and true score covariance. The results are consistent with the systematic distortion hypothesis presented by R. A. Shweder (see record 1976-07240-001). The association between conceptual similarity and rating covariance was significantly greater when Ss lacked sufficient job and/or ratee knowledge. Moreover, the degree of halo was also significantly greater when Ss lacked relevant job and ratee knowledge. The advantages of using objective measures of actual performance as true score estimates in the study of rater cognitive processes are discussed. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Organizational communication and performance.

Five independent objective measures of overall organization performance (i.e., number of clients served, cost of operation) were examined as correlates of 4 perceived communication variables (adequacy of information, information exchange, supervisor communication, and performance feedback) among 12 district organizations within a federally funded social services system. Ss included 457 employees (mean age 34 yrs) in 3 major job categories (administrators, service providers, and clerical workers), 60% of whom were women. Results indicate that 2 perceived communication variables, the quality of supervisory communication and information exchange within the peer workgroup, were strongly related to critical revenue and workload measures of overall organization performance. (17 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)