Comparative analysis of the reliability of job performance ratings.

This study used meta-analytic methods to compare the interrater and intrarater reliabilities of ratings of 10 dimensions of job performance used in the literature; ratings of overall job performance were also examined. There was mixed support for the notion that some dimensions are rated more reliably than others. Supervisory ratings appear to have higher interrater reliability than peer ratings. Consistent with H. R. Rothstein (1990), mean interrater reliability of supervisory ratings of overall job performance was found to be .52. In all cases, interrater reliability is lower than intrarater reliability, indicating that the inappropriate use of intrarater reliability estimates to correct for biases from measurement error leads to biased research results. These findings have important implications for both research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impact of job experience and ability on job knowledge, work sample performance, and supervisory ratings of job performance.

Based on data from 4 independent studies reported by R. Vineberg and E. N. Taylor (1972) with a total sample size of 1,474, path analysis was used to examine the causal impact of job experience on job knowledge, performance capability as measured by job sample tests, and supervisory ratings of job performance. Findings support the conclusion that (1) when mean job experience is 2–3 yrs, there is substantial variance in job experience and (2) when the jobs are of an intermediate complexity level, job experience has a substantial direct impact on job knowledge and a smaller direct impact on performance capabilities as assessed by job sample measures. Job experience also has a substantial indirect effect on work sample performance through its effect on job knowledge, which, in turn, was found to be the strongest determinant of work sample performance. The pattern and magnitude of causal effects of general mental ability were similar to those of job experience. The effect of job knowledge on supervisory ratings was several times stronger than the effect of job sample performance, confirming the findings of J. E. Hunter (1983). When job experience was held constant, the direct impact of ability on the acquisition of job knowledge increased substantially, and this, in turn, increased the indirect effect of ability on job sample performance. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Conceptual similarity as a source of illusory halo in job performance ratings.

Tested whether a possible source of difficulty in materially reducing illusory halo in job performance ratings is raters' beliefs that rating categories are conceptually similar and hence covary, thereby inflating observed correlation matrices. 11 graduate business administration students evaluated the conceptual similarities among job dimensions within 3 jobs. The previously observed interdimension correlation matrices were successfully predicted by Ss' mean conceptual similarity scores. When the observed correlation matrix obtained by W. C. Borman (see record 1980-26801-001) was compared with the normative true score matrix, the conceptual similarity scores were found to be inferior predictors of the observed correlation matrix compared with the superior predictive ability of the normative true score matrix. It is suggested that conceptual similarities among job dimensions represent one potentially recalcitrant source of illusory halo in performance ratings, particularly when ratings are based on encoded observations that have decayed in memory. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Convergent job expectations and ratings of industrial foremen.

A significant positive correlation was hypothesized between evaluations of foremen made by superiors and subordinates and the degree to which foremen share and accurately predict superior-subordinate expectations regarding the foreman's job behavior. The Ss, (8 superiors, 32 foremen, and 377 subordinates) responded to a questionnaire composed of "consideration" and "structure" items yielding "expectation-convergence scores" which were correlated with evaluation measures. In a department providing close face-to-face interaction, a significant proportion of the variance in superior evaluations was accounted for by the hypothesis of expectation convergence. Implications for common personnel practices are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Internal analyses of task ratings by job incumbents.

A total of 411 middle-level managers in three Civil Service occupational groups responded to 111 items of a task inventory using Time Spent and Difficulty rating scales. Factor and item analyses were used to form two homogeneous subsets of tasks labeled general administrative (75 items) and supervisory tasks (18 items). Analyses of variance conducted on each item indicated there were few differences between respondents of different demographic subgroups, although there were a sizable number of occupational differences. Sex subgroup differences were primarily due to fewer women than men reporting that they were involved in tasks of a budgetary or financial nature and tasks that involved speaking or interacting with large groups of people outside the organization. Ethnic group differences were due to non-White job incumbents more frequently reporting that they did not perform a task than did their White colleagues. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe.

The interrater reliabilities of ratings of 9,975 ratees from 79 organizations were examined as a function of length of exposure to the ratee. It was found that there was a strong, nonlinear relationship between months of exposure and interrater reliability. The correlation between a logarithmic transformation of months of experience and reliability was .73 for one type of ratings and .65 for another type. The relationship was strongest during the first 12 months on the job. Changes in reliability were accounted for mostly by changes in criterion variance. Asymptotic levels of reliability were only about .60, even with 10–20 yrs of experience. Implications for estimating reliabilities in individual and meta-analytic studies and for performance appraisal were presented, and possible explanations of the reliability–variance relationship were advanced. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of disability, gender, and job level on ratings of job applicants.

Objective: To examine the effects of applicant disability, gender, and job level on ratings of job applicants. Design and Participants: Full-time workers (n?=?88) and undergraduates (n?=?98) provided ratings of hypothetical job applicants who differed on the 3 factors of interest. Measures: Job applicants were evaluated on the basis of competence, overall recommendation, potency, activity, and starting salary. Results: Applicants with disabilities were generally rated significantly higher in activity and potency than the applicant without a disability. Additional analyses revealed a significant Gender X Job Level interaction for applicants with a disability. Conclusions: Consistent with D. T. Wegener and R. E. Petty's (1997) flexible correction model, the results of this study suggest that evaluations of job applicants with disabilities may depend on the amount of cognitive resources raters have available at the time the evaluations are made. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationship between organizational context and job analysis task ratings.

This research examined whether differences among job incumbents in their task ratings could be attributed to systematic as well as random variation. Data collected from 83 respondents showed that there was a significant degree of interrater agreement regarding task importance but not for ratings of task time spent. Moreover, measures of job context were significantly correlated with ratings of time spent but not task importance. These results suggest that within-job variation among raters can have both systematic and random components. Thus, practitioners should collect data on the job context as well as on incumbents' demographic characteristics and experience. Personnel practices based on task analyses may need to be examined to ensure that they are compatible with local variation in the job. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach.

A review of research on job performance suggests 3 broad components: task, citizenship, and counterproductive performance. This study examined the relative importance of each component to ratings of overall performance by using an experimental policy-capturing design. Managers in 5 jobs read hypothetical profiles describing employees' task, citizenship, and counterproductive performance and provided global ratings of performance. Within-subjects regression analyses indicated that the weights given to the 3 performance components varied across raters. Hierarchical cluster analyses indicated that raters' policies could be grouped into 3 homogeneous clusters: (a) task performance weighted highest, (b) counterproductive performance weighted highest, and (c) equal and large weights given to task and counterproductive performance. Hierarchical linear modeling indicated that demographic variables were not related to raters' weights. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationship of initiating structure and job performance as moderated by consideration.

Investigated the hypothesis that the effect of initiating structure on the rated productivity and quality of work-group performance in an industrial manufacturing plant is moderated by considerate leader behavior (rated and self-report). Ss were 133 foremen who were administered the leadership opinion questionnaire, and rated by at least 4 managers on 7-point likert scales. Results support the hypothesis when quality but not productivity was the criterion measure. Results are consistent with E. Fleishman and E. Harris's (see pa, vol. 37:3953) finding that grievances and turnover are less affected by initiation of structure when foremen are rated high on consideration. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of professional versus student ratings of job interviewee behavior.

Designed a replication of research by T. McGovern and H. Tinsley (1978) which asked professional personnel representatives to rate a videotaped job interviewee. Nonprofessional undergraduates were used as the raters of job interviewee behavior. Content of the interviews was constant; nonverbal behavior was experimentally manipulated. The results obtained with the student raters closely parallel those of professionals. The student Ss were able to discriminate nonverbal behaviors such as eye contact, voice tone, and body movement in making their evaluations. The implications for future research with students as Ss and the development of peer counseling programs for job interview preparation are discussed. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Variability across sources of performance ratings.

Examined the comparability of peer ratings, supervisor ratings, and self-ratings of counseling performance early in training. Earlier studies of the relationship of performance ratings from different sources have indicated some comparability across source of rating, particularly late in the training process. The results of the present study with 30 21–50 yr old graduate students enrolled in a prepracticum counseling methods course indicate considerable variability across sources of ratings used early in training. Implications and practical training strategies are discussed. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further investigation of common knowledge effects on job analysis ratings.

Previous research (Smith & Hakel, 1979) raised the possibility that the Position Analysis Questionnaire (PAQ) only captures common knowledge, or stereotypes, about jobs. Cornelius, DeNisi, and Blencoe (1984) presented data to refute this, but found that the number of PAQ items rated does not apply (DNA) was related to the agreement between naive raters and expert raters. The present study used data from 87 analysts and 24 jobs. Naive ratings were those obtained from analysts who had not studied the job, whereas expert ratings were those obtained from raters who had observed the job, interviewed incumbents, and written task statements describing the job. These ratings were then compared to target-score profiles obtained from PAQ services. Results confirmed earlier hypotheses that large numbers of DNA items artifactually inflate correlational estimates of agreement between expert and naive raters. In addition to this artifact, results also supported the view that the PAQ is less appropriate as a job analysis tool for some types of jobs. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Initial decisions and subsequent performance ratings.

Examined how information initially encountered for one decision was later used for evaluating ratee performance. 101 undergraduates viewed videotaped performances of 4 male carpenters performing 4 tasks. Half of the Ss were instructed to view the performance with the intent of rating the deservedness of each worker for outside contracting work; the other half were asked to designate the one best worker for this work. Two days later, 70 of the 101 Ss were asked to rate (from memory) each worker's performance. Findings indicate that raters initially designating one worker for a positive outcome rated all workers higher than raters making initial deservedness ratings. This elevation in ratings, which occurred for both subsequent overall evaluations and task ratings, may have reflected both the tendency to inflate ratings given the individual receiving the initial treatment and leniency toward the other ratees for whom representations of ability may not have been well established or remembered. Overall delayed performance ratings were influenced by these initial contextual factors. Results suggest that the pattern in which information is presented to raters and the nature of previous decisions may affect memory-based performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationships between job difficulty, employee's attitude toward his job, and supervisory ratings of the employee effectiveness.

Using correlational techniques, an investigation was made of the relationships between job difficulty (estimated by job evaluation factors), employee attitudes toward his job, and job environment, and supervisory ratings of employee performance. As job difficulty increased, employee attitudes were significantly more positive toward the job, management, and communication, and opportunity for advancement. Partial correlations showed that the relationship between job satisfaction and job difficulty increased when the effects of general morale were eliminated. Supervisory ratings of employee effectiveness were significantly rated (negatively) to employee salary and job tenure. Correlations between employee's attitude dimensions indicate increasing complexity of job content and increased content with people as a part of the job, are positively related to an employee's attitude toward his job. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Motivational determinants of job performance.

Studied V. Vroom's expectancy theory model in relation to work performance. 138 incentive workers employed in a steel fabricating plant participated in the study. The data indicate that the total expectancy model is a valid indicator of job performance, although the variances explained by the various parts of the theory were low. Several operational problems in defining the basic measures used in testing Vroom's theory are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)