Relationship of internal-external control to work motivation and performance in an expectancy model.

Conducted a study of 80 officer and 127 enlisted Naval personnel to investigate the hypotheses that internality (perception of much control) is positively related to (a) the concept of instrumentality from expectancy theory, (b) work motivation as measured by ratings and by expectancy theory measures, (c) job performance as measured by ratings, and (d) rank. These hypotheses were confirmed at or beyond the .05 significance level. A multitrait-multirater matrix showed convergent but no discriminant validity for ratings by supervisors, peers, and the Ss themselves on the Ss' job effort and performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty.

Meta-analysis was used to cumulate the correlations between supervisor and peer ratings for different dimensions of job performance. The general pattern in the data suggested that raters from the same organizational level disagree as much as raters from different levels. Methods were used to separate the effects of rating difficulty and lack of construct-level convergence on the correlation between supervisor and peer ratings. The authors found complete construct-level convergence for ratings of overall job performance, productivity, effort, job knowledge, quality, and leadership but not for ratings of administrative competence, interpersonal competence, and compliance or acceptance of authority. Higher rating difficulty was more strongly associated with lower mean observed peer-supervisor correlations than were construct-level disagreements between peers and supervisors. Implications for research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student, peer, and self evaluations of college instructors.

Compared student, peer (or colleague), and self ratings in terms of item statistics, convergent and discriminant validity, and relation to student learning. Ratings from the 3 sources (involving 263 students and 14 instructors) were similar in range and distribution, although colleagues tended to give the most favorable ratings, students the least favorable. Individual student and colleague reliabilities were also similar; composite student reliabilities were considerably higher than composite colleague reliabilities, only partly because of differing sample sizes. Student and self ratings and rankings were quite good in terms of convergent and discriminant validity, but no student, peer, or self rating was significantly related to residualized student achievement. (32 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multimethod factor analysis of behavioral and objective measures of psychopathology.

Evaluated the extent to which certain objective tests, ward behavior ratings, recorded verbalization scores, and interview behavior ratings possessed convergent and discriminant validity for 10 psychotic syndromes in a varied group of 125 psychotics drawn from 5 hospitals. A multimethod factor analysis was made of the 10 interview behavior and 12 ward behavior syndrome scores, and a similar analysis made of the entire set of 48 variables. The 1st analysis yielded 10 multimethod factors that corresponded closely to the original 10 interview syndromes. In the 2nd analysis, 12 interpretable factors were obtained, all of which exhibited convergent validity for 2 or 3 method sources. 1 or more objective test variables were meaningfully associated with each of the factors except 1. It is concluded that the convergent and discriminant validity of the syndromes is strongly supported. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluation of students' clinical competence.

In this article we describe a clinical proficiency review required of 3rd-year graduate students, and we present data from a study of its psychometric properties and utility. Students' oral and written case presentations were evaluated on seven dimensions by a six-member panel composed of faculty and students. Moderately high interjudge reliabilities were achieved, although the level of agreement between rater pairs was related to the level of similarity of their theoretical orientation. Significant correlations were obtained between panel ratings and those of current field placement supervisors, but not with supervisor's ratings from prior or subsequent placements. Performance on the clinical proficiency review was relatively independent of academic performance measures such as course grades and scores on a six-part comprehensive examination. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interpersonal facilitation and job dedication as separate facets of contextual performance.

This study attempts to refine the construct of contextual performance by dividing it into 2 narrower constructs, interpersonal facilitation and job dedication. Supervisors rated 975 U.S. Air Force mechanics on at least 1 of 4 aspects of job performance (different supervisors rated each aspect of performance), and 515 of these mechanics also completed self-report individual difference measures. Correlations between performance ratings and individual difference variables support distinguishing task performance from interpersonal facilitation but not from job dedication. Thus this study suggests the need to redefine task performance to include motivational elements of job dedication. Then task performance would include task proficiency and motivation to perform one's own tasks effectively, and contextual performance would include interpersonal skills, the motivation to maintain good working relationships and help others perform their tasks. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distinguishing contextual performance from task performance for managerial jobs.

The purpose of this study was to extend previous research on the contextual and task performance distinction to managerial jobs. It was hypothesized that, unlike results for nonmanagerial work, the job dedication facet of contextual performance would contribute uniquely to overall managerial performance. The interpersonal facilitation facet of contextual performance was expected to be redundant with leadership task performance and therefore not to make a unique contribution. A multitrait-multirater correlation matrix was developed on the basis of meta-analysis. Structural equation modeling results generally supported the hypotheses, although there was some evidence of a unique contribution by interpersonal facilitation. Results also suggested that peers paid more attention to interpersonal facilitation when making overall performance ratings, whereas supervisors paid more attention to task performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of using high- versus low-performing job incumbents as sources of job-analysis information.

We investigated the relation between incumbent performance level and job-analysis information using three different methods. Separate groups of high- and low-performing incumbents generated lists of tasks and of knowledges, skills, and abilities (KSAs) for the job of youth officer for a large metropolitan police force. These lists were virtually the same for all groups. In addition, those tasks and KSAs omitted by any of the groups were found to be unimportant in later ratings. Group differences in the ratings of the inventories on a number of scales as well as the ratings of the 18 cognitive Fleishman Ability Scales by the entire incumbent population (N?=?179) and their supervisors (N?=?9) were examined. The individual scales were factor analyzed, and discriminant analyses were applied to the factor scores to identify any differences in the ratings of high and low performers and supervisors. No differences were found. The limitations of this study are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Teenage Inventory of Social Skills: Development, reliability, and validity.

Describes the development and the evaluation of reliability and convergent and discriminant validity of a newly designed self-report questionnaire for the assessment of adolescent social competence: the Teenage Inventory of Social Skills (TISS). Two-week test–retest reliabilities for positive and negative behavior scales were .90 and .72; internal consistencies were .88. Convergent validity was assessed by comparing TISS scales with self-monitoring data, ratings by peers, and sociometric data. Discriminant validity was examined by investigating correlations between scores on the TISS and social desirability, SES, and another paper-and-pencil self-report instrument (Conflict Behavior Questionnaire) thought not to be necessarily related to adolescent social behavior. Results provided adequate evidence for both the convergent and discriminant validity of the TISS scales. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Influence of Self-Ratings versus Peer Ratings on Supervisors' Performance Judgments

This study investigated the extent to which supervisors use feedback from outside sources in making performance judgments. A simulation was conducted in which participants with organizational supervisory experience made an initial performance judgment about a profiled employee. Participants then received additional information that was discrepant from their initial judgment (positive or negative) from one of two sources (the profiled employee himself or one of his peers). The direction of the discrepant information and its source interacted in determining final ratings, such that, participants were more likely to use discrepant information to alter their performance judgments in a consistent direction when the source was a peer than when the source was the employee himself. Furthermore, participants' opinions about the usefulness of peer information for performance judgments moderated this interaction. Specifically, participants who believed that information from an employee's peers was useful, were more likely to use discrepant information provided by a peer when making final performance judgments than were participants who did not believe that information from an employee's peers was useful. Copyright 1998 Academic Press.     

The identification of traits related to managerial success.

To evaluate the effectiveness of a test battery in discriminating between successful and unsuccessful managers, a comprehensive battery was administered to 76 supervisors of an autonomous, medium-sized operating division. Results from the Guilford-Zimmerman Aptitude Survery, California Test of Mental Maturity, and Structured-Objective (S-O) Rorschach Test were related to ratings of overall performance, promotability, and versatility by peers and superiors. The test battery showed significant correlations with peer ratings in all three of these areas (.46-.68), but showed a significant correlation (.44) only with the supervisors' evaluation of versatility. The S-O Rorschach Test proved to be the most effective test in the battery. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparisons of self- and superior ratings of managerial performance.

Self- and superior ratings of performance on 9 dimensions were obtained from an interorganizational sample of 102 managers by a mail questionnaire. It was found that self-ratings possessed less leniency, restriction of range, and halo error than did superior ratings. The ratings only partially met the requirements of convergent and discriminant validity. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationship between measures of effort and job performance.

Effort expended and job performance are considered to be different, although not independent, constructs in the industrial environment. The relationship between these 2 variables was investigated using the multitrait-multimethod and multitrait-multirater approaches. 202 engineers completed self-ratings and were rated by their supervisors (n = 41), using global and dimensional rating methods. Convergent validity was found for the measures of effort and the measures of performance, but only the measures of performance demonstrated discriminant validity when compared with the measures of effort. Raters demonstrated convergent validity for each variable, but only some discriminant validity on the performance measures. Implications of the results are discussed in terms of appropriateness of the dimensional measure of effort. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development and validation of the Supervisory Styles Inventory.

Conducted a series of studies to develop and validate the Supervisory Styles Inventory (SSI) with parallel versions for supervisors and trainees. The instrument and validation procedures were based on a conceptual model of interrelated sources of variability among supervisors. Four separate analyses consistently revealed 3 factors among the perceptions of heterogeneous samples of trainees and experienced supervisors, which suggests that supervisory style is multidimensional. Scales constructed from these factors—Attractive, Interpersonally Sensitive, Task Oriented—demonstrated robust reliabilities and construct validity. In one study, SSI scales discriminated within and between expert supervisors with different theoretical orientations working with the same supervisee. These results were consistent with other evidence that a highly task-oriented style is endorsed by cognitive-behavioral supervisors, a highly interpersonal style by psychodynamic and humanistic supervisors. Furthermore, these supervisory styles seem to be differentially related to trainees' level of experience—supervisors are more task oriented with beginners, more attractive and interpersonally sensitive with interns. All 3 styles were strongly associated with trainees' reported willingness to work with different model supervisors and satisfaction with supervision. (39 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A new look at within-source interrater reliability of 360-degree feedback ratings.

This study investigated within-source interrater reliability of supervisor, peer, and subordinate feedback ratings made for managerial development. Raters provided 360-degree feedback ratings on a sample of 153 managers. Using generalizability theory, results indicated that little within-source agreement exists; a large portion of the error variance is attributable to the combined rater main effect and Rater X Ratee effect; more raters are needed than currently used to reach acceptable levels of reliability; supervisors are the most reliable with trivial differences between peers and subordinates when the numbers of raters and items are held constant; and peers are the most reliable, followed by subordinates, followed by supervisors, under conditions commonly encountered in practice. Implications for the validity, design, and maintenance of 360-degree feedback systems are discussed along with directions for future research in this area. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Situational specificity in assessment center ratings: A confirmatory factor analysis.

Assessment center ratings of eight abilities from each of five situational exercises were examined for their cross-situational consistency and discriminant validity. A series of confirmatory factor analyses revealed that the ratings were largely (if not totally) situation specific, and that assessors failed to distinguish among the eight target abilities. These results combined with previous research suggest that the assessment center method measures mainly situation-specific performance, not cross-situational managerial abilities. We suggest that the intended constructs might be better measured if more ability-related behaviors were elicited within each exercise and if the cognitive demands placed on assessors were reduced. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The five-factor model of personality and managerial performance: Validity gains through the use of 360 degree performance ratings.

This study investigated the usefulness of the five-factor model (FFM) of personality in predicting two aspects of managerial performance (task vs. contextual) assessed by utilizing the 360 degree performance rating system. The authors speculated that one reason for the low validity of the FFM might be the failure of single-source (e.g., supervisor) ratings to comprehensively capture the construct of managerial performance. The operational validity of personality was found to increase substantially (50%–74%) across all of the FFM personality traits when both peer and subordinate ratings were added to supervisor ratings according to the multitrait–multimethod approach. Furthermore, the authors responded to the recent calls to validate tests via a multivariate (e.g., multitrait–multimethod) approach by decomposing overall managerial performance into task and contextual performance criteria and by using multiple rating perspectives (sources). Overall, this study contributes to the evidence that personality may be even more useful in predicting managerial performance if the performance criteria are less deficient. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Specification of the criterion construct space: An application of hierarchical confirmatory factor analysis.

Used hierarchical confirmatory factor analysis to test alternative latent structures underlying performance in multiple dimensions as measured by work samples and 3 rating sources. The model receiving strongest support was one that specified multidimensional trait (performance dimension) and measurement method second-order factors (SOFs). This model exhibited significant convergent and discriminant validity, showed significant measurement method effects, and indicated that different measurement methods assess somewhat different aspects of the total criterion construct space. Significant correlations between measurement method SOFs and performance predictors suggest the need to reconsider the interpretation of method factors, not as bias, but as possibly representing different overall perspectives on performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multitrait (ADHD–IN, ADHD–HI, ODD toward adults, academic and social competence) by multisource (mothers and fathers) evaluation of the invariance and convergent/discriminant validity of the Child and Adolescent Disruptive Behavior Inventory with Thai adolescents.

Confirmatory factor analysis was used with a multitrait (attention-deficit/hyperactivity disorder—inattention, attention-deficit/hyperactivity disorder—hyperactivity/impulsivity, oppositional defiant disorder toward adults, academic competence, and social competence) by multisource (mothers and fathers) matrix to test the invariance and convergent/discriminant validity of the 5-factor model between mothers’ and fathers’ ratings of Thai adolescents (Year 1: n = 872; Year 2: n = 903; Year 3: n = 700; Year 4: n = 984) with the Child and Adolescent Disruptive Behavior Inventory (G. L. Burns, T. Taylor, & J. Rusby, 2001). The results showed equality of like-item loadings, intercepts, and residuals, as well as like-factor variances, covariances, and means between mothers’ and fathers’ ratings within each of the 4 yearly samples. In addition, the between-parent factor correlations showed convergent and discriminant validity with the within-parent factor correlations, showing discriminant validity for each year as well. These results for Thai adolescents and similar results (G. L. Burns et al., 2008) for mothers’ and fathers’ ratings of Brazilian, Thai, and American children provide broader support for the construct validity of the scale. The confirmatory factor analysis invariance and convergent/discriminant validity procedure with multiple sources is considered to provide a much more sophisticated procedure to evaluate the construct validity of attention-deficit/hyperactivity disorder and oppositional defiant disorder rating scales than a single-source approach. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of instructions to avoid halo error on reliability and validity of performance evaluation ratings.

Investigated the effects of a short training session designed to reduce halo error in performance ratings. 90 low and middle managers rated 1 of 6 hypothetical first-line supervisors on 6 performance dimensions according to behavior displayed in a prepared vignette. Ratings were taken prior to and following the 5-min training session, with rater-ratee combinations counterbalanced. The vignettes were developed to contain previously scaled behavior examples, thus enabling the calculation of "true" criterion scores for each dimension. Comparisons between these "true" criterion and the performance ratings revealed that the training session significantly reduced halo, while leaving validity of the ratings generally unaffected. Performance ratings completed after training possessed lower reliability, although raters provided somewhat more accurate performance profiles. (PsycINFO Database Record (c) 2010 APA, all rights reserved)