Testing the constancy of validity with computer-generated sampling distributions of the multiplicative model variance estimate: Results for petroleum industry validation research.

A comparison of computer-generated sampling distributions showed that increases in total N resulting from greater sample size/validity study reduced the sampling variation of multiplicative model estimates of true validity variance more than did increases in total N resulting from a larger number of studies. Validity coefficients, range restriction, and criterion reliability data for 5 tests predicting job performance and training performance were obtained from studies conducted at petroleum industry plants. Those tests included the Richardson-Bellows-Henry Ability, Arithmetic Reasoning, and Chemical Comprehension Tests; the Bennett Mechanical Comprehension Tests; and the Mathematics scale of the California Achievement Tests. Studies for 3 job categories (operation, maintenance, and laboratory) were combined for analysis to maximize sensitivity to job and plant differences. Sampling distributions of true validity variance estimates were generated for each test–criterion type combination. A comparison of actual variance estimates with critical values indicated that 3 of the 5 tests did not vary significantly in true validities for job performance or for training criteria. Results for 2 arithmetic reasoning tests support the existence of variation in true validities for job performance and for training criteria. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Personality scale validities increase throughout medical school.

Admissions and personnel decisions rely on stable predictor–criterion relationships. The authors studied the validity of Big Five personality factors and their facets for predicting academic performance in medical school across multiple years, investigating whether criterion-related validities change over time. In this longitudinal investigation, an entire European country’s 1997 cohort of medical students was studied throughout their medical school career (Year 1, N = 627; Year 7, N = 306). Over time, extraversion, openness, and conscientiousness factor and facet scale scores showed increases in operational validity for predicting grade point averages. Although there may not be any advantages to being open and extraverted for early academic performance, these traits gain importance for later academic performance when applied practice increasingly plays a part in the curriculum. Conscientiousness, perhaps more than any other personality trait, appears to be an increasing asset for medical students: Operational validities of conscientiousness increased from .18 to .45. In assessing the utility of personality measures, relying on early criteria might underestimate the predictive value of personality variables. Implications for personality measures to predict work performance are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Adding a dimension: Time as a factor in the generalizability of predictive relationships.

Trends in predictive validity coefficients across time or repeated performance assessments show significant trends. Commonly used predictors show decreasing validities for the prediction of temporally more remote performance assessments. Within-study corrections for statistical artifacts across performance assessments increased the negative slopes of the regressions of predictive validities on ordinal position of performance assessment. The average corrected validity decrement from initial to final performance assessment was –.60; the trimmed average decrement was –.45; the median decrement was –.29. The average within-study correlation between predictive validity and time of performance assessment was –.80. An analysis of stability coefficients of Time Period?×?Time Period performance assessments revealed similar findings. Several theoretical explanations are discussed as reasons for predictive validity decrements. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Test-taking dispositions: A missing link?

Test-taking dispositions and motivation have been proposed as moderators of employment test validity. In a simulated, multiorganization employment system, scores on the Test Attitude Survey were found to moderate the validities of both an ability and a personality test used to predict a performance criterion. The criterion-related validity of the personality test was found to be higher for a subsample with less positive test-taking motivation than for a subsample with higher test-taking motivation. An opposite effect was found among the subgroups for the ability test. Range restriction and the predictability of individuals with high and low test-taking dispositions were examined as explanations for the findings. Practice and negative feedback effects on test-taking motivation and dispositions were assessed. Implications for examining predictive vs concurrent validation designs are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Synthetic validity: A conceptual and comparative review.

Describes synthetic validity as the logical process of inferring test-battery validity from predetermined validities of the tests for basic work components. Two programmatic approaches to synthetic validity are discussed: the J-coefficient and the job component model. The J-coefficient is a mathematical index of the test and job-performance relationships that has been shown to yield results comparable to intertest and criterion correlation methods of assembling test batteries. The job component model associates mean test-scores with job components and is related to criterion-related validation while the J-coefficient approach is more related to content validation. Other approaches have used ratings as criteria for component validities, but they must be free of the usual rater biases to be useful. It is suggested that the most important aspect of synthetic validity is in making selection decisions. (55 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Policy capturing in the employment interview.

Employed an information-processing perspective to analyze the judgments of individual employment interviewers in a corporate setting. Linear policy-capturing equations were estimated from 3 interviewers' ratings of 120 job applicants in live and audiotaped interviews. The equations were evaluated across interviewers to identify sources of predictive validity and consistency in information use. In competition with the interviewers from whom they were derived, regression models displayed higher predictive validities in a majority of instances. Following training on selected rating dimensions, interviewers' predictive validities improved. After interviewer training, the regression models of the training dimensions yielded higher predictive validities than all 3 interviewers. Results suggest specific directions for enhancing the effectiveness of interviewing in the employee-selection process. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Meta-Analytic Study of General Mental Ability Validity for Different Occupations in the European Community.

A comprehensive meta-analysis of the validity of general mental ability (GMA) measures across 12 occupational categories in the European Community (EC) is presented. GMA measures showed that there is validity generalization and large operational validities for job performance and training success in 11 occupational groups. Results also showed that job complexity moderated the magnitude of the operational validity of GMA tests across three levels of job complexity: low, medium, and high. In general, results were similar to those found in the United States, although the European findings showed a slightly larger magnitude of operational validity in some cases. Theoretical and practical implications of these findings for personnel selection are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Positive manifold limits the relevance of content-matching strategies for validating selection test batteries.

Inferences about the relationships between scores on selection tests and measures of job performance are often made on the basis of an assessment of the match between the content of the test and the content of the job. However, there is little evidence that these test-to-job comparisons have any bearing on the criterion-related validity of selection tests. The authors show that conclusions reached in analyses of cognitive tests—that content matching is largely irrelevant to criterion-related validity—can be generalized to most sets of selection tests (e.g., psychomotor and performance tests, interview ratings, biodata scores, knowledge tests, work sample tests) that are positively correlated with one another and with the criterion. When the universe of potential predictors shows positive manifold, almost all possible sets of test batteries will yield similar outcomes and show similar validities, regardless of whether the content of these tests matches the content of the job. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Task differences as moderators of aptitude test validity in selection: A red herring.

Two studies, with a total sample size of 400,000 Ss and with the US Department of Labor's Dictionary of Occupational Job Titles (1977), examined the traditional belief that between-job task differences cause aptitude tests to be valid for some jobs but not for others. Results indicate that aptitude tests are valid across jobs, since the moderating effect of tasks(a) is negligible even when jobs differ grossly in task makeup and (b) is probably nonexistent when task differences are less extreme. Findings have implications for validity generalization, the use of task-oriented job analysis in selection research, criterion construction, moderator research, and proper interpretation of the US's Uniform Guidelines on Employee Selection Procedures. It is concluded that the belief that tasks are important moderators of test validities can be traced to behaviorist assumptions introduced into personnel psychology in the early 1960's and that, in retrospect, these assumptions are false. (44 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are you interested? A meta-analysis of relations between vocational interests and employee performance and turnover.

A common belief among researchers is that vocational interests have limited value for personnel selection. However, no comprehensive quantitative summaries of interests validity research have been conducted to substantiate claims for or against the use of interests. To help address this gap, we conducted a meta-analysis of relations between interests and employee performance and turnover using data from 74 studies and 141 independent samples. Overall validity estimates (corrected for measurement error in the criterion but not for range restriction) for single interest scales were .14 for job performance, .26 for training performance, –.19 for turnover intentions, and –.15 for actual turnover. Several factors appeared to moderate interest–criterion relations. For example, validity estimates were larger when interests were theoretically relevant to the work performed in the target job. The type of interest scale also moderated validity, such that corrected validities were larger for scales designed to assess interests relevant to a particular job or vocation (e.g., .23 for job performance) than for scales designed to assess a single, job-relevant realistic, investigative, artistic, social, enterprising, or conventional (i.e., RIASEC) interest (.10) or a basic interest (.11). Finally, validity estimates were largest when studies used multiple interests for prediction, either by using a single job or vocation focused scale (which tend to tap multiple interests) or by using a regression-weighted composite of several RIASEC or basic interest scales. Overall, the results suggest that vocational interests may hold more promise for predicting employee performance and turnover than researchers may have thought. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.

This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Moderating effects of decision-making/information-processing job dimensions on test validities.

Analyzed selection test data from the General Aptitude Test Battery and job dimension data from the Position Analysis Questionnaire for 111 diverse jobs. Test validities were statistically corrected for restriction of range for each job and correlated with each of 3 information-processing/decision-making (IP/DM) job dimensions and 2 manually oriented job dimensions to determine possible moderating effects of these dimensions. Results show that the IP/DM dimensions generally moderated the validities of the general intelligence test, the verbal ability test, and the numerical test, with positive correlations being demonstrated, and the validities of the finger and manual dexterity tests, with negative correlations demonstrated. The manually oriented job dimensions did not, however, reveal any significant moderating effects. Results are discussed in terms of their implications for the validity generalization model extended by F. L. Schmidt and J. E. Hunter (see record 1978-11448-001). (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Creativity tests and achievement in high school science.

This study investigated the relative validities of a battery of "creativity tests" and an IQ tests for predicting several indices of achievement in high school science. Criteria included grade-point average in science courses, percentile rank on the STEP Science Achievement Test, teacher rating of overall scientific potential, number of high school science courses taken, and a measure of involvement with science. Results indicated that the creativity tests did have considerable predictive validity against each criterion for each sex and that the criterion variance accounted for by the creativity tests is to a substantial degree independent of IQ. Contrary to findings of other investigators, teachers did not discriminate against highly creative pupils in their ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Subgroup differences in predictor and criterion variances and differential validity.

Investigated whether findings of differential validity could be explained by taking account of subgroup differences in predictor and/or criterion variances. The Thorndike equations for restriction in range were used to "correct" subgroup validities for both predictor and criterion for all race and sex subgroup data sets in which subgroup validities and standard deviations were available. The data indicate that corrections for differences in predictor variances had little effect on differences in subgroup validities. Equating criterion variances in those cases in which female validities exceeded male validities had the effect of virtually eliminating the difference in validity. It is suggested that findings of differential validity may be due to methodological problems associated with the criteria used, although in the data reported here corrections for differences in variances would increase the difference in male and female validities. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Role of social desirability in personality testing for personnel selection: The red herring.

Response bias continues to be the most frequently cited criticism of personality testing for personnel selection. The authors meta-analyzed the social desirability literature, examining whether social desirability functions as a predictor for a variety of criteria, as a suppressor, or as a mediator. Social desirability scales were found not to predict school success, task performance, counterproductive behaviors, and job performance. Correlations with the Big Five personality dimensions, cognitive ability, and years of education are presented along with empirical evidence that (a) social desirability is not as pervasive a problem as has been anticipated by industrial-organizational psychologists, (b) social desirability is in fact related to real individual differences in emotional stability and conscientiousness, and (c) social desirability does not function as a predictor, as a practically useful suppressor, or as a mediator variable for the criterion of job performance. Removing the effects of social desirability from the Big Five dimensions of personality leaves the criterion-related validity of personality constructs for predicting job performance intact. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Extension of the Schmidt and Hunter validity generalization procedure to the prediction of absenteeism behavior from knowledge of job satisfaction and organizational commitment.

Examined whether the weak relationship between job satisfaction and absenteeism could be explained by the statistical artifacts of I. L. Schmidt et al (see record 1978-11448-001) (criterion unreliability, predictor unreliability, range restriction, sampling error, computational and typographical errors, criterion contamination and deficiency, and slight differences in factor structures). Data were collected on job satisfaction and organizational commitment from 242 sales employees. Evidence of differences were found on variable means, standard deviations, reliabilities, and predictive validities. However, when 6 artifacts were controlled, little variability in validities remained. (32 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization results for two job groups in the petroleum industry.

Estimated the transituational generalizability (or "transportability") of the validities of 4 types of cognitive tests (Bennett Mechanical Comprehension Test, Richardson-Bellows-Henry Chemical Comprehension and RBH Arithmetic Reasoning Tests, and general intelligence) and a weighted biographical information blank for performance in 2 petroleum industry job groups. Generalizability was strongly supported for mechanical and chemical comprehension tests for both jobs. In the case of the chemical comprehension tests, virtually all variance of observed validity coefficients was accounted for by artifacts, and thus the hypothesis of situational specificity was rejected. Support for generalizability was substantial for general mental ability and arithmetic reasoning tests. It was found, however, that corrections for variance due to sampling error accounted for an average of 90% of all variance due to artifacts, indicating the relative unimportance of differences between sudies in criterion reliability and in range restriction in accounting for variation in observed validities. Generalizable multivariate validities were estimated for various test batteries using beta and unit weights. Finally, true score beta weights were used to estimate the causal role of the 4 cognitive abilities in job performance. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Joint relation of experience and ability with job performance: Test of three hypotheses.

Data from four different jobs (N?=?1,474) were used to evaluate three hypotheses of the joint relation of job experience and general mental ability to job performance as measured by (a) work sample measures, (b) job knowledge measures, and (c) supervisory ratings of job performance. The divergence hypothesis predicts an increasing difference and the convergence hypothesis predicts a decreasing difference in the job performance of high- and low-mental-ability employees as employees gain increasing experience on the job. The noninteractive hypothesis, by contrast, predicts that the performance difference will be constant over time. For all three measures of job performance, results supported the noninteractive hypothesis. Also, consistent with the noninteractive hypothesis, correlational analyses showed essentially constant validities for general mental ability (measured earlier) out to 5 years of experience on the job. In addition to their theoretical implications, these findings have an important practical implication: They indicate that the concerns that employment test validities may decrease over time, complicating estimates of selection utility, are probably unwarranted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The impact of corrections for faking on the validity of noncognitive measures in selection settings.

In selection research and practice, there have been many attempts to correct scores on noncognitive measures for applicants who may have faked their responses somehow. A related approach with more impact would be identifying and removing faking applicants from consideration for employment entirely, replacing them with high-scoring alternatives. The current study demonstrates that under typical conditions found in selection, even this latter approach has minimal impact on mean performance levels. Results indicate about .1 SD change in mean performance across a range of typical correlations between a faking measure and the criterion. Where trait scores were corrected only for suspected faking, and applicants not removed or replaced, the minimal impact the authors found on mean performance was reduced even further. By comparison, the impact of selection ratio and test validity is much larger across a range of realistic levels of selection ratios and validities. If selection researchers are interested only in maximizing predicted performance or validity, the use of faking measures to correct scores or remove applicants from further employment consideration will produce minimal effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Biographical data in employment selection: Can validities be made generalizable?

The hypothesis was examined that organizational specificity of biodata validity results from the methods typically used to select and key items. In this study, items were initially screened for job relevance, keying was based on large samples from multiple organizations, and items were retained only if they showed validity across organizations. Cross-validation was performed on approximately 11,000 first-line supervisors in 79 organizations. The resulting validities were meta-analyzed across organizations, age levels, sex, and levels of education, supervisory experience, and company tenure. In all cases, validities were generalizable. Validities were also stable across time and did not appear to stem from measurement of knowledge, skills, or abilities acquired through job experience. Finally, these results provide additional evidence against the hypothesis of situational specificity of validities, the first large-sample evidence in a noncognitive domain. (PsycINFO Database Record (c) 2010 APA, all rights reserved)