A comparison of economic utility estimates for alternative SDy estimation procedures.

Compared the per-selectee and total utility estimates for an assessment center that was used to select midlevel sales managers when 2 consensus-seeking procedures, the F. L. Schmidt et al (see record 1981-02231-001) procedure and 40% and 70% of mean salary, were used for estimating the standard deviation of job performance in dollars (SDy). Using the Schmidt et al procedure as the base, the overall utility estimates varied from 21 to 125% of the Schmidt et al values. For most cases, the different SDy estimation procedures produced somewhat similar utility estimates. The resulting estimated dollar gains from the use of the assessment center to select sales managers were substantial in all cases. The dollar gains from the current selection strategy were, at a maximum, only 63% of estimated top-down selection utility gains. Findings illustrate the value of economic utility analysis for evaluating human resource programs regardless of the type of SDy estimation procedure used. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Quantifying the effects of psychological interventions on employee job performance and work-force productivity.

Quantifying the economic impact of psychological programs in organizations requires determination of (a) the size and variability of the resulting increase in job performance and (b) the economic value of the increase in job performance. New methods of meta-analysis allow attainment of the 1st of these; and in relation to the 2nd, utility analysis methods provide the ability to translate job performance increases into estimates of the economic value of the program. In the area of personnel selection, many meta-analytic studies have resulted in precise and generalizable estimates of the validity of cognitive ability tests and other selection procedures. Utility analyses show that the job performance increases resulting from use of valid selection methods have substantial economic value. Valid selection produces major increases in work-force productivity. It is concluded that the combined effects of selection and nonselection interventions can be expected to produce substantial increases in workforce productivity. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Refinements in validity generalization methods: Implications for the situational specificity hypothesis.

Using a large database, this study examined 3 refinements of validity generalization procedures: (1) a more accurate procedure for correcting the residual standard deviation (SD) for range restriction to estimate SDp, (2) use of r? instead of study-observed rs in the formula for sampling error variance, and (3) removal of non-Pearson rs. The 1st procedure does not affect the amount of variance accounted for by artifacts. The addition of the 2nd and 3rd procedures increased the mean percentage of validity variance accounted for by artifacts from 70 to 82%, a 17% increase. The cumulative addition of all 3 procedures decreased the mean SDp estimate from .150 to .106, a 29% decrease. Six additional variance-producing artifacts were identified that could not be corrected for. In light of these it was concluded that the obtained estimates of mean SDp and mean validity variance accounted for were consistent with the hypothesis that the true mean SDp value is close to zero. These findings provide further evidence against the situational specificity hypothesis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical comparison of the results of five validity generalization procedures.

With data originally obtained by the 3rd author and colleagues (see record 1980-31533-001), comparative results are presented for the noninteractive, interactive, independent multiplicative and Taylor Series Approximations 1 and 2 validity generalization procedures when there is nonzero sampling error. Findings indicate that the 5 validity generalization procedures yielded similar estimates of the fully corrected mean and variance of true validity coefficients. It is concluded that the 5 validity generalization procedures will lead to the same general conclusions regarding the effectiveness of a predictor measure. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on validity generalization procedures.

Reviews validity generalization (VG) procedures and finds them to be subject to the logical fallacy of "affirming the consequent." It is argued that alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the VG approach. Moreover, some of the assumptions that form the statistical foundation of VG work are open to question. It is suggested that Fisher z transformations, which remove most of these problems and usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations, be used. It is also recommended that a more stringent criterion than the 75% decision rule for rejecting the situational specificity hypothesis be used. (64 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Obituaries: John E. Hunter (1939-2002).

Reports the death of John E. Hunter. His various contributions to the areas of validity generalization, differential validity/selection fairness, and selection utility are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Utility of the assessment center as a selection device.

Evaluated the utility of the assessment center by means of the Brogden-Cronbach-Gleser continuous variable utility model. After specifying several cost assumptions, 6 parameters were varied systematically: the validity and cost of the assessment center, the validity of the ordinary selection procedure, the selection ratio, the standard deviation of the criterion, and the number of assessment centers. The largest impacts on assessment center payoffs were exerted by the size of the criterion standard deviation, the selection ratio, and the difference in validity between the assessment center and the ordinary selection procedure. Even assessment centers with validities as low as .10 showed positive gains in utility over random selection. (39 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Empirical distributions of range restricted SDx in validity studies.

The Schmidt–Hunter validity generalization procedure (Schmidt & Hunter, 1977) uses a hypothetical distribution of range restricted standard deviations and the expected value obtained from that distribution. In this article we investigate 12 empirically derived distributions, using the General Aptitude Test Battery, Law School Admissions Test, and Scholastic Aptitude Test validity studies. These empirical distributions of range restricted standard deviations are compared with both the assumed distribution used in most validity generalization research and with those few other empirical distributions reported in the literature. The findings are discussed in terms of differences in setting and ways these empirical results could be used in validity generalization analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Two modified procedures for estimating standard deviations in utility analyses.

F. L. Schimdt et al (see record 1981-02231-001) developed a procedure for obtaining rational estimates of the standard deviation of job performance in dollars (SDy). A problem in using their procedure is that it can yield large variation in percentile point estimates across judges. Two modified procedures using consensual feedback were tested with 26 high-level managers to determine if the new procedures could reduce this variability. The normality assumption of the utility estimates yielded by the Schmidt et al and modified procedures was also investigated, and SDy estimates from the Schmidt et al and 2 modified procedures were compared to each other and to a performance and salary distribution. Results indicate that one of the modified procedures offers an improved practical means of obtaining rational SDy estimates in utility analyses. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings.

This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63). A further advantage of the latter 2 combinations is that they can be used for both entry level selection and selection of experienced employees. The practical utility implications of these summary findings are substantial. The implications of these research findings for the development of theories of job performance are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bootstrap estimates of standard errors in validity generalization.

Bootstrapping is introduced as a method for approximating the standard errors of validity generalization (VG) estimates. A Monte Carlo study was conducted to evaluate the accuracy of bootstrap validity-distribution parameter estimates, bootstrap standard error estimates, and nonparametric bootstrap confidence intervals. In the simulation study the authors manipulated the sample sizes per correlation coefficient, the number of coefficients per VG analysis, and the variance of the distribution of true correlation coefficients. The results indicate that the standard error estimates produced by the bootstrapping procedure were very accurate. It is recommended that the bootstrap standard-error estimates and confidence intervals be used in the interpretation of the results of VG analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study.

Validity generalization methods require accurate estimates of the sampling variance in the correlation coefficient when the range of variation in the data is restricted. This article presents the results of computer simulations examining the accuracy of the sampling variance estimator under sample range restrictions. Range restriction is assumed to occur by direct selection on the predictor. Sample sizes of 25, 60, and 100 are used, with the selection ratio ranging from .10 to 1.0 and the population correlation ranging from .10 to .90. The estimator is found to have a slight negative bias in unrestricted data. In restricted data, the bias is substantial in sample sizes of 60 or less. In all sample sizes, the negative bias increases as the selection ratio becomes smaller. Implications of the results for studies of validity generalization are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis, level of analysis, and best estimates of population correlations: Cautions for interpreting meta-analytic results in organizational behavior.

Meta-analytic procedures allow for determining best estimates of the individual-level, the within-organization, and the organizational-level population correlations. In most validity generalization work, meta-analytic procedures have been used to provide best estimates of the within-organization correlation. However, in many other organizational domains, researchers often do not clearly specify which population parameter is of interest. Further, researchers often combine studies in which data were collected at different levels of analysis or with mixed (single- and multiple-organization) sampling schemes, making it difficult to interpret unambiguously the meta-analytic ρ?. The authors focus on how to make appropriate inferences from meta-analytic studies by integrating a levels-of-analysis framework with meta-analytic techniques, highlighting how meta-analytic procedures can aid researchers in better understanding multilevel relationships among organizational constructs. The authors provide recommendations for clearer specifications of populations and levels issues in future meta-analytic studies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fairness of psychological tests: Implications of four definitions for selection utility and minority hiring.

Expected utility losses in moving from T. A. Cleary's (1968), R. L. Thorndike's (1971), and R. B. Darlington's (1971) Definition 3 selection fairness models to the quota model were assessed on an interval scale for various combinations of validity, minority base rate, and selection ratio. Expected changes in minority selection ratios across conditions were also determined. Utility losses were shown to be large enough to be of practical significance in many commonly occurring selection situations, although when considered as a percentage of maximum value, utilities remained quite high for the Thorndike and Darlington models. Increases in minority selection ratios across models were more striking than utility losses. Because no accepted method for converting minority selection ratios to utility units exists, and in light of the fact that the legal status of all 4 models is as yet unclear, it is concluded that each personnel researcher or organization must consider the trade-off between utility and the minority selection ratio subjectively and choose the model of selection fairness most consistent with his or its values. (18 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization and situational specificity: A second look at the 75% rule and Fisher's z transformation.

We analyzed the James, Demaree, and Mulaik (1986) critique of validity generalization, contending that the work (a) is not relevant to the real-world use of validity generalization in organizations, (b) has overlooked the bulk of the evidence against the situational specificity hypothesis, and (c) has confused the processes of hypothesis testing and parameter estimation in validity generalization and has made incorrect statements about the assumptions underlying both. In addition, the critique of the 75% rule is a statistical power argument; the procedures for use of confidence intervals are erroneous; there is no double correction of artifacts in validity generalization; the bias in the correlation (r) and the sampling error formula for r is well-known, trivial in magnitude, and has no empirical significance; and the use of the Fisher's z transformation of r in validity generalization studies and other meta-analyses creates an unnecessary inflationary bias in estimates of true validities and provides no benefits. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

What is the interpretation of the validity generalization estimate S2p=S2r-S2e?

The estimate S–2ρ is difficult to interpret; it is not a simple variance estimate. Its expectation under the hypothesis that all population correlation coefficients are equal is about zero, but it can be negative and depends on the unknown constant population correlation coefficient. The probability distribution of S–2ρ can have large negative probability mass even if its expectation is positive. Observed estimates of S–2, even if negative, cannot be taken as unqualified evidence supporting the validity generalization hypothesis because expectations under certain alternative hypotheses may have smaller expectations than under certain validity generalization hypotheses. It appears virtually impossible to provide a general hypothesis-testing framework that distinguishes between validity specificity and generalizability based on S–2. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impact of Consent Procedures on Reported Levels of Adolescent Alcohol Use.

This study examined sample bias related to parental consent procedures in school-based survey research on alcohol-related behavior. Alcohol prevalence and severity of use estimates from a high school student survey using traditional-active parental consent (N = 1,429) were compared with estimates from 2 student surveys using alternative parental consent procedures (passive consent; N = 2,210 and N = 3,231). Traditional-active parental consent procedures resulted in underrepresentation of lifetime drinkers. Furthermore, traditional-active parental consent procedures resulted in lower levels of high-risk drinking, and this bias was most evident for Caucasians and both boys and girls. Findings accentuate the need for administrators, policymakers, and researchers to consider the impact consent procedure related bias may have on results and interpretation of findings from school-based substance use research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An unbiased correction for sampling error in validity generalization studies.

A correction for variance due to sampling error is part of all the statistical procedures used to study validity generalization. This correction is analogous to estimation of a variance component in a random effects analysis of variance. The correction usually has a large influence on the results of validity generalization analyses. James, Demaree, and Mulaik (1986) recently questioned the correction for sampling error, suggesting that several of the assumptions required for its derivation are questionable. In this article, I provide an alternative correction for sampling error that is shown to be (exactly) unbiased. A comparison of the calculations involved in the computation of the unbiased correction with the calculations involved in the calculation of the usual correction suggests that the 2 corrections will rarely differ substantially. Hence, the usual correction is not seriously biased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impact of valid selection procedures on work-force productivity.

Used decision theoretic equations to estimate the impact of the Programmer Aptitude Test (PAT) on productivity if used to select new computer programers for 1 yr in the federal government and the national economy. A newly developed technique was used to estimate the standard deviation of the dollar value of employee job performance, which in the past has been the most difficult and expensive item of required information. For the federal government and the US economy separately, results are presented for different selection ratios and for different assumed values for the validity of previously used selection procedures. The impact of the PAT on programmer productivity was substantial for all combinations of assumptions. Results support the conclusion that hundreds of millions of dollars in increased productivity could be realized by increasing the validity of selection decisions in this occupation. Similarities between computer programers and other occupations are discussed. It is concluded that the impact of valid selection procedures on work-force productivity is considerably greater than most personnel psychologists have believed. (37 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A procedure for combining sample standardized mean differences and vote counts to estimate the population standardized mean difference in fixed event models.

Missing effect-size estimates pose a difficult problem in meta-analysis. Conventional procedures for dealing with this problem include discarding studies with missing estimates and imputing single values for missing estimates (e.g., 0, mean). An alternative procedure, which combines effect-size estimates and vote counts, is proposed for handling missing estimates. The combined estimator has several desirable features: (a) It uses all the information available from studies in a research synthesis, (b) it is consistent, (c) it is more efficient than other estimators, (d) it has known variance, and (e) it gives weight to all studies proportional to the Fisher information they provide. The combined procedure is the method of choice in a research synthesis when some studies do not provide enough information to compute effect-size estimates but do provide information about the direction or statistical significance of results. (PsycINFO Database Record (c) 2010 APA, all rights reserved)