Bias in validity generalization variance estimates: A reply to Hoben Thomas.

In a recent article H. Thomas (see record 1989-10691-001) derived the expected value of the true validity variance estimate used in validity generalization studies. Based on computations of the expected values for certain scenarios, Thomas made a number of critical assertions regarding the variance estimate. This article shows that Thomas's arguments regarding deficiencies in the variance estimate used in the validity generalization studies are misleading. Contrary to Thomas's extremely negative assessment of the situation, there is no really convincing reason to doubt or abandon the estimates of true validity variance obtained in applied research from the Callender-Osborn and other closely related methods. Rather, there is strong evidence to indicate that populations of true validities with meaningful differences in mean and variance can be reliably distinguished, provided that a sufficient amount of base data are available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Random assignment of available cases: Bootstrap standard errors and confidence intervals.

A frequently used experimental design in psychological research randomly divides a set of available cases, a local population, between 2 treatments and then applies an independent-samples t test to either test a hypothesis about or estimate a confidence interval (CI) for the population mean difference in treatment response. C. S. Reichardt and H. F. Gollob (1999) established that the t test can be conservative for this design--yielding hypothesis test P values that are too large or CIs that are too wide for the relevant local population. This article develops a less conservative approach to local population inference, one based on the logic of B. Efron's (1979) nonparametric bootstrap. The resulting randomization bootstrap is then compared with an established approach to local population inference, that based on randomization or permutation tests. Finally, the importance of local population inference is established by reference to the distinction between statistical and scientific inference. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A likelihood-based model for validity generalization.

The pair (ri, ni) is an observed correlation coefficient, ri, with sample size ni. Given s independent pairs, each arising from a setting in which the test and criterion measurement are bivariate normal, a mixture model may be used to make the following inferences about the unknown population parameters: (a) The number t of different population correlation coefficients, ρj, j?=?1,…, t, supported by the data, (b) point values of the ρj, (c) the proportion of ri associated with each ρj, and (d) the variance among the ρj all may be estimated from the s pairs. Maximum likelihood estimation equations are given. Approximate confidence intervals and tests may be constructed. Monte Carlo examples and a real-data example are provided. Procedures for correcting artifacts of range restriction and unreliability are briefly discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on validity generalization procedures.

Reviews validity generalization (VG) procedures and finds them to be subject to the logical fallacy of "affirming the consequent." It is argued that alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the VG approach. Moreover, some of the assumptions that form the statistical foundation of VG work are open to question. It is suggested that Fisher z transformations, which remove most of these problems and usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations, be used. It is also recommended that a more stringent criterion than the 75% decision rule for rejecting the situational specificity hypothesis be used. (64 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Two new procedures for studying validity generalization.

Several Monte Carlo studies examined the accuracy of 2 new procedures in estimating population true validity mean and variance. Results indicate that 1 of the new procedures provided slightly more accurate estimates than the procedures of F. L. Schmidt and J. E. Hunter (see record 1978-11448-001) and J. C. Callender and H. G. Osburn (see record 1981-00257-001). From a practical point of view, however, the estimates from the various procedures were quite comparable. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An unbiased correction for sampling error in validity generalization studies.

A correction for variance due to sampling error is part of all the statistical procedures used to study validity generalization. This correction is analogous to estimation of a variance component in a random effects analysis of variance. The correction usually has a large influence on the results of validity generalization analyses. James, Demaree, and Mulaik (1986) recently questioned the correction for sampling error, suggesting that several of the assumptions required for its derivation are questionable. In this article, I provide an alternative correction for sampling error that is shown to be (exactly) unbiased. A comparison of the calculations involved in the computation of the unbiased correction with the calculations involved in the calculation of the usual correction suggests that the 2 corrections will rarely differ substantially. Hence, the usual correction is not seriously biased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of ignoring random sire and dam effects on estimates and standard errors of breed comparisons

Data were weights of F1 calves and weaning weights of top-cross progeny from sires and maternal grandsires of 13 breeds. Three analyses were performed on each trait to obtain estimates and standard errors of breed effects needed to calculate across-breed EPD and accuracies. Model (R) for records of F1 progeny contained fixed effects for birth year and date of birth, sex, age and breed of dam, and breed of sire, and a random residual effect. The second analysis included random effects for sires (RS), and the third analysis included random effects for sires and dams (RSD). In maternal analysis of top-cross progeny, model (Rm) contained fixed effects for cycle of experiment, age of dam, year of birth, sex, breeds of maternal grandam and grandsire, and breed of sire, and a random residual effect. In addition, the second and third analyses fit random effects for maternal grandsires (RSm) and for maternal grandsires and daughters of maternal grandsires (RSDm). Estimates of breed of sire effects changed only slightly for different models. Total variance increased in RSD and RS relative to R. Standard errors of breed of sire comparisons were underestimated with Model R, compared to Models RS and RSD. Standard errors of other contrasts were generally not affected. Variance components, breed effects, and standard errors followed patterns for Rm, RSm, and RSDm similar to those for R, RS, and RSD. Ignoring random variation due to sires and dams underestimated standard errors of breed of sire comparisons.     

Development of a general solution to the problem of validity generalization.

Personnel psychologists have traditionally believed that employment test validities are situation specific. This study presents a Bayesian statistical model that is based on the alternate hypothesis that variation in validity outcomes from study to study for similar jobs and tests is artifactual in nature. Certain outcomes using this model permit validity generalization to new settings without carrying out a validation study of any kind. Where such generalization is not justified, the procedure is considered to provide an improved method of data analysis and decision making for the necessary situational validity study. Application to 4 distributions of empirical validity coefficients is presented to demonstrate the power of the model. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical comparison of the results of five validity generalization procedures.

With data originally obtained by the 3rd author and colleagues (see record 1980-31533-001), comparative results are presented for the noninteractive, interactive, independent multiplicative and Taylor Series Approximations 1 and 2 validity generalization procedures when there is nonzero sampling error. Findings indicate that the 5 validity generalization procedures yielded similar estimates of the fully corrected mean and variance of true validity coefficients. It is concluded that the 5 validity generalization procedures will lead to the same general conclusions regarding the effectiveness of a predictor measure. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Refinements in validity generalization methods: Implications for the situational specificity hypothesis.

Using a large database, this study examined 3 refinements of validity generalization procedures: (1) a more accurate procedure for correcting the residual standard deviation (SD) for range restriction to estimate SDp, (2) use of r? instead of study-observed rs in the formula for sampling error variance, and (3) removal of non-Pearson rs. The 1st procedure does not affect the amount of variance accounted for by artifacts. The addition of the 2nd and 3rd procedures increased the mean percentage of validity variance accounted for by artifacts from 70 to 82%, a 17% increase. The cumulative addition of all 3 procedures decreased the mean SDp estimate from .150 to .106, a 29% decrease. Six additional variance-producing artifacts were identified that could not be corrected for. In light of these it was concluded that the obtained estimates of mean SDp and mean validity variance accounted for were consistent with the hypothesis that the true mean SDp value is close to zero. These findings provide further evidence against the situational specificity hypothesis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Progress in validity generalization: Comments on Callender and Osburn and further developments.

Examined the simulation study of J. C. Callender and H. G. Osburn (see record 1981-27033-001) aimed at determining the relative accuracy of equations used in validity generalization (transportability) research to estimate the variance of true validities. Certain incorrect statements about equations derived and used by the present authors and their associates (1980, 1981) shown to be predicated on the mistaken impression that their derivational assumptions were the same as those of Callender and Osburn. The following conclusions are made: There is only 1 model for validity generalization, although there are different equations for estimating its parameters. There are now 3 accurate equations available for estimating the variance of true validities: All 3 equations are correctly derived and are of necessity approximations rather than perfect reflections of reality, although for different reasons. Validity generalization is such a robust phenomenon that it is possible based on only corrections for simple sampling error. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Large sample standard errors of kappa and weighted kappa.

2 statistics, kappa and weighted kappa, are available for measuring agreement between 2 raters on a nominal scale. Formulas for the standard errors of these 2 statistics are in error in the direction of overestimation, so that their use results in conservative significance tests and confidence intervals. Valid formulas for the approximate large-sample variances are given, and their calculation is illustrated using a numerical example. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of distribution of true validities.

Used simulation techniques to study the accuracy of estimates of the mean, variance, and lower credibility value of true validities produced by the independent multiplicative model and by a modified dependent model that takes the correlation between range restriction and criterion reliability artifacts into account. Sample sizes (n?=?50 or 100) and the number of studies/analyses (50) were selected to be consistent with the typical parameters found in 129 validity generalization analyses. The mean, standard deviation, and shape of the true validity distributions were systematically varied. It is concluded that the independent and modified dependent estimates were typically accurate, although the credibility estimates were affected by the extremely skewed distributions. The mean and variance estimates from both models were not affected by distribution shape. Results support the applicability of the 1st 2 authors' (see record 1981-27033-001) models and F. L. Schmidt and J. E. Hunter's (see record 1978-11448-001) noninteractive model. Some limitations of the "bare-bones" sampling-error-only approach are noted. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An investigation of the validity of expert true score estimates in appraisal research.

Researchers of appraisal rating accuracy have defined ratee true scores as the mean ratings given by experts provided with enhanced opportunities to observe performance. The external validity of accuracy research depends on the relevance of these expert estimates to true performance. In our study, we provided expert raters with enhanced opportunities to view videotapes of five ratees under conditions of high true dimension intercorrelation or low true intercorrelation. The accuracy of expert ratings was compared with the accuracy of nonexpert ratings of the same tapes viewed under more typical rating conditions. Subjects' ratings were compared with ratee true scores defined in terms of objective worker output. Results indicated that experts were more accurate than nonexperts, regardless of the true dimension intercorrelations. Accuracy indices computed by using objective true scores were highly correlated with indices computed by using the mean expert ratings as true score estimates. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The validity of violence risk estimates: An issue of item performance.

Typically, research conducted on the cross-validation or generalization of risk assessment schemes focuses on the aggregate score accuracy of the schemes within the new sample or population. Often overlooked when the schemes are examined in their aggregate form is the performance of the individual items. This study looks at the association between the items of the HCR-20 (C. D. Webster, K. S. Eaves, D. Douglas, & S. D. Wintrup, 1995) and the Violence Risk Appraisal Guide (VRAG; C. D. Webster, G. T. Harris, M. E. Rice, C. Cormier, & V. L. Quinsey, 1994) and violent recidivism in a sample of predominantly violent offenders. The results show that a number of the items from each scale do not distinguish between violent recidivists and nonrecidivists and that the presence of these items potentially reduces the predictive accuracy of the instruments. In addition, the inclusion of items that do not discriminate between recidivists and nonrecidivists potentially undermines the validity of the risk assessment process. Discussion centers on the application of prediction schemes and their individual risk factors in forensic practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization.

The authors conducted Monte Carlo simulations to investigate whether indirect range restriction (IRR) on 2 variables X and Y increases the sampling error variability in the correlation coefficient between them. The manipulated parameters were (a) IRR on X and Y (i.e., direct restriction on a third variable Z), (b) population correlations ρxy, ρxz, and ρyz and (c) sample size. IRR increased the sampling error variance in rxy to values as high as 8.50% larger than the analytically derived expected values. Thus, in the presence of IRR, validity generalization users need to make theory-based decisions to ascertain whether the effects of IRR are artifactual or caused by situational-specific moderating effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Another view of progress in validity generalization: Reply to Schmidt, Hunter, and Pearlman.

Refutes the F. L. Schmidt et al (see record 1983-07150-001) argument that the present authors' (see record 1981-27033-001) work on validity generalization models contained erroneous conclusions. Modifications to past practices in computing and reporting variance estimates are recommended. (3 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization.

Compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and validity generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, Vr ?=?Vr/K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ρs are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Evaluated the statistical power of the Callender-Osburn method for testing the situational specificity hypothesis in validity generalization studies. The Schmidt-Hunter 75% rule for testing the situational specificity hypothesis was also studied with regard to its sensitivity for detecting both Type I and Type II errors. Results show that both the Callender-Osburn procedure and Schmidt-Hunter 75% rule lacked sufficient statistical power to detect low-to-moderate true validity variance when sample size was below 100. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)