A note on validity generalization procedures.

Reviews validity generalization (VG) procedures and finds them to be subject to the logical fallacy of "affirming the consequent." It is argued that alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the VG approach. Moreover, some of the assumptions that form the statistical foundation of VG work are open to question. It is suggested that Fisher z transformations, which remove most of these problems and usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations, be used. It is also recommended that a more stringent criterion than the 75% decision rule for rejecting the situational specificity hypothesis be used. (64 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization.

The authors conducted Monte Carlo simulations to investigate whether indirect range restriction (IRR) on 2 variables X and Y increases the sampling error variability in the correlation coefficient between them. The manipulated parameters were (a) IRR on X and Y (i.e., direct restriction on a third variable Z), (b) population correlations ρxy, ρxz, and ρyz and (c) sample size. IRR increased the sampling error variance in rxy to values as high as 8.50% larger than the analytically derived expected values. Thus, in the presence of IRR, validity generalization users need to make theory-based decisions to ascertain whether the effects of IRR are artifactual or caused by situational-specific moderating effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bias in validity generalization variance estimates: A reply to Hoben Thomas.

In a recent article H. Thomas (see record 1989-10691-001) derived the expected value of the true validity variance estimate used in validity generalization studies. Based on computations of the expected values for certain scenarios, Thomas made a number of critical assertions regarding the variance estimate. This article shows that Thomas's arguments regarding deficiencies in the variance estimate used in the validity generalization studies are misleading. Contrary to Thomas's extremely negative assessment of the situation, there is no really convincing reason to doubt or abandon the estimates of true validity variance obtained in applied research from the Callender-Osborn and other closely related methods. Rather, there is strong evidence to indicate that populations of true validities with meaningful differences in mean and variance can be reliably distinguished, provided that a sufficient amount of base data are available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An unbiased correction for sampling error in validity generalization studies.

A correction for variance due to sampling error is part of all the statistical procedures used to study validity generalization. This correction is analogous to estimation of a variance component in a random effects analysis of variance. The correction usually has a large influence on the results of validity generalization analyses. James, Demaree, and Mulaik (1986) recently questioned the correction for sampling error, suggesting that several of the assumptions required for its derivation are questionable. In this article, I provide an alternative correction for sampling error that is shown to be (exactly) unbiased. A comparison of the calculations involved in the computation of the unbiased correction with the calculations involved in the calculation of the usual correction suggests that the 2 corrections will rarely differ substantially. Hence, the usual correction is not seriously biased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimation of sampling error variance in the meta-analysis of correlations: Use of average correlation in the homogeneous case.

The estimate of the population correlation used in the formula for sampling error variance of a correlation is typically the observed correlation, but in meta-analysis the average of the observed correlations can be used. For the case in which there is no variation in the study population correlations or sample sizes and the number of studies is very large, the authors found that use of the average correlation estimator is more accurate than use of the traditional, individual correlation estimator, except in those rare cases in which the uncorrected population correlation is greater than .60. For typical sample sizes, when the uncorrected population correlation is between -.40 and .40, there is virtually no error in the meta-analysis credibility interval based on the average correlation estimator. On the other hand, the amount of the error in the individual correlation estimator is qualitatively important if the sample is 25 or less and the population correlation is less than .40. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on variance explained in the mixed analysis of variance models.

Examines assumptions about the general linear model for interaction terms in the mixed analysis of variance. Some well-known results of S. R. Searle (1971) demonstrate that the inconsistencies between J. H. Dwyer's (see record 1975-02166-001) technique and that of G. M. Vaughn and M. C. Corballis (see record 1969-16617-001) in estimating the magnitude of effect for a mixed interaction are the direct result of specific assumptions made. If it is assumed that the interaction source of variance is a random variable, then the equations obtained by Vaughn and Corballis are correct; however, if an alternative assumption is made (i.e., that the iteraction term is fixed in one direction), then Dwyer's equations are correct. Researchers are called on to be cognizant of these two sets of assumptions and to be aware of the dramatic effects they may have on estimates of magnitude of effect for mixed interactions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Evaluated the statistical power of the Callender-Osburn method for testing the situational specificity hypothesis in validity generalization studies. The Schmidt-Hunter 75% rule for testing the situational specificity hypothesis was also studied with regard to its sensitivity for detecting both Type I and Type II errors. Results show that both the Callender-Osburn procedure and Schmidt-Hunter 75% rule lacked sufficient statistical power to detect low-to-moderate true validity variance when sample size was below 100. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A likelihood-based model for validity generalization.

The pair (ri, ni) is an observed correlation coefficient, ri, with sample size ni. Given s independent pairs, each arising from a setting in which the test and criterion measurement are bivariate normal, a mixture model may be used to make the following inferences about the unknown population parameters: (a) The number t of different population correlation coefficients, ρj, j?=?1,…, t, supported by the data, (b) point values of the ρj, (c) the proportion of ri associated with each ρj, and (d) the variance among the ρj all may be estimated from the s pairs. Maximum likelihood estimation equations are given. Approximate confidence intervals and tests may be constructed. Monte Carlo examples and a real-data example are provided. Procedures for correcting artifacts of range restriction and unreliability are briefly discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Perceptual generalization: A note on the peak shift.

Contends that a recent review of the literature on peak shift by R. B. Purtle (see record 1974-08409-001) fails to attend to the influence of underlying perceptual categories. The argument for such an influence (e.g., in wavelength generalization) is presented, and examples from several animal species are reported. (33 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of the relation between personality traits and leadership perceptions: An application of validity generalization procedures.

Reexamines, via meta-analysis, the relation between personality traits and leadership perceptions or extent of leader emergence, arguing that prior research on trait theories and leadership has been misinterpreted as applying to a leader's effect on performance when it actually pertains to the relation of leadership traits to leadership emergence. Further, based on current theories of social perceptions, several traits were expected to be strongly related to leadership perceptions. The meta-analytic technique of validity generalization was used with the 15 articles identified by R. D. Mann (see record 1960-04194-001) as investigating the relationship between personality traits and leadership. These studies were then pooled with 9 subsequent studies in an additional set of meta-analyses. Results support the expectation in that intelligence, masculinity–femininity, and dominance were significantly related to leadership perceptions. Findings show that variability across studies in the relation of these traits to leadership perceptions could be explained largely by methodological factors, indicating that contingency theories of leadership perceptions may not be needed. Both of these results contrast with the conclusions of earlier nonquantitative literature reviews on traits and leadership perceptions and with conventional thinking in the leadership area. (62 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiplicative validity generalization model: Accuracy of estimates as a function of sample size and mean, variance, and shape of distribution of true validities.

Used simulation techniques to study the accuracy of estimates of the mean, variance, and lower credibility value of true validities produced by the independent multiplicative model and by a modified dependent model that takes the correlation between range restriction and criterion reliability artifacts into account. Sample sizes (n?=?50 or 100) and the number of studies/analyses (50) were selected to be consistent with the typical parameters found in 129 validity generalization analyses. The mean, standard deviation, and shape of the true validity distributions were systematically varied. It is concluded that the independent and modified dependent estimates were typically accurate, although the credibility estimates were affected by the extremely skewed distributions. The mean and variance estimates from both models were not affected by distribution shape. Results support the applicability of the 1st 2 authors' (see record 1981-27033-001) models and F. L. Schmidt and J. E. Hunter's (see record 1978-11448-001) noninteractive model. Some limitations of the "bare-bones" sampling-error-only approach are noted. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the dynamic correlation coefficient.

Examined the use of the dynamic correlation coefficient as a test of spuriousness in longitudinal designs. It is shown that given conditions of spuriousness and perfect stationarity, the dynamic correlation coefficient was positively, rather then inversely, related to spuriousness. It is also shown that very high dynamic correlations were indicative of spuriousness and that low to moderate dynamic correlations may reflect a number of conditions. It is concluded that the dynamic correlation is, in many instances, a highly fallible indicator of spuriousness. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A cautionary note on the detection of method variance in multitrait-multimethod data.

Method variance in multitrait–multimethod (MTMM) data is often assessed by comparing the monomethod–heterotrait correlations to the heteromethod–heterotrait correlations in the MTMM matrix. Alternatively, method variance can be assessed within a confirmatory factor analysis of the MTMM data. It is shown that these two approaches are dissimilar theoretically and may give inconsistent assessments of method variance in practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on experimenter sampling in the graduate department of psychology.

A reasonable and economic solution to the cost of experimenter sampling and the problem of experimenter bias may be the formation of a pool of graduate students who are currently running either their own or their sponsor's research and who would equitably run a portion of each other's Ss. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparative meta-analysis of Rorschach and MMPI validity.

Two previous meta-analyses concluded that average validity coefficients for the Rorschach and the MMPI have similar magnitudes (L. Atkinson, 1986; K. C. H. Parker et al, see record 1989-14153-001), but methodological problems in both meta-analyses may have impeded acceptance of these results (H. N. Garb et al, see record 1998-11225-011). We conducted a new meta-analysis comparing criterion-related validity evidence for the Rorschach and the MMPI. The unweighted mean validity coefficients (r?s) were .30 for MMPI and .29 for Rorschach, and they were not reliably different (p = .76 under fixed-effects model, p = .89 under random-effects model). The MMPI had larger validity coefficients than the Rorschach for studies using psychiatric diagnoses and self-report measures as criterion variables, whereas the Rorschach had larger validity coefficients than the MMPI for studies using objective criterion variables. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bootstrap estimates of standard errors in validity generalization.

Bootstrapping is introduced as a method for approximating the standard errors of validity generalization (VG) estimates. A Monte Carlo study was conducted to evaluate the accuracy of bootstrap validity-distribution parameter estimates, bootstrap standard error estimates, and nonparametric bootstrap confidence intervals. In the simulation study the authors manipulated the sample sizes per correlation coefficient, the number of coefficients per VG analysis, and the variance of the distribution of true correlation coefficients. The results indicate that the standard error estimates produced by the bootstrapping procedure were very accurate. It is recommended that the bootstrap standard-error estimates and confidence intervals be used in the interpretation of the results of VG analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on concurrent and predictive validity designs: A critical reanalysis.

Comments on an article in which G. Barrett et al (see record 1981-11631-001) argued that the conceptual distinction between predictive and concurrent validity has been exaggerated. The present authors offer a more complete classification of validity designs and illustrate sources of differences in the quality of the validity estimates obtained. Five kinds of predictive designs for selection research are identified: (a) follow-up with random selection, (b) follow-up with selection by existing systems, (c) selection based on test to be validated, (d) selection preceding testing, and (e) shelf research. For some of these, and for corresponding concurrent designs, problems arise from the lack of data for informed estimates of appropriate population parameters. It is pointed out that severe range restriction intensifies parameter estimation and the risk of Type II error. Corrections suggested for the effects of contaminations such as age, tenure, and experience are shown to be oversimplifications. It is concluded that on both conceptual and practical bases different validity designs are not equivalent or interchangeable. (10 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the economic utility of generalized validity coefficients in personnel selection.

This study compared economic utility estimates that were based on noninteractive, interactive, independent multiplicative, and Taylor Series Approximation (TSA) 1 and 2 validity generalization results for clerical selection procedures at a large international manufacturing company. On the basis of estimates of the mean true validity and lower bound 90% credibility value, magnitude and percentage differences in resulting utility estimates across validity generalization procedures were relatively small for almost all comparisons. Regardless of the specific validity generalization parameter estimate used in estimating a utility value, the change in economic utility, going from the organization's current selection procedure (i.e., a verbal ability test) to an alternative procedure, was sizable in most cases. These results clearly demonstrate the practical similarity in utility terms of alternative validity generalization procedure results as well as the sizable economic value of minimum-level generalized validity coefficients. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The reliability of relationship satisfaction: A reliability generalization meta-analysis.

We conducted a reliability-generalization meta-analysis of 7 of the most frequently used measures of relationship satisfaction: the Locke–Wallace Marital Adjustment Test (LWMAT), the Kansas Marital Satisfaction Scale (KMS), the Quality of Marriage Index, the Relationship Assessment Scale, the Marital Opinion Questionnaire, Karney and Bradbury's (1997) semantic differential scale, and the Couples Satisfaction Index. Six hundred thirty-nine reliability coefficients from 398 articles and 636,806 individuals provided internal consistency reliability estimates for this meta-analysis. We present the average score reliabilities for each measure, characterize the variance in score reliabilities across studies, and consider sample and study characteristics that are predictive of score reliability. Overall, the KMS and the LWMAT appear to be the strongest and weakest measures, respectively, from a reliability perspective. We discuss the importance of considering reliability invariance when making cross-group comparisons and provide recommendations for researchers when electing a measure of relationship satisfaction. (PsycINFO Database Record (c) 2011 APA, all rights reserved)