Unbiased estimation of sampling variance of correlations.

In a recent article, P. E. Spector and E. L. Levine (1987) asserted that the estimate of sampling error variance used in validity generalization studies is biased when the number of correlations is relatively small. In addition, Spector and Levine maintained that the bias is such that the sampling error variance estimate seriously overestimates the actual variance of observed correlations. A partial replication of Spector and Levine's study showed that the alleged bias was due to a distributional artifact and that the sampling error estimate is not seriously biased. We review evidence from several Monte Carlo studies indicating that the sampling error estimate is quite accurate. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the sampling variance of intraclass correlations and genetic correlations

Widely used standard expressions for the sampling variance of intraclass correlations and genetic correlation coefficients were reviewed for small and large sample sizes. For the sampling variance of the intraclass correlation, it was shown by simulation that the commonly used expression, derived using a first-order Taylor series performs better than alternative expressions found in the literature, when the between-sire degrees of freedom were small. The expressions for the sampling variance of the genetic correlation are significantly biased for small sample sizes, in particular when the population values, or their estimates, are close to zero. It was shown, both analytically and by simulation, that this is because the estimate of the sampling variance becomes very large in these cases due to very small values of the denominator of the expressions. It was concluded, therefore, that for small samples, estimates of the heritabilities and genetic correlations should not be used in the expressions for the sampling variance of the genetic correlation. It was shown analytically that in cases where the population values of the heritabilities are known, using the estimated heritabilities rather than their true values to estimate the genetic correlation results in a lower sampling variance for the genetic correlation. Therefore, for large samples, estimates of heritabilities, and not their true values, should be used.     

A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization.

Compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and validity generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, Vr ?=?Vr/K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ρs are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Allowing for correlations between correlations in random-effects meta-analysis of correlation matrices.

Practical meta-analysis of correlation matrices generally ignores covariances (and hence correlations) between correlation estimates. The authors consider various methods for allowing for covariances, including generalized least squares, maximum marginal likelihood, and Bayesian approaches, illustrated using a 6-dimensional response in a series of psychological studies concerning prediction of exercise behavior change. Quantities of interest include the overall population mean correlation matrix, the contrast between the mean correlations, the predicted correlation matrix in a new study, and the conflict between the existing studies and a new correlation matrix. The authors conclude that accounting for correlations between correlations is unnecessary when interested in individual correlations but potentially important if concerned with a composite measure involving 2 or more correlations. A simulation study indicates the asymptotic normal assumption appears reasonable. Because of potential instability in the generalized least squares methods, they recommend a model-based approach, either the maximum marginal likelihood approach or a full Bayesian analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correction for variance restriction in point-biserial correlations.

A correction procedure is proposed for adjusting point-biserial correlations for attenuation produced by inopportune splits in the dichotomous variable. The correction procedure permits estimation of the point-biserial correlation that would have been seen had equal proportions been present. Monte Carlo simulation evidence is provided for the accuracy of the correction procedure. Also, an example is provided from the employee turnover literature to illustrate how the correction procedure may be used. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimation of the reliability of average of rankings.

A formula for estimating the average reliability of a set of rankings, based on n sets, each reduced to standard score form, is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of error management training: A meta-analysis.

Error management training (EMT) is a training method that involves active exploration as well as explicit encouragement for learners to make errors during training and to learn from them. Past evaluation studies, which compared skill-based training outcomes of EMT with those of proceduralized erroravoidant training or of exploratory training without error encouragement, have yielded considerable variation in effect sizes. The present meta-analysis compiles the results of the existing studies and seeks to explain this variation. Although the mean effect of EMT across all 24 identified studies (N = 2,183) was positive and significant (Cohen's d = 0.44), there were several moderators. Moderator analyses showed effect sizes to be larger (a) for posttraining transfer (d = 0.56) than for within-training performance and (b) for performance tasks that were structurally distinct (adaptive transfer; d = 0.80) than for tasks that were similar to training (analogical transfer). In addition, both active exploration and error encouragement were identified as effective elements in EMT. Results suggest that EMT may be better suited than error-avoidant training methods for promotion of transfer to novel tasks. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in attenuated correlation coefficients: A Monte Carlo study.

Research in validity generalization has generated renewed interest in the sampling error of the Pearson correlation coefficient. The standard estimator for the sampling variance of the correlation was derived under assumptions that do not consider the presence of measurement error or range restriction in the data. The accuracy of the estimator in attenuated or restricted data has not been studied. This article presented the results of computer simulations that examined the accuracy of the sampling variance estimator in data containing measurement error. Sample sizes of n?=?25, n?=?60, and n?=?100 are used, with the reliability ranging from .10 to 1.00, and the population correlation ranging from .10 to 0.90. Results demonstrated that the estimator has a slight negative bias, but may be sufficiently accurate for practical applications if the sample size is at least 60. In samples of this size, the presence of measurement error does not add greatly to the inaccuracy of the estimator. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Within-study meta-analysis: Pooling the significance of doubly nonindependent (nonoverlapping) correlations.

The Stouffer method of adding Zs is the most familiar and widely used method for pooling the significance levels of multiple hypothesis tests. However, this method requires the assumption under the null hypothesis that the outcomes of the tests are statistically independent. A method for pooling the significance of nonoverlapping, or "doubly nonindependent," correlations within a single study is presented in this article. Monte Carlo studies demonstrate that the method controls the Type I error rate even with small samples and does so better than (a) ignoring nonindependence and using Stouffer's method or (b) adjusting for nonindependence in the dependent variable set only using Strube's method (M. J. Strube, 1985). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study.

Validity generalization methods require accurate estimates of the sampling variance in the correlation coefficient when the range of variation in the data is restricted. This article presents the results of computer simulations examining the accuracy of the sampling variance estimator under sample range restrictions. Range restriction is assumed to occur by direct selection on the predictor. Sample sizes of 25, 60, and 100 are used, with the selection ratio ranging from .10 to 1.0 and the population correlation ranging from .10 to .90. The estimator is found to have a slight negative bias in unrestricted data. In restricted data, the bias is substantial in sample sizes of 60 or less. In all sample sizes, the negative bias increases as the selection ratio becomes smaller. Implications of the results for studies of validity generalization are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimation of cross-validated multiple correlation: A clarification.

Suggests that by the logic of their derivations, the formulas recently proposed by N. Schmitt et al (see record 1978-07042-001) for estimating cross-validated multiple correlation most closely approximate a measure of generalized predictive accuracy distinct from any traditional correlation statistic. Interpreted correlationally, these formulas have a negative bias that can become appreciable in parameter regions not sampled by their Monte Carlo tests. It is concluded that although less biased and more cogently derived alternatives are available, one of the formulas proposed by Schmitt et al works reasonably well in practice. (5 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under indirect range restriction: Implications for validity generalization.

The authors conducted Monte Carlo simulations to investigate whether indirect range restriction (IRR) on 2 variables X and Y increases the sampling error variability in the correlation coefficient between them. The manipulated parameters were (a) IRR on X and Y (i.e., direct restriction on a third variable Z), (b) population correlations ρxy, ρxz, and ρyz and (c) sample size. IRR increased the sampling error variance in rxy to values as high as 8.50% larger than the analytically derived expected values. Thus, in the presence of IRR, validity generalization users need to make theory-based decisions to ascertain whether the effects of IRR are artifactual or caused by situational-specific moderating effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An unbiased correction for sampling error in validity generalization studies.

A correction for variance due to sampling error is part of all the statistical procedures used to study validity generalization. This correction is analogous to estimation of a variance component in a random effects analysis of variance. The correction usually has a large influence on the results of validity generalization analyses. James, Demaree, and Mulaik (1986) recently questioned the correction for sampling error, suggesting that several of the assumptions required for its derivation are questionable. In this article, I provide an alternative correction for sampling error that is shown to be (exactly) unbiased. A comparison of the calculations involved in the computation of the unbiased correction with the calculations involved in the calculation of the usual correction suggests that the 2 corrections will rarely differ substantially. Hence, the usual correction is not seriously biased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimation of dominance variance in purebred Yorkshire swine

We used 179,485 Yorkshire reproductive and 239,354 Yorkshire growth records to estimate additive and dominance variances by Method Fraktur R. Estimates were obtained for number born alive (NBA), 21-d litter weight (LWT), days to 104.5 kg (DAYS), and backfat at 104.5 kg (BF). The single-trait models for NBA and LWT included the fixed effects of contemporary group and regression on inbreeding percentage and the random effects mate within contemporary group, animal permanent environment, animal additive, and parental dominance. The single-trait models for DAYS and BF included the fixed effects of contemporary group, sex, and regression on inbreeding percentage and the random effects litter of birth, dam permanent environment, animal additive, and parental dominance. Final estimates were obtained from six samples for each trait. Regression coefficients for 10% inbreeding were found to be -.23 for NBA, -.52 kg for LWT, 2.1 d for DAYS, and 0 mm for BF. Estimates of additive and dominance variances expressed as a percentage of phenotypic variances were, respectively, 8.8 +/- .5 and 2.2 +/- .7 for NBA, 8.1 +/- 1.1 and 6.3 +/- .9 for LWT, 33.2 +/- .4 and 10.3 +/- 1.5 for DAYS, and 43.6 +/- .9 and 4.8 +/- .7 for BF. The ratio of dominance to additive variances ranged from .78 to .11.     

Evidence for response bias as a source of error variance in applied assessment.

After 100 years of discussion, response bias remains a controversial topic in psychological measurement. The use of bias indicators in applied assessment is predicated on the assumptions that (a) response bias suppresses or moderates the criterion-related validity of substantive psychological indicators and (b) bias indicators are capable of detecting the presence of response bias. To test these assumptions, we reviewed literature comprising investigations in which bias indicators were evaluated as suppressors or moderators of the validity of other indicators. This review yielded only 41 studies across the contexts of personality assessment, workplace variables, emotional disorders, eligibility for disability, and forensic populations. In the first two contexts, there were enough studies to conclude that support for the use of bias indicators was weak. Evidence suggesting that random or careless responding may represent a biasing influence was noted, but this conclusion was based on a small set of studies. Several possible causes for failure to support the overall hypothesis were suggested, including poor validity of bias indicators, the extreme base rate of bias, and the adequacy of the criteria. In the other settings, the yield was too small to afford viable conclusions. Although the absence of a consensus could be used to justify continued use of bias indicators in such settings, false positives have their costs, including wasted effort and adverse impact. Despite many years of research, a sufficient justification for the use of bias indicators in applied settings remains elusive. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Statistical and empirical examination of the chi-square test for homogeneity of correlations in meta-analysis.

This article statistically and empirically examines the chi-square test of homogeneity of effect sizes when that test is applied to correlation coefficients and to their Fisher's r-to-z transformation equivalents. Shows that the test on untransformed rs has excessively high Type I error rates as a result of distribution characteristics of r but that the test performs nominally for Fisher's r-to-z transformation values. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimating correlations from epidemiological data in the presence of measurement error

Analysis of a major multi-site epidemiologic study of heart disease has required estimation of the pairwise correlation of several measurements across subpopulations. Because the measurements from each subpopulation were subject to sampling variability, the Pearson product moment estimator of these correlations produces biased estimates. This paper proposes a model that takes into account within and between sub-population variation, provides algorithms for obtaining maximum likelihood estimates of these correlations and discusses several approaches for obtaining interval estimates.     

Effect of error variance heterogeneity on the power of tests for regression slope differences.

In moderated regression analysis with both a continuous predictor and nominal-level (group membership) variables, there are conditions in which the hypothesis of equal slopes of the regression of Y onto X across groups is equivalent to the hypothesis of equality of X–Y correlations across groups. This research uses those conditions to investgate the impact of heterogeneity of error variance on the power accuracy of the F test for equality of regression slopes. The results show that even when sample sizes are equal, the test is not robust and, under unequal sample sizes, the pattern of excessively high or excessively low rejection rates can be severe. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Correcting effect sizes computed from factor analysis of variance for use in meta-analysis.

In meta-analysis, it is possible to average results across multiple studies because the effect sizes estimated from each study are in the same metric (e.g., the standardized mean difference). However, when effect sizes are computed from a factorial analysis of variance, these estimates are influenced by the other factors in the design. A correction developed by G. V. Glass, B. McGaw, and M. L. Smith (1981) solves this problem; however, it requires information (e.g., sums of squares) that is often not available in published research. A reformulated version of the correction is presented, which requires only F values and degrees of freedom. The impact of the correction on effect size estimates is examined. (PsycINFO Database Record (c) 2010 APA, all rights reserved)