A procedure for combining sample correlation coefficients and vote counts to obtain an estimate and a confidence interval for the population correlation coefficient.

Missing effect-size estimates pose a particularly difficult problem in meta-analysis. Rather than discarding studies with missing effect-size estimates or setting missing effect-size estimates equal to 0, the meta-analyst can supplement effect-size procedures with vote-counting procedures if the studies report the direction of results or the statistical significance of results. By combining effect-size and vote-counting procedures, the meta-analyst can obtain a less biased estimate of the population effect size and a narrower confidence interval for the population effect size. This article describes 3 vote-counting procedures for estimating the population correlation coefficient in studies with missing sample correlations. Easy-to-use tables, based on equal sample sizes, are presented for the 3 procedures. More complicated vote-counting procedures also are given for unequal sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple correlation: Exact power and sample size calculations.

This article discusses power and sample size calculations for observational studies in which the values of the independent variables cannot be fixed in advance but are themselves outcomes of the study. It reviews the mathematical framework applicable when a multivariate normal distribution can be assumed and describes a method for calculating exact power and sample sizes using a series expansion for the distribution of the multiple correlation coefficient. A table of exact sample sizes for level .05 tests is provided. Approximations to the exact power are discussed, most notably those of J. Cohen (1977). A rigorous justification of Cohen's approximations is given. Comparisons with exact answers show that the approximations are quite accurate in many situations of practical interest. More extensive tables and a computer program for exact calculations can be obtained from the authors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sample Size for Multiple Regression: Obtaining Regression Coefficients That Are Accurate, Not Simply Significant.

An approach to sample size planning for multiple regression is presented that emphasizes accuracy in parameter estimation (AIPE). The AIPE approach yields precise estimates of population parameters by providing necessary sample sizes in order for the likely widths of confidence intervals to be sufficiently narrow. One AIPE method yields a sample size such that the expected width of the confidence interval around the standardized population regression coefficient is equal to the width specified. An enhanced formulation ensures, with some stipulated probability, that the width of the confidence interval will be no larger than the width specified. Issues involving standardized regression coefficients and random predictors are discussed, as are the philosophical differences between AIPE and the power analytic approaches to sample size planning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Comment on Sampling Error in the Standardized Mean Difference With Unequal Sample Sizes: Avoiding Potential Errors in Meta-Analytic and Primary Research.

The authors discuss potential confusion in conducting primary studies and meta-analyses on the basis of differences between groups. First, the authors show that a formula for the sampling error of the standardized mean difference (d) that is based on equal group sample sizes can produce substantially biased results if applied with markedly unequal group sizes. Second, the authors show that the same concerns are present when primary analyses or meta-analyses are conducted with point-biserial correlations, as the point-biserial correlation (r) is a transformation of d. Third, the authors examine the practice of correcting a point-biserial r for unequal sample sizes and note that such correction would also increase the sampling error of the corrected r. Correcting rs for unequal sample sizes, but using the standard formula for sampling error in uncorrected r, can result in bias. The authors offer a set of recommendations for conducting meta-analyses of group differences. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Comparison of five rules for determining the number of components to retain.

Investigated the performance of 5 methods for determining the number of components to retain—J. L. Horn's (see record 1965-13273-001) parallel analysis, W. F. Velicer's (see record 1977-00166-001) minimum average partial (MAP), R. B. Cattell's (see PA, Vol 41:969) scree test, M. S. Bartlett's (1950) chi-square test, and H. F. Kaiser's (see record 1960-06772-001) eigenvalue greater than 1 rule—across 7 systematically varied conditions (sample size, number of variables, number of components, component saturation, equal or unequal numbers of variables for each component, and the presence or absence of unique and complex variables). Five sample correlation matrices were generated at each of 2 sample sizes from the 48 known population correlation matrices representing 6 levels of component pattern complexity. Results indicate that the performance of the parallel analysis and MAP methods was generally the best across all situations; the scree test was generally accurate but variable; and Bartlett's chi-square test was less accurate and more variable than the scree test. Kaiser's method tended to severely overestimate the number of components. (65 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Simple formula aids for understanding the joint action of two predictors.

"It was demonstrated that the ratio of any two fixed validity coefficients will locate a value for the intercorrelation term in the two-predictor, multiple correlation formula such that the variable having the lower validity will contribute nothing to the variable having the higher validity. The resulting multiple correlation is thereby a minimum. A second formula is presented that determines two intercorrelation values for any fixed pair of validity coefficients which will result in a multiple correlation of 1.00. This second formula does not hold when both validity coefficients are exactly zero, and is restricted to determination of a single intercorrelation value when the two validity coefficients are exactly equal and not zero. From Psyc Abstracts 36:05:5AG85M. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Averaging correlation coefficients: Should Fisher's z transformation be used?

Averaging correlations leads to underestimation because the sampling distribution of the correlation coefficient is skewed. It is also known that if correlations are transformed by Fisher's z prior to averaging, the resulting average overestimates the population value of z. The behavior of these procedures for averaging correlations was investigated via Monte Carlo simulation, both in terms of bias (under- and overestimation) and precision (standard errors). It was found that average z backtransformed to r is less biased positively than average r is biased negatively. The standard error of average r was smaller than that of average z when the population correlation was small; however, the reverse was true when the population correlation exceeded .5. Regardless of sample size, back transformed average z was always less biased; therefore, the use of the z transformation is recommended when averaging correlation coefficients, particularly when sample size is small. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the sampling variance of intraclass correlations and genetic correlations

Widely used standard expressions for the sampling variance of intraclass correlations and genetic correlation coefficients were reviewed for small and large sample sizes. For the sampling variance of the intraclass correlation, it was shown by simulation that the commonly used expression, derived using a first-order Taylor series performs better than alternative expressions found in the literature, when the between-sire degrees of freedom were small. The expressions for the sampling variance of the genetic correlation are significantly biased for small sample sizes, in particular when the population values, or their estimates, are close to zero. It was shown, both analytically and by simulation, that this is because the estimate of the sampling variance becomes very large in these cases due to very small values of the denominator of the expressions. It was concluded, therefore, that for small samples, estimates of the heritabilities and genetic correlations should not be used in the expressions for the sampling variance of the genetic correlation. It was shown analytically that in cases where the population values of the heritabilities are known, using the estimated heritabilities rather than their true values to estimate the genetic correlation results in a lower sampling variance for the genetic correlation. Therefore, for large samples, estimates of heritabilities, and not their true values, should be used.     

Alternative procedures for testing regression slope homogeneity when group error variances are unequal.

The test for regression slope homogeneity across groups (e.g., sex, race, and treatments) is used in such varied settings as the analysis of covariance, the study of aptitude by treatment interactions, and bias detection in differential prediction research. The accuracy of this test requires the seldom-considered assumption of equality of within-group error variances. This research studies the effect of violating that assumption on the power of the F test for regression slope equality and finds that the test may be substantially affected when sample sizes are equal and severely affected when sample sizes are unequal. Alternative procedures based on R. A. Alexander's (see record 1994-39680-001) normalized-t approximation, G. S. James's (1951) second-order approximation, the Welch-Aspin approximation, and the chi-square test are described and evaluated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical study of Hedge's homogeneity test.

The behavior of the L. V. Hedges's (see record 1983-00213-001) Q test for the fixed-effects meta-analytic model was investigated for small and unequal study sample sizes paired with larger numbers of studies, nonnormal score distributions, and unequal variances. The results of a Monte Carlo study indicate that the hypothesis of equal effect sizes tends to be rejected less than expected if smaller study sample sizes are paired with larger numbers of studies; pairing smaller variances with larger sample sizes (or vice versa) leads to this hypothesis being rejected more than expected. The power of the Q test is also less than expected when small study sample sizes are paired with larger numbers of studies. These findings suggest conditions for which the Q test should be used cautiously. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of methods for analyzing binary data arising from two-sample twin studies

The study of twins is widely used for research into genetic and environmental influences on human outcome measurements. For the study design in which independent samples of monozygotic and dizygotic twins are compared with respect to their similarity on a binary trait, several statistical methods have been proposed. Using a Monte Carlo simulation, we compare the five following procedures: 1) goodness-of-fit method based on the common correlation model, 2) normal approximation of the maximum likelihood estimators of the common correlation coefficients, 3) Ramakrishnan et al. [(1992) Genet Epidemiol 9:273-282] method of odds ratio comparison, 4) generalized estimating equations method of odds ratio estimation, and 5) tetrachoric correlation method. The results show that the goodness-of-fit approach has similar or better performance in both type-one error rates and power than the other methods in all parameter settings. Its advantage with respect to type-one error rates is particularly clear under conditions of small sample sizes, extreme prevalences, or high values of the intraclass correlation coefficients. Therefore, the goodness-of-fit method is recommended for the two-sample twin study design.     

Bootstrapping correlation coefficients using univariate and bivariate sampling.

A new univariate sampling approach for bootstrapping correlation coefficients is proposed and evaluated. Bootstrapping correlations to define confidence intervals or to test hypotheses has previously relied on repeated bivariate sampling of observed (x,y) values to create an empirical sampling distribution. Bivariate sampling matches the logic of confidence interval construction, but hypothesis testing logic suggests that x and y should be sampled independently. This study uses Monte Carlo methods to compare the univariate bootstrap with 3 bivariate bootstrap procedures and with the traditional parametric procedure, using various sample sizes, population correlations, and population distributions. Results suggest that the univariate bootstrap is superior to other bootstrap procedures in many hypothesis testing settings, and even improves on parametric hypothesis testing in certain cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for regression homogeneity under variance heterogeneity.

One of the assumptions underlying the F test of parallelism of 2 or more regression lines is that the within-group residual variances are homogeneous. In the present study, a 2-group Monte Carlo investigation examined the effect of violating this assumption for F, a large-sample chi-square approximation (U?), and an approximate F test (F*). In terms of Type I error probabilities, the standard F test performed acceptably well as long as sample sizes were equal. This was not true when sample sizes were unequal, with F* proving to be clearly superior. The pattern of results parallel exactly what is known about the robustness of the F test when testing for mean differences in the presence of unequal variances. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of Brazelton demonstrations on later parenting: a meta-analysis

Meta-analysis of parenting interventions based on the Neonatal Behavioral Assessment Scale (NBAS) was conducted. Only published studies (n = 13) were included in this analysis, with one effect size entered for each study. The studies contained a total of 668 families, an average of about 51 per study. Effect sizes are reported in terms of the correlation coefficient (r) as well as the difference between experimental and control group means divided by the pooled standard deviation (Cohen's d). Analyses were conducted by weighting each study equally (unit weighting) and also by sample size. Similar average effect size were obtained for both weighting procedures (r's of about.2, d's of about.4), indicating that Brazelton-based interventions during the neonatal period have a small-moderate beneficial effect on the quality of later parenting. The probability of obtaining these findings by chance approached zero. The potential factors influencing these results are discussed, as well as directions for future research.     

What is the interpretation of the validity generalization estimate S2p=S2r-S2e?

The estimate S–2ρ is difficult to interpret; it is not a simple variance estimate. Its expectation under the hypothesis that all population correlation coefficients are equal is about zero, but it can be negative and depends on the unknown constant population correlation coefficient. The probability distribution of S–2ρ can have large negative probability mass even if its expectation is positive. Observed estimates of S–2, even if negative, cannot be taken as unqualified evidence supporting the validity generalization hypothesis because expectations under certain alternative hypotheses may have smaller expectations than under certain validity generalization hypotheses. It appears virtually impossible to provide a general hypothesis-testing framework that distinguishes between validity specificity and generalizability based on S–2. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Errata to Larzelere and Mulaik.

The article, "Single-Sample Tests for Many Correlations," by Robert E. Larzelere and Stanley A. Mulaik (Psychological Bulletin, 1977, Vol. 84, No. 3, pp. 557-569) contains the following errors: On page 558, the sentence that includes Equation 1 should read: "Assuming that the X and Y variables have a joint multivariate normal distribution, one then regards the multiple correlation as significantly different from zero if: (1)F=R2(N-p-1)/[(1-R2)p] is greater than Fα, the corresponding critical value of the F distribution with p and N-p-1 degrees of freedom at the 100(1-α) percentile level, with N the sample size." On page 559, the sentence that includes Equation 2 should read: "The population correlation p(W, Y) between W and Y is then regarded as significantly different from zero if: (2) F=rwy2(N-p-1)/[1-rwy2]p) is greater than the critical value of Fα of the F distribution with p and N-p-1 degrees of freedom at the 100(1-α) percentile level." On page 559, the sentence that reads, "If p(W, Y) is regarded as equal to zero, then any variable Xι with a nonzero weight wι in the linear combination W is also considered to have a zero correlation with Y," should be deleted. Thanks are due to Paul A. Games. (The following abstract originally appeared in record 1978-00149-001) When more than one correlation coefficient is tested for significance in a study, the probability of making at least one Type I error rises rapidly as the number of tests increases, and the probability of making a Type I error after a Type I error on a previous test is usually greater than the nominal significance level used in each test. To avoid excessive Type I errors with multiple tests of correlations, it is noted that researchers should use procedures that answer research questions with a single statistical test and/or should use special multiple-test procedures. A review of simultaneous-test and multiple-test procedures for correlations (e.g. Bartlett and Rajalakshman's test, multistage Bonferroni procedure, union-intersection tests, and the rank adjusted method) is presented, and several new procedures are described. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of error variance heterogeneity on the power of tests for regression slope differences.

In moderated regression analysis with both a continuous predictor and nominal-level (group membership) variables, there are conditions in which the hypothesis of equal slopes of the regression of Y onto X across groups is equivalent to the hypothesis of equality of X–Y correlations across groups. This research uses those conditions to investgate the impact of heterogeneity of error variance on the power accuracy of the F test for equality of regression slopes. The results show that even when sample sizes are equal, the test is not robust and, under unequal sample sizes, the pattern of excessively high or excessively low rejection rates can be severe. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Tests of significance in stepwise regression.

Tests of significance of the sample squared multiple correlation (R–2) in stepwise multiple regression have not been possible because its distribution is unknown. The present study used Monte Carlo simulation and least squares smoothing to construct tables of the upper 95th and 99th percentage points of the sample R–2 distribution in forward selection. A survey of published psychological research that used stepwise regression found a substantial inflation of reported significance levels when compared to the tabled values. Recommendations are given for use of these tables in evaluating results from forward selection and other stepwise methods. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study.

Validity generalization methods require accurate estimates of the sampling variance in the correlation coefficient when the range of variation in the data is restricted. This article presents the results of computer simulations examining the accuracy of the sampling variance estimator under sample range restrictions. Range restriction is assumed to occur by direct selection on the predictor. Sample sizes of 25, 60, and 100 are used, with the selection ratio ranging from .10 to 1.0 and the population correlation ranging from .10 to .90. The estimator is found to have a slight negative bias in unrestricted data. In restricted data, the bias is substantial in sample sizes of 60 or less. In all sample sizes, the negative bias increases as the selection ratio becomes smaller. Implications of the results for studies of validity generalization are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Application of GEE procedures for sample size calculations in repeated measures experiments

Derivation of the minimum sample size is an important consideration in an applied research effort. When the outcome is measured at a single time point, sample size procedures are well known and widely applied. The corresponding situation for longitudinal designs, however, is less well developed. In this paper, we adapt the generalized estimating equation (GEE) approach of Liang and Zeger to sample size calculations for discrete and continuous outcome variables. The non-central version of the Wald Chi 2 test is considered. We use the damped exponential family of correlation structures described in Mu?oz et al. for the 'working' correlation matrix among the repeated measures. We present a table of minimum sample sizes for binary outcomes, and discuss extensions that account for unequal allocation, staggered entry and loss to follow-up.