A Comment on Sampling Error in the Standardized Mean Difference With Unequal Sample Sizes: Avoiding Potential Errors in Meta-Analytic and Primary Research.

The authors discuss potential confusion in conducting primary studies and meta-analyses on the basis of differences between groups. First, the authors show that a formula for the sampling error of the standardized mean difference (d) that is based on equal group sample sizes can produce substantially biased results if applied with markedly unequal group sizes. Second, the authors show that the same concerns are present when primary analyses or meta-analyses are conducted with point-biserial correlations, as the point-biserial correlation (r) is a transformation of d. Third, the authors examine the practice of correcting a point-biserial r for unequal sample sizes and note that such correction would also increase the sampling error of the corrected r. Correcting rs for unequal sample sizes, but using the standard formula for sampling error in uncorrected r, can result in bias. The authors offer a set of recommendations for conducting meta-analyses of group differences. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Estimation of sampling error variance in the meta-analysis of correlations: Use of average correlation in the homogeneous case.

The estimate of the population correlation used in the formula for sampling error variance of a correlation is typically the observed correlation, but in meta-analysis the average of the observed correlations can be used. For the case in which there is no variation in the study population correlations or sample sizes and the number of studies is very large, the authors found that use of the average correlation estimator is more accurate than use of the traditional, individual correlation estimator, except in those rare cases in which the uncorrected population correlation is greater than .60. For typical sample sizes, when the uncorrected population correlation is between -.40 and .40, there is virtually no error in the meta-analysis credibility interval based on the average correlation estimator. On the other hand, the amount of the error in the individual correlation estimator is qualitatively important if the sample is 25 or less and the population correlation is less than .40. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of Brazelton demonstrations on later parenting: a meta-analysis

Meta-analysis of parenting interventions based on the Neonatal Behavioral Assessment Scale (NBAS) was conducted. Only published studies (n = 13) were included in this analysis, with one effect size entered for each study. The studies contained a total of 668 families, an average of about 51 per study. Effect sizes are reported in terms of the correlation coefficient (r) as well as the difference between experimental and control group means divided by the pooled standard deviation (Cohen's d). Analyses were conducted by weighting each study equally (unit weighting) and also by sample size. Similar average effect size were obtained for both weighting procedures (r's of about.2, d's of about.4), indicating that Brazelton-based interventions during the neonatal period have a small-moderate beneficial effect on the quality of later parenting. The probability of obtaining these findings by chance approached zero. The potential factors influencing these results are discussed, as well as directions for future research.     

Bootstrap Standard Error and Confidence Intervals for the Correlation Corrected for Range Restriction: A Simulation Study.

The standard Pearson correlation coefficient is a biased estimator of the true population correlation, ρ, when the predictor and the criterion are range restricted. To correct the bias, the correlation corrected for range restriction, rc, has been recommended, and a standard formula based on asymptotic results for estimating its standard error is also available. In the present study, the bootstrap standard-error estimate is proposed as an alternative. Monte Carlo simulation studies involving both normal and nonnormal data were conducted to examine the empirical performance of the proposed procedure under different levels of ρ, selection ratio, sample size, and truncation types. Results indicated that, with normal data, the bootstrap standard-error estimate is more accurate than the traditional estimate, particularly with small sample size. With nonnormal data, performance of both estimates depends critically on the distribution type. Furthermore, the bootstrap bias-corrected and accelerated interval consistently provided the most accurate coverage probability for ρ. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessment of ecologic regression in the study of lung cancer and indoor radon

Ecologic regression studies conducted to assess the cancer risk of indoor radon to the general population are subject to methodological limitations, and they have given seemingly contradictory results. The authors use simulations to examine the effects of two major methodological problems that affect these studies: measurement error and misspecification of the risk model. In a simulation study of the effect of measurement error caused by the sampling process used to estimate radon exposure for a geographic unit, both the effect of radon and the standard error of the effect estimate were underestimated, with greater bias for smaller sample sizes. In another simulation study, which addressed the consequences of uncontrolled confounding by cigarette smoking, even small negative correlations between county geometric mean annual radon exposure and the proportion of smokers resulted in negative average estimates of the radon effect. A third study considered consequences of using simple linear ecologic models when the true underlying model relation between lung cancer and radon exposure is nonlinear. These examples quantify potential biases and demonstrate the limitations of estimating risks from ecologic studies of lung cancer and indoor radon.     

Clustering estimates of effect magnitude from independent studies.

One method of combining results of a series of studies is to calculate the average of the estimates of effect magnitude obtained from each study. The average estimate of effect magnitude may be misleading, however, when all studies do not share a common effect-magnitude parameter. When the effect-magnitude parameters (correlation coefficients or standardized mean differences) are heterogeneous across studies, it is often desirable to cluster studies into groups that are homogeneous with respect to the effect-size parameter. The present paper presents 2 procedures for clustering correlation coefficients and standardized mean differences when each estimator is based on the same number of observations. One procedure yields disjoint clusters and the other yields possibly overlapping clusters. In each case a method for determining the statistical significance level of the clusterings is given. Preliminary tests of homogeneity of a set of correlations or standardized mean differences are also given. The accuracy of the significance levels when estimators are based on different sample sizes is also studied. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for regression homogeneity under variance heterogeneity.

One of the assumptions underlying the F test of parallelism of 2 or more regression lines is that the within-group residual variances are homogeneous. In the present study, a 2-group Monte Carlo investigation examined the effect of violating this assumption for F, a large-sample chi-square approximation (U?), and an approximate F test (F*). In terms of Type I error probabilities, the standard F test performed acceptably well as long as sample sizes were equal. This was not true when sample sizes were unequal, with F* proving to be clearly superior. The pattern of results parallel exactly what is known about the robustness of the F test when testing for mean differences in the presence of unequal variances. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization.

An empirical Monte Carlo study with a large data base (N?=?84,808) was conducted to determine the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization (VG). This study has resulted in the following three major conclusions: (1) The correlation between sampling errors and population parameters appears to be close to zero; (2) when the sampling error is the only artifact, all three models do well in estimating the relevant parameters across three sample sizes and 10 different numbers of validity studies per VG study; and (3) when the predictor reliability, criterion reliability, and range restriction are also included as artifacts, the accuracy of estimation depends on how closely the hypothetical and true distributions of artifacts match. All three models performed inadequately when the match between the two sets of distributions was poor. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An analysis of correlations corrected for attenuation and range restriction.

An empirical attempt at demonstrating the bias in correlation coefficients that are corrected for both attenuation and range restriction was recently presented by R. Lee et al (see record 1983-02451-001). Using asymptotic methods, the present article analytically derives properties of the double-corrected correlation. It is shown that the double-corrected correlation is negatively biased. This negative bias decreases with increasing sample size and/or selection ratio. An expression for the standard error of the corrected correlation, useful for confidence interval estimation, is presented. Although the standard error of the corrected correlation is larger than that of the uncorrected correlation, the increase is proportionately smaller than the respective increase in the point estimate. Findings represent progress toward the request for full information about corrected correlations set forth in the Standards for Educational and Psychological Tests. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The predictive power of ridge regression: Some quasi-simulation results.

Reviews research indicating that ridge regression tends to improve the mean square error of prediction obtained with ordinary least squares (OLS) regression in a wide range of conditions. MMPI profiles of 861 psychiatric patients and diagnoses from 29 psychologists, as collected by P. E. Meehl (see record 1960-04396-001), were used to examine the gains in cross-validated multiple correlation obtained with ridge regression compared with OLS and equal weights as a function of sample size and ridge constant. A simple formula for estimating the ridge constant was also evaluated. Results that are related to recent developments concerning the use of Bayesian regression procedures show that ridge regression improves both the mean square error of prediction and the cross-validated multiple correlation obtained with OLS when the ratio sample size to number of predictor variables is relatively small. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bootstrap estimates of standard errors in validity generalization.

Bootstrapping is introduced as a method for approximating the standard errors of validity generalization (VG) estimates. A Monte Carlo study was conducted to evaluate the accuracy of bootstrap validity-distribution parameter estimates, bootstrap standard error estimates, and nonparametric bootstrap confidence intervals. In the simulation study the authors manipulated the sample sizes per correlation coefficient, the number of coefficients per VG analysis, and the variance of the distribution of true correlation coefficients. The results indicate that the standard error estimates produced by the bootstrapping procedure were very accurate. It is recommended that the bootstrap standard-error estimates and confidence intervals be used in the interpretation of the results of VG analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Which Behaviors Do Attitudes Predict? Meta-Analyzing the Effects of Social Pressure and Perceived Difficulty.

A meta-analysis of 797 studies and 1,001 effect sizes tested a theoretical hypothesis that situational constraints, such as perceived social pressure and perceived difficulty, weaken the relationship between attitudes and behavior. This hypothesis was confirmed for attitudes toward performing behaviors and for attitudes toward issues and social groups. Meta-analytic estimates of attitude-behavior correlations served to quantify these moderating effects. The present results indicated that the mean attitude-behavior correlation was .41 when people experienced a mean level of social pressure to perform a behavior of mean difficulty. The mean correlation was .30 when people experienced social pressure 1 standard deviation above the mean to perform a behavior that was 1 standard deviation more difficult than the mean. The results suggest a need for increased attention to the "behavior" side of the attitude-behavior equation. Attitudes predict some behaviors better than others. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical study of Hedge's homogeneity test.

The behavior of the L. V. Hedges's (see record 1983-00213-001) Q test for the fixed-effects meta-analytic model was investigated for small and unequal study sample sizes paired with larger numbers of studies, nonnormal score distributions, and unequal variances. The results of a Monte Carlo study indicate that the hypothesis of equal effect sizes tends to be rejected less than expected if smaller study sample sizes are paired with larger numbers of studies; pairing smaller variances with larger sample sizes (or vice versa) leads to this hypothesis being rejected more than expected. The power of the Q test is also less than expected when small study sample sizes are paired with larger numbers of studies. These findings suggest conditions for which the Q test should be used cautiously. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The predictive power of ridge regression: Some quasi-simulation results: Correction to Cattin.

Reports an error in the original article by Philippe Cattin (Journal of Applied Psychology, 1981, Vol. 66, No. 3, pp. 282-290). The fifth sentence in the first paragraph on page 284 contains an error. The sentence should read: "For intermediate values of k?, the ridge regression weights are 'weighted sums' of the OLS regression weights [not models] and of the zero-order sample correlations and tend to decrease in absolute value as k? increases." (The following abstract of this article originally appeared in record 1981-27117-001.) Reviews research indicating that ridge regression tends to improve the mean square error of prediction obtained with ordinary least squares (OLS) regression in a wide range of conditions. MMPI profiles of 861 psychiatric patients and diagnoses from 29 psychologists, as collected by P. E. Meehl (see record 1960-04396-001), were used to examine the gains in cross-validated multiple correlation obtained with ridge regression compared with OLS and equal weights as a function of sample size and ridge constant. A simple formula for estimating the ridge constant was also evaluated. Results that are related to recent developments concerning the use of Bayesian regression procedures show that ridge regression improves both the mean square error of prediction and the cross-validated multiple correlation obtained with OLS when the ratio sample size to number of predictor variables is relatively small. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Synthesis of pharmacokinetic parameters of vancomycin via bootstrap methods

OBJECTIVE: For the adjustment of individual vancomycin dosages, we estimate the important pharmacokinetic quantities half-life, clearance, and volume of distribution. MATERIAL: To obtain reliable information 293 observations from 244 patients were extracted from 23 published studies on vancomycin. Information about vancomycin's pharmacokinetics out of different sources represents an increase in sample size and, therefore, interpretive power. METHODS: Once the whole of the data had been stratified into a small number of homogeneous clusters based on cofactors, different (robust) estimators (mean, median, Winsorized, and trimmed mean) were calculated for the expected value of the pharmacokinetic parameters of vancomycin within the clusters. Measures of the statistical accuracy such as standard error, bias, mean square error, and confidence interval were estimated via bootstrap methods from large bootstrap sample sizes to compare the quality of the estimators. RESULTS: Due to the homogenization of the data all individual estimator functions yield very similar results and the empirical mean works fairly well as an estimate. The most frequently used estimator with the smallest estimated mean square error was the Winsorized mean.     

A more realistic look at the robustness and Type II error properties of the t test to departures from population normality.

The Type I and II error properties of the t test were evaluated by means of a Monte Carlo study that sampled 8 real distribution shapes identified by T. Micceri (1986, 1989) as being representative of types encountered in psychology and education research. Results showed the independent-samples t tests to be reasonably robust to Type I error when (1) sample sizes are equal, (2) sample sizes are fairly large, and (3) tests are 2-tailed rather than 1-tailed. Nonrobust results were obtained primarily under distributions with extreme skew. The t test was robust to Type II error under these nonnormal distributions, but researchers should not overlook robust nonparametric competitors that are often more powerful than the t test when its underlying assumptions are violated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sampling variance in the correlation coefficient under range restriction: A Monte Carlo study.

Validity generalization methods require accurate estimates of the sampling variance in the correlation coefficient when the range of variation in the data is restricted. This article presents the results of computer simulations examining the accuracy of the sampling variance estimator under sample range restrictions. Range restriction is assumed to occur by direct selection on the predictor. Sample sizes of 25, 60, and 100 are used, with the selection ratio ranging from .10 to 1.0 and the population correlation ranging from .10 to .90. The estimator is found to have a slight negative bias in unrestricted data. In restricted data, the bias is substantial in sample sizes of 60 or less. In all sample sizes, the negative bias increases as the selection ratio becomes smaller. Implications of the results for studies of validity generalization are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Simultaneous pairwise multiple comparison procedures for means when sample sizes are unequal.

Pairwise multiple comparison procedures (MCPs) are appropriate when the behavioral researcher is interested in comparing all possible pairwise mean differences. An exposition of the various simultaneous MCPs is presented that classifies those procedures as either (a) nonrobust to combinations of variance heterogeneity and unequal sample sizes because a pooled within-cell estimate of error variability is used to obtain the standard error of the contrast; or (b) robust to the homogeneity assumption because the standard error of the contrast is obtained via the Behrens-Fisher solution, and various approximate and/or conservative critical values that maintain the overall level of Type I error at "alpha" are used. A numerical example illustrating the latter MCPs is given. A choice among P. A. Games and J. F. Howell's (1976), C. W. Dunnett's (1980), and W. G. Cochran's (1964) procedures is recommended. (27 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Overall job satisfaction: How good are single-item measures?

A meta-analysis of single-item measures of overall job satisfaction (28 correlations from 17 studies with 7,682 people) found an average uncorrected correlation of .63 (SD?=?.09) with scale measures of overall job satisfaction. The overall mean correlation (corrected only for reliability) is .67 (SD?=?.08), and it is moderated by the type of measurement scale used. The mean corrected correlation for the best group of scale measures ( 8 correlations, 1,735 people) is .72 (SD?=?.05). The correction for attenuation formula was used to estimate the minimum level of reliability for a single-item measure. These estimates range from .45 to .69, depending on the assumptions made. (PsycINFO Database Record (c) 2010 APA, all rights reserved)