Is the Meta-Analysis of Correlation Coefficients Accurate When Population Correlations Vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of methods to test mediation and other intervening variable effects.

A Monte Carlo study compared 14 methods to test the statistical significance of the intervening variable effect. An intervening variable (mediator) transmits the effect of an independent variable to a dependent variable. The commonly used R. M. Baron and D. A. Kenny (1986) approach has low statistical power. Two methods based on the distribution of the product and 2 difference-in-coefficients methods have the most accurate Type I error rates and greatest statistical power except in 1 important case in which Type I error rates are too high. The best balance of Type I error and statistical power across all cases is the test of the joint significance of the two effects comprising the intervening variable effect. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Combining studies using effect sizes and quality scores: application to bone loss in postmenopausal women

This article presents a random effects model that uses effect sizes (ES) and quality scores to integrate results from investigations. An empirical example is given with data obtained from a meta-analysis on the effectiveness of physical activity in the prevention of bone loss in healthy postmenopausal women. A Medline search was performed to locate relevant studies published in French or English between January 1966 and May 1996. Three independent reviewers extracted data from studies. Effect sizes were calculated according to the method of Hedges and Olkin. A modified version of Chalmers' scale was utilized to calculate quality scores. DerSimonian and Laird's method with incorporation of the quality scores was used to estimate the overall effect size. Quality scores and the inverse of the variances were included as weights when combining studies. The overall estimate and standard error (SE) of the effect of physical activity on spinal bone mineral density loss in healthy postmenopausal women was ESoverall = 0.4263 (1.1361). When compared to other meta-analysis methods such as the fixed effects model and the model of DerSimonian and Laird without the quality score (DL), the new model generated comparable estimators (fixed effects model's ESoverall (SE) = 1.2724 (0.0139), DLs ESoverall (SE) = 0.3958 (1.2370)). Due to the heterogeneity that existed between studies, a random effects model was more appropriate then a fixed effects model. However, it resulted in wider confidence intervals, as expected. It was shown empirically that the model using quality scores generated narrower confidence intervals than the model of DL alone. The inclusion of covariates such as quality scores in meta-analyses permits the quantification of the variation between studies.     

Appropriate critical percentages for the Schmidt and Hunter meta-analysis procedure: Comparative evaluation of Type I error rate and power.

Recent investigations have shown that the Schmidt and Hunter 75% meta-analysis procedure (S&H–75%) does not have adequate control of the Type I error rate. This lack of control has caused two problems: First, the S&H–75% displays an erratic relationship between the likelihood to detect moderators and the number of studies included in the meta-analysis. Second, it has precluded meaningful power comparisons of the S&H–75% procedure with alternate procedures. In the present study we first determine appropriate critical percentages for the Schmidt and Hunter procedure that maintain a fixed Type I error rate. Then we compare this procedure, using the appropriate percentages, with an alternative statistic, U. When the correct percentages are used, the Schmidt and Hunter procedure shows equivalent power to the U statistic. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors.

AMonte Carlo study was conducted to determine Types I and II error rates of the Schmidt and Hunter (S&H) meta-analysis method and the U statistic for assessing homogeneity within a set of correlations. One thousand samples of correlations were generated randomly to fill each of 450 cells of an 18?×?5?×?5 (Underlying Population Correlations?×?Numbers of Correlations Compared?×?Sample Size Per Correlation) design. To assess Type I error rates, correlations were drawn from the same population. To assess power, correlations were drawn from two different populations. As compared with U, which was uniformly robust, the Type I error rate for the S&H method was unacceptably high in many cells, particularly when the criterion for determining homogeneity was set at a highly conservative level. Power for the S&H method increased with increasing size of population differences, sample size per correlation, and in some cases, number of correlations compared. The U statistic did more poorly in most conditions in protecting from Type II errors. (14 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The role of method in treatment effectiveness research: Evidence from meta-analysis.

A synthesis of 319 meta-analyses of psychological, behavioral, and educational treatment research was conducted to assess the influence of study method on observed effect sizes relative to that of substantive features of the interventions. An index was used to estimate the proportion of effect size variance associated with various study features. Study methods accounted for nearly as much variability in study outcomes as characteristics of the interventions. Type of research design and operationalization of the dependent variable were the method features associated with the largest proportion of variance. The variance as a result of sampling error was about as large as that associated with the features of the interventions studied. These results underscore the difficulty of detecting treatment outcomes, the importance of cautiously interpreting findings from a single study, and the importance of meta-analysis in summarizing results across studies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of Psychological Interventions for Child Maltreatment: A Meta-Analysis.

A meta-analysis tested the effectiveness of psychological treatments for child maltreatment (CM) using weighted least squares methods (e.g., L. V. Hedges & I. Olkin, 1985). A mean effect size of d+ = 0.54 (SE = .03, 95% CI = .39- .69) was observed, indicating that on average, treated participants were better off than 71% of those in control groups. Partitioning by type and target of outcome assessment yielded homogeneous effects within each of 5 different outcomes. Treatment effects were weakest when linked to objective behavioral observations of the family (d+ = .21) and strongest when associated with parent self-reported parenting attitudes and behaviors (d+ = .53). Results of other moderator analyses are presented, along with limitations of current CM treatment research; implications for future research, practice, and social policy are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Power and Measures of Effect Size in Analysis of Variance With Fixed Versus Random Nested Factors.

Ignoring a nested factor can influence the validity of statistical decisions about treatment effectiveness. Previous discussions have centered on consequences of ignoring nested factors versus treating them as random factors on Type I errors and measures of effect size (B. E. Wampold & R. C. Serlin, see record 2000-16737-003). The authors (a) discuss circumstances under which the treatment of nested provider effects as fixed as opposed to random is appropriate; (b) present 2 formulas for the correct estimation of effect sizes when nested factors are fixed; (c) present the results of Monte Carlo simulations of the consequences of treating providers as fixed versus random on effect size estimates, Type I error rates, and power; and (d) discuss implications of mistaken considerations of provider effects for the study of differential treatment effects in psychotherapy research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effect of mindfulness-based therapy on anxiety and depression: A meta-analytic review.

Objective: Although mindfulness-based therapy has become a popular treatment, little is known about its efficacy. Therefore, our objective was to conduct an effect size analysis of this popular intervention for anxiety and mood symptoms in clinical samples. Method: We conducted a literature search using PubMed, PsycINFO, the Cochrane Library, and manual searches. Our meta-analysis was based on 39 studies totaling 1,140 participants receiving mindfulness-based therapy for a range of conditions, including cancer, generalized anxiety disorder, depression, and other psychiatric or medical conditions. Results: Effect size estimates suggest that mindfulness-based therapy was moderately effective for improving anxiety (Hedges’s g = 0.63) and mood symptoms (Hedges’s g = 0.59) from pre- to posttreatment in the overall sample. In patients with anxiety and mood disorders, this intervention was associated with effect sizes (Hedges’s g) of 0.97 and 0.95 for improving anxiety and mood symptoms, respectively. These effect sizes were robust, were unrelated to publication year or number of treatment sessions, and were maintained over follow-up. Conclusions: These results suggest that mindfulness-based therapy is a promising intervention for treating anxiety and mood problems in clinical populations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of two random-effects methods of meta-analysis.

Two studies compared the Schmidt-Hunter method of meta-analysis (J. E. Hunter & F. L. Schmidt, 1990) with the method described by L. V. Hedges and J. L. Vevea (1998). Study 1 evaluated estimates of ρ?, ?ρ, and resulting credibility intervals for both models through Monte Carlo methods. Results showed slight differences between the 2 methods. In Study 2, a reanalysis of published meta-analyses using both methods with several artifact distributions showed that although both choice of technique and type of correction could matter, the technique of meta-analysis used is less influential on the study outcome than is the choice of artifact correction. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relation between Type A behavior and chronic emotional distress: A meta-analysis.

A meta-analysis was conducted of studies examining the relation between Type A behavior and chronic emotional distress as measured by standard psychological scales. Aggregating across all studies, the average effect size was .27, indicating a positive association between Type A and chronic dysphoria; however, there was considerable variability in the size of the relation among studies. Partitioning by Type A measure revealed that Structured Interview-assessed Type A was unrelated to chronic dysphoric emotions; however, most of the self-report measures of Type A behavior were moderately correlated with upset. The Framingham Type A Scale and the Bortner Scale showed the strongest relations. Thus, contrary to the traditional view, Type A measured by self-report does have some emotional concomitants, although they are not in the pathological range. Also discussed are how the results bear on the proposal that the maladjusted personality confers coronary risk, the implications for reported associations between Type A and illness complaints, and for the study of the Type A as a social psychological construct. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On seeking moderator variables in the meta-analysis of correlational data: A Monte Carlo investigation of statistical power and resistance to Type I error.

A series of Monte Carlo computer simulations was conducted to investigate (a) the likelihood that meta-analysis will detect true differences in effect sizes rather than attributing differences to methodological artifact and (b) the likelihood that meta-analysis will suggest the presence of moderator variables when in fact differences in effect sizes are due to methodological artifact. The simulations varied the magnitude of the true population differences between correlations, the number of studies included in the meta-analysis, and the average sample size. Simulations were run both correcting and not correcting for measurement error. The power of 3 indices—the Schmidt-Hunter ratio of expected to observed variance, the Callender-Osburn procedure, and a chi-square test—to detect true differences was investigated. Results show that small true differences were not detected regardless of sample size and the number of studies and that moderate true differences were not detected with small numbers of studies or small sample sizes. Hence, there is a need for caution in attributing observed variation across studies to artifact. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

How Meta-Analysis Increases Statistical Power.

One of the most frequently cited reasons for conducting a meta-analysis is the increase in statistical power that it affords a reviewer. This article demonstrates that fixed-effects meta-analysis increases statistical power by reducing the standard error of the weighted average effect size (T?.) and, in so doing, shrinks the confidence interval around T?.. Small confidence intervals make it more likely for reviewers to detect nonzero population effects, thereby increasing statistical power. Smaller confidence intervals also represent increased precision of the estimated population effect size. Computational examples are provided for 3 effect-size indices: d (standardized mean difference), Pearson's r, and odds ratios. Random-effects meta-analyses also may show increased statistical power and a smaller standard error of the weighted average effect size. However, the authors demonstrate that increasing the number of studies in a random-effects meta-analysis does not always increase statistical power. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Clinical supervision research from 1981 to 1993: A methodological critique.

The empirical studies in clinical supervision published from 1981 through 1993 were investigated to assess scientific rigor and to test whether the quality of methodology had improved since the review by R. K. Russell, A. M. Crimmings, and R. W. Lent (1984). The 144 studies were evaluated according to 49 threats to validity (T. D. Cook & D. T. Campbell, 1979; R. K. Russell et al., 1984; B. E. Wampold, B. Davis, & R. H. Good III; see record 1990-28928-001) and 8 statistical variables (e.g., effect size, statistical power, and Type I and Type II error rates). The data revealed a shift to realistic field studies, unchecked Type I and Type II error rates, medium effect sizes, and inattention to hypothesis validity. Recommendations for designing and conducting a feasible and well-designed supervision study are offered. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes.

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measurement error, instrumental variables and corrections for attenuation with applications to meta-analyses

MacMahon et al. present a meta-analysis of the effect of blood pressure on coronary heart disease, as well as new methods for estimation in measurement error models for the case when a replicate or second measurement is made of the fallible predictor. The correction for attenuation used by these authors is compared to others already existing in the literature, as well as to a new instrumental variable method. The assumptions justifying the various methods are examined and their efficiencies are studied via simulation. Compared to the methods we discuss, the method of MacMahon et al. may have bias in some circumstances because it does not take into account: (i) possible correlations among the predictors within a study; (ii) possible bias in the second measurement; or (iii) possibly differing marginal distributions of the predictors or measurement errors across studies. A unifying asymptotic theory using estimating equations is also presented.     

Evidence for the Type in Type A behavior: A taxometric analysis.

Four taxometric procedures were applied to the self-report responses of 1,239 Ss who completed the Jenkins Activity Survey (JAS). All 4 procedures provided clear evidence for a latent class variable. A continuous model simulation that mimicked the item characteristics of the JAS correctly rejected the presence of a latent class variable. Using an external validation procedure, I reexamined 5 previously published studies to determine if the simple Type A–B dichotomy was as predictive of outcome measures as the use of continuous JAS scores. The presence of a latent class variable predicts no gain in predictive power in moving from a simple dichotomy to continuous scores. Across 5 studies, there was a slight decrease in the size of the relation between Type A-B and outcome for the continuous JAS scores relative to the simple Type A-B dichotomy. Taken together, these results suggest that the Type A-B distinction is based on a latent typology. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects.

The growing popularity of meta-analysis has focused increased attention on the statistical models analysts are using and the assumptions underlying these models. Although comparisons often have been limited to fixed-effects (FE) models, recently there has been a call to investigate the differences between FE and random-effects (RE) models, differences that may have substantial theoretical and applied implications (National Research Council, 1992). Three FE models (including L. V. Hedges & I. Olkin's, 1985, and R. Rosenthal's, 1991, tests) and 2 RE models were applied to simulated correlation data in tests for moderator effects. The FE models seriously underestimated and the RE models greatly overestimated sampling error variance when their basic assumptions were violated, which caused biased confidence intervals and hypothesis tests. The implications of these and other findings are discussed as are methodological issues concerning meta-analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behavior therapy better than placebo treatments: Fact or artifact?

N. Brody's (see record 1990-12934-001) comment on our meta-analysis (T. G. Bowers and G. A. Clum; see record 1989-16477-001) seems to suggest that the efficacy of behavior therapy has not been established relative to placebo control conditions, especially for "neurotic conditions." The comments appear directed at defending an earlier meta-analysis (Priouleau, Murdock, & Brody, 1983) that concluded that psychotherapy was not more efficacious than a placebo control. We agree with Brody regarding the need for increased use of the heteromethod approach and longer follow-up for psychotherapy studies. However, we do not agree that the 10 studies Brody selected do not support the effectiveness of behavior therapy relative to placebo controls. Although a set of 10 studies is probably too small to allow robust conclusions, we noted a median effect size of .63 for those studies of neurotic conditions, relative to a placebo control. These results were very similar to our overall findings from 69 studies. Furthermore, available follow-up data suggest moderate effect sizes exist for those studies. We also comment on the existence of Type I and Type II errors of inference in reviews and meta-analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correlation of eyewitness accuracy and confidence: Optimality hypothesis revisited.

Inasmuch as a completely satisfactory estimate of effect size for the eyewitness accuracy-confidence relation does not exist, we conducted a meta-analysis of 35 staged-event studies. Estimated r?=?.25 (d?=?.52), with a 95% confidence interval of .08 to .42. Sampling error accounted for 52% of the variation in r, leaving room for measurement error and possibly moderator variables to account for the remaining variation. Further analysis identified duration of target face exposure as a moderator variable, providing support for Deffenbacher's (1980) optimality hypothesis. When corrected for the attenuating effect of sampling error in the accuracy-confidence correlations, the correlation of exposure duration and the accuracy-confidence correlation was .51: Longer exposures allowed for greater predictability of accuracy from confidence. Even through correlation for unreliability in the confidence measure produces a higher estimate of the population correlation of accuracy and confidence, .34, one must be cautious in assessing the utility of confidence for predicting accuracy in actual cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)