A likelihood approach to meta-analysis with random effects

In a meta-analysis of a set of clinical trials, a crucial but problematic component is providing an estimate and confidence interval for the overall treatment effect theta. Since in the presence of heterogeneity a fixed effect approach yields an artificially narrow confidence interval for theta, the random effects method of DerSimonian and Laird, which incorporates a moment estimator of the between-trial components of variance sigma B2, has been advocated. With the additional distributional assumptions of normality, a confidence interval for theta may be obtained. However, this method does not provide a confidence interval for sigma B2, nor a confidence interval for theta which takes account of the fact that sigma B2 has to be estimated from the data. We show how a likelihood based method can be used to overcome these problems, and use profile likelihoods to construct likelihood based confidence intervals. This approach yields an appropriately widened confidence interval compared with the standard random effects method. Examples of application to a published meta-analysis and a multicentre clinical trial are discussed. It is concluded that likelihood based methods are preferred to the standard method in undertaking random effects meta-analysis when the value of sigma B2 has an important effect on the overall estimated treatment effect.     

Links between physical fitness and cardiovascular reactivity and recovery to psychological stressors: A meta-analysis.

A meta-analysis of published studies with adult human participants was conducted to evaluate whether physical fitness attenuates cardiovascular reactivity and improves recovery from acute psychological stressors. Thirty-three studies met selection criteria; 18 were included in recovery analyses. Effect sizes and moderator influences were calculated by using meta-analysis software. A fixed effects model was fit initially; however, between-studies heterogeneity could not be explained even after inclusion of moderators. Therefore, to account for residual heterogeneity, a random effects model was estimated. Under this model, fit individuals showed significantly attenuated heart rate and systolic blood pressure reactivity and a trend toward attenuated diastolic blood pressure reactivity. Fit individuals also showed faster heart rate recovery, but there were no significant differences in systolic blood pressure or diastolic blood pressure recovery. No significant moderators emerged. Results have important implications for elucidating mechanisms underlying effects of fitness on cardiovascular disease and suggest that fitness may be an important confound in studies of stress reactivity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A random effects model for effect sizes.

Recent interest in quantitative research synthesis has led to the development of rigorous statistical theory for some of the methods used in meta-analysis. Statistical theory proposed previously has stressed the estimation of fixed but unknown population effect sizes (standardized mean differences). Theoretical considerations often suggest that treatment effects are not fixed but vary across different implementations of a treatment. The present author presents a random effects model (analogous to random effects ANOVA) in which the population effect sizes are not fixed but are sample realizations from a distribution of possible population effect sizes. An analogy to variance component estimation is used to derive an unbiased estimator of the variance of the effect-size distribution. An example shows that these methods may suggest insights that are not available from inspection of means and standard deviation of effect-size estimates. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Is implicit sequence learning impaired in Parkinson's disease? A meta-analysis.

The aim of the present study was to examine impairment of implicit learning in Parkinson's disease (PD) by means of a meta-analysis of studies that used the serial reaction time (SRT) task. The authors performed a systematic review and meta-analysis of published journal articles (1987-2005) that used the SRT task with patients with PD. The principal outcome measures used to compare studies were (a) the difference in reaction time between the last block of ordered sequence trials and the randomized block for PD and control participants and (b) fixed and random effects pooled estimates by the inverse weighting method. Six studies, including 67 patients with PD, met the inclusion criteria. The meta-analysis showed that implicit learning was impaired in PD, relative to healthy controls, with a standardized mean difference of 0.73 (95% confidence interval = 0.38, 1.07). Implicit sequence learning appears to be impaired in patients with PD. Some common methodological weaknesses and limitations in the reporting of statistical data are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analytic interval estimation for standardized and unstandardized mean differences.

The fixed-effects (FE) meta-analytic confidence intervals for unstandardized and standardized mean differences are based on an unrealistic assumption of effect-size homogeneity and perform poorly when this assumption is violated. The random-effects (RE) meta-analytic confidence intervals are based on an unrealistic assumption that the selected studies represent a random sample from a large superpopulation of studies. The RE approach cannot be justified in typical meta-analysis applications in which studies are nonrandomly selected. New FE meta-analytic confidence intervals for unstandardized and standardized mean differences are proposed that are easy to compute and perform properly under effect-size heterogeneity and nonrandomly selected studies. The proposed meta-analytic confidence intervals may be used to combine unstandardized or standardized mean differences from studies having either independent samples or dependent samples and may also be used to integrate results from previous studies into a new study. An alternative approach to assessing effect-size heterogeneity is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Ancient Pygmalion joins contemporary management: A meta-analysis of the result.

To determine the existence and relative magnitude of the Pygmalion effect among adults within management contexts, a meta-analysis of 17 relevant studies (58 effect sizes; N? = ?2874) was conducted to provide an estimate of the population mean effect size and variation from Pygmalion interventions. Performance measures in these studies included exam scores, performance appraisals, and physical output. Results indicate that Pygmalion effects can be fairly strong within some management contexts (average d? = ?1.13) but may vary widely depending on moderating variables. Specifically, results were stronger in the military, with men, and when involving persons for whom low expectations were initially held. Findings also highlight the need for more research in ongoing work contexts (vs. training) and in for-profit organizations (vs. the military). Suggestions are given to help overcome possible hurdles in conducting research in these areas. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effects of psychological therapies under clinically representative conditions: A meta-analysis.

Recently, concern has arisen that meta-analyses overestimate the effects of psychological therapies and that those therapies may not work under clinically representative conditions. This meta-analysis of 90 studies found that therapies are effective over a range of clinical representativeness. The projected effects of an ideal study of clinically representative therapy are similar to effect sizes in past meta-analyses. Effects increase with larger dose and when outcome measures are specific to treatment. Some clinically representative studies used self-selected treatment clients who were more distressed than available controls, and these quasi-experiments underestimated therapy effects. This study illustrates the joint use of fixed and random effects models, use of pretest effect sizes to study selection bias in quasi-experiments, and use of regression analysis to project results to an ideal study in the spirit of response surface modeling. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Controversies in meta-analysis: the case of the trials of serum cholesterol reduction

There has recently been disagreement in the literature on the results and interpretation of meta-analyses of the trials of serum cholesterol reduction, both in terms of the quantification of the effect on ischaemic heart disease and as regards the evidence of any adverse effect on other causes of death. This paper describes statistical aspects of a recent meta-analysis of these trials, and draws some more general conclusions about the methods used in meta-analysis. Tests of an overall null hypothesis are shown to have a basis clearly distinct from the more extensive assumptions needed to provide an overall estimate of effect. The fixed effect approach to estimation relies on the implausible assumption of homogeneity of treatment effects across the trials, and is therefore likely to yield confidence intervals which are too narrow and conclusions which are too dogmatic. However the conventional random effects method relies on its own set of unrealistic assumptions, and cannot be regarded as a robust solution to the problem of statistical heterogeneity. The random effects method is more usefully regarded as a type of sensitivity analysis in which the weights allocated to each study in estimating the overall effect are modified. However, rather than using a statistical model for the 'unexplained' heterogeneity, greater insight and scientific understanding of the results of a set of trials may be obtained by a careful exploration of potential sources of heterogeneity. In the context of the cholesterol trials, the heterogeneity according to the extent and duration of cholesterol reduction are of prime concern and are investigated using logistic regression. It is concluded that the long-term benefits of serum cholesterol reduction on the risk of heart disease have been seriously underestimated in some previous meta-analyses, while the evidence for adverse effects on other causes of death have been misleadingly exaggerated.     

Meta-analysis of psychological interventions for chronic low back pain.

The purpose of this meta-analysis of randomized controlled trials was to evaluate the efficacy of psychological interventions for adults with noncancerous chronic low back pain (CLBP). The authors updated and expanded upon prior meta-analyses by using broad definitions of CLBP and psychological intervention, a broad data search strategy, and state-of-the-art data analysis techniques. All relevant controlled clinical trials meeting the inclusion criteria were identified primarily through a computer-aided literature search. Two independent reviewers screened abstracts and articles for inclusion criteria and extracted relevant data. Cohen's d effect sizes were calculated by using a random effects model. Outcomes included pain intensity, emotional functioning, physical functioning (pain interference or pain-specific disability, health-related quality of life), participant ratings of global improvement, health care utilization, health care provider visits, pain medications, and employment/disability compensation status. A total of 205 effect sizes from 22 studies were pooled in 34 analyses. Positive effects of psychological interventions, contrasted with various control groups, were noted for pain intensity, pain-related interference, health-related quality of life, and depression. Cognitive-behavioral and self-regulatory treatments were specifically found to be efficacious. Multidisciplinary approaches that included a psychological component, when compared with active control conditions, were also noted to have positive short-term effects on pain interference and positive long-term effects on return to work. The results demonstrated positive effects of psychological interventions for CLBP. The rigor of the methods used, as well as the results that reflect mild to moderate heterogeneity and minimal publication bias, suggest confidence in the conclusions of this review. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Does expressive writing reduce health care utilization? A meta-analysis of randomized trials.

This meta-analysis examined whether writing about stressful experiences affects health care utilization (HCU) compared with writing on neutral topics or no-writing control groups. Randomized controlled trials of 30 independent samples representing 2,294 participants were located that contained sufficient information to calculate effect sizes. After omitting one study as an outlier, the effects were combined within 3 homogeneous groups: healthy samples (13 studies), samples with preexisting medical conditions (6 studies), and samples prescreened for psychological criteria (10 studies). Combined effect sizes, Hedges's g (95% confidence interval), with random effects estimation were 0.16 (0.02, 0.31), 0.21 (-0.02, 0.43), and 0.06 (-0.12, 0.24), respectively. Writing about stressful experiences reduces HCU in healthy samples but not in samples defined by medical diagnoses or exposure to stress or other psychological factors. The significance of these effects for individuals' health is unknown. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Varying coefficient meta-analytic methods for alpha reliability.

The conventional fixed-effects (FE) and random-effects (RE) confidence intervals that are used to assess the average alpha reliability across multiple studies have serious limitations. The FE method, which is based on a constant coefficient model, assumes equal reliability coefficients across studies and breaks down under minor violations of this assumption. The RE method, which is based on a random coefficient model, assumes that the selected studies are a random sample from a normally distributed superpopulation. The RE method performs poorly in typical meta-analytic applications where the studies have not been randomly sampled from a normally distributed superpopulation or have been randomly sampled from a nonnormal superpopulation. A new confidence interval for the average reliability coefficient of a specific measurement scale is based on a varying coefficient statistical model and is shown to perform well under realistic conditions of reliability heterogeneity and nonrandom sampling of studies. New methods are proposed for assessing reliability moderator effects. The proposed methods are especially useful in meta-analyses that involve a small number of carefully selected studies for the purpose of obtaining a more accurate reliability estimate or to detect factors that moderate the reliability of a scale. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Confidence intervals for the cross-validated multiple correlation in predictive regression models.

Presents improved procedures to approximate confidence intervals for ρ–2 and ρc–2 in both fixed and random predictor models. These approximations require neither point estimates nor variance estimates and are analytically shown to be precise enough for most practical prediction purposes. An application of confidence intervals in regression model development is also given. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Adult Age and Digit Symbol Substitution Performance: A Meta-Analysis.

This article reports the results of a meta-analysis of the effects of age, education, and estimated year of measurement on scores from the Wechsler Adult Intelligence Scale and the Wechsler Adult Intelligence Scale-Revised Digit Symbol Substitution Test. Analysis of effect sizes for age reported in 141 studies published between 1986 and 2002 indicated a mean standardized difference of -2.07. Age accounted for 86% of the variance in a regression model using age, education, and year submitted as predictors of Digit Symbol scores. There was no association between years of education or year submitted and Digit Symbol scores for younger adults or older adults. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

How Meta-Analysis Increases Statistical Power.

One of the most frequently cited reasons for conducting a meta-analysis is the increase in statistical power that it affords a reviewer. This article demonstrates that fixed-effects meta-analysis increases statistical power by reducing the standard error of the weighted average effect size (T?.) and, in so doing, shrinks the confidence interval around T?.. Small confidence intervals make it more likely for reviewers to detect nonzero population effects, thereby increasing statistical power. Smaller confidence intervals also represent increased precision of the estimated population effect size. Computational examples are provided for 3 effect-size indices: d (standardized mean difference), Pearson's r, and odds ratios. Random-effects meta-analyses also may show increased statistical power and a smaller standard error of the weighted average effect size. However, the authors demonstrate that increasing the number of studies in a random-effects meta-analysis does not always increase statistical power. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Is the Meta-Analysis of Correlation Coefficients Accurate When Population Correlations Vary?

One conceptualization of meta-analysis is that studies within the meta-analysis are sampled from populations with mean effect sizes that vary (random-effects models). The consequences of not applying such models and the comparison of different methods have been hotly debated. A Monte Carlo study compared the efficacy of Hedges and Vevea's random-effects methods of meta-analysis with Hunter and Schmidt's, over a wide range of conditions, as the variability in population correlations increases. (a) The Hunter-Schmidt method produced estimates of the average correlation with the least error, although estimates from both methods were very accurate; (b) confidence intervals from Hunter and Schmidt's method were always slightly too narrow but became more accurate than those from Hedges and Vevea's method as the number of studies included in the meta-analysis, the size of the true correlation, and the variability of correlations increased; and (c) the study weights did not explain the differences between the methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A procedure for combining sample correlation coefficients and vote counts to obtain an estimate and a confidence interval for the population correlation coefficient.

Missing effect-size estimates pose a particularly difficult problem in meta-analysis. Rather than discarding studies with missing effect-size estimates or setting missing effect-size estimates equal to 0, the meta-analyst can supplement effect-size procedures with vote-counting procedures if the studies report the direction of results or the statistical significance of results. By combining effect-size and vote-counting procedures, the meta-analyst can obtain a less biased estimate of the population effect size and a narrower confidence interval for the population effect size. This article describes 3 vote-counting procedures for estimating the population correlation coefficient in studies with missing sample correlations. Easy-to-use tables, based on equal sample sizes, are presented for the 3 procedures. More complicated vote-counting procedures also are given for unequal sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods.

The efficacy of the Hedges and colleagues, Rosenthal-Rubin, and Hunter-Schmidt methods for combining correlation coefficients was tested for cases in which population effect sizes were both fixed and variable. After a brief tutorial on these meta-analytic methods, the author presents 2 Monte Carlo simulations that compare these methods for cases in which the number of studies in the meta-analysis and the average sample size of studies were varied. In the fixed case the methods produced comparable estimates of the average effect size; however, the Hunter-Schmidt method failed to control the Type I error rate for the associated significance tests. In the variable case, for both the Hedges and colleagues and Hunter-Schmidt methods, Type I error rates were not controlled for meta-analyses including 15 or fewer studies and the probability of detecting small effects was less than .3. Some practical recommendations are made about the use of meta-analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fixed- and random-effects models in meta-analysis.

There are 2 families of statistical procedures in meta-analysis: fixed- and random-effects procedures. They were developed for somewhat different inference goals: making inferences about the effect parameters in the studies that have been observed versus making inferences about the distribution of effect parameters in a population of studies from a random sample of studies. The authors evaluate the performance of confidence intervals and hypothesis tests when each type of statistical procedure is used for each type of inference and confirm that each procedure is best for making the kind of inference for which it was designed. Conditionally random-effects procedures (a hybrid type) are shown to have properties in between those of fixed- and random-effects procedures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Psyllium-enriched cereals lower blood total cholesterol and LDL cholesterol, but not HDL cholesterol, in hypercholesterolemic adults: results of a meta-analysis

We conducted a meta-analysis to determine the effect of consumption of psyllium-enriched cereal products on blood total cholesterol (TC), LDL cholesterol (LDL-C) and HDL cholesterol (HDL-C) levels and to estimate the magnitude of the effect among 404 adults with mild to moderate hypercholesterolemia (TC of 5.17-7.8 mmol/L) who consumed a low fat diet. Studies of psyllium cereals were identified by a computerized search of MEDLINE and Current Contents and by contacting United States-based food companies involved in psyllium research. Published and unpublished studies were reviewed by one author and considered eligible for inclusion in the meta-analysis if they were conducted in humans, were randomized, controlled experiments, and included a control group that ate cereal providing /=50 y) on blood lipids. The meta-analysis showed that subjects who consumed a psyllium cereal had lower TC and LDL-C concentrations [differences of 0.31 mmol/L (5%) and 0.35 mmol/L (9%), respectively] than subjects who ate a control cereal; HDL-C concentrations were unaffected in subjects eating psyllium cereal. There was no effect of sex, age or menopausal status on blood lipids. Results indicate that consuming a psyllium-enriched cereal as part of a low fat diet improves the blood lipid profile of hypercholesterolemic adults over that which can be achieved with a low fat diet alone.