Incorporation of sequential trials into a fixed effects meta-analysis

The technique of meta-analysis provides a systematic and quantitative approach to the summary of results from a collection of similar randomized studies. Comprehensive methodology exists for analysis when all trials are of a fixed sample size design, but this is based on assumptions which are no longer valid when incorporating sequentially designed studies. In this paper the effect of incorporating one or more sequential trials into a meta-analysis otherwise consisting of fixed sample size trials is investigated. The results from simulation studies of a variety of plausible scenarios are presented. The aim is to quantify the extent to which bias is introduced. It was found that when incorporating two alternative sequential designs, the triangular test and the O'Brien and Fleming procedure, the results of a conventional meta-analysis remain accurate.     

How significant is a significant difference? Problems with the measurement of magnitude of effect.

The use of measures of magnitude of effect (MOE) has been advocated as a way to go beyond statistical tests of significance and to identify effects of a practical size and MOE has been used in meta-analysis to combine results of different studies. Some problems associated with measures of MOE are described, and implications for researchers are discussed. It is demonstrated that one commonly used measure of MOE is heavily dependent on a study's number of treatments and sample size. Such a measure should be used with caution for comparing results of studies of different sizes. (31 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Random versus nonrandom assignment in controlled experiments: Do you get the same answer?

Psychotherapy meta-analyses commonly combine results from controlled experiments that use random and nonrandom assignment without examining whether the 2 methods give the same answer. Results from this article call this practice into question. With the use of outcome studies of marital and family therapy, 64 experiments using random assignment yielded consistently higher mean posttest effects and less variable posttest effects than 36 studies using nonrandom assignment. This difference was reduced by about half by taking into account various covariates, especially pretest effect size levels and various characteristics of control groups. The importance of this finding depends on (a) whether one is discussing meta-analysis or primary experiments, (b) how precise an answer is desired, and (c) whether some adjustment to the data from studies using nonrandom assignment is possible. It is concluded that studies using nonrandom assignment may produce acceptable approximations to results from randomized experiments under some circumstances but that reliance on results from randomized experiments as the gold standard is still well founded. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Quantitative synthesis in systematic reviews

The final common pathway for most systematic reviews is a statistical summary of the data, or meta-analysis. The complex methods used in meta-analyses should always be complemented by clinical acumen and common sense in designing the protocol of a systematic review, deciding which data can be combined, and determining whether data should be combined. Both continuous and binary data can be pooled. Most meta-analyses summarize data from randomized trials, but other applications, such as the evaluation of diagnostic test performance and observational studies, have also been developed. The statistical methods of meta-analysis aim at evaluating the diversity (heterogeneity) among the results of different studies, exploring and explaining observed heterogeneity, and estimating a common pooled effect with increased precision. Fixed-effects models assume that an intervention has a single true effect, whereas random-effects models assume that an effect may vary across studies. Meta-regression analyses, by using each study rather than each patient as a unit of observation, can help to evaluate the effect of individual variables on the magnitude of an observed effect and thus may sometimes explain why study results differ. It is also important to assess the robustness of conclusions through sensitivity analyses and a formal evaluation of potential sources of bias, including publication bias and the effect of the quality of the studies on the observed effect.     

Meta-analysis of studies assessing the efficacy of projective techniques in discriminating child sexual abuse

OBJECTIVE: This meta-analysis of 12 studies assesses the efficacy of projective techniques to discriminate between sexually abused children and nonsexually abused children. METHOD: A literature search was conducted to identify published studies that used projective instruments with sexually abused children. Those studies that reported statistics that allowed for an effect size to be calculated, were then included in the meta-analysis. There were 12 studies that fit the criteria. The projectives reviewed include The Rorschach, The Hand Test, The Thematic Apperception Test (TAT), the Kinetic Family Drawings, Human Figure Drawings, Draw Your Favorite Kind of Day, The Rosebush: A Visualization Strategy, and The House-Tree-Person. RESULTS: The results of this analysis gave an over-all effect size of d = .81, which is a large effect. Six studies included only a norm group of nondistressed, nonabused children with the sexual abuse group. The average effect size was d = .87, which is impressive. Six studies did include a clinical group of distressed nonsexually abused subjects and the effect size lowered to d = .76, which is a medium to large effect. CONCLUSION: This indicates that projective instruments can discriminate distressed children from nondistressed subjects, quite well. In the studies that included a clinical group of distressed children who were not sexually abused, the lower effect size indicates that the instruments were less able to discriminate the type of distress. This meta-analysis gives evidence that projective techniques have the ability to discriminate between children who have been sexually abused and those who were not abused sexually. However, further research that is designed to include clinical groups of distressed children is needed in order to determine how well the projectives can discriminate the type of distress.     

Examining psychokinesis: The interaction of human intention with random number generators--A meta-analysis.

Séance-room and other large-scale psychokinetic phenomena have fascinated humankind for decades. Experimental research has reduced these phenomena to attempts to influence (a) the fall of dice and, later, (b) the output of random number generators (RNGs). The meta-analysis combined 380 studies that assessed whether RNG output correlated with human intention and found a significant but very small overall effect size. The study effect sizes were strongly and inversely related to sample size and were extremely heterogeneous. A Monte Carlo simulation revealed that the small effect size, the relation between sample size and effect size, and the extreme effect size heterogeneity found could in principle be a result of publication bias. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Grouped random effects models for Bayesian meta-analysis

Meta-analysis refers to quantitative methods to combine results from independent studies so as to draw overall conclusions. Frequently, results from dissimilar studies are inappropriately combined, resulting in suspect inferential synthesis. We present a straightforward method to identify and address this problem through the development of grouped random effect models for meta-analysis. We examine 15 comparative studies that investigate the efficacy of a new anti-epileptic drug, progabide. The flexibility of this modelling scheme is exemplified by the result that the open studies support the efficacy of progabide while the closed studies support the reverse hypothesis. Bayesian approaches for meta-analysis are preferable because of the small number of studies prevalent in meta-analysis. We specify diffuse proper prior and hyperprior distributions to assure posterior propriety. We investigate sensitivity of the posterior to choice of prior. We use Gibbs sampling and the Metropolis algorithm to generate samples from the relevant posteriors. We analyse posterior summaries and plots of model parameters to suggest solutions to questions of interest.     

Threat-related attentional bias in anxious and nonanxious individuals: A meta-analytic study.

This meta-analysis of 172 studies (N = 2,263 anxious,N = 1,768 nonanxious) examined the boundary conditions of threat-related attentional biases in anxiety. Overall, the results show that the bias is reliably demonstrated with different experimental paradigms and under a variety of experimental conditions, but that it is only an effect size of d = 0.45. Although processes requiring conscious perception of threat contribute to the bias, a significant bias is also observed with stimuli outside awareness. The bias is of comparable magnitude across different types of anxious populations (individuals with different clinical disorders, high-anxious nonclinical individuals, anxious children and adults) and is not observed in nonanxious individuals. Empirical and clinical implications as well as future directions for research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Roaming through the methodology. VIII. Pilot studies: sense and nonsense

A pilot study is often performed before the start of a large study. Its aim is improving the methodological quality and evaluating the feasibility. The results of a pilot study are also often used to gain an impression of the efficacy of an intervention. For this purpose a pilot is absolutely unsuited. However, the estimate of the effect of an intervention in small studies such as pilot studies is determined to a large extent by chance. When conducting small studies the chance of publication bias is large. Small studies of adequate methodological quality should be published, whether the results are positive or negative, significant or non-significant, because combination in a later cumulative meta-analysis may lead to sufficient power to assess the efficacy of an experimental intervention.     

Why what we teach depends on when: Grade and reading intervention modality moderate effect size.

Despite impressive advances in the science of reading intervention, how to best help at-risk readers remains a point of contention. Because reading represents the synthesis of background factors and language and reading skills—all of which develop with age and experience—this meta-analysis investigated whether development (as approximated by grade) and intervention modality are key moderators of intervention effect size for disadvantaged readers. Eighty-five experimental or quasi-experimental studies with 116 treatment–control groups (N = 7,522) were selected from preschool to Grade 7. Analyses accounted for intervention length, instructor-to-student ratio, measure design, experimental design, attrition, intervention language, and publication bias. Between-group comparisons suggested that effect sizes were larger for older students, comprehension interventions, quasi-experimental studies, and samples at greater risk. In hierarchical regression analyses, intervention modality alone did not explain additional variance in effect size; however, when interacting with grade, intervention modality did explain additional variance. Phonics interventions were more effective until Grade 1, after which comprehension and mixed interventions, in particular, tended to be associated with greater effect sizes. These results highlight the importance of a developmental understanding of reading remediation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effectiveness of psychodramatic techniques: A meta-analysis.

Questions concerning the scientific basis of the clinical application of psychodramatic techniques have been raised primarily because of the infrequency of research publications that validate the clinical observations. A meta-analysis conducted on the basis of 25 experimentally designed studies showed an overall effect size that points to a large size improvement effect similar to or better than that commonly reported for group psychotherapy in general. The techniques of role reversal and doubling emerged as the most effective interventions. Of the 4 techniques investigated, 3 were significantly different from each other. There was no difference between the techniques' effectiveness when used with clinical versus student populations or between their use in single versus multiple sessions. The meaning of the findings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing the null hypothesis in meta-analysis: A comparison of combined probability and confidence interval procedures.

Combined significance tests (combined p values) and tests of the weighted mean effect size are used to combine information across studies in meta-analysis. A combined significance test (Stouffer test) is compared with a test based on the weighted mean effect size as tests of the same null hypothesis. The tests are compared analytically in the case in which the within-group variances are known and compared through large-sample theory in the more usual case in which the variances are unknown. Generalizations suggested are then explored through a simulation study. This work demonstrates that the test based on the average effect size is usually more powerful than the Stouffer test unless there is a substantial negative correlation between within-study sample size and effect size. Thus, the test based on the average effect size is generally preferable, and there is little reason to also calculate the Stouffer test. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analyses of studies of the diagnostic accuracy of laboratory tests: a review of the concepts and methods

OBJECTIVE: To provide practicing pathologists and other laboratory professionals with the necessary background for reading and evaluating published reports of meta-analyses of studies of the diagnostic accuracy of laboratory tests. STUDY SELECTION: English language literature, 1980 to present, pertaining to the rationale, objectives, and interpretation of meta-analyses of randomized controlled trials, and meta-analyses of studies of the diagnostic accuracy of laboratory tests. CONCLUSIONS: Meta-analysis has several applications in the investigation of the diagnostic accuracy of laboratory tests. It can improve the quality of future primary studies by drawing attention to the methodologic deficiencies of existing reports; it can identify reasons for the variation in the results of those reports; and it can generate valid summary estimates of the diagnostic accuracy of laboratory tests based on all completed investigations, providing that the available primary studies are of high scientific validity. Several statistical techniques for integrating data from reports on diagnostic test accuracy have either been developed or are under development, but meta-analysis is often limited by the poor quality of the primary studies and the effect of publication bias. Meta-analysis can evolve into a reliable tool for assessing the accuracy of laboratory tests if both investigators and editors strive to improve the quality of the primary studies and to reduce the extent of publication bias in this area of the literature.     

Meta-analysis of facial identification studies.

Presents a meta-analysis of 128 eyewitness identification and facial recognition studies involving 960 experimental conditions and 16,950 Ss. The meta-analysis was designed to determine what knowledge has been accumulated on factors that influence facial identification performance and what areas of facial identification research would benefit from further research. Two techniques were used: an effect size analysis, which integrates the effect sizes of independent variables across studies, and a study-characteristics analysis, which integrates the influence of study characteristics on performance. A number of variables operating at the encoding and retrieval stages yielded large effects on performance. These variables included context reinstatement, transformations in the appearance of faces, depth of processing strategies, target distinctiveness, and elaboration at encoding. Additional variables yielding strong effects on recognition performance were exposure time, cross-racial identification, and retention interval. There was little correspondence between a variable's impact on hit rate and false-alarm rate. A list of the studies used in the meta-analysis is appended. (49 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Factors influencing the relation between extraversion and pleasant affect.

Factors that influence the relation between extraversion and pleasant affect were examined in 6 studies. In Studies 1 through 5, the authors used structural equation modeling techniques to test whether different extraversion and pleasant affect scales and the use of multiple methods of assessment influenced the strength of the relation. In Study 6, the authors conducted a meta-analysis of previous literature to calculate an average effect size and to assess the influence of moderator variables. Results from both the structural equation models and the meta-analysis showed that with only a few exceptions, the use of different extraversion and affect scales resulted in moderate to strong correlations. The use of "on-line" methods of mood assessment (moment reports or daily-diary reports) resulted in lower and more homogeneous correlations than did the use of global, retrospective measures of mood. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments?

This meta-analysis compares effect size estimates from 51 randomized experiments to those from 47 nonrandomized experiments. These experiments were drawn from published and unpublished studies of Scholastic Aptitude Test coaching, ability grouping of students within classrooms, presurgical education of patients to improve postsurgical outcome, and drug abuse prevention with juveniles. The raw results suggest that the two kinds of experiments yield very different answers. But when studies are equated for crucial features (which is not always possible), nonrandomized experiments can yield a reasonably accurate effect size in comparison with randomized designs. Crucial design features include the activity level of the intervention given the control group, pretest effect size, selection and attrition levels, and the accuracy of the effect-size estimation method. Implications of these results for the conduct of meta-analysis and for the design of good nonrandomized experiments are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Where oh where are the specific ingredients? A meta-analysis of component studies in counseling and psychotherapy.

Component studies, which involve comparisons between a treatment package and the treatment package without a theoretically important component or the treatment package with an added component, use experimental designs to test whether the component is necessary to produce therapeutic benefit. A meta-analysis was conducted on 27 component studies culled from the literature. It was found that the effect size for the difference between a package with and without the critical components was not significantly different from zero, indicating that theoretically purported important components are not responsible for therapeutic benefits. Moreover, the effect sizes were homogeneous, which suggests that there were no important variables moderating effect sizes. The results cast doubt on the specificity of psychological treatments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of fluoxetine outcome in the treatment of depression

A meta-analysis was conducted on all the double-blind, placebo-controlled efficacy trials of the antidepressant fluoxetine (Prozac). This drug has become the antidepressant most frequently prescribed by psychiatrists and has been hailed by the media as a "wonder drug." Results produced a relatively modest overall effect size that was no greater than effect sizes obtained by previous meta-analyses of tricyclic antidepressants. This study also examined the possibility that bias may have contaminated study outcome ratings. Because past studies suggest that the greater frequency of side effects in active drug groups unblinds study participants, we examined the relationship between study effect sizes and the percentage of patients reporting side effects. As predicted, both clinician and patient outcome ratings correlated significantly with the percentage of patients experiencing side effects. Questions are raised about the role of side effects in mediating drug outcome results.     

Electron microscopic study of long-term denervated rat skeletal muscle

This article reports a secondary analysis of past therapy outcome meta-analysis. Fifteen meta-analysis provided effect sizes from 56 studies in previous reviews that met 1 of 3 increasingly stringent levels of criteria for clinical representativeness. The effect sizes were synthesized and compared with results from the original meta-analyses. Effect sizes from more clinically representative studies are the same size at all 3 criteria levels as in past meta-analyses. Almost no studies exist that meet the most stringent level of criteria. Results are interpreted cautiously because of controversy about what criteria best capture the notion of clinical representativeness, because so few experiments have tested therapy in clinical conditions, and because other models for exploring the generalizability of therapy outcome research to clinical conditions might yield different results.     

Quinine for nocturnal leg cramps: a meta-analysis including unpublished data

OBJECTIVE: With respect to the use of quinine for the treatment of nocturnal leg cramps, to determine whether the findings of a previously performed meta-analysis of published data are altered with the addition of unpublished data, and whether publication bias is present in this area. DESIGN: A meta-analysis of eight (four published and four unpublished) randomized, double-blind, placebo-controlled trials, seven of which had a crossover design. SETTING: Randomized trials that were available as of July 1997. SUBJECTS: Ambulatory patients (659) who suffered from regular nocturnal leg cramps. MAIN RESULTS: When individual patient data from all crossover studies were pooled, persons had 3.60 (95% confidence interval [CI] 2.15, 5.05) fewer cramps in a 4-week period when taking quinine compared with placebo. This compared with an estimate of 8.83 fewer cramps (95% CI 4.16, 13.49) from pooling published studies alone. The corresponding relative risk reductions were 21% (95% CI 12%, 30%) and 43% (95% CI 21%, 65%), respectively. Compared with placebo, the use of quinine was associated with an increased incidence of side effects, particularly tinnitus. Publication bias is present in the reporting of the efficacy of quinine for this indication, as almost all published studies reported larger estimates of its efficacy than did unpublished studies. CONCLUSIONS: This study confirms that quinine is efficacious in the prevention of nocturnal leg cramps. However, its benefit may not be as large as reported from the pooling of published studies alone. Given the side effect profile of quinine, nonpharmacologic therapy (e.g., regular passive stretching of the affected muscle) is the best first-line treatment. For persons who find this ineffective and whose quality of life is significantly affected, a trial of quinine is warranted. Prescribing physicians must closely monitor the risks and benefits in individual patients. Publication bias is present in this area even though there is controversy about the role of quinine in the treatment of leg cramps. To minimize the possibility of this bias, persons performing medication-related meta-analyses should seek high-quality unpublished data from drug regulatory agencies and pharmaceutical companies.