Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis.

Presents a framework for studying the influence of reporting quality on meta-analytic results in which 3 sources of reporting deficiency are identified: quality (adequacy) of publicizing, quality of macrolevel reporting, and quality of the review process in a different way. To assess the influence of reporting quality empirically, 25 reports were sampled from the psychotherapy meta-analysis reported by M. L. Smith et al (1980) and recorded by the present authors. Two sources of information pertinent to reporting quality were established: interrater reliabilities and confidence judgments. Reanalyses incorporating reliability corrections and confidence judgments suggested that deficient reporting injects considerable noise into meta-analytic data and can lead to spurious conclusions. (43 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues.

Meta-analysis has mostly been used to summarize the size of an effect averaged over multiple studies, but meta-analysis has not been much applied to the study of causal mediating processes through which an effect is produced. This lacuna has limited the contribution of meta-analysis to the explanatory theories that play such a key role in science. Fortunately, meta-analysts can explore causal processes. This article reviews several examples of how this has been done in past meta-analyses, using these examples to introduce the methodological, statistical, and conceptual problems that are raised when meta-analysis is applied to the task. Meta-analysts are encouraged to adapt such methods to their work to improve the capacity of their work to contribute to scientific theory, and statisticians are encouraged to solve the remaining statistical problems that current meta-analytic mediational analyses incur. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Summing up evidence: one answer is not always enough

Are meta-analyses the brave new world, or are the critics of such combined analyses right to say that the biases inherent in clinical trials make them uncombinable? Negative trials are often unreported, and hence can be missed by meta-analysts. And how much heterogeneity between trials is acceptable? A recent major criticism is that large randomised trials do not always agree with a prior meta-analysis. Neither individual trials nor meta-analyses, reporting as they do on population effects, tell how to treat the individual patient. Here we take a more rounded approach to meta-analyses, arguing that their strengths outweigh their weaknesses, although the latter must not be brushed aside.     

Cohesion and Performance in Groups: A Meta-Analytic Clarification of Construct Relations.

Previous meta-analytic examinations of group cohesion and performance have focused primarily on contextual factors. This study examined issues relevant to applied researchers by providing a more detailed analysis of the criterion domain. In addition, the authors reinvestigated the role of components of cohesion using more modern meta-analytic methods and in light of different types of performance criteria. The results of the authors' meta-analyses revealed stronger correlations between cohesion and performance when performance was defined as behavior (as opposed to outcome), when it was assessed with efficiency measures (as opposed to effectiveness measures), and as patterns of team workflow became more intensive. In addition, and in contrast to B. Mullen and C. Copper's (1994) meta-analysis, the 3 main components of cohesion were independently related to the various performance domains. Implications for organizations and future research on cohesion and performance are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods.

The efficacy of the Hedges and colleagues, Rosenthal-Rubin, and Hunter-Schmidt methods for combining correlation coefficients was tested for cases in which population effect sizes were both fixed and variable. After a brief tutorial on these meta-analytic methods, the author presents 2 Monte Carlo simulations that compare these methods for cases in which the number of studies in the meta-analysis and the average sample size of studies were varied. In the fixed case the methods produced comparable estimates of the average effect size; however, the Hunter-Schmidt method failed to control the Type I error rate for the associated significance tests. In the variable case, for both the Hedges and colleagues and Hunter-Schmidt methods, Type I error rates were not controlled for meta-analyses including 15 or fewer studies and the probability of detecting small effects was less than .3. Some practical recommendations are made about the use of meta-analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The role of judgment calls in meta-analysis.

The use of meta-analysis in literature reviews is increasing at a geometric rate, even though many researchers are still not familiar with the technique. Although most of the criticism of meta-analysis has focused on the formulas used, we believe there should be concern with the many judgment calls that must be made and how they affect the results of a meta-analysis. In recent years four topics have each been independently meta-analyzed twice. These four pairs are reexamined, and the differences in results between each of the pairs are diagnosed as resulting from judgment calls. Future users of meta-analysis are advised to be explicit about the judgments they make, and a summary of judgment calls is offered as a guide for reporting them. In addition to reporting judgment calls, their effects on the estimates of the population mean effect size and standard deviation should be tested. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Combining and comparing significance levels from nonindependent hypothesis tests.

Describes procedures necessary for adjusting for nonindependence in meta-analysis. The assumption of independence among significance levels that are analyzed is often violated in practice. Because the failure to adjust for nonindependence is likely to produce bias on the meta-analytic level, ignoring the assumption of independence is not of minor consequence and requires careful attention. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A narrative and meta-analytic review of helping skills training: Time to revive a dormant area of inquiry.

The authors review previous narrative and meta-analytic reviews on the effectiveness of overall helping skills training programs. The authors then review narrative reviews and conduct a new meta-analysis of specific methods used to teach helping skills within these programs. Our meta-analysis found that, in the aggregate, training methods substantially outperformed no training conditions, and that effect sizes did not vary as a function of trainee educational level (graduate vs. undergraduate students) or the type of criterion measure (interview-based vs. analogue-based empathy measures). Direct comparison of the training methods revealed that modeling outperformed instruction and feedback, and multimethod outperformed single-method training. The authors critique the literature and suggest that the studies in the helping skills literature generally fail to meet contemporary methodological standards, thereby limiting the conclusions that can be drawn. The authors appeal for better research on helping skills training, especially as it is currently practiced. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Under the meta-scope: possibilities and limits of meta-analyses

Critical review and statistical synthesis of the best evidence that is available on the benefits and risks of medical interventions is an important component of evidence-based medicine. There is, however, controversy regarding the merits and perils of this approach. In this article the potential and limitations of meta-analysis are illustrated and the problems of applying meta-analytical results to the individual patient are discussed. Whereas the classical narrative review article often reflects the personal opinion of the author, systematic reviews are characterized by clearly defined objectives and reproducible methods. If appropriate and feasible, individual studies are combined in a meta-analysis which will lead to enhanced precision of effect estimates. The potential of meta-analysis is illustrated by considering a patient who suffered a myocardial infarction in 1981. After consulting a misleading narrative review, this patient was discharged without beta-blockade for secondary prevention. Meta-analyses are, however, also liable to bias. For example, a meta-analysis of trials of magnesium infusions in acute myocardial infarction showed a substantial reduction in mortality. The beneficial effect of magnesium was not confirmed in the large ISIS-4 trial published in 1995. The occurrence of misleading meta-analyses is not surprising considering the selective publication of "positive" findings and the often inadequate quality of component studies. Misleading meta-analyses may also result from the inappropriate combination of heterogeneous studies. Every patient is unique regarding the exact localization and severity of the lesion, preexisting abilities, co-morbidity, social environment, personality and emotional response. The appropriate application of meta-analytic findings to the individual patient is difficult. It requires consideration of both quantitative and qualitative approaches to decision-making. Emphatic integration of these approaches characterizes the experienced, scientifically and socially competent physician.     

Probability of the superior outcome of one treatment over another.

An intuitively appealing indicator of magnitude of effect in applied research is an estimate of the probability of the superior outcome of one treatment over another. Parametric and nonparametric estimates are discussed, as is a meta-analytic estimate. Estimates from values of t, the point-biserial correlation, and standardized effect size are presented. A new perspective on J. Cohen's ( 1988) standards for small, medium, and large effect sizes is provided. Psychologists who are conducting applied primary research or meta-analyses are urged to include such estimation in their reports. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Updating meta-analytic research findings: Bayesian approaches versus the medical model.

The authors examine 3 methods of combining new studies into existing meta-analyses: (a) adding the new study or studies to the database and recalculating the meta-analysis (the medical model); (b) using the Bayesian procedure advocated by F. L. Schmidt and J. E. Hunter (1977) and F. L. Schmidt, J. E. Hunter, K. Pearlman, and G. S. Shane (1979) to update the meta-analysis; and (c) using the Bayesian methods advocated by these authors and M. T. Brannick (2001) and M. T. Brannick, S. M. Hall, and Y. Liu (2002) to estimate study-specific parameters. Method b was found to severely overweight new studies relative to the previous studies contained in the meta-analysis, and Method c was found to do the same while also requiring an assumption with a low prior probability of being correct, causing the method to violate Bayesian principles. The authors present an alternative Bayesian procedure that does not suffer from these drawbacks and yields meta-analytic results very similar to those obtained with the medical model. They recommend use of the medical model or this alternative Bayesian procedure. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of the relation between personality traits and leadership perceptions: An application of validity generalization procedures.

Reexamines, via meta-analysis, the relation between personality traits and leadership perceptions or extent of leader emergence, arguing that prior research on trait theories and leadership has been misinterpreted as applying to a leader's effect on performance when it actually pertains to the relation of leadership traits to leadership emergence. Further, based on current theories of social perceptions, several traits were expected to be strongly related to leadership perceptions. The meta-analytic technique of validity generalization was used with the 15 articles identified by R. D. Mann (see record 1960-04194-001) as investigating the relationship between personality traits and leadership. These studies were then pooled with 9 subsequent studies in an additional set of meta-analyses. Results support the expectation in that intelligence, masculinity–femininity, and dominance were significantly related to leadership perceptions. Findings show that variability across studies in the relation of these traits to leadership perceptions could be explained largely by methodological factors, indicating that contingency theories of leadership perceptions may not be needed. Both of these results contrast with the conclusions of earlier nonquantitative literature reviews on traits and leadership perceptions and with conventional thinking in the leadership area. (62 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Overcoming the limitations of current meta-analysis of randomised controlled trials

For a meta-analysis to give definitive information, it should meet at least the minimum standards that would be expected of a well-designed, adequately powered, and carefully conducted randomised controlled trial. These minimum standards include both qualitative characteristics--a prospective protocol, comparable definitions of key outcomes, quality control of data, and inclusion of all patients from all trials in the final analysis--and quantitative standards--an assessment of whether the total sample is large enough to provide reliable results and the use of appropriate statistical monitoring guidelines to indicate when the results of the accumulating data of a meta-analysis are conclusive. We believe that rigorous meta-analyses undertaken according to these principles will lead to more reliable evidence about the efficacy and safety of interventions than either retrospective meta-analysis or individual trials.     

Review of developments in meta-analytic method.

Presents a brief history of the development of meta-analysis, which is quickly establishing itself as a useful tool for the social sciences. Five approaches to meta-analytic method are distinguished: Glassian meta-analysis, study effect meta-analysis, the combined probability method, approximate data pooling with tests of homogeneity, and approximate data pooling with sampling error correction. It is concluded that the systematic differences in meta-analysis should not be taken as evidence of some inherent weakness of meta-analysis but as a reflection of the natural evolution of a new scientific tool. (80 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The five factor model of personality and job performance in the European Community.

In 3 prior meta-analyses, the relationship between the Big Five factors of personality and job criteria was investigated. However, these meta-analyses showed different findings. Furthermore, these reviews included studies carried out only in the United States and Canada. This study reports meta-analytic research on the same topic but with studies conducted in the European Community, which were not included in the prior reviews. The results indicate that Conscientiousness and Emotional Stability are valid predictors across job criteria and occupational groups. The remaining factors are valid only for some criteria and for some occupational groups. Extraversion was a predictor for 2 occupations, and Openness and Agreeableness were valid predictors of training proficiency. These findings are consistent with M. R. Barrick and M. K. Mount (see record 1991-22928-001) and L. M. Hough et al (see record 1991-06268-001). Implications of the results for future research and the practice of personnel selection are suggested. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cyclophosphamide pulse therapy for refractory rheumatic diseases

Meta-analysis was proposed more than 20 years ago as an innovative technique for pooling the results of a series of clinical studies. This technique has a two-fold purpose: i) to retrieve pertinent clinical information without introducing selection biases; ii) to combine these clinical data obtained from different sources into a single synthetic index that summarises all available information. This review highlights the current perspectives in the application of meta-analysis and places particular emphasis on discussing the limitations of this technique that have been identified in recent years. Specific controversies in this area include: i) comparison between meta-analysis of the literature and meta-analysis of individual patient data; ii) heterogeneity of the clinical material introduced in the meta-analytic pooling; and iii) survival meta-analysis as a method for pooling long-term outcome data. After debating these three controversial points, this review examines the newest uses of meta-analysis that have been proposed for the evaluation of cost-effectiveness data and tries to identify the meta-analytic applications that will probably be expanded in the near future as opposed to those that will instead be developed less and less.     

Meta-analytic interval estimation for standardized and unstandardized mean differences.

The fixed-effects (FE) meta-analytic confidence intervals for unstandardized and standardized mean differences are based on an unrealistic assumption of effect-size homogeneity and perform poorly when this assumption is violated. The random-effects (RE) meta-analytic confidence intervals are based on an unrealistic assumption that the selected studies represent a random sample from a large superpopulation of studies. The RE approach cannot be justified in typical meta-analysis applications in which studies are nonrandomly selected. New FE meta-analytic confidence intervals for unstandardized and standardized mean differences are proposed that are easy to compute and perform properly under effect-size heterogeneity and nonrandomly selected studies. The proposed meta-analytic confidence intervals may be used to combine unstandardized or standardized mean differences from studies having either independent samples or dependent samples and may also be used to integrate results from previous studies into a new study. An alternative approach to assessing effect-size heterogeneity is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of a new outlier statistic for meta-analytic data.

This article describes the development of a new technique for identifying outlier coefficients in meta-analytic data sets. Denoted as the sample-adjusted meta-analytic deviancy statistic or SAMD, this technique takes into account the sample size on which each study is based when determining outlier status. An empirical test of the SAMD statistic with an actual meta-analytic data set resulted in a substantial reduction in residual variabilities and a corresponding increase in the percentage of variance accounted for by statistical artifacts after removal of outlier study coefficients. Moreover, removal of these coefficients helped to clarify what was a confusing and difficult-to-explain finding in this meta-analysis. It is suggested that analysis for outliers become a routine part of meta-analysis methodology. Limitations and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)