Meta-analytic procedures for combining studies with multiple effect sizes.

Presents a general set of meta-analytic procedures for combining and comparing research results from studies yielding multiple effect sizes based on multiple dependent variables. These require, in addition to the individual effect sizes or significance levels, only the degrees of freedom in the study and the typical intercorrelation among the variables. Older methods are reviewed, and a new method for obtaining a single summary effect size estimate from multiple effect sizes is presented. Significant testing of this summary effect size estimate is described. Procedures for computing the effect size for a contrast, and its significance level, among the multiple effect sizes of a single study are also described. Methods for dealing with problems of heterogeneous intercorrelations among the dependent variables are presented. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing hypotheses about ordinal interactions: Simulations and further comments.

This paper elaborates on several issues related to testing for the presence of ordinal interactions, as described by Bobko (1986). First, the philosophy underlying Bobko's approach is explicitly stated and compared with the traditional approach to testing for the presence of interactions. Second, two modifications of Bobko's approach are described. Third, the procedures for testing ordinal interactions are compared (on the basis of Type I and Type II error rates) with each other as well as to the traditional analysis of variance (ANOVA) approach. All variants of Bobko's procedure have comparable power across different sample sizes and experimental effect sizes. These procedures differ, however, in their likelihood of falsely concluding that an ordinal pattern is present. The traditional ANOVA approach (a) is noticeably lacking in power for detecting ordinal interactions and (b) commonly identifies significant main effects but not an interaction when, in fact, an ordinal interaction is present in the population. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Confidence intervals for the overall effect size in random-effects meta-analysis.

One of the main objectives in meta-analysis is to estimate the overall effect size by calculating a confidence interval (CI). The usual procedure consists of assuming a standard normal distribution and a sampling variance defined as the inverse of the sum of the estimated weights of the effect sizes. But this procedure does not take into account the uncertainty due to the fact that the heterogeneity variance (τ2) and the within-study variances have to be estimated, leading to CIs that are too narrow with the consequence that the actual coverage probability is smaller than the nominal confidence level. In this article, the performances of 3 alternatives to the standard CI procedure are examined under a random-effects model and 8 different τ2 estimators to estimate the weights: the t distribution CI, the weighted variance CI (with an improved variance), and the quantile approximation method (recently proposed). The results of a Monte Carlo simulation showed that the weighted variance CI outperformed the other methods regardless of the τ2 estimator, the value of τ2, the number of studies, and the sample size. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impact of experimental design on effect size: Findings from the research literature on training.

This article examines whether differences in the equations commonly used to calculate effect size for single group pretest-posttest (SGPP) designs versus those for control group designs can account for the finding that SGPP designs yield larger mean effect sizes (e.g., M. S. Lipsey & D. B. Wilson, 1993). It was found that the assumptions of no control group effect and the equivalence of pretraining and posttraining dependent variable standard deviations required for these equations to produce equivalent estimates of effect size were violated for some dependent variable types. Results indicate that control group effects and inflation in the standard deviation of the posttraining dependent variable measure account for most of the observed difference in effect size. The most severe violations occurred when the dependent variable was a knowledge assessment. Methods for including data from SGPP designs in meta-analyses that minimize potential biases are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Repeated measurements of dichotomous variables: Q and F tests.

Examined 4 procedures for analyzing 0–2 data in repeated measurements designs for various combinations of sample size, number of treatments, and degree of heterogeneity of covariance. The test statistics were Cochran's Q test, the univariate F test, and Q and F statistics adjusted for heterogeneous covariances. Type I and Type II error rates based on computer simulations indicated problems with sample sizes of less than 16; for larger samples, Q+ and F+ gave honest Type I error rates, even under conditions of extreme heterogeneity of covariance. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Two-sample T–2 procedure and the assumption of homogeneous covariance matrices.

Reports on an empirical investigation of the robustness of H. Hotelling's (1931) 2-sample T–2 test with respect to violation of the assumption of homogeneity of covariance matrices. In a Monte Carlo study, empirical sampling distributions of the T–2 statistic were obtained from a large number of sets, each consisting of 2,000 samples drawn from multivariate normal parent populations. Average sample size (n), extent of inequality of sample sizes, number of variables (p), and degree of inequality of covariance matrices were combined to 108 different conditions. Actual proportions of values that exceeded nominal α levels are presented. For equal ns, the procedure was found to be generally robust. With unequal ns, the procedure is shown to become increasingly less robust as covariance matrix heterogeneity and p increase. The results are related to earlier findings, and implications for the proper use of the T–2 procedure are noted. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distinguishing between moderator and quadratic effects in multiple regression.

Moderated regression analysis is commonly used to test for multiplicative influences of independent variables in regression models. D. Lubinski and L. G. Humphreys (1990) have shown that significant moderator effects can exist even when stronger quadratic effects are present. They recommend comparing effect sizes associated with both effect types and selecting the model that yields the strongest effect. The authors show that this procedure of comparing effect sizes is biased in favor of the moderated model when multicollinearity is high because of the differential reliability of the quadratic and multiplicative terms in the regression models. Fortunately, levels of multicollinearity under which this bias is most problematic may be outside the range encountered in many empirical studies. The authors discuss causes and implications of this phenomenon as well as alternative procedures for evaluating structural relationships among variables. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

How significant is a significant difference? Problems with the measurement of magnitude of effect.

The use of measures of magnitude of effect (MOE) has been advocated as a way to go beyond statistical tests of significance and to identify effects of a practical size and MOE has been used in meta-analysis to combine results of different studies. Some problems associated with measures of MOE are described, and implications for researchers are discussed. It is demonstrated that one commonly used measure of MOE is heavily dependent on a study's number of treatments and sample size. Such a measure should be used with caution for comparing results of studies of different sizes. (31 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analytic interval estimation for standardized and unstandardized mean differences.

The fixed-effects (FE) meta-analytic confidence intervals for unstandardized and standardized mean differences are based on an unrealistic assumption of effect-size homogeneity and perform poorly when this assumption is violated. The random-effects (RE) meta-analytic confidence intervals are based on an unrealistic assumption that the selected studies represent a random sample from a large superpopulation of studies. The RE approach cannot be justified in typical meta-analysis applications in which studies are nonrandomly selected. New FE meta-analytic confidence intervals for unstandardized and standardized mean differences are proposed that are easy to compute and perform properly under effect-size heterogeneity and nonrandomly selected studies. The proposed meta-analytic confidence intervals may be used to combine unstandardized or standardized mean differences from studies having either independent samples or dependent samples and may also be used to integrate results from previous studies into a new study. An alternative approach to assessing effect-size heterogeneity is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further meta-analytic procedures for assessing cognitive gender differences.

Outlined procedures for assessing the heterogeneity of a set of effect sizes derived from a meta-analysis, testing for trends with contrasts among the effect sizes obtained, and evaluating the practical importance of the average effect size obtained. These procedures were applied to data presented by J. S. Hyde (1981) regarding cognitive gender differences. The authors conclude that (a) for all 4 areas of cognitive skill investigated, effect sizes for gender differences differed significantly across studies; (b) recent studies of gender differences show a substantial gain in cognitive performance by females relative to males; and (c) studies of gender differences show male vs female effect sizes of practical importance equivalent to outcome rates of 60 vs 40%. (6 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fixed- and random-effects models in meta-analysis.

There are 2 families of statistical procedures in meta-analysis: fixed- and random-effects procedures. They were developed for somewhat different inference goals: making inferences about the effect parameters in the studies that have been observed versus making inferences about the distribution of effect parameters in a population of studies from a random sample of studies. The authors evaluate the performance of confidence intervals and hypothesis tests when each type of statistical procedure is used for each type of inference and confirm that each procedure is best for making the kind of inference for which it was designed. Conditionally random-effects procedures (a hybrid type) are shown to have properties in between those of fixed- and random-effects procedures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Controlling for stimulus dominance in dichotic listening tests: A modification of !l.

Dichotic listening procedures have been used to assess cerebral lateralization in normal Ss. One particularly useful technique is the use of stimuli that fuse into a single percept. Although this procedure has many advantages over other dichotic listening methods, it is particularly susceptible to stimulus dominance, which acts as noise in a S's response data, thus reducing the power of any statistical test of the ear advantage. It is proposed that the solution to this problem is a log-linear analysis of the response data to yield a λ-type index (λ*) that is a measure of ear dominance independent of stimulus dominance. Details of the analysis are provided, as well as a sample analysis of data collected from 104 right-handed and 30 left-handed Ss. Comparisons are drawn between the log-linear analysis and other methods that have been proposed to control for stimulus dominance in this single-response dichotic fusion procedure. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Choosing the number of clusters in Κ-means clustering.

Steinley (2007) provided a lower bound for the sum-of-squares error criterion function used in K-means clustering. In this article, on the basis of the lower bound, the authors propose a method to distinguish between 1 cluster (i.e., a single distribution) versus more than 1 cluster. Additionally, conditional on indicating there are multiple clusters, the procedure is extended to determine the number of clusters. Through a series of simulations, the proposed methodology is shown to outperform several other commonly used procedures for determining both the presence of clusters and their number. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

A procedure for combining sample correlation coefficients and vote counts to obtain an estimate and a confidence interval for the population correlation coefficient.

Missing effect-size estimates pose a particularly difficult problem in meta-analysis. Rather than discarding studies with missing effect-size estimates or setting missing effect-size estimates equal to 0, the meta-analyst can supplement effect-size procedures with vote-counting procedures if the studies report the direction of results or the statistical significance of results. By combining effect-size and vote-counting procedures, the meta-analyst can obtain a less biased estimate of the population effect size and a narrower confidence interval for the population effect size. This article describes 3 vote-counting procedures for estimating the population correlation coefficient in studies with missing sample correlations. Easy-to-use tables, based on equal sample sizes, are presented for the 3 procedures. More complicated vote-counting procedures also are given for unequal sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Estimation of effect size from a series of independent experiments.

Extends statistical theory for procedures based on the Glass estimator of effect size for methods used in the quantitative synthesis of research. An unbiased estimator of effect size is given. A weighted estimator of effect size based on data from several experiments is defined and shown to be optimal (asymptotically efficient). An approximate (large-sample) test for homogeneity of effect size across experiments is also given. The results of an empirical sampling study show that the large-sample distributions of the weighted estimator and the homogeneity statistic are quite accurate when the experimental and control group sample sizes exceed 10 and the effect sizes are smaller than about 1.5. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fully Bayesian estimation of Gibbs hyperparameters for emission computed tomography data

In recent years, many investigators have proposed Gibbs prior models to regularize images reconstructed from emission computed tomography data. Unfortunately, hyperparameters used to specify Gibbs priors can greatly influence the degree of regularity imposed by such priors and, as a result, numerous procedures have been proposed to estimate hyperparameter values from observed image data. Many of these procedures attempt to maximize the joint posterior distribution on the image scene. To implement these methods, approximations to the joint posterior densities are required, because the dependence of the Gibbs partition function on the hyperparameter values is unknown. In this paper, we use recent results in Markov chain Monte Carlo (MCMC) sampling to estimate the relative values of Gibbs partition functions and using these values, sample from joint posterior distributions on image scenes. This allows for a fully Bayesian procedure which does not fix the hyperparameters at some estimated or specified value, but enables uncertainty about these values to be propagated through to the estimated intensities. We utilize realizations from the posterior distribution for determining credible regions for the intensity of the emission source. We consider two different Markov random field (MRF) models-the power model and a line-site model. As applications we estimate the posterior distribution of source intensities from computer simulated data as well as data collected from a physical single photon emission computed tomography (SPECT) phantom.     

Experience and Materials Affect Combinatorial Construction in Tufted Capuchin Monkeys (Cebus apella).

Three movement procedures can combine nesting cups into seriated structures. Reliance on these procedures changes with age in human children, and the putatively most advanced emerges as a predominant procedure at 3 or more years. Six monkeys' (Cebus apella) combinatorial procedures and successes at nesting seriated cups were evaluated. The current study examined whether the procedures used (a) shift toward more efficient procedures after unguided experience, (b) are dependent on the type of object being combined, and (c) can be altered by specific training history. All factors produced a change in procedure for some individuals, suggesting that combinatorial procedure is a product of the dynamic influences of preexisting tendencies to act in certain ways, of environmental circumstances, and of prior experiences. Some monkeys preferred the putatively most cognitively complex procedure. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparing squared multiple correlation coefficients: Examination of a confidence interval and a test significance.

I. Olkin and J. D. Finn (1995) presented 2 methods for comparing squared multiple correlation coefficients for 2 independent samples. In 1 method, the researcher constructs a confidence interval for the difference between 2 population squared coefficients; in the 2nd method, a Fisher-type transformation of the sample squared correlation coefficient is used to obtain a test statistic. Both methods are based on asymptotic theory and use approximations to the sampling variance. The approximations are incorrect when the population multiple correlation coefficient is zero. The 2 procedures were examined for equal and unequal population multiple correlation coefficients in combination with equal and unequal sample sizes. As expected, the procedures were inaccurate when the population multiple correlation coefficients were zero or very small and, in some conditions, were inaccurate when sample sizes and coefficients were unequal. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparing effect sizes of independent studies.

Procedures for comparing the effect sizes of 2 or more independent studies include a method for calculating the approximate significance level for the heterogeneity of effect sizes of studies and a method for calculating the approximate significance level of a contrast among the effect sizes. Although the focus is on effect size as measured by the standardized difference between the means, the procedures can be applied to any measure of effect size having an estimated variance. (10 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A random effects model for effect sizes.

Recent interest in quantitative research synthesis has led to the development of rigorous statistical theory for some of the methods used in meta-analysis. Statistical theory proposed previously has stressed the estimation of fixed but unknown population effect sizes (standardized mean differences). Theoretical considerations often suggest that treatment effects are not fixed but vary across different implementations of a treatment. The present author presents a random effects model (analogous to random effects ANOVA) in which the population effect sizes are not fixed but are sample realizations from a distribution of possible population effect sizes. An analogy to variance component estimation is used to derive an unbiased estimator of the variance of the effect-size distribution. An example shows that these methods may suggest insights that are not available from inspection of means and standard deviation of effect-size estimates. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)