The Persistence of Underpowered Studies in Psychological Research: Causes, Consequences, and Remedies.

Underpowered studies persist in the psychological literature. This article examines reasons for their persistence and the effects on efforts to create a cumulative science. The "curse of multiplicities" plays a central role in the presentation. Most psychologists realize that testing multiple hypotheses in a single study affects the Type I error rate, but corresponding implications for power have largely been ignored. The presence of multiple hypothesis tests leads to 3 different conceptualizations of power. Implications of these 3 conceptualizations are discussed from the perspective of the individual researcher and from the perspective of developing a coherent literature. Supplementing significance tests with effect size measures and confidence intervals is shown to address some but not necessarily all problems associated with multiple testing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validation and statistical power: Implications for applied research.

Comments that criterion-related validity studies are often not technically feasible because sample sizes are inadequate for necessary statistical power (e.g., .90). Effect sizes are frequently overestimated because of a failure to consider the combined effects of range restriction and criterion unreliability, both of which attenuate validity coefficients. Restricted validities must therefore be estimated by applying appropriate correction formulas. In this study the corrections are made for the multiple prediction case. Required sample sizes, determined using the univariate power model, are presented for a range of unit-weighted predictors, for varying degrees of restriction, and for power levels of .50 and .90. The advantage of multiple predictors is shown by comparing their required sample size to that of the best single predictor. For a given power, effect size is clearly the major determinant of required sample size. Implications for applied research are discussed. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple comparison procedures within experimental research.

Discusses 2 approaches in the use of multiple comparison procedures. One is that of the practical-research-oriented investigator, who emphasizes the importance of the Type II error and associated power. The other is exemplified by mathematical statisticians who emphasize "pure" mathematical aspects and concentrate on the importance of controlling Type I errors. Two important issues are discussed: emphasis on Type I errors within single experiments vs emphasis on Type II errors within multiple experiments. (French abstract) (36 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The pairwise multiple comparison multiplicity problem: An alternative approach to familywise and comparison wise Type I error control.

When undertaking many tests of significance, researchers are faced with the problem of how best to control the probability of committing a Type I error. The familywise approach deals directly with multiplicity problems by setting a level of significance for an entire set of related hypotheses; the comparison approach ignores the issue by setting the rate of error on each individual hypothesis. A new formulation of control, the false discovery rate, does not provide control as stringent as that of the familywise rate, but concomitant with this relaxation in stringency is an increase in sensitivity to detect effects relative to the sensitivity of familywise control. Type I error and power rates for 4 relatively powerful and easily computed pairwise multiple comparison procedures were compared with the false discovery rate procedure for various 1-way layouts by use of test statistics that do not assume variance homogeneity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Analysis of variance with ipsative measures.

Measures with more than 1 score per participant, when the total for each participant equals the same constant, are said to be ipsative. Ipsativity occurs when data are percentages, with each participant's total equal to 100%, or when data are ranks, with each participant's total equal to the sum of the ranks. When ipsative measures are analyzed with analysis of variance (ANOVA), certain sums of squares equal 0, and the average intercorrelation among measures is negative. These characteristics of ipsativity may result in violations of ANOVA assumptions, producing an inflated Type I error rate and affecting power. The purpose of this Monte Carlo study was to empirically examine the extent to which ANOVA is affected by ipsative data. Findings indicated that, with few exceptions, ANOVA worked quite well with ipsative data. Not only were Type I error rates well preserved, but power was nearly equivalent to that with nonipsative data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A confidence interval for ordinal multiple regression weights.

A confidence interval (CI) for a population predictor weight for use with N. Cliff's (1994) method of ordinal multiple regression (OMR) is presented. The OMR CI is based on an estimated standard error of a weight derived from a fixed-effects model. A simulation was performed to examine the sampling properties of the OMR CI. The results show that the OMR CI had good Type I error rate and coverage. The OMR CI had lower power than the least-squares multiple regression (LSMR) CI when predictors were not correlated but had higher power when predictor correlations were moderate to high. In addition to discussing the simulation results, it is pointed out that the OMR CI can have superior sampling properties when the fixed-effects assumptions are violated. The OMR CI is recommended when a researcher wants to consider only ordinal information in multivariate prediction, when predictor correlations are moderate to high, and when the assumptions of fixed-effects LSMR are violated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Hypothesis testing with multiple outcomes requires adjustments to control Type I error inflation, which reduces power to detect significant differences. Maintaining the prechosen Type I error level is challenging when outcomes are correlated. This problem concerns many research areas, including neuropsychological research in which multiple, interrelated assessment measures are common. Standard p value adjustment methods include Bonferroni-, Sidak-, and resampling-class methods. In this report, the authors aimed to develop a multiple hypothesis testing strategy to maximize power while controlling Type I error. The authors conducted a sensitivity analysis, using a neuropsychological dataset, to offer a relative comparison of the methods and a simulation study to compare the robustness of the methods with respect to varying patterns and magnitudes of correlation between outcomes. The results lead them to recommend the Hochberg and Hommel methods (step-up modifications of the Bonferroni method) for mildly correlated outcomes and the step-down minP method (a resampling-based method) for highly correlated outcomes. The authors note caveats regarding the implementation of these methods using available software. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing hypotheses about ordinal interactions: Simulations and further comments.

This paper elaborates on several issues related to testing for the presence of ordinal interactions, as described by Bobko (1986). First, the philosophy underlying Bobko's approach is explicitly stated and compared with the traditional approach to testing for the presence of interactions. Second, two modifications of Bobko's approach are described. Third, the procedures for testing ordinal interactions are compared (on the basis of Type I and Type II error rates) with each other as well as to the traditional analysis of variance (ANOVA) approach. All variants of Bobko's procedure have comparable power across different sample sizes and experimental effect sizes. These procedures differ, however, in their likelihood of falsely concluding that an ordinal pattern is present. The traditional ANOVA approach (a) is noticeably lacking in power for detecting ordinal interactions and (b) commonly identifies significant main effects but not an interaction when, in fact, an ordinal interaction is present in the population. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using Lancaster's mid-P correction to the Fisher's exact test for adverse impact analyses.

Adverse impact is often assessed by evaluating whether the success rates for 2 groups on a selection procedure are significantly different. Although various statistical methods have been used to analyze adverse impact data, Fisher's exact test (FET) has been widely adopted, especially when sample sizes are small. In recent years, however, the statistical field has expressed concern regarding the default use of the FET and has proposed several alternative tests. This article reviews Lancaster's mid-P (LMP) test (Lancaster, 1961), an adjustment to the FET that tends to have increased power while maintaining a Type I error rate close to the nominal level. On the basis of Monte Carlo simulation results, the LMP test was found to outperform the FET across a wide range of conditions typical of adverse impact analyses. The LMP test was also found to provide better control over Type I errors than the large-sample Z-test when sample size was very small, but it tended to have slightly lower power than the Z-test under some conditions. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Advances in testing the statistical significance of mediation effects.

P. A. Frazier, A. P. Tix, and K. E. Barron (2004) highlighted a normal theory method popularized by R. M. Baron and D. A. Kenny (1986) for testing the statistical significance of indirect effects (i.e., mediator variables) in multiple regression contexts. However, simulation studies suggest that this method lacks statistical power relative to some other approaches. The authors describe an alternative developed by P. E. Shrout and N. Bolger (2002) based on bootstrap resampling methods. An example and step-by-step guide for performing bootstrap mediation analyses are provided. The test of joint significance is also briefly described as an alternative to both the normal theory and bootstrap methods. The relative advantages and disadvantages of each approach in terms of precision in estimating confidence intervals of indirect effects, Type I error, and Type II error are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A test of four theories of coalition formation: Effects of payoffs and resources.

Used 3 coalition games to test the minimum resource, minimum power, and bargaining theories against each other and against equal excess theory; 144 male undergraduates were Ss. In Game X, all winning coalitions had the same payoffs but players had different resources; in Game Y, winning coalitions had different payoffs and players had different resources. The characteristic functions of the games (payoffs for the coalitions) were the same for Games Y and Z, and the resource distributions were the same for Games X and Z. Coalition behavior was virtually the same in Games Y and Z, but coalition behavior in these games differed from that in Game X. Thus, when there were differences in both coalition payoffs and individual resources, the payoffs rather than the resources tended to influence coalition behavior. Coalition behavior in Games Y and Z was best accounted for by equal excess theory, coalition behavior in Game X by bargaining theory. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for adverse impact when sample size is small.

Adverse impact evaluations often call for evidence that the disparity between groups in selection rates is statistically significant, and practitioners must choose which test statistic to apply in this situation. To identify the most effective testing procedure, the authors compared several alternate test statistics in terms of Type I error rates and power, focusing on situations with small samples. Significance testing was found to be of limited value because of low power for all tests. Among the alternate test statistics, the widely-used Z-test on the difference between two proportions performed reasonably well, except when sample size was extremely small. A test suggested by G. J. G. Upton (1982) provided slightly better control of Type I error under some conditions but generally produced results similar to the Z-test. Use of the Fisher Exact Test and Yates's continuity-corrected chi-square test are not recommended because of overly conservative Type I error rates and substantially lower power than the Z-test. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Structural Equation Models of Latent Interactions: Evaluation of Alternative Estimation Strategies and Indicator Construction.

Interactions between (multiple indicator) latent variables are rarely used because of implementation complexity and competing strategies. Based on 4 simulation studies, the traditional constrained approach performed more poorly than did 3 new approaches-unconstrained, generalized appended product indicator, and quasi-maximum-likelihood (QML). The authors' new unconstrained approach was easiest to apply. All 4 approaches were relatively unbiased for normally distributed indicators, but the constrained and QML approaches were more biased for nonnormal data; the size and direction of the bias varied with the distribution but not with the sample size. QML had more power, but this advantage was qualified by consistently higher Type I error rates. The authors also compared general strategies for defining product indicators to represent the latent interaction factor. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Mixed-model pairwise multiple comparisons of repeated measures means.

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of the data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach was evaluated for testing all possible pairwise differences among repeated measures marginal means in a Between-Subjects?×?Within-Subjects design. Specifically, the authors investigated Type I error and power rates for a number of simultaneous and stepwise multiple comparison procedures using SAS (1999) PROC MIXED in unbalanced designs when normality and covariance homogeneity assumptions did not hold. J. P. Shaffer's (1986) sequentially rejective step-down and Y. Hochberg's (1988) sequentially acceptive step-up Bonferroni procedures, based on an unstructured covariance structure, had superior Type I error control and power to detect true pairwise differences across the investigated conditions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bivariate median splits and spurious statistical significance.

Despite pleas from methodologists, researchers often continue to dichotomize continuous predictor variables. The primary argument against this practice has been that it underestimates the strength of relationships and reduces statistical power. Although this argument is correct for relationships involving a single predictor, a different problem can arise when multiple predictors are involved. Specifically, dichotomizing 2 continuous independent variables can lead to false statistical significance. As a result, the typical justification for using a median split as long as results continue to be statistically significant is invalid, because such results may in fact be spurious. Thus, researchers who dichotomize multiple continuous predictor variables not only may lose power to detect true predictor–criterion relationships in some situations but also may dramatically increase the probability of Type I errors in other situations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Closed reduction of a rare type III dislocation of the first metatarsophalangeal joint

OBJECTIVE: To discuss a rare Type III dislocation of the first metatarsophalangeal (MP) joint, without fracture, that used a closed reduction technique for correction. CLINICAL FEATURES: A 43-yr-old man suffered from an acute severe dislocation of his great toe as the result of acute forceful motion applied to the toe as his foot was depressed onto a brake pedal to avoid a motor vehicle accident. Physical examination and X-rays revealed the dislocation, muscle spasm, edema and severely restricted range of motion. INTERVENTION AND OUTCOME: The dislocation was corrected using a closed reduction technique, in this case a chiropractic manipulation. Fourteen months after reduction, the joint was intact, muscle strength was graded +5 normal, ranges of motion were within normal limits and no crepitation was noted. X-rays revealed normal intact joint congruency. The patient experienced full weight bearing, range of motion and function of the joint. CONCLUSION: Although a Type III dislocation of the great toe has only once been cited briefly in the literature, this classification carries a recommended surgical treatment protocol for correction. No literature describes a closed reduction of a Type III dislocation as described in this case report. It is apparent that a closed reduction technique using a chiropractic manipulation may be considered a valid alternative correction technique for Type III dislocations of the great toe.     

Postperformance attributions and task persistence among Type A and B individuals: A clarification.

Tested the hypothesis that Type A (coronary-prone) Ss would be more self-serving than Type B (noncoronary-prone) Ss in their attributions for success and failure. It was also hypothesized that task persistence would differ among Type A's and B's and would be dependent on task difficulty and perceived task diagnosticity. 78 undergraduates classified on the basis of scores on the Jenkins Activity Survey as Type A's and B's attempted multiple sets of anagrams that were either easy or difficult. Persistence was measured by the number of anagram sets attempted, and, after task performance, attributions for success and failure were assessed. Results support both hypotheses. Type A's took more credit for success than for failure, whereas Type B's did not provide reliably different attributions for success and failure. Furthermore, Type A's persisted longer at the task when it was difficult and when it was viewed as relatively low in information value. Type B's persisted longer at the task when it was difficult but viewed as relatively high in information value. Results are discussed in the context of current debates regarding the responses of Type A's and B's to performance settings. (47 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Improved control of smelting in blast furnaces

Attention focuses on the highly efficient heat and mass transfer in the counterflow of batch and hot reducing gases, which determines the duration of the blast-furnace process and increases the unit power of the furnace. On that basis, the control of the internal furnace processes may be reappraised. Improvement in control systems for local processes is proposed, by establishing internal relations with the fast-moving input parameters of the blast-furnace process. With specified batch and fuel conditions, the mixed hot blast may be regarded as the determining high-speed factor, which is functionally related to all the physicochemical transformations between the batch, the fuel, the hot reducing gases, and the blast-furnace gas. A mathematical apparatus is developed to describe the relation between the local transformations, whose use in regulating the processes through the mixed hot blast improves blast-furnace regulation and practically eliminates time lost in transport delays. Correspondingly, energy losses may be reduced on account of timely correction of local processes over the whole working space of the furnace.     

Directional decisions for two-tailed tests: Power, error rates, and sample size.

One imposing directional decisions on nondirectional tests will overestimate power, underestimate sample size, and ignore the risk of Type III error (getting the direction wrong) if traditional calculations—those applying to nondirectional decisions—are used. Usually trivial with the z test, the errors might be important where α is large and effect size is small or with tests using other distributions. One can avoid the errors by using calculations that apply to directional decisions or by using a directional two-tailed test at the outset, a conceptually simpler solution. With a revised concept of power, this article shows calculations for the test; explains how to find its power, Type III error risk, and sample size in statistical tables for traditional tests; compares it to conventional one- and two-tailed tests and to one- and two-sided confidence intervals; and concludes that when a significance test is planned it is the best choice for most purposes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)