A Monte Carlo evaluation of Bobko's ordinal interaction analysis technique.

In a two-factor design, interactions are typically analyzed by analysis of variance (ANOVA). Bobko (1986) has suggested an alternative ordinal-interaction technique that might avoid spurious main effects and show more power than the classical ANOVA. In this study I (a) compared the classical and ordinal techniques in terms of Type I error rate and power under normally distributed homogeneous and heterogeneous populations, and (b) determined the effect of a population non-null main effect on the ordinal technique's Type I error rate. The ordinal technique showed a substantial power superiority over the classical technique under variance homogeneity, although it did have a power cap of less than 100%. Its Type I error rate under variance heterogeneity, however, was not stable and was susceptible to non-null main effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing hypotheses about ordinal interactions: Simulations and further comments.

This paper elaborates on several issues related to testing for the presence of ordinal interactions, as described by Bobko (1986). First, the philosophy underlying Bobko's approach is explicitly stated and compared with the traditional approach to testing for the presence of interactions. Second, two modifications of Bobko's approach are described. Third, the procedures for testing ordinal interactions are compared (on the basis of Type I and Type II error rates) with each other as well as to the traditional analysis of variance (ANOVA) approach. All variants of Bobko's procedure have comparable power across different sample sizes and experimental effect sizes. These procedures differ, however, in their likelihood of falsely concluding that an ordinal pattern is present. The traditional ANOVA approach (a) is noticeably lacking in power for detecting ordinal interactions and (b) commonly identifies significant main effects but not an interaction when, in fact, an ordinal interaction is present in the population. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Protecting the overall rate of Type I errors for pairwise comparisons with an omnibus test statistic.

Compares 2 procedures for protecting the number of false rejections for a set of all possible pairwise comparisons. The 2-stage strategy of computing pairwise comparisons, conditional on a significant omnibus test, is compared with the multiple comparison strategy that sets a "familywise" critical value directly. The ANOVA test, the Brown and Forsythe test, and the Welch omnibus test, as well as 3 procedures for assessing the significance of pairwise comparisons, are combined into 9 2-stage testing strategies. The data from this study establish that the common strategy of following a significant ANOVA F with Student's t tests on pairs of means results in a substantially inflated rate of Type I error when variances are heterogeneous. Type I error control, however, can be obtained with other 2-stage procedures, and the authors tentatively consider the Welch F″ Welch t″ combination desirable. In addition, the 2 techniques for controlling Type I error do not substantially differ as much as might be expected; some 2-stage procedures are comparable to simultaneous techniques. (18 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Mixed-model pairwise multiple comparisons of repeated measures means.

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of the data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach was evaluated for testing all possible pairwise differences among repeated measures marginal means in a Between-Subjects?×?Within-Subjects design. Specifically, the authors investigated Type I error and power rates for a number of simultaneous and stepwise multiple comparison procedures using SAS (1999) PROC MIXED in unbalanced designs when normality and covariance homogeneity assumptions did not hold. J. P. Shaffer's (1986) sequentially rejective step-down and Y. Hochberg's (1988) sequentially acceptive step-up Bonferroni procedures, based on an unstructured covariance structure, had superior Type I error control and power to detect true pairwise differences across the investigated conditions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of ANOVA alternatives under variance heterogeneity and specific noncentrality structures.

A Monte Carlo study compared the Type I error properties and power of 4 commonly recommended analysis of variance (ANOVA) alternatives for testing mean differences under variance heterogeneity that were developed by B. L. Welch (1951), M. B. Brown and A. B. Forsythe (1974), W. H. Kruskal and W. A. Wallis (1952), and B. L. van der Waerden (1952). On the basis of superior control of Type I errors and greater power, the Welch test proved to be the procedure of choice when means were equally spaced, when extreme means were paired with small variances, and when 2 identical means were situated midway between 2 extreme means. When extreme means were paired with large variances, the Brown and Forsythe test was optimal, though less clearly so. (42 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for adverse impact when sample size is small.

Adverse impact evaluations often call for evidence that the disparity between groups in selection rates is statistically significant, and practitioners must choose which test statistic to apply in this situation. To identify the most effective testing procedure, the authors compared several alternate test statistics in terms of Type I error rates and power, focusing on situations with small samples. Significance testing was found to be of limited value because of low power for all tests. Among the alternate test statistics, the widely-used Z-test on the difference between two proportions performed reasonably well, except when sample size was extremely small. A test suggested by G. J. G. Upton (1982) provided slightly better control of Type I error under some conditions but generally produced results similar to the Z-test. Use of the Fisher Exact Test and Yates's continuity-corrected chi-square test are not recommended because of overly conservative Type I error rates and substantially lower power than the Z-test. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Selection of pairwise multiple comparison procedures for parametric and nonparametric analysis of variance models.

Discusses 2 issues that are often overlooked in the evaluation of pairwise multiple comparison procedures (MCPs): Comparisons of powers for competing MCPs are not useful unless the procedures have identical nominal experimentwise error rates; and "protected" MCPs do not control the experimentwise Type I error rate at the nominal alpha except in the complete null case. Also, procedures based on the ranks of all K groups do not test the same hypothesis as MCPs that involve reranking the data for each pairwise comparison. (41 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Hypothesis testing with multiple outcomes requires adjustments to control Type I error inflation, which reduces power to detect significant differences. Maintaining the prechosen Type I error level is challenging when outcomes are correlated. This problem concerns many research areas, including neuropsychological research in which multiple, interrelated assessment measures are common. Standard p value adjustment methods include Bonferroni-, Sidak-, and resampling-class methods. In this report, the authors aimed to develop a multiple hypothesis testing strategy to maximize power while controlling Type I error. The authors conducted a sensitivity analysis, using a neuropsychological dataset, to offer a relative comparison of the methods and a simulation study to compare the robustness of the methods with respect to varying patterns and magnitudes of correlation between outcomes. The results lead them to recommend the Hochberg and Hommel methods (step-up modifications of the Bonferroni method) for mildly correlated outcomes and the step-down minP method (a resampling-based method) for highly correlated outcomes. The authors note caveats regarding the implementation of these methods using available software. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A more realistic look at the robustness and Type II error properties of the t test to departures from population normality.

The Type I and II error properties of the t test were evaluated by means of a Monte Carlo study that sampled 8 real distribution shapes identified by T. Micceri (1986, 1989) as being representative of types encountered in psychology and education research. Results showed the independent-samples t tests to be reasonably robust to Type I error when (1) sample sizes are equal, (2) sample sizes are fairly large, and (3) tests are 2-tailed rather than 1-tailed. Nonrobust results were obtained primarily under distributions with extreme skew. The t test was robust to Type II error under these nonnormal distributions, but researchers should not overlook robust nonparametric competitors that are often more powerful than the t test when its underlying assumptions are violated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Transformations for within-subject designs: A Monte Carlo investigation.

Explored the use of transformations to improve power in within-S designs in which multiple observations are collected for each S in each condition, such as reaction time (RT) and psychophysiological experiments. Often, the multiple measures within a treatment are simply averaged to yield a single number, but other transformations have been proposed. Monte Carlo simulations were used to investigate the influence of those transformations on the probabilities of Type I and Type II errors. With normally distributed data, Z and range correction transformations led to substantial increases in power over simple averages. With highly skewed distributions, the optimal transformation depended on several variables, but Z and range correction performed well across conditions. Correction for outliers was useful in increasing power, and trimming was more effective than eliminating all points beyond a criterion. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Monte Carlo study of three approaches to nonorthogonal analysis of variance.

Several alternative procedures have been advocated for analyzing nonorthogonal ANOVA data. Two in particular, J. E. Overall and D. K. Spiegel's (see record 1970-01534-001) Methods 1 and 2, have been the focus of controversy. A Monte Carlo study was undertaken to explore the relative sensitivity and error rates of these 2 methods, in addition to M. I. Applebaum and E. M. Cramer's (see record 1974-28956-001) procedure. Results of 2,250 3?×?3 ANOVAs conducted with each method and involving 3 underlying groups of population effects supported 3 hypotheses raised in the study: (a) Method 2 was more powerful than Method 1 in the absence of interaction; (b) Method 2 was biased upwards in the presence of interaction; and (c) Methods 1 and 2 both had Type I error rates close to those expected in the absence of interaction. In addition, it was found that in the absence of interaction, the Appelbaum and Cramer procedure was more powerful than Method 2 but slightly increased the Type I error rate. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis for integrating study outcomes: A Monte Carlo study of its susceptibility to Type I and Type II errors.

AMonte Carlo study was conducted to determine Types I and II error rates of the Schmidt and Hunter (S&H) meta-analysis method and the U statistic for assessing homogeneity within a set of correlations. One thousand samples of correlations were generated randomly to fill each of 450 cells of an 18?×?5?×?5 (Underlying Population Correlations?×?Numbers of Correlations Compared?×?Sample Size Per Correlation) design. To assess Type I error rates, correlations were drawn from the same population. To assess power, correlations were drawn from two different populations. As compared with U, which was uniformly robust, the Type I error rate for the S&H method was unacceptably high in many cells, particularly when the criterion for determining homogeneity was set at a highly conservative level. Power for the S&H method increased with increasing size of population differences, sample size per correlation, and in some cases, number of correlations compared. The U statistic did more poorly in most conditions in protecting from Type II errors. (14 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Monte Carlo study of the inferential properties of three methods of shape comparison

Three inferential morphometric methods, Euclidean distance matrix analysis (EDMA), Bookstein's edge-matching method (EMM), and the Procrustes method, were applied to facial landmark data. A Monte Carlo simulation was conducted with three sample sizes, ranging from n = 10 to 50, to assess type I error rates and the power of the tests to detect group differences for two- and three-dimensional representations of forms. Type I error rates for EMM were at or below nominal levels in both two and three dimensions. Procrustes in 2D and EDMA in 2D and 3D produced inflated type I error rates in all conditions, but approached acceptable levels with moderate cell sizes. Procrustes maintained error rates below the nominal levels in 2D. The power of EMM was high compared with the other methods in both 2D and 3D, but, conflicting EMM decisions were provided depending on which pair (2D) or triad (3D) of landmarks were selected as reference points. EDMA and Procrustes were more powerful in 2D data than for 3D data. Interpretation of these results must take into account that the data used in this simulation were selected because they represent real data that might have been collected during a study or experiment. These data had characteristics which violated assumptions central to the methods here with unequal variances about landmarks, correlated errors, and correlated landmark locations; therefore these results may not generalize to all conditions, such as cases with no violations of assumptions. This simulation demonstrates, however, limitations of each procedure that should be considered when making inferences about shape comparisons.     

Prefabricated Vertical Drains Flow Resistance under Vacuum Conditions

The results of experimental research are presented and discussed with focus on the internal well resistance of prefabricated vertical drains (PVD) under vacuum-induced water flow. Measured results included fluid flow rates for two different cross-sectional hydraulic profiles (Types I and II PVDs). Experimental results indicated linear relationships, independent of the PVD widths, between extracted fluid velocity and the applied hydraulic gradient. Data showed a laminar flow regime to predominate for test velocities corresponding to hydraulic gradients <0.5. The larger nominal hydraulic radius of the Type II PVD is credited with providing a flow rate equal to approximately 3.2 times that of the Type I PVD at approximately the same operating total head. There was no apparent dependency of the transmissivity θ on the width or lengths (3, 4, and 5 m) of the PVDs tested. In the case of the 100-mm-wide Type I PVD, θ = 618 mm2∕s was estimated from the measured data versus θ = 1,996 mm2∕s for Type II PVD with the same dimensions.     

Condom use as a dependent variable: measurement issues relevant to HIV prevention programs

The value of condoms in efforts to slow the spread of HIV infection has been well established in the literature. Behavioral science faces the challenge of promoting condom use through intervention programs. As these programs are evaluated, multiple issues should be considered in relation to measuring participant use of condoms for the purposes of preventing HIV infection. Lack of attention to these issues is likely to create a large number of Type I and Type II errors. Ten common sources of error are described and corresponding recommendations for eliminating these errors are offered. A review of published studies shows that there is little consistency relevant to controlling for these sources of error. Incorporation of standardized methodology will allow for more accurate program evaluation and benefit researchers by facilitating comparisons across studies.     

Analysis of Likert-scale data: A reinterpretation of Gregoire and Driver.

Most studies that have investigated the use of coarsely grained scales have indicated that the accuracy of statistics calculated on such scales is not compromised as long as the scales have about 5 or more points. Gregoire and Driver (1987), however, found serious perturbances of the Type I and Type II error rates using a 5-point scale. They carried out three computer simulation experiments in which continuous data were transformed to Likert-scale values. Two of the three experiments are shown to be flawed because the authors incorrectly specified the population mean in their simulation. This article corrects the flaw and demonstrates that the Type I and Type II error rates are not seriously compromised by the use of ordinal-scale data. Furthermore, Gregoire and Driver's results are reinterpreted to show that in most cases, the parametric test of location equality shows a power superiority to the nonparametric tests. Only in their most nonnormal simulation does a nonparametric test show a power superiority. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for association in 2?×?2 contingency tables with very small sample sizes.

Applied statistics textbooks generally recommend the use of the chi-square tests of homogeneity and independence with 2?×?2 contingency tables only when the expected frequency of each cell is 5 or more. Recent research has shown this rule-of-thumb criterion to be unnecessarily restrictive, but has not explored the accuracy of the chi-square tests when the total number of observations is less than 20 or when the expected frequencies fall well below one—the primary issues considered in this article. The chi-square tests of homogeneity and independence were found to provide reasonably accurate estimates of Type I error probability for N?≥?8. Certain alternatives to the chi-square tests are considered. (61 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of methods to test mediation and other intervening variable effects.

A Monte Carlo study compared 14 methods to test the statistical significance of the intervening variable effect. An intervening variable (mediator) transmits the effect of an independent variable to a dependent variable. The commonly used R. M. Baron and D. A. Kenny (1986) approach has low statistical power. Two methods based on the distribution of the product and 2 difference-in-coefficients methods have the most accurate Type I error rates and greatest statistical power except in 1 important case in which Type I error rates are too high. The best balance of Type I error and statistical power across all cases is the test of the joint significance of the two effects comprising the intervening variable effect. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Testing for robustness in Monte Carlo studies.

Monte Carlo studies provide the information needed to help researchers select appropriate analytical procedures under design conditions in which the underlying assumptions of the procedures are not met. In Monte Carlo studies, the 2 errors that one could commit involve (a) concluding that a statistical procedure is robust when it is not or (b) concluding that it is not robust when it is. In previous attempts to apply standard statistical design principles to Monte Carlo studies, the less severe of these errors has been wrongly designated the Type I error. In this article, a method is presented for controlling the appropriate Type I error rate; the determination of the number of iterations required in a Monte Carlo study to achieve desired power is described; and a confidence interval for a test's true Type I error rate is derived. A robustness criterion is also proposed that is a compromise between W. G. Cochran's (1952) and J. V. Bradley's (1978) criteria. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using Lancaster's mid-P correction to the Fisher's exact test for adverse impact analyses.

Adverse impact is often assessed by evaluating whether the success rates for 2 groups on a selection procedure are significantly different. Although various statistical methods have been used to analyze adverse impact data, Fisher's exact test (FET) has been widely adopted, especially when sample sizes are small. In recent years, however, the statistical field has expressed concern regarding the default use of the FET and has proposed several alternative tests. This article reviews Lancaster's mid-P (LMP) test (Lancaster, 1961), an adjustment to the FET that tends to have increased power while maintaining a Type I error rate close to the nominal level. On the basis of Monte Carlo simulation results, the LMP test was found to outperform the FET across a wide range of conditions typical of adverse impact analyses. The LMP test was also found to provide better control over Type I errors than the large-sample Z-test when sample size was very small, but it tended to have slightly lower power than the Z-test under some conditions. (PsycINFO Database Record (c) 2011 APA, all rights reserved)