Power of univariate pairwise multiple comparison procedures.

Used Monte Carlo simulation to evaluate the power of various procedures under the constraint of experimentwise Type I error control. Both the configuration of population means and the method of assessing power influenced the relative powers of the procedures. The Shaffer-Welsch or one of several Q-test procedures had the greatest any-pair power or power to detect at least true difference. Either Peritz's F-test procedure or a model testing procedure had the greatest all-pairs power or power to detect all true differences. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Alternative procedures for testing regression slope homogeneity when group error variances are unequal.

The test for regression slope homogeneity across groups (e.g., sex, race, and treatments) is used in such varied settings as the analysis of covariance, the study of aptitude by treatment interactions, and bias detection in differential prediction research. The accuracy of this test requires the seldom-considered assumption of equality of within-group error variances. This research studies the effect of violating that assumption on the power of the F test for regression slope equality and finds that the test may be substantially affected when sample sizes are equal and severely affected when sample sizes are unequal. Alternative procedures based on R. A. Alexander's (see record 1994-39680-001) normalized-t approximation, G. S. James's (1951) second-order approximation, the Welch-Aspin approximation, and the chi-square test are described and evaluated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Selection of pairwise multiple comparison procedures for parametric and nonparametric analysis of variance models.

Discusses 2 issues that are often overlooked in the evaluation of pairwise multiple comparison procedures (MCPs): Comparisons of powers for competing MCPs are not useful unless the procedures have identical nominal experimentwise error rates; and "protected" MCPs do not control the experimentwise Type I error rate at the nominal alpha except in the complete null case. Also, procedures based on the ranks of all K groups do not test the same hypothesis as MCPs that involve reranking the data for each pairwise comparison. (41 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of closed testing procedures for pairwise testing of means.

A Monte Carlo simulation was conducted to compare 9 pairwise multiple comparison procedures. Procedures were evaluated on the basis of any-pair power and all-pairs power. No procedure was found to be uniformly most powerful. A modification due to A. J. Hayter (1986) of Fisher's least significant difference was found to provide the best combination of ease of use and moderately high any-pair power in most cases. Pilot or exploratory studies can expect good power results with this relatively simple procedure. The greatest all-pairs power was usually provided by 1 of 2 partition-based versions of E. Peritz's (1970) procedure. Confirmatory studies will require such complex methods but may also need larger sample sizes than have been customary in psychological research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Robustness of the independent samples Hotelling's T2 to variance-covariance heteroscedasticity when sample sizes are unequal and in small ratios.

Type I error rates for the independent samples Hotelling's T–2 were estimated using simulated data for conditions in which the variance–covariance matrices were heteroscedastic and the ratios of unequal sample sizes were small (n?:n??=?1:1.25, 1.25:1, 1:1.1, and 1.1:1). Type I error rates were estimated for various combinations of the number of variables, sample-size-to-variables ratio, and degree and type of heteroscedasticity. Even with a sample size ratio as small as 1.1:1, the T–2 procedure can be seriously nonrobust. As expected, the procedure became less robust as the number of variables and degree of heteroscedasticity increased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple comparison procedures for trimmed means.

Stepwise multiple comparison procedures (MCPs) based on least squares and trimmed estimators were compared for their rates of Type I error and their ability to detect true pairwise group differences. The MCPs were compared in unbalanced one-way completely randomized designs when normality and homogeneity of variance assumptions were violated. Results indicated that MCPs based on trimmed means and Winsorized variances controlled rates of Type I error, whereas MCPs based on least squares estimators typically could not, particularly when the data were highly skewed. However, MCPs based on least squares estimators were substantially more powerful than their counterparts based on trimmed means and Winsorized variances when the data were only moderately skewed, a finding which qualifies recommendations on the use of trimmed estimators offered in the literature. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Mixed-model pairwise multiple comparisons of repeated measures means.

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of the data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach was evaluated for testing all possible pairwise differences among repeated measures marginal means in a Between-Subjects?×?Within-Subjects design. Specifically, the authors investigated Type I error and power rates for a number of simultaneous and stepwise multiple comparison procedures using SAS (1999) PROC MIXED in unbalanced designs when normality and covariance homogeneity assumptions did not hold. J. P. Shaffer's (1986) sequentially rejective step-down and Y. Hochberg's (1988) sequentially acceptive step-up Bonferroni procedures, based on an unstructured covariance structure, had superior Type I error control and power to detect true pairwise differences across the investigated conditions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analytic procedures for combining studies with multiple effect sizes.

Presents a general set of meta-analytic procedures for combining and comparing research results from studies yielding multiple effect sizes based on multiple dependent variables. These require, in addition to the individual effect sizes or significance levels, only the degrees of freedom in the study and the typical intercorrelation among the variables. Older methods are reviewed, and a new method for obtaining a single summary effect size estimate from multiple effect sizes is presented. Significant testing of this summary effect size estimate is described. Procedures for computing the effect size for a contrast, and its significance level, among the multiple effect sizes of a single study are also described. Methods for dealing with problems of heterogeneous intercorrelations among the dependent variables are presented. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of two procedures for testing multiple contrasts.

Scheffe's test for contrasts is frequently used when a relatively small set of contrasts is of interest. When these contrasts have been planned prior to the data analysis, the Bonferroni t statistic provides a reasonable alternative. Comparisons are made of critical values for the 2 tests. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Pairwise multiple comparison procedures: A review.

Reviews the literature on the use of pairwise multiple comparisons in terms of between-S and repeated measures experimental designs and the optimality of experimental conditions. Tests are recommended for use in conditions in which the assumptions of normality, homogeneous population variances, and equal sample sizes are violated. (65 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Basic statistics and the inconsistency of multiple comparison procedures.

This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD[5%]). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several well-known MCPs by focussing on a particular operating known characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments... (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Corrected formulas for computing sample sizes under indirect range restriction.

Asserts that the study of P. R. Sackett and B. E. Wade (see record 1983-31752-001) added significantly to the understanding of the interaction of statistical power, reliability, predictor intercorrelation, and selection ratio in determining the sample sizes required in validation studies where indirect range restriction is assumed. It is argued that, although Sackett and Wade's conclusions were correct, their results were marred by computational problems. The present note provides correct formulas for determining sample sizes under indirect range restriction. (4 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The pairwise multiple comparison multiplicity problem: An alternative approach to familywise and comparison wise Type I error control.

When undertaking many tests of significance, researchers are faced with the problem of how best to control the probability of committing a Type I error. The familywise approach deals directly with multiplicity problems by setting a level of significance for an entire set of related hypotheses; the comparison approach ignores the issue by setting the rate of error on each individual hypothesis. A new formulation of control, the false discovery rate, does not provide control as stringent as that of the familywise rate, but concomitant with this relaxation in stringency is an increase in sensitivity to detect effects relative to the sensitivity of familywise control. Type I error and power rates for 4 relatively powerful and easily computed pairwise multiple comparison procedures were compared with the false discovery rate procedure for various 1-way layouts by use of test statistics that do not assume variance homogeneity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for adverse impact when sample size is small.

Adverse impact evaluations often call for evidence that the disparity between groups in selection rates is statistically significant, and practitioners must choose which test statistic to apply in this situation. To identify the most effective testing procedure, the authors compared several alternate test statistics in terms of Type I error rates and power, focusing on situations with small samples. Significance testing was found to be of limited value because of low power for all tests. Among the alternate test statistics, the widely-used Z-test on the difference between two proportions performed reasonably well, except when sample size was extremely small. A test suggested by G. J. G. Upton (1982) provided slightly better control of Type I error under some conditions but generally produced results similar to the Z-test. Use of the Fisher Exact Test and Yates's continuity-corrected chi-square test are not recommended because of overly conservative Type I error rates and substantially lower power than the Z-test. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Improved Bonferroni-type multiple testing procedures.

[Correction Notice: An erratum for this article was reported in Vol 104(2) of Psychological Bulletin (see record 2008-06007-001). An error was made in the author note on page 145. Correspondence should be addressed to Burt S. Holland, Department of Statistics, Temple University, Speakman Hall (006-00), Philadelphia, Pennsylvania 19122. Margaret DiPonzio Copenhaver is now at Merck Sharp & Dohme Research Laboratories, West Point, Pennsylvania.] The Bonferroni multiple comparisons procedure is customarily used when doing several simultaneous tests of significance in relatively nonstandard situations in which other methods do not apply. We review some new and improved competitors to the Bonferroni procedure, that although constraining generalized Type I error probability to be at most α, afford increased power in exchange for increased complexity in implementation. An improvement to the weighted form of the Bonferroni procedure is also presented. Several data sets are reanalyzed with the new methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of three procedures for classifying addicts.

2 prevalent techniques plus a 3rd technique which compared individual to group profiles were employed to analyze the profiles of 2,179 narcotic addicts, using the Sixteen Personality Factor Questionnaire. It is concluded that inferences from classification studies can be related to the techniques used. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Information criteria for pairwise comparisons.

A new approach is presented for the interpretation of differences among means and proportions. Post hoc techniques, such as Tukey's honestly significant difference procedure, have interpretive problems related to intransitive decisions and technical issues arising from unequal sample sizes or heterogeneity of variance. These concerns can be avoided by considering ordered subsets of means and by using information criterion to select among competing models. This paired-comparisons information-criterion (PCIC) approach is wholistic in nature and does not depend on interpreting a series of statistical tests. Simulation results suggest that a protected version of the PCIC procedure is desirable to minimize failures to detect the null case. This technique is illustrated for independent means, proportions, and means from repeated measures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple comparison procedures within experimental research.

Discusses 2 approaches in the use of multiple comparison procedures. One is that of the practical-research-oriented investigator, who emphasizes the importance of the Type II error and associated power. The other is exemplified by mathematical statisticians who emphasize "pure" mathematical aspects and concentrate on the importance of controlling Type I errors. Two important issues are discussed: emphasis on Type I errors within single experiments vs emphasis on Type II errors within multiple experiments. (French abstract) (36 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Power of recent multiple comparison procedures as applied to a complete set of planned orthogonal contrasts.

Four recent multiple comparison strategies (Y. Hochberg, 1988; S. Holm, 1979; G. Hommel, 1988; and J. P. Shaffer, 1986), as well as a stagewise protected procedure proposed herein, were compared as tests of significance for a complete set of planned orthogonal contrasts. The Holm, Shaffer, and stagewise protected procedures were further modified using an adjustment suggested by B. S. Holland and M. D. Copenhaver (1987, 1988). The 2 F-protected methods and 3 unprotected methods all maintain experimentwise error rate at or below α. Simulation results indicate that the relative power of the 5 procedures depends on the configuration of the treatment effects contained in all contrasts. Specifically, the unprotected tests favor configurations with relatively small amounts of variability that are due to treatment effects, but the protected tests have more power in cases with a relatively large amount of treatment variability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of two group brainstorming procedures.

A modified brainstorming procedure that required Ss (male introductory psychology students) to identify psychologically with significant components of the task (called synectics) was compared to standard brainstorming over 3 sessions and 9 different problems. Each treatment condition was also divided into high- and low-interpersonal-effectiveness groups. The synectics groups were superior to the brainstorming groups on all 9 problems, but the differences were statistically significant for only 4. There were no significant main effects due to group composition, nor were there any interactions. It is concluded that synectics is a more effective group-problem-solving strategy than brainstorming. (PsycINFO Database Record (c) 2010 APA, all rights reserved)