Null hypothesis significance testing: On the survival of a flawed method.

Null hypothesis significance testing (NHST) is the researcher's workhorse for making inductive inferences. This method has often been challenged, has occasionally been defended, and has persistently been used through most of the history of scientific psychology. This article reviews both the criticisms of NHST and the arguments brought to its defense. The review shows that the criticisms address the logical validity of inferences arising from NHST, whereas the defenses stress the pragmatic value of these inferences. The author suggests that both critics and apologists implicitly rely on Bayesian assumptions. When these assumptions are made explicit, the primary challenge for NHST—and any system of induction—can be confronted. The challenge is to find a solution to the question of replicability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Null hypothesis significance testing: A review of an old and continuing controversy.

Null hypothesis significance testing (NHST) is arguably the most widely used approach to hypothesis evaluation among behavioral and social scientists. It is also very controversial. A major concern expressed by critics is that such testing is misunderstood by many of those who use it. Several other objections to its use have also been raised. In this article the author reviews and comments on the claimed misunderstandings as well as on other criticisms of the approach, and he notes arguments that have been advanced in support of NHST. Alternatives and supplements to NHST are considered, as are several related recommendations regarding the interpretation of experimental data. The concluding opinion is that NHST is easily misunderstood and misused but that when applied with good judgment it can be an effective aid to the interpretation of experimental data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

What are the fundamental facts of a comparison of two treatments' outcomes?

Responds to comments by Stefan G. Hofmann (see record 2011-19228-003), Edward A. Wise (see record 2011-19228-004), Michael J. Lambert (see record 2011-19228-005), and William H. Gottdiener (see record 2011-19228-006) on the authors original article "Statistical significance testing and clinical trials" (see record 2011-19228-002). The original article is one very narrowly focused effort at studying the implications of relying on the null hypothesis significance test (NHST) for determining which psychotherapy randomized clinical trial (RCT) findings to take seriously for clinical purposes. Although there are several approaches for faulting the NHST, the matter is important and complicated enough to justify dealing, in detail, with one approach at a time. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Bayes rules.

Responds to the F. Schmidt and J. Hunter (see record 2002-10575-012), J. L. Brand (see record 2002-10575-013), R. K. Guenther (see record 2002-10575-014), K. A. Markus (see record 2002-10575-015), and S. G. Hofmann (see record 2002-10575-016) comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). Krueger responds to each of the criticisms in turn. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A further look at wrong reasons to abandon statistical testing.

Responds to comments by W. W. Tryon, R. E. McGrath, R. G. Malgady, R. Falk, B. Thompson, and M. M. Granaas (see records 1998-04417-011, 1998-04417-012, 1998-04417-013, 1998-04417-014, 1998-04417-015, and 1998-04417-016, respectively) on the author's article (see record 1997-02239-002) defending use of the null hypothesis statistical test (NHST). The logic of NHST has been challenged by 3 claims: (1) the null hypothesis is always false; therefore, a test of the null hypothesis is only a search for what is already known to be true; (2) the form of logic on which NHST rests is flawed; and (3) NHST does not tell one what one wants to know. In attempting to rebut these claims, while there may be good reasons to give up NHST, these particular points are not the reason why. Key points of each commentary are addressed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the logic and purpose of significance testing.

There has been much recent attention given to the problems involved with the traditional approach to null hypothesis significance testing (NHST). Many have suggested that, perhaps, NHST should be abandoned altogether in favor of other bases for conclusions such as confidence intervals and effect size estimates (e.g., F. L. Schmidt; see record 83-24994) . The purposes of this article are to (a) review the function that data analysis is supposed to serve in the social sciences, (b) examine the ways in which these functions are performed by NHST, (c) examine the case against NHST, and (d) evaluate interval-based estimation as an alternative to NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Significance testing: Is there something Better?

Comments on the article by R. L. Hagen (see record 1997-02239-002) supporting use of the null hypothesis statistical test (NHST). Hagen did an admirable job of reminding readers that the NHST represents a brilliant and useful innovation, but does not offer a strong case for its continued use as the primary inferential strategy in psychology. The question is not "Is it useless?" but "Is there something better?" Popular opinion holds that interval estimation represents a superior strategy to NHST in many ways. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating statistical difference, equivalence, and indeterminancy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests.

Null hypothesis statistical testing (NHST) has been debated extensively but always successfully defended. The technical merits of NHST are not disputed in this article. The widespread misuse of NHST has created a human factors problem that this article intends to ameliorate. This article describes an integrated, alternative inferential confidence interval approach to testing for statistical difference, equivalence, and indeterminacy that is algebraically equivalent to standard NHST procedures and therefore exacts the same evidential standard. The combined numeric and graphic tests of statistical difference, equivalence, and indeterminacy are designed to avoid common interpretive problems associated with NHST procedures. Multiple comparisons, power, sample size, test reliability, effect size, and cause-effect ratio are discussed. A section on the proper interpretation of confidence intervals is followed by a decision rule summary and caveats. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The inscrutable null hypothesis.

Comments on the article by R. L. Hagen (see record 1997-02239-002) praising the null hypothesis statistical test (NHST). Hagen's praise of the NHST may be supported on purely technical grounds but it is unfortunate if it prolongs primary reliance on NHST to evaluate quantitative difference and equivalence given the prominent human factors problem of widespread and intractable interpretation errors. Alternative methods are available for these purposes that are far less subject to misinterpretation. The science of psychology can openly benefit by supplementing, if not replacing, NHST practices with these methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fisher's fallacy and NHST's flawed logic.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author contends that the underlying philosophical problems with NHST are considerably more complicated and serious than Krueger seemed to realize. Hofman describes how the logic of NHST is based on a misapplication of deductive sylogistic reasoning, because probabilistic statements are incompatible with the rules of deductive reasoning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Postscript: Theory development should not end (but always begins) with good empirical fits: Response to Roberts and Pashler's (2002) reply.

Presents a response to Roberts and Pashler's reply (see record 2002-13781-009) to the current author's original article (see record 2002-13781-008). Roberts and Pashler (2002) have shifted their original criterion to make it appear that we "have failed to come up with even one clear counterexample" (Roberts & Pashler, 2002, p. 607). However, no one ever argued that goodness of fit stands alone (note the title of our original criticism). Their original challenge was, exactly, to find "theories originally supported mainly or entirely by good fits to data that eventually found support from other sources" (Roberts & Pashler, 2000, p. 362). Roberts and Pashler (2002) rejected our analogy between criticism of null hypothesis significance testing (NHST) and their criticism of using goodness of fit as a starting point in the development of theories. We believe that Roberts and Pashler (2000, 2002) made important statements about theory development. Although those statements are embedded in an extreme, unreasonable, and unacceptable position as regards goodness of fit, we can still extract great value from their principles for theory development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Beyond objectivity and subjectivity.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author comments on the general treatment of objectivity and subjectivity that underlies Krueger's review. Despite repeated appeals to a pragmatic basis for NHST, Krueger seems to have missed a key implication of a pragmatic view of scientific inference in this regard. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Trait memory and behavior memory: The effects of alternative pathways on impression judgment response times.

The 1st author (1980) and J. H. Lingle (see record 1984-15004-001) argued that remembered behaviors, previous trait inferences, or both may be accessed and used in making new trait inferences. This argument is related to a spreading activation model of memory, and factors that should affect the relative accessibility of inferences and behaviors during trait judgment processes are suggested. A study with 112 undergraduates varied several of these factors and assessed accessibility, using response-time methods. Results support the model's prediction that prompting inference formation facilitates subsequent trait judgment response times, but only when relevant behavior memories have not been recently primed. It is theorized that the inference manipulations used in this study strengthened the direct pathway to a relevant trait concept, but that the strength of this pathway was immaterial to judgment response times when a "proximal prime" directed retrieval efforts along an alternative "behavioral" route to the trait information. Results also suggest that the proximal behavior prime facilitated trait responses among Ss who had not been induced to make trait inferences, but slowed trait responses among Ss who had previously been induced to make trait inferences. (29 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Why chance is a good theory.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author states that Krueger carelessly included a dubious claim that weakened at least some of his contentions: that the widespread use of NHST represents a ubiquitous ignorance of its logical pitfalls. Contrary to Krueger's claims, the current author believes that within a larger causal framework, the null hypothesis remains the best theory available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

In praise of the null hypothesis statistical test.

Jacob Cohen (see record 1995-12080-001) raised a number of questions about the logic and information value of the null hypothesis statistical test (NHST). Specifically, he suggested that: (1) The NHST does not tell us what we want to know; (2) the null hypothesis is always false; and (3) the NHST lacks logical integrity. It is the author's view that although there may be good reasons to give up the NHST, these particular points made by Cohen are not among those reasons. When addressing these points, the author also attempts to demonstrate the elegance and usefulness of the NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Whither subliminal psychodynamic activation? A reply to Silverman.

The authors reply to comments by L. H. Silverman (see record 1982-21579-001) on their earlier criticisms (see record 1980-26153-001) of subliminal symbiotic stimulation as a clinical adjunct to systematic desensitization. The authors contend that Silverman credits them with an overly nebulous "challenge" that was not made, and cites new data that purportedly contradict this overgeneralized conclusion. None of this "contradictory" evidence deals with desensitization or any other treatment for phobic anxiety. The alternative explanations he derives from these new data are based on an arbitrary and simplistic method of data aggregation that lacks consistency across investigations. The resulting selective bias severely reduces the explanatory power of these alternative possibilities and does not negate the authors' contention that stimulation of unconscious merging fantasies is superfluous in desensitization. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Values and Science: Reply.

Responds to Kendler (2004), Johnson (2004), and Zuriff (2004) who directed a number of criticisms against the authors' original article (see record 2003-03405-003) examining the psychological data and policy debates surrounding affirmative action. Kendler and Zuriff both chided the authors for interjecting values into the realm of science. The authors, however, state that both seriously misread the argument, imagining that they talked about "morality" when they did not. The authors claim to hold the same view as Kendler and Zuriff about the dichotomy between data and values and revisit portions of their argument further questioned. To Johnson, they state that the issues are not as clear cut as he has suggested. The authors do, however, reevaluate their original article in light of his criticisms. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Model fitting: A better approach.

Comments on the article by R. L. Hagen (see record 1997-02239-002) defending the logic and practice of null hypothesis statistical testing (NHST). It is argued that model fitting provides an approach to data analysis that is more appropriate to the cognitive needs of the researcher than is NHST. Model fitting combines the NHST ability to falsify hypotheses with the parameter-estimation characteristic of confidence intervals in an approach that is simpler to learn, understand, and use. Effect size estimation is central to the approach, and power calculations are vastly simplified relative to NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are there benefits from NHST?

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The validity and accuracy of significance testing is addressed by the current author. (PsycINFO Database Record (c) 2010 APA, all rights reserved)