The proof of the pudding: An illustration of the relative strengths of null hypothesis, meta-analysis, and Bayesian analysis.

Some methodologists have recently suggested that scientific psychology's overreliance on null hypothesis significance testing (NHST) impedes the progress of the discipline. In response, a number of defenders have maintained that NHST continues to play a vital role in psychological research. Both sides of the argument to date have been presented abstractly. The authors take a different approach to this issue by illustrating the use of NHST along with 2 possible alternatives (meta-analysis as a primary data analysis strategy and Bayesian approaches) in a series of 3 studies. Comparing and contrasting the approaches on actual data brings out the strengths and weaknesses of each approach. The exercise demonstrates that the approaches are not mutually exclusive but instead can be used to complement one another. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the logic and purpose of significance testing.

There has been much recent attention given to the problems involved with the traditional approach to null hypothesis significance testing (NHST). Many have suggested that, perhaps, NHST should be abandoned altogether in favor of other bases for conclusions such as confidence intervals and effect size estimates (e.g., F. L. Schmidt; see record 83-24994) . The purposes of this article are to (a) review the function that data analysis is supposed to serve in the social sciences, (b) examine the ways in which these functions are performed by NHST, (c) examine the case against NHST, and (d) evaluate interval-based estimation as an alternative to NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating statistical difference, equivalence, and indeterminancy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests.

Null hypothesis statistical testing (NHST) has been debated extensively but always successfully defended. The technical merits of NHST are not disputed in this article. The widespread misuse of NHST has created a human factors problem that this article intends to ameliorate. This article describes an integrated, alternative inferential confidence interval approach to testing for statistical difference, equivalence, and indeterminacy that is algebraically equivalent to standard NHST procedures and therefore exacts the same evidential standard. The combined numeric and graphic tests of statistical difference, equivalence, and indeterminacy are designed to avoid common interpretive problems associated with NHST procedures. Multiple comparisons, power, sample size, test reliability, effect size, and cause-effect ratio are discussed. A section on the proper interpretation of confidence intervals is followed by a decision rule summary and caveats. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Model fitting: A better approach.

Comments on the article by R. L. Hagen (see record 1997-02239-002) defending the logic and practice of null hypothesis statistical testing (NHST). It is argued that model fitting provides an approach to data analysis that is more appropriate to the cognitive needs of the researcher than is NHST. Model fitting combines the NHST ability to falsify hypotheses with the parameter-estimation characteristic of confidence intervals in an approach that is simpler to learn, understand, and use. Effect size estimation is central to the approach, and power calculations are vastly simplified relative to NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The inscrutable null hypothesis.

Comments on the article by R. L. Hagen (see record 1997-02239-002) praising the null hypothesis statistical test (NHST). Hagen's praise of the NHST may be supported on purely technical grounds but it is unfortunate if it prolongs primary reliance on NHST to evaluate quantitative difference and equivalence given the prominent human factors problem of widespread and intractable interpretation errors. Alternative methods are available for these purposes that are far less subject to misinterpretation. The science of psychology can openly benefit by supplementing, if not replacing, NHST practices with these methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Why chance is a good theory.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author states that Krueger carelessly included a dubious claim that weakened at least some of his contentions: that the widespread use of NHST represents a ubiquitous ignorance of its logical pitfalls. Contrary to Krueger's claims, the current author believes that within a larger causal framework, the null hypothesis remains the best theory available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Significance testing: Is there something Better?

Comments on the article by R. L. Hagen (see record 1997-02239-002) supporting use of the null hypothesis statistical test (NHST). Hagen did an admirable job of reminding readers that the NHST represents a brilliant and useful innovation, but does not offer a strong case for its continued use as the primary inferential strategy in psychology. The question is not "Is it useless?" but "Is there something better?" Popular opinion holds that interval estimation represents a superior strategy to NHST in many ways. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Null hypothesis significance testing: On the survival of a flawed method.

Null hypothesis significance testing (NHST) is the researcher's workhorse for making inductive inferences. This method has often been challenged, has occasionally been defended, and has persistently been used through most of the history of scientific psychology. This article reviews both the criticisms of NHST and the arguments brought to its defense. The review shows that the criticisms address the logical validity of inferences arising from NHST, whereas the defenses stress the pragmatic value of these inferences. The author suggests that both critics and apologists implicitly rely on Bayesian assumptions. When these assumptions are made explicit, the primary challenge for NHST—and any system of induction—can be confronted. The challenge is to find a solution to the question of replicability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On Confidence Intervals for Within-Subjects Designs.

Confidence intervals (CIs) for means are frequently advocated as alternatives to null hypothesis significance testing (NHST), for which a common theme in the debate is that conclusions from CIs and NHST should be mutually consistent. The authors examined a class of CIs for which the conclusions are said to be inconsistent with NHST in within-subjects designs and a class for which the conclusions are said to be consistent. The difference between them is a difference in models. In particular, the main issue is that the class for which the conclusions are said to be consistent derives from fixed-effects models with subjects fixed, not mixed models with subjects random. Offered is mixed model methodology that has been popularized in the statistical literature and statistical software procedures. Generalizations to different classes of within-subjects designs are explored, and comments on the future direction of the debate on NHST are offered. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fisher's fallacy and NHST's flawed logic.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author contends that the underlying philosophical problems with NHST are considerably more complicated and serious than Krueger seemed to realize. Hofman describes how the logic of NHST is based on a misapplication of deductive sylogistic reasoning, because probabilistic statements are incompatible with the rules of deductive reasoning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

In praise of the null hypothesis statistical test.

Jacob Cohen (see record 1995-12080-001) raised a number of questions about the logic and information value of the null hypothesis statistical test (NHST). Specifically, he suggested that: (1) The NHST does not tell us what we want to know; (2) the null hypothesis is always false; and (3) the NHST lacks logical integrity. It is the author's view that although there may be good reasons to give up the NHST, these particular points made by Cohen are not among those reasons. When addressing these points, the author also attempts to demonstrate the elegance and usefulness of the NHST. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Beyond objectivity and subjectivity.

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The current author comments on the general treatment of objectivity and subjectivity that underlies Krueger's review. Despite repeated appeals to a pragmatic basis for NHST, Krueger seems to have missed a key implication of a pragmatic view of scientific inference in this regard. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A further look at wrong reasons to abandon statistical testing.

Responds to comments by W. W. Tryon, R. E. McGrath, R. G. Malgady, R. Falk, B. Thompson, and M. M. Granaas (see records 1998-04417-011, 1998-04417-012, 1998-04417-013, 1998-04417-014, 1998-04417-015, and 1998-04417-016, respectively) on the author's article (see record 1997-02239-002) defending use of the null hypothesis statistical test (NHST). The logic of NHST has been challenged by 3 claims: (1) the null hypothesis is always false; therefore, a test of the null hypothesis is only a search for what is already known to be true; (2) the form of logic on which NHST rests is flawed; and (3) NHST does not tell one what one wants to know. In attempting to rebut these claims, while there may be good reasons to give up NHST, these particular points are not the reason why. Key points of each commentary are addressed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correction to Krueger (2001).

Reports an error in the original article by J. Krueger (American Psychologist, 2001, Vol 56[1], pp. 16-26). In Figure 2, on page 22, two of the curves are labeled "p(H0)=.9" and "p(H0)=.1." These labels should have been reversed. (The following abstract of this article originally appeared in record 2001-16601-002.) Null hypothesis significance testing (NHST) is the researcher's workhorse for making inductive inferences. This method has often been challenged, has occasionally been defended, and has persistently been used through most of the history of scientific psychology. This article reviews both the criticisms of NHST and the arguments brought to its defense. The review shows that the criticisms address the logical validity of inferences arising from NHST, whereas the defenses stress the pragmatic value of these inferences. The author suggests that both critics and apologists implicitly rely on Bayesian assumptions. When these assumptions are made explicit, the primary challenge for NHST--and any system of induction--can be confronted. The challenge is to find a solution to the question of replicability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The epistemology of mathematical and statistical modeling: A quiet methodological revolution.

A quiet methodological revolution, a modeling revolution, has occurred over the past several decades, almost without discussion. In contrast, the 20th century ended with contentious argument over the utility of null hypothesis significance testing (NHST). The NHST controversy may have been at least partially irrelevant, because in certain ways the modeling revolution obviated the NHST argument. I begin with a history of NHST and modeling and their relation to one another. Next, I define and illustrate principles involved in developing and evaluating mathematical models. Following, I discuss the difference between using statistical procedures within a rule-based framework and building mathematical models from a scientific epistemology. Only the former is treated carefully in most psychology graduate training. The pedagogical implications of this imbalance and the revised pedagogy required to account for the modeling revolution are described. To conclude, I discuss how attention to modeling implies shifting statistical practice in certain progressive ways. The epistemological basis of statistics has moved away from being a set of procedures, applied mechanistically, and moved toward building and evaluating statistical and scientific models. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

What are the fundamental facts of a comparison of two treatments' outcomes?

Responds to comments by Stefan G. Hofmann (see record 2011-19228-003), Edward A. Wise (see record 2011-19228-004), Michael J. Lambert (see record 2011-19228-005), and William H. Gottdiener (see record 2011-19228-006) on the authors original article "Statistical significance testing and clinical trials" (see record 2011-19228-002). The original article is one very narrowly focused effort at studying the implications of relying on the null hypothesis significance test (NHST) for determining which psychotherapy randomized clinical trial (RCT) findings to take seriously for clinical purposes. Although there are several approaches for faulting the NHST, the matter is important and complicated enough to justify dealing, in detail, with one approach at a time. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Are there benefits from NHST?

Comments on the J. Krueger (see record 2001-16601-002) discussion on null hypothesis significance testing (NHST). The validity and accuracy of significance testing is addressed by the current author. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The hidden observer as an experimental creation

Laurence, Perry, and Kihlstrom appear to have misunderstood the theoretical positions advanced both by Hilgard and by Spanos and Hewitt. All of their criticisms of Spanos and Hewitt's "hidden observer" experiment are shown to be either misleading or invalid. The available data provide no support for a dissociation hypothesis of the hidden observer phenomenon. On the other hand, these data indicate strongly that this phenomenon is an experimental creation.     

Postscript: Theory development should not end (but always begins) with good empirical fits: Response to Roberts and Pashler's (2002) reply.

Presents a response to Roberts and Pashler's reply (see record 2002-13781-009) to the current author's original article (see record 2002-13781-008). Roberts and Pashler (2002) have shifted their original criterion to make it appear that we "have failed to come up with even one clear counterexample" (Roberts & Pashler, 2002, p. 607). However, no one ever argued that goodness of fit stands alone (note the title of our original criticism). Their original challenge was, exactly, to find "theories originally supported mainly or entirely by good fits to data that eventually found support from other sources" (Roberts & Pashler, 2000, p. 362). Roberts and Pashler (2002) rejected our analogy between criticism of null hypothesis significance testing (NHST) and their criticism of using goodness of fit as a starting point in the development of theories. We believe that Roberts and Pashler (2000, 2002) made important statements about theory development. Although those statements are embedded in an extreme, unreasonable, and unacceptable position as regards goodness of fit, we can still extract great value from their principles for theory development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

In criticism of the null hypothesis statistical test.

Comments on the article by R. L. Hagen (see record 1997-02239-002) in praise of the null hypothesis statistical test (NHST). NHST, is, in fact, a probabilistic imitation of modus tollens (or of the mathematical procedure of proof by contradiction). However, once the reasoning is made probabilistic, the inference is no longer valid. (PsycINFO Database Record (c) 2010 APA, all rights reserved)