Trade-offs in the design of experiments.

This comment supplements and clarifies issues raised by J. C. Schank and T. J. Koehnle (2009) in their critique of experimental design. First, the pervasiveness of trade-offs in the design of experiments is emphasized (Wiley, 2003). Particularly germane to Schank and Koehnle’s discussion are the inevitable trade-offs in any decisions to include blocking or to standardize conditions in experiments. Second, the interpretation of multiple tests of a hypothesis is clarified. Only when interest focuses on any, rather than each, of N possible responses is it appropriate to adjust criteria for statistical significance of the results. Finally, a misunderstanding is corrected about a disadvantage of large experiments (Wiley, 2003). Experiments with large samples raise the possibility of small, but statistically significant, biases even after randomization of treatments. Because these small biases are difficult for experimenters and readers to notice, large experiments demonstrating small effects require special scrutiny. Such experiments are justified only when they involve minimal human intervention and maximal standardization. Justifications for the inevitable trade-offs in experimental design require careful attention when reporting any experiment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The "post-mortem" testing of experimental comparisons.

In an experimental design the investigator frequently wishes to go beyond an over-all analysis of variance F test, and seeks to examine specific means or totals from certain groups of treatments. In the a posteriori ("post-mortem") situation, where comparisons are suggested by examining the data, and have not been pre-planned, it is inappropriate to use the tabular F and t values, since these values are not based upon randomly selected observation. A procedure is recommended to handle this situation, one that avoids the danger of erroneous inferences (spuriously significant comparisons). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative susceptibility of two rating scales to disturbances resulting from shifts in stimulus context.

A simple and a complex rating scale were compared with respect to their resistance to distorting effects produced by limited and shifting stimulus contexts. These 9-point scales assessed "shifting in the value of common stimuli as a function of context" and "loss of refinement or correlational accuracy." Results favored the detailed rating scale with judgmental equivalence and accuracy. Utility of creating experimental stress tests for rating scale evaluation is implied. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Additional "post mortem" tests of experimental comparisons.

Scheffe's technique for assessing all possible combinations of mean comparisons after the analysis of variance is judged to be overly conservative for most comparisons of interest to psychologists. Techniques yielding more sensitive tests of comparisons are presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The automatic aesthetic evaluation of different art and architectural styles.

The subject under discussion concerns the existence of an automatic aesthetic evaluation. When we encounter an object like an artwork or an architectural structure that activates an aesthetic response, does the associated evaluation appear in our mind as an automatic process? From the broad field of aesthetic appraisal, we will be considering a specific aspect that refers only to the positive and negative affects related to an individual's preference between two art styles (figurative vs. abstract) and two architectural styles (classic vs. contemporary). The hypothesis is that there is a preexistent preference within the visual arts and architecture that can clearly be identified using implicit measurements. Results from two experiments that were conducted with the use of the Implicit Association Test (IAT) showed that participants' reaction times were faster in associating positive words to figurative art and classical architecture (the so-called compatible task) than to abstract art and contemporary architecture (the so-called incompatible task). The results are in line with the hypothesis that aesthetic preferences can also be experienced automatically. Prototypicality (i.e., the degree to which an object is representative of a general class of object), familiarity and the relative simplicity of figurative art and classical architecture (compared to abstract art and modern architecture) can explain the shorter reaction time and as a consequence, an implicit aesthetic preference for these kinds of stimuli. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Using analysis of covariance (ANCOVA) with fallible covariates.

Analysis of covariance (ANCOVA) is used widely in psychological research implementing nonexperimental designs. However, when covariates are fallible (i.e., measured with error), which is the norm, researchers must choose from among 3 inadequate courses of action: (a) know that the assumption that covariates are perfectly reliable is violated but use ANCOVA anyway (and, most likely, report misleading results); (b) attempt to employ 1 of several measurement error models with the understanding that no research has examined their relative performance and with the added practical difficulty that several of these models are not available in commonly used statistical software; or (c) not use ANCOVA at all. First, we discuss analytic evidence to explain why using ANCOVA with fallible covariates produces bias and a systematic inflation of Type I error rates that may lead to the incorrect conclusion that treatment effects exist. Second, to provide a solution for this problem, we conduct 2 Monte Carlo studies to compare 4 existing approaches for adjusting treatment effects in the presence of covariate measurement error: errors-in-variables (EIV; Warren, White, & Fuller, 1974), Lord's (1960) method, Raaijmakers and Pieters's (1987) method (R&P), and structural equation modeling methods proposed by S?rbom (1978) and Hayduk (1996). Results show that EIV models are superior in terms of parameter accuracy, statistical power, and keeping Type I error close to the nominal value. Finally, we offer a program written in R that performs all needed computations for implementing EIV models so that ANCOVA can be used to obtain accurate results even when covariates are measured with error. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

More Powerful Tests of Predictor Subsets in Regression Analysis Under Nonnormality.

It is well-known that for normally distributed errors parametric tests are optimal statistically, but perhaps less well-known is that when normality does not hold, nonparametric tests frequently possess greater statistical power than parametric tests, while controlling Type I error rate. However, the use of nonparametric procedures has been limited by the absence of easily performed tests for complex experimental designs and analyses and by limited information about their statistical behavior for realistic conditions. A Monte Carlo study of tests of predictor subsets in multiple regression analysis indicates that various nonparametric tests show greater power than the F test for skewed and heavy-tailed data. These nonparametric tests can be computed with available software. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The accuracy of reference lists in five experimental psychology journals.

Reference lists appended to published journals are of critical importance to readers of academic literature. They allow readers to retrieve cited sources, either to increase their own knowledge or to substantiate the authors' claims, and in so doing help to establish the judgment and credibility of the authors (M. F. McLellan et al, 1992). However, erroneous references detract from the intended function of reference lists by impeding retrieval searches and by providing misleading bibliographic information (G. de Lacey et al, 1985; McLellan et al, 1992). The present study aims to assess the accuracy of reference lists in five experimental psychology journals published by the American Psychological Association (APA) in 1999 (Journal of Experimental Psychology: Animal Behavior Processes, Journal of Experimental Psychology: Applied, Journal of Experimental Psychology: Learning, Memory, and Cognition, Journal of Experimental Psychology: Human Perception and Performance, and Journal of Experimental Psychology: General). Perhaps the most disturbing result from this study is the finding that authors were incorrectly cited in 12% of reference list citations, with errors in the initials of the authors being the most common. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluation of six multinomial models of conscious and unconscious processes with the recall-recognition paradigm.

Six multinomial processing-tree models (W. H. Batchelder & D. M. Riefer, 1999), which include parameters representing conscious and unconscious memory processes, were tested using the recall-recognition paradigm. Data from 2 experiments were fit equally well by 3 of the 6 models. One model recognition was an extension of the generate-recognize model (L. L. Jacoby, 1998), and another was an extension of the non-high-threshold model (D. M. McBride & B. A. Dosher, 1999). The 3rd model was the source evaluation model (D. M. McBride & B. A. Dosher, 1999). Values of the parameters of 2 of these 3 models, excepting the non-high-threshold model, responded to experimental manipulations in accordance with the meaning of the parameters. The equivalence of models with regard to goodness-of-fit tests is discussed as is how experiments can be designed to demonstrate the superiority of one model over another. The potential usefulness of these models in the study of amnesia is considered. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Three Faces of Evelyn: A Case Report. III. Reactions to the Blind Analysis.

This article discusses the blind analysis of the test data of a case of multiple personality (Gina). I am glad that the blind analysis reveals Evelyn to be a fusion of Gina and Mary. This confirms my view of what happens in the successful treatment of a case of multiple personality: The prejudice of each ego state against the others is made weaker and weaker until the barriers between them disappear and the individual becomes psychologically whole again. As to the comparative validity of intimate clinical versus remote objective interpretation, I have learned as a result of this experience that they are complementary, and that there is nothing to lose and much to be gained by their complementarity. The pressure of practice and a certain confidence in my clinical ability had prevented me from drawing as much meaning from my data as I could when confronted with the results of the blind analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An experimental manipulation of Rorschach form: The effect of making indistinct the ambiguous.

Systematic variations of the stimulus properties of the Rorschach inkblots are relatively rare. In this study, form was altered by projecting slides of the blots out of focus. Since this blurring more or less duplicated the situation to be expected in some types of poor vision, there are practical implications; to make the application to the clinic as direct as possible within the limitations of the experimental design, the degree of blurring was adjusted to that associated with a commonly encountered level of defective eyesight. In an otherwise standard group-administration procedure, every other slide was presented out of focus, the even numbered to 1 group of 20 Ss, the odd to another. Marked changes occurred in location, P, F%, F + %, and other, nonform determinants. The magnitude and importance of these changes lead to the caution that the Rorschach might well be preceded by a test of visual acuity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Review of The nature of nondirective group psychotherapy. An experimental investigation.

Reviews the book, The nature of nondirective group psychotherapy. An experimental investigation by Leon Gorlow, Erasmus L. Hoch, and Earl F. Telschow (1952). This study was designed to investigate with quantitative precision various parameters of the process of nondirective group psychotherapy. Acknowledging the tentative, exploratory nature of the investigation, the authors outline a threefold objective: (a) analysis of the uniformity of group patterns of verbal behavior as a function of variables like time, differing membership, and leadership; (b) delineation of the therapeutic value of the group, per se; and (c) evaluation of the role of the group leader. This book is successful in providing interesting hypotheses and techniques for analyzing protocols. The limitations of its experimental design, however, prevent an evaluation of either the unique characteristics or therapeutic value of nondirective group psychotherapy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Concerns regarding the traditional paradigm for causal research: The unified paradigm and causal research in scientific psychology.

The traditional paradigm refers to the assumption held by most methodologists and researchers that causal research must be defined in terms of the causal powers evident in a closed system. The traditional paradigm does not concord, however, with the nature of scientific theories often cited in the methodological and research literature. The unified paradigm is introduced and causal research defined in terms of the causal powers evident in an open system. Notable implications of the unified paradigm are that experimental methods do not provide a better opportunity than modeling methods to conduct a causal analysis and that the nomenclature often used to describe the validity of causal conclusions must be amended. Additional implications of the unified paradigm are discussed and includes a comparison of the traditional paradigm and the unified paradigm when applied to treatment-outcome research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A critique and experimental design for the study of the relationship between productivity and job satisfaction.

"It is argued that the present approach to the study of the relationship between employee output and job satisfaction is not fruitful. What is needed is an examination of the characteristics of workers who are operating at a satisfactory level of both output and job satisfaction with workers and groups of workers who are not operating at such a level. A procedure is described which will permit the location of the workers who are operating at this 'optimal' level." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The converse inequality argument against tests of statistical significance.

Critics have put forth several arguments against the use of tests of statistical significance (TOSSes). Among these, the converse inequality argument stands out but remains sketchy, as does criticism of it. The argument states that we want P(HΔ) (where H and D represent hypothesis and data, respectively), we get P(DΗ), and the 2 do not equal one another. Each of the terms in 'P(DΗ)?≠?P(HΔ)' requires clarification. Furthermore, the argument as a whole allows for multiple interpretations. If the argument questions the logic of TOSSes, then defenses of TOSSes fall into 2 distinct types. Clarification and analysis of the argument suggest more moderate conclusions than previously offered by friends and critics of TOSSes. Furthermore, the general method of clarification through formalization may offer a way out of the current impasse. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Auditory Scene Analysis and the Role of Phenomenology in Experimental Psychology.

This paper is about two topics: a) the role of subjectivity in psychological research; and, b) my research on the perceptual organization of sound, in which subjectivity has played an important role. Audio demonstrations that appeal to the subjective experience of the reader are presented in lieu of objective research data to support the claims made about auditory organization. It is argued that all psychological research depends on an underlying framework of intuition (unformalized knowledge acquired through everyday experience), and that this intuition plays a role in the design of the experiment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The p-value fallacy and how to avoid it.

Null hypothesis significance tests are commonly used to provide a link between empirical evidence and theoretical interpretation. However, this strategy is prone to the "p-value fallacy" in which effects and interactions are classified as either "noise" or "real" based on whether the associated p value is greater or less than .05. This dichotomous classification can lead to dramatic misconstruals of the evidence provided by an experiment. For example, it is quite possible to have similar patterns of means that lead to entirely different patterns of significance, and one can easily find the same patterns of significance that are associated with completely different patterns of means. Describing data in terms of an inventory of significant and nonsignificant effects can thus completely misrepresent the results. An alternative analytical technique is to identify competing interpretations of the data and then use likelihood ratios to assess which interpretation provides the better account. Several different methods of calculating the likelihood ratios are illustrated. It is argued that this approach satisfies a principle of "graded evidence," according to which similar data should provide similar evidence. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Pooling data from multiple longitudinal studies: The role of item response theory in integrative data analysis.

There are a number of significant challenges researchers encounter when studying development over an extended period of time, including subject attrition, the changing of measurement structures across groups and developmental periods, and the need to invest substantial time and money. Integrative data analysis is an emerging set of methodologies that allows researchers to overcome many of the challenges of single-sample designs through the pooling of data drawn from multiple existing developmental studies. This approach is characterized by a host of advantages, but this also introduces several new complexities that must be addressed prior to broad adoption by developmental researchers. In this article, the authors focus on methods for fitting measurement models and creating scale scores using data drawn from multiple longitudinal studies. The authors present findings from the analysis of repeated measures of internalizing symptomatology that were pooled from three existing developmental studies. The authors describe and demonstrate each step in the analysis and conclude with a discussion of potential limitations and directions for future research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)