Multiway frequency analysis for experimental psychologists.

Many research designs in experimental psychology generate data that are fundamentally discrete or categorical in nature, and produce multiway tables of frequencies. Despite an extensive and, more recently, accessible literature on the topic, multiway frequency analysis is rarely used in experimental psychology. A reason may be the form of exposition in the literature, with emphases and concerns far removed from those of the typical experimental psychologist. An approach to multiway frequency analysis for experimental psychologists is described that has the features we want: asymmetrical designs, factors assessed for their respective main and interactive effects in a manner analogous to ANOVA, and the ability to handle within-subject designs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Individual differences methods for randomized experiments.

Experiments allow researchers to randomly vary the key manipulation, the instruments of measurement, and the sequences of the measurements and manipulations across participants. To date, however, the advantages of randomized experiments to manipulate both the aspects of interest and the aspects that threaten internal validity have been primarily used to make inferences about the average causal effect of the experimental manipulation. This article introduces a general framework for analyzing experimental data to make inferences about individual differences in causal effects. Approaches to analyzing the data produced by a number of classical designs and 2 more novel designs are discussed. Simulations highlight the strengths and weaknesses of the data produced by each design with respect to internal validity. Results indicate that, although the data produced by standard designs can be used to produce accurate estimates of average causal effects of experimental manipulations, more elaborate designs are often necessary for accurate inferences with respect to individual differences in causal effects. The methods described here can be diversely applied by researchers interested in determining the extent to which individuals respond differentially to an experimental manipulation or treatment and how differential responsiveness relates to individual participant characteristics. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Design of experiments with multiple independent variables: A resource management perspective on complete and reduced factorial designs.

An investigator who plans to conduct an experiment with multiple independent variables must decide whether to use a complete or reduced factorial design. This article advocates a resource management perspective on making this decision, in which the investigator seeks a strategic balance between service to scientific objectives and economy. Considerations in making design decisions include whether research questions are framed as main effects or simple effects; whether and which effects are aliased (confounded) in a particular design; the number of experimental conditions that must be implemented in a particular design and the number of experimental subjects the design requires to maintain the desired level of statistical power; and the costs associated with implementing experimental conditions and obtaining experimental subjects. In this article 4 design options are compared: complete factorial, individual experiments, single factor, and fractional factorial. Complete and fractional factorial designs and single-factor designs are generally more economical than conducting individual experiments on each factor. Although relatively unfamiliar to behavioral scientists, fractional factorial designs merit serious consideration because of their economy and versatility. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Double-blind versus deceptive administration of a placebo.

Subjects were given varying doses of a placebo, consisting of decaffeinated coffee, with double-blind or deceptive instructions. Deceptive administration simulated clinical situations in that subjects were led to believe that they were receiving an active drug. In contrast, subjects in double-blind conditions were aware that they might receive a placebo. Double-blind and deceptive administration of the placebo produced different, and in some instances, opposite effects on pulse rate, systolic blood pressure, and subjective mood. Deceptive administration produced an increase in pulse rate, whereas double-blind administration did not. A theoretically predicted curvilinear effect on systolic blood pressure, alertness, tension, and certainty of having consumed caffeine was confirmed with deceptive administration, but not with double-blind administration. Double-blind administration produced curves in the opposite direction on each of these variables. The effects of the placebo on motor performance varied as a function of subjects' beliefs about the effects of caffeine. These data challenge the validity of double-blind experimental designs and suggest that this common method of drug assessment may lead to spurious conclusions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Diffusion effects: Control group contamination threats to the validity of teacher-administered interventions.

Within-class experimental designs (with experimental and control groups in the same classroom) are subject to diffusion effects whereby both experimental and control students benefit from the intervention, thereby contaminating the control group and biasing evaluations of intervention effects. Although the problem has been recognized, most previous demonstrations have been anecdotal. In support of diffusion effects, we show that a classroom intervention resulted in higher academic self-concepts for internal (within-class) controls compared with external (between-class) controls. The construct validity of the interpretation of this difference as a diffusion effect was supported by observer and teacher comments and ratings of teacher success in focusing the intervention on experimental students and by different patterns of results for teachers who were more or less successful in maintaining this focus. Potential dangers in sole reliance on internal within-class control groups may outweigh advantages of this expedient experimental design. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Preventing adolescent smoking: What have we learned about treatment construct validity?

Reviews the treatment construct validity of programs to prevent adolescent smoking to determine how and why programs reduce adolescent smoking. The issue of why beneficial effects are obtained is explored in terms of whether a treatment affects outcome; whether each variable in a proposed causal chain affects the variable that follows it after controlling for all prior variables, including treatment; and whether treatment has an effect on outcome when the mediating variables are controlled. Studies investigating the mediational effects of subjective norms such as social consequences and peer vs adult leaders, social competence, and intervention are reviewed. It is concluded that little is presently known about the construct validity of successful programs, a problem that results primarily from the neglect of process assessment and analyses. The advantages and disadvantages of several future research approaches are discussed, including utilization of process measures within large-scale treatment/no-treatment designs, small-scale studies to test the effects of prevention components on process measures (e.g., attitudes, intentions to smoke), and combinations of these approaches. (4 p ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reorganization of variables in analysis of variance and multidimensional contingency tables.

Appropriate reorganization of variables in some analysis of variance designs may make the obtained results more easily interpretable and may also expand the range of experimental designs that can easily be analyzed by standard procedures. A rule is given for determining equivalences of effects in terms of original and reorganized variables, and an example illustrates the usefulness for theoretical purposes of such restructuring. The potential of reorganization for broadening the range of readily applicable experimental designs, and the implications of the possibility of reorganization for nonorthogonal analysis of variance, are explored. Applications of restructuring in multidimensional contingency table analysis are noted. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on concurrent and predictive validity designs: A critical reanalysis.

Comments on an article in which G. Barrett et al (see record 1981-11631-001) argued that the conceptual distinction between predictive and concurrent validity has been exaggerated. The present authors offer a more complete classification of validity designs and illustrate sources of differences in the quality of the validity estimates obtained. Five kinds of predictive designs for selection research are identified: (a) follow-up with random selection, (b) follow-up with selection by existing systems, (c) selection based on test to be validated, (d) selection preceding testing, and (e) shelf research. For some of these, and for corresponding concurrent designs, problems arise from the lack of data for informed estimates of appropriate population parameters. It is pointed out that severe range restriction intensifies parameter estimation and the risk of Type II error. Corrections suggested for the effects of contaminations such as age, tenure, and experience are shown to be oversimplifications. It is concluded that on both conceptual and practical bases different validity designs are not equivalent or interchangeable. (10 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Learning the control of variables strategy in higher and lower achieving classrooms: Contributions of explicit instruction and experimentation.

Students (n = 797) from 36 4th-grade classrooms were taught the control of variables strategy for designing experiments. In the instruct condition, classes were taught in an interactive lecture format. In the manipulate condition, students worked in groups to design and run experiments to determine the effects of four variables. In the both condition, classes received the interactive lecture and also designed and ran experiments. We assessed students' understanding using a written test of their ability to distinguish valid from invalid experimental comparisons. Performance on this test improved from the pretest to the immediate posttest in all conditions, and gains were maintained at a 5-month delay. For students from both higher and lower achieving schools, gains ordered as follows: both > instruct > manipulate. However, students from higher achieving schools showed greater gains in all conditions. Item analyses showed that the interactive lecture improved students' understanding of the need to control irrelevant variables, and experimentation improved students' understanding of the need to vary the focal variable. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Incremental Validity of Psychological Testing and Assessment: Conceptual, Methodological, and Statistical Issues.

There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Is time-out from positive reinforcement an aversive event? A review of the experimental evidence.

Experiments concerning the aversive properties of time-out (TO) from positive reinforcement are reviewed. A discussion of experimental designs employed and problems encountered in defining aversive stimuli precedes the review. The major topics covered are: avoidance of TO, escape from TO, escape from stimuli which previously signalled TO, punishment with TO, effects on ongoing behavior of pre-TO stimuli, escape from conditions of positive reinforcement into TO. In general, TO satisfied sufficient criteria to conclude that it belongs to the class of stimuli called "aversive." However, this conclusion is only tentative. (2 p. ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Within-subjects designs: To use or not to use?

Examines several factors pertinent to deciding whether a within- or between-Ss design should be employed for a research application. A general principle favoring within-Ss designs is the statistical efficiency afforded by removing S variance from error terms used to test treatment effects. Within-Ss designs, however, are often faulted for being subject to context effects of practice, sensitization, and carry-over that may limit interpretation of results. At the same time, between-Ss designs are not devoid of context effects, but rather have the context that a single treatment affords itself. Since ecological validity of results depends on the correspondence of the research context to the generalization context, within-Ss designs may be preferred when the generalization context includes the equivalent of several concurrent treatments. Procedures to minimize practice, sensitization and carry-over effects in within-Ss designs when they are not desired, and means of using these effects to advantage in research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relation between psychometric and experimental research in psychology.

A discussion of the distinction between experimental research (variables determined by experimental operations) and psychometric research (variables determined by psychometric operations). Indications of how these two traditionally distinct methods may be combined and the resultant advantages obtained are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Worksite physical activity interventions

BACKGROUND: National objectives for public health have targeted worksite as important settings for interventions to increase physical activity. However, expert reviews reveal no scientific consensus about the effectiveness of worksite interventions for increasing physical activity or fitness. METHODS: We judged the quantity and quality of existing evidence against scientific standards for the internal and external validity of the research design and the validity of measurements. Meta-analytic methods were used to quantify the size of effects expressed as Pearson correlation coefficients (r). Variation in effect was examined in relation to several features of the studies deemed important for implementing successful worksite interventions. Pre-experimental cohort studies were excluded because they are sensitive to secular trends in physical activity. RESULTS: Twenty-six studies involving nearly 9,000 subjects yielded 45 effects. The mean effect was heterogeneous and small, r = 0.11 (95% CI, -0.20 to 0.40), approximating 1/4 S.D., or an increase in binomial success rate from 50% to 56%. Although effects varied slightly according to some of the study features we examined, effects were heterogeneous within levels of these features. Hence, the moderating variables examined did not explain variation in the effects (P > 0.05). The exception was that effects were smaller in randomized studies compared with studies using quasi-experimental designs (P < 0.05). CONCLUSIONS: Our results indicate that the typical worksite intervention has yet to demonstrate a statistically significant increase in physical activity or fitness. The few studies that have used an exemplary sample, research design, and outcome measure have also yielded small or no effects. The generally poor scientific quality of the literature on this topic precludes the judgment that interventions at worksites cannot increase physical activity or fitness, but such an increase remains to be demonstrated by studies using valid research designs and measures.     

The French system of occupation medicine. II. Evolution of the French system of occupational medicine]

The validity of the 1991 Finnish Medical Birth Registry data was assessed, with special emphasis on the effects of changes made to the data collection form in 1990. Data abstracted from medical records for all births occurring in 49 hospitals during a five-day sample period (n = 865) were compared to the register information. Good or satisfactory validity was found for 32 of 33 variables, when minor error was tolerated in variables with continuous scales. For diagnoses and procedures, recorded in check-box format, satisfactory validity was found for 10 of 45 variables. Validity could not be assessed for 18 variables because of insufficient number of cases (13 items) or definition problems (5 items). When the results were compared to a 1987 data quality study, many of the variables that had been changed to the check-box format showed improvement in validity. In addition, in some cases a small change in question alternatives or instructions caused a noticeable change in validity.     

The two disciplines of scientific psychology.

"No man can be acquainted with all of psychology today." The past and future place within psychology of 2 historic streams of method, thought, and affiliation—experimental psychology and correlational psychology—is discussed in this address of the President at the 65th annual convention of the APA. "The well-known virtue of the experimental method is that it brings situational variables under tight control… . The correlation method, for its part, can study what man has not learned to control or can never hope to control… . A true federation of the disciplines is required. Kept independent, they can give only wrong answers or no answers at all regarding certain important problems… . Correlational psychology studies only variance among organisms; experimental psychology studies only variance among treatments. A united discipline will study both of these, but it will also be concerned with the otherwise neglected interactions between organismic and treatment variables. Our job is to invent constructs and to form a network of laws which permits prediction." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Constructs and their vicissitudes in educational and psychological measurement.

Examines implications for educational and psychological measurement of 3 ontological perspectives on the nature of mediating variables underlying consistencies in test and nontest behaviors: (a) intervening variables operationally tied to real causal entities, such as personality traits or environmental contingencies; (b) hypothetical constructs that organize and summarize behavioral consistencies but have no reality outside the theoretical system; and (c) manifestations of real entities that are understood only in terms of constructs that summarize their empirical properties in relation to a theoretical network. All 3 apply to personality traits, situational forces, and their interactions; the summary of power of constructs that led to the predominance of construct validity principles in trait measurement implies that these principles should hold with equal cogency for situational and interactive measurement. (48 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Physiological effects of "hypnotic suggestions': A critical review of recent research (1960-64).

Recent studies are reviewed which were concerned with the effectiveness of suggestions given under "hypnosis" and "waking" experimental treatments in alleviating allergies, ichthyosis, myopia, and other conditions and in eliciting deafness, blindness, hallucinations, analgesia, cardiac acceleration and deceleration, emotional responses, urine secretion to sham water ingestion, narcotic-like drug effects, and other phenomena. The review indicates that a wide variety of physiological functions can be influenced by suggestions administered under either hypnosis or waking experimental treatments, and direct and indirect suggestions to show the particular physiological manifestations are crucial variables in producing the effects. (3 p. ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Designs, participants, and measurement methods in psychological research.

Critical reviews of psychological scholarship suggest that self-report questionnaires, experimental designs, and college students dominate psychological research. Although researchers within specific subfields of psychology have the requisite knowledge to assess the generality of these concerns, novices of psychological research do not. To provide such knowledge, we surveyed a random sample of 200 journal articles in the PsycINFO database coding for psychological content area, research design, measurement method, and participant type. Results indicate self-report questionnaires, experimental designs, and college students each appeared in a minority of studies and these study characteristics often varied significantly by content area. Results also suggest no single dominant "typical" study in psychological research but rather that the characteristics of such research exhibit remarkable diversity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)