External validity and scientific psychology.

Comments on the article by Stanley Sue (December 1999; see record 1999-15532-003), in which he eloquently described a persistent problem in psychological research: the relative lack of research on ethnic minorities. Sue traced the source of this problem to how science is practiced and, in particular, to scientific psychology's emphasis on internal validity over external validity. He argued that researchers' assumption that causal inferences drawn from a given study are generalizable across individuals from different ethnic backgrounds ultimately masks true differences among diverse ethnic groups and hinders research to determine whether such differences exist. Sue recommended that researchers increase their emphasis on external validity in study designs and embrace methodological pluralism in adopting more qualitative and ethnographic approaches to complement traditional scientistic methods used in psychological research. If scientific psychology is to serve the public interest, its findings must be valid and generalizable and must promote the translation of research into informed public policy. This will be possible only when psychologists stop assuming generality across population subgroups, across settings, and across times. Judicious prioritization of external validity above internal validity will build a better scientific psychology and inform public policy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Factors relevant to the validity of experiments in social settings.

"In analyzing the extraneous variables which experimental designs for social settings seek to control, seven categories have been distinguished: history, maturation, testing, instrument decay, regression, selection, and mortality. In general, the simple or main effects of these variables jeopardize the internal validity of the experiment and are adequately controlled in standard experimental designs. The interactive effects of these variables and of experimental arrangements affect the external validity or generalizability of experimental results. Standard experimental designs vary in their susceptibility to these interactive effects." 37 references. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some issues of ethics and social psychology.

Reviews the implications of tightening ethical standards for social psychology, and contends that there is a current shift in the field toward investigating neutral or positive, rather than negative, aspects of human behavior. The major methodological approaches (laboratory experiments, field experiments, role playing, and complex correlational methods) are discussed with respect to the issues of validity of causal inference, ecological validity, and ethics. It is argued that there is a fundamental difference in the perception of the nature of man on the part of humanistic critics and traditional experimental social psychologists that leads to different assessments of the long-term effects of deception research on Ss. Long-term follow-up studies are proposed as a source of empirical information on which investigators and ethics committees may base decisions about the acceptability of the level of risk associated with experimental manipulations. (40 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A plea for consideration of ecological validity in the experimental psychology of mental retardation: a guest editorial

The point was raised that, despite considerable experimental effort, laboratory research concerning learning, memory, and cognition, more generally, has not produced a very remarkable increase in our understanding of retarded behavior. Our principal contention in this paper is that the experimental psychology of mental retardation, while basically seeking causal relations between theoretical constructs and retarded behavior, is suffering from some metatheoretical and methodological shortcomings. These include, basically, a prevalent failure to consider the ecological aspects of the phenomenon of mental retardation. Implications of ecological validity are important with respect to the basis upon which subjects are selected for experimentation, the rationale underlying manipulation of independent variables, the choice of dependent variables, and the definition of the boundaries that limit generalizations. Some suggestions were offered for the purpose of guiding experimental research toward more meaningful and socially relevant goals.     

The validity of validity: An analysis of validation study designs.

Draws a distinction between validity (logic of inferences) and validation (the process of research using a specific design) to evaluate 11 criterion-related validation designs that differ in the timing of measurement of test behavior and job performance and in the selection procedure (random, existing tests, and experimental tests). Comparisons are based on T. D. Cook and D. T. Campbell's (1949) 4 interrelated criteria: statistical conclusion, internal, construct, and external validity. The analysis marks the selection procedure as a major design property that influences the validity of validation studies. Important differences between concurrent and predictive validation are identified for several aspects of validity. The fact that no specific design is considered most appropriate for all purposes suggests the use of validation research programs, based on a series of studies with different designs, to enhance the validity of validation research. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing Mediational Models With Longitudinal Data: Questions and Tips in the Use of Structural Equation Modeling.

R. M. Baron and D. A. Kenny (1986; see record 1987-13085-001) provided clarion conceptual and methodological guidelines for testing mediational models with cross-sectional data. Graduating from cross-sectional to longitudinal designs enables researchers to make more rigorous inferences about the causal relations implied by such models. In this transition, misconceptions and erroneous assumptions are the norm. First, we describe some of the questions that arise (and misconceptions that sometimes emerge) in longitudinal tests of mediational models. We also provide a collection of tips for structural equation modeling (SEM) of mediational processes. Finally, we suggest a series of 5 steps when using SEM to test mediational processes in longitudinal designs: testing the measurement model, testing for added components, testing for omitted paths, testing the stationarity assumption, and estimating the mediational effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Causal Learning Across Domains.

Five studies investigated (a) children's ability to use the dependent and independent probabilities of events to make causal inferences and (b) the interaction between such inferences and domain-specific knowledge. In Experiment 1, preschoolers used patterns of dependence and independence to make accurate causal inferences in the domains of biology and psychology. Experiment 2 replicated the results in the domain of biology with a more complex pattern of conditional dependencies. In Experiment 3, children used evidence about patterns of dependence and independence to craft novel interventions across domains. In Experiments 4 and 5, children's sensitivity to patterns of dependence was pitted against their domain-specific knowledge. Children used conditional probabilities to make accurate causal inferences even when asked to violate domain boundaries. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

All data collection and analysis methods have limitations: Reply to Rabbitt (2011) and Raz and Lindenberger (2011).

The commentaries on my article contain a number of points with which I disagree but also several with which I agree. For example, I continue to believe that the existence of many cases in which between-person variability does not increase with age indicates that greater variance with increased age is not inevitable among healthy individuals up to about 80 years of age. I also do not believe that problems of causal inferences from correlational information are more severe in the cognitive neuroscience of aging than in other research areas; I contend instead that neglect of these problems has led to confusion about neurobiological underpinnings of cognitive aging. I agree that researchers need to be cautious in extrapolating from cross-sectional to longitudinal relations, but I also note that even longitudinal data are limited with respect to their ability to support causal inferences. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Analogical and category-based inference: A theoretical integration with Bayesian causal models.

A fundamental issue for theories of human induction is to specify constraints on potential inferences. For inferences based on shared category membership, an analogy, and/or a relational schema, it appears that the basic goal of induction is to make accurate and goal-relevant inferences that are sensitive to uncertainty. People can use source information at various levels of abstraction (including both specific instances and more general categories), coupled with prior causal knowledge, to build a causal model for a target situation, which in turn constrains inferences about the target. We propose a computational theory in the framework of Bayesian inference and test its predictions (parameter-free for the cases we consider) in a series of experiments in which people were asked to assess the probabilities of various causal predictions and attributions about a target on the basis of source knowledge about generative and preventive causes. The theory proved successful in accounting for systematic patterns of judgments about interrelated types of causal inferences, including evidence that analogical inferences are partially dissociable from overall mapping quality. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Transfer among phonological manipulation skills.

Skills that require the manipulations of sounds that make up words are closely associated with the process of learning to read. This study involved an experimental analysis of the relations among phonological manipulation skills. Several skills were taught to 35 Head Start preschool children (mean age 5.2 yrs), and the degree to which learning 1 of these skills resulted in improved skill performance and accelerated learning of a 2nd skill was investigated. Instruction produced robust gains in skill performance and generalization of skills to novel instances. Posttest scores and data on students' efficiency of learning, however, offered no evidence of transfer across phonological manipulation skills. These results imply that the class of phonological manipulation skills does not have a simple structure and suggest that task and subject characteristics must be considered in predicting transfer among such skills. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The causal generalization paradox: The case of treatment outcome research.

Data from psychological experiments pose a causal generalization paradox. Unless the experimental results have some generality, they contribute little to scientific knowledge. Yet, because most experiments use convenience samples rather than probability-based samples, there is almost never a formal justification, or set of rigorous guidelines, for generalizing the study's findings to other populations. This article discusses the causal generalization paradox in the context of outcome findings from experimental evaluations of psychological treatment programs and services. In grappling with the generalization paradox, researchers often make misleading (or at least oversimplified) assumptions. The article analyzes 10 such assumptions, including the belief that a significant experimental treatment effect is likely to be causally generalizable and the belief that the magnitude of a significant experimental effect provides a sound effect size estimate for causal generalization. The article then outlines 10 constructive strategies for assessing and enhancing causal generality. They include strategies involving the scaling level of outcome measures, variable treatment dosages, effectiveness designs, multiple measures, corroboration from observational designs, and the synthesis of multiple studies. Finally, the article's discussion section reviews the conditions under which causal generalizations are justified. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Incremental Validity of Psychological Testing and Assessment: Conceptual, Methodological, and Statistical Issues.

There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Non-infectious pulmonary complications in cystic fibrosis in the adult

This paper reviews the logic of causal inference from epidemiological data. I maintain that the clearest causal statements can be made when the philosophical causal principles of association, direction and isolation are upheld in epidemiological research. After reviewing the argument by Holland that only experimental manipulation affords clear causal claims, I examine the utility of structural equation models and longitudinal methods for making causal claims from non-experimental data. This examination leads to the conclusion that mental health epidemiologists should begin to incorporate intervention trials into the last phases of their research programmes when they want to make strong causal claims.     

Correlation, causation, motivation, and second language acquisition.

Examines 4 truisms about correlation, demonstrating that they are not always true. For example, under certain conditions, correlation can imply causation, though these conditions are seldom satisfied in most applications. Nonetheless, many people are interested in investigating individual differences, and in making inferences of the type that this individual difference variable is related to, mediates, moderates, or even causes or influences that individual difference variable. Generally speaking, the analytic procedures used involve the correlation coefficient in one form or another. The author proposes 4 steps that researchers can follow to accumulate evidence that increases one's confidence in the validity of a particular causal model. These steps are illustrated by reviewing research on individual differences in 2nd language acquisition. This approach is not conclusive, but it does force the examination of the implications of the model, thus leading to further insights and research. Although the focus here is on 2nd language acquisition, the generalizations apply to other areas of research that are concerned with individual differences. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Blickets and babies: The development of causal reasoning in toddlers and infants.

Previous research has suggested that preschoolers possess a cognitive system that allows them to construct an abstract, coherent representation of causal relations among events. Such a system lets children reason retrospectively when they observe ambiguous data in a rational manner (e.g., D. M. Sobel, J. B. Tenenbaum, & A. Gopnik, 2004). However, there is little evidence that demonstrates whether younger children possess similar inferential abilities. In Experiment 1, the authors extended previous findings with older children to examine 19- and 24-month-olds' causal inferences. Twenty-four-month-olds' inferences were similar to those of preschoolers, but younger children lacked the ability to make retrospective causal inferences, perhaps because of performance limitations. In Experiment 2, the authors designed an eye-tracking paradigm to test younger participants that eliminated various manual search demands. Eight-month-olds' anticipatory eye movements, in response to retrospective data, revealed inferences similar to those of 24-month-olds in Experiment 1 and preschoolers in previous research. These data are discussed in terms of associative reasoning and causal inference. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of inference necessity and reading goal on children's inferential generation.

Examined inferential processing during reading. Adults and 3rd, 5th, and 8th graders read stories implying a consequence and answered questions. Some of the inferences were more necessary than others for comprehension according to T. Trabasso and P. Van den Broek's (see record 1987-18271-001) causal criteria, and the readers engaged in either superficial or integrative reading. Results showed that elaborative inferences were not as likely to be generated as those more necessary for comprehension. The necessity manipulation also produced a similar pattern of responding among the 4 groups, suggesting that even the young children were sensitive to the causal criteria. In addition, the integrative reading condition prompted a general slowdown for the 2 younger groups, whereas the 2 older groups were not as affected by reading condition. This pattern was interpreted in terms of differences in attention demands and working memory capacity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Causal learning mechanisms in very young children: Two-, three-, and four-year-olds infer causal relations from patterns of variation and covariation.

Three studies investigated whether young children make accurate causal inferences on the basis of patterns of variation and covariation. Children were presented with a new causal relation by means of a machine called the "blicket detector." Some objects, but not others, made the machine light up and play music. In the first 2 experiments, children were told that "blickets make the machine go" and were then asked to identify which objects were "blickets." Two-, 3-, and 4-year-old children were shown various patterns of variation and covariation between two different objects and the activation of the machine. All 3 age groups took this information into account in their causal judgments about which objects were blickets. In a 3rd experiment, 3- and 4-year-old children used the information when they were asked to make the machine stop. These results are related to Bayes-net causal graphical models of causal learning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)