Covariance adjustment versus gain scores—revisited.

There are 2 main ways of estimating the average treatment effect in a 2-group pretest-posttest study: the gain score and the covariance adjustment estimator. The difficulty of the estimation problem arises from the fact that it involves missing posttest data. The gain score and the covariance adjustment estimator both use the pretest to predict these missing data, but in different ways: The gain score estimator treats the pretest as a baseline and the covariance adjustment estimator treats it as a covariate. Using a result by D. B. Rubin (1977), it is shown that, if the assignment is not on the basis of the pretest, there is no basis for preferring the covariance adjustment estimator over the gain score estimator. Contrary to what is sometimes suggested, regression toward the mean is not a reason for not using the gain score estimator; neither is measurement error in the pretest, unless the assignment is affected by the pretest. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Use of the nonequivalent control group design when subjects are growing.

Many experimental programs aim to accelerate the growth of individuals on some dimension of interest. Typically, a group exposed to the program is compared with a control group. When it is feasible, random assignment of Ss to the program and control groups assures an unbiased comparison between treatments. Without randomization there may be important differences between groups, in terms of pretreatment characteristics and growth potential. In the nonequivalent control group design, pretest and posttest data on both groups are obtained. Statistical methods are used to adjust posttest comparisons, based mainly on pretest information. A variety of statistical techniques have been proposed, but there is much disagreement among methodologists as to which, if any, are adequate. This article examines the adequacy of these techniques, from an individual growth perspective. The performance of various commonly used methods is examined under alternative assumptions about the nature of growth. It is concluded that statistical adjustments are generally inadequate in the face of nonequivalent growth systems across treatment groups. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Statistical treatment of the Solomon four-group design: A meta-analytic approach.

One of the causes of the underuse of the Solomon four-group design may be that the complete details for the statistical analysis have not previously been presented. The primary issue previously unaddressed was how to combine an analysis of the effect of the treatment in the posttest-only groups with the same effect in the pre- and posttest groups (after an earlier phase of the analysis has shown no evidence of pretest sensitization.) A meta-analytic solution for this problem is proposed, and the entire analysis is presented, complete with flowchart and example. It is shown that the analysis has adequate statistical power even if the total N is not increased from that of a posttest-only design, removing the last of the serious objections to the Solomon design. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group

In observational studies, investigators have no control over the treatment assignment. The treated and non-treated (that is, control) groups may have large differences on their observed covariates, and these differences can lead to biased estimates of treatment effects. Even traditional covariance analysis adjustments may be inadequate to eliminate this bias. The propensity score, defined as the conditional probability of being treated given the covariates, can be used to balance the covariates in the two groups, and therefore reduce this bias. In order to estimate the propensity score, one must model the distribution of the treatment indicator variable given the observed covariates. Once estimated the propensity score can be used to reduce bias through matching, stratification (subclassification), regression adjustment, or some combination of all three. In this tutorial we discuss the uses of propensity score methods for bias reduction, give references to the literature and illustrate the uses through applied examples.     

Nonrandom assignment and the analysis of covariance.

Discusses the appropriate use of the analysis of covariance for cases in which groups differ substantially on a variable that is entered as a covariate. The erroneous notions that groups must not differ significantly on the covariate and that covariates must be measured without error are rejected. Selective nonrandom assignment of Ss to groups on the basis of an observed variable that is measured with error can result in groups that differ substantially, but it is shown that conventional analysis of covariance provides unbiased estimates of the true treatment effects, in spite of the initial group differences. In other cases, correction for attenuation due to measurement error is required to obtain unbiased estimates of true treatment effects. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Alpha, beta, and gamma change in evaluation research: A structural equation approach.

Effects of organizational interventions are often studied within a nonequivalent control group design, with the pretest and posttest variables being self-report measures. Changes in these measures from pretest to posttest have multiple interpretations, depending on the processes believed to be responsible for the change. Golembiewski, Billingsley, and Yeager (1976) classified observed change in three categories as alpha, beta, or gamma change. Alpha change corresponds to absolute quantitative change. Beta change results from the respondent's subjective recalibration of the measurement scale. Gamma change results from the respondent's reconceptualization of the measured variable. A structural equation method is presented for the evaluation of intervention effects in the nonequivalent control group design when either alpha, beta, or gamma change may have occurred. The method uses an analysis of covariance in the latent variables underlying the pretest and posttest measures. Use of the method in distinguishing the three types of change, and linking them to the intervention, is discussed. The method is illustrated in examples using simulated data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative power of training evaluation designs under different cost configurations.

After noting that the statistical power of training evaluation designs is a complex function of sample size, the reliability of the dependent measure, the correlation between pre- and posttest measures, and whether a randomized pretest–posttest or randomized posttest-only design is used, the authors show that the costs of conducting an evaluation are important considerations that also affect the relative power of the designs. Specifically, S costs, administrative costs, and item development costs are different components that can absorb resources when training evaluations are conducted. When total cost resources are fixed, these separate costs affect the relative power of pretest–posttest and posttest-only designs differently, and the posttest-only design may be the more preferred design under many different conditions. In other words, a variety of design and parameter tradeoffs affect power when total costs are fixed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Issues for covariance analysis of dichotomous and ordered categorical data from randomized clinical trials and non-parametric strategies for addressing them

Analysis of covariance is an effective method for addressing two considerations for randomized clinical trials. One is reduction of variance for estimates of treatment effects and thereby the production of narrower confidence intervals and more powerful statistical tests. The other is the clarification of the magnitude of treatment effects through adjustment of corresponding estimates for any random imbalances between the treatment groups with respect to the covariables. The statistical basis of covariance analysis can be either non-parametric, with reliance only on the randomization in the study design, or parametric through a statistical model for a postulated sampling process. For non-parametric methods, there are no formal assumptions for how a response variable is related to the covariables, but strong correlation between response and covariables is necessary for variance reduction. Computations for these methods are straightforward through the application of weighted least squares to fit linear models to the differences between treatment groups for the means of the response variable and the covariables jointly with a specification that has null values for the differences that correspond to the covariables. Moreover, such analysis is similarly applicable to dichotomous indicators, ranks or integers for ordered categories, and continuous measurements. Since non-parametric covariance analysis can have many forms, the ones which are planned for a clinical trial need careful specification in its protocol. A limitation of non-parametric analysis is that it does not directly address the magnitude of treatment effects within subgroups based on the covariables or the homogeneity of such effects. For this purpose, a statistical model is needed. When the response criterion is dichotomous or has ordered categories, such a model may have a non-linear nature which determines how covariance adjustment modifies results for treatment effects. Insight concerning such modifications can be gained through their evaluation relative to non-parametric counterparts. Such evaluation usually indicates that alternative ways to compare treatments for a response criterion with adjustment for a set of covariables mutually support the same conclusion about the strength of treatment effects. This robustness is noteworthy since the alternative methods for covariance analysis have substantially different rationales and assumptions. Since findings can differ in important ways across alternative choices for covariables (as opposed to methods for covariance adjustment), the critical consideration for studies with covariance analyses planned as the primary method for comparing treatments is the specification of the covariables in the protocol (or in an amendment or formal plan prior to any unmasking of the study.     

Power in randomized group comparisons: The value of adding a single intermediate time point to a traditional pretest-posttest design.

Adding a pretest as a covariate to a randomized posttest-only design increases statistical power, as does the addition of intermediate time points to a randomized pretest-posttest design. Although typically 5 waves of data are required in this instance to produce meaningful gains in power, a 3-wave intensive design allows the evaluation of the straight-line growth model and may reduce the effect of missing data. The authors identify the statistically most powerful method of data analysis in the 3-wave intensive design. If straight-line growth is assumed, the pretest-posttest slope must assume fairly extreme values for the intermediate time point to increase power beyond the standard analysis of covariance on the posttest with the pretest as covariate, ignoring the intermediate time point. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Maximizing power in skewed distributions: Analysis and assignment.

Power and Type I error control were compared in a Monte Carlo simulation of the combinations of 2 methods of assignment to groups and 2 methods of analysis. Assignment to treatment groups was either random or systematic on the basis of alternating ranks on a concomitant variable. Analysis was either with a randomized block or analysis of covariance using the concomitant variable. The correlation between the concomitant and outcome variables was set at .2, .5, and .8. The skew of the concomitant and outcome were separately set at γ? values of 0.0, 0.75, 1.25, and 1.75. Overall the most powerful design was an analysis of covariance with systematic assignment of subjects. This superiority was particularly noticeable when the correlation was large and the distribution of the outcome variable was exponentially skewed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the power of multivariate latent growth curve models to detect correlated change.

We evaluated the statistical power of single-indicator latent growth curve models (LGCMs) to detect correlated change between two variables (covariance of slopes) as a function of sample size, number of longitudinal measurement occasions, and reliability (measurement error variance). Power approximations following the method of Satorra and Saris (1985) were used to evaluate the power to detect slope covariances. Even with large samples (N=500) and several longitudinal occasions (4 or 5), statistical power to detect covariance of slopes was moderate to low unless growth curve reliability at study onset was above .90. Studies using LGCMs may fail to detect slope correlations because of low power rather than a lack of relationship of change between variables. The present findings allow researchers to make more informed design decisions when planning a longitudinal study and aid in interpreting LGCM results regarding correlated interindividual differences in rates of development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correction to Venter et al. (2002).

Reports an error in the original article by A. Venter et al (Psychological Methods, 2002[Jun], Vol No. 7[2], 194-209. On page 202, there were 2 errors. Appendix B correctly shows that Equation 17 and 18 should read as indicated here. (The following abstract of this article originally appeared in record 2002-13431-003.) Adding a pretest as a covariate to a randomized posttest-only design increases statistical power, as does the addition of intermediate time points to a randomized pretest-posttest design. Although typically 5 waves of data are required in this instance to produce meaningful gains in power, a 3-wave intensive design allows the evaluation of the straight-line growth model and may reduce the effect of missing data. The authors identify the statistically most powerful method of data analysis in the 3-wave intensive design. If straight-line growth is assumed, the pretest-posttest slope must assume fairly extreme values for the intermediate time point to increase power beyond the standard analysis of covariance on the posttest with the pretest as covariate, ignoring the intermediate time point. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The reliability of self-image change scores in psychotherapy research: An application of generalizability theory.

The aim of this study was to estimate the reliability of the pre- to posttreatment change scores for 3 different self-image aspects, Attack, Love, and Control. To measure self-image, we used the Norwegian version of the introject surface of Benjamin's (1974) structural analysis of social behavior. The article introduces Generalizability (G-) theory, combined with the recent concept of tolerance for error, as a framework for estimating the reliability and precision of change scores in 1- and 2-facet designs. Data were obtained from the Norwegian Multi-Site Study of Process and Outcome in Psychotherapy, including 291 outpatients. The mean number of treatment sessions was 47. The results show that change scores may be highly reliable. Generalizability coefficients resting on the relative and absolute score interpretations, respectively, for both the Love and Attack change scores reached acceptable levels. The reliability of the Control change score was, however, poor. G-theory combined with the error-tolerance concept proved to be a helpful framework for assessing the dependability of change scores in a psychotherapy research setting. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of transformational leadership training on attitudinal and financial outcomes: A field experiment.

A pretest-posttest control-group design (N ?=?20) was used to assess the effects of transformational leadership training, with 9 and 11 managers assigned randomly to training and control groups, respectively. Training consisted of a 1-day group session and 4 individual booster sessions thereafter on a monthly basis. Multivariate analyses of covariance, with pretest scores as the covariate, showed that the training resulted in significant effects on subordinates' perceptions of leaders' transformational leadership, subordinates' own organizational commitment, and 2 aspects of branch-level financial performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of alcohol on perceptual field dependence.

The stability of perceptual field dependence, as determined from rod-and-frame test (RFT) performance, when alcohol is ingested by nonalcoholic Ss was investigated. Ss randomly assigned to the experimental (Alcohol) and the control (No Alcohol) groups were found not to differ significantly on pretest RFT scores. A significant increase in perceptual field dependence occurred on the posttest for the Alcohol group, while the No Alcohol group showed no significant change. The difference between the change scores for the 2 groups was significant at p     

Improving breast cancer control among Latinas: evaluation of a theory-based educational program

The study evaluated a theory-based breast cancer control program specially developed for less acculturated Latinas. The authors used a quasi-experimental design with random assignment of Latinas into experimental (n = 51) or control (n = 37) groups that completed one pretest and two posttest surveys. The experimental group received the educational program, which was based on Bandura's self-efficacy theory and Freire's empowerment pedagogy. Outcome measures included knowledge, perceived self-efficacy, attitudes, breast self-examination (BSE) skills, and mammogram use. At posttest 1, controlling for pretest scores, the experimental group was significantly more likely than the control group to have more medically recognized knowledge (sum of square [SS] = 17.0, F = 6.58, p < .01), have less medically recognized knowledge (SS = 128.8, F = 39.24, p < .001), greater sense of perceived self-efficacy (SS = 316.5, F = 9.63, p < .01), and greater adeptness in the conduct of BSE (SS = 234.8, F = 153.33, p < .001). Cancer control programs designed for less acculturated women should use informal and interactive educational methods that incorporate skill-enhancing and empowering techniques.     

Planned and post hoc methods for multiple-sample McNemar (1947) tests with missing data.

Pre-post experimental designs with dichotomous dependent variables are encountered frequently in behavioral research. If there is only one group, the McNemar (1947) test can be used to test the hypothesis of no change in parameters. The McNemar test has been extended to cover multiple groups. The problem with these tests is that complete data must be available for all subjects. If post- or pretest data are missing, subjects must be discarded. Ekbohm (1982) provided a solution for the one-group model that has good statistical properties. We extend this method to the multiple-group case. We describe and illustrate procedures for planned and post hoc contrasts. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A general approach for estimating scale score reliability for panel survey data.

Scale score measures are ubiquitous in the psychological literature and can be used as both dependent and independent variables in data analysis. Poor reliability of scale score measures leads to inflated standard errors and/or biased estimates, particularly in multivariate analysis. Reliability estimation is usually an integral step to assess data quality in the analysis of scale score data. Cronbach’s α is a widely used indicator of reliability but, due to its rather strong assumptions, can be a poor estimator (L. J. Cronbach, 1951). For longitudinal data, an alternative approach is the simplex method; however, it too requires assumptions that may not hold in practice. One effective approach is an alternative estimator of reliability that relaxes the assumptions of both Cronbach’s α and the simplex estimator and thus generalizes both estimators. Using data from a large-scale panel survey, the benefits of the statistical properties of this estimator are investigated, and its use is illustrated and compared with the more traditional estimators of reliability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Physical fitness training and counseling as treatment for youthful offenders.

Two groups of 30 randomly selected males (15 yrs 5 mo to 18 yrs 6 mo) in a state industrial school for youthful offenders were administered pre- and posttreatment a battery of physiological and psychological measures by an exercise physiologist and a psychometrist. Measures included tests of cardiovascular fitness and muscle strength/endurance as well as the Self-Esteem Inventory, State-Trait Anxiety Inventory, and Beck Depression Inventory. The experimental group received a systematic physical fitness program delivered by counselors for 1.5 hrs/day, 3 days/wk, for 20 wks. Treatment included a counseling model previously used with delinquent adolescents. MANOVAs revealed significant differences between the groups on pretest measures in favor of controls. Significant differences on the posttest measures were found in favor of experimental Ss. Univariate analysis identified the areas of difference both physiologically and psychologically. (44 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measuring the impact of patient counseling in the outpatient pharmacy setting: the research design of the Kaiser Permanente/USC patient consultation study

This article describes the research method used to measure the impact of three alternative models of patient counseling in the outpatient pharmacy setting. The study was conducted in pharmacies operated by the Southern California region Kaiser Permanente Medical Care Program. Both random assignment and large-scale geographic area research designs were used. The presentation of the research design includes discussions of data collection and patient sampling methods; the measurement of patient outcomes, including measures of health care costs and utilization, patient functional status, and quality of life. Demographic data are presented for the study population, including an analysis of potential biased selection of patients electing to participate in random assignment. Data are also presented documenting potential selection bias across geographically determined treatment groups in the geographic area design arm. Finally, the article presents the analysis plan for the study and discusses study limitations.